在使用GBM算法时如何估计内存的使用

Question

我想估计一下我的算法XGBOOST和LightGBM如何使用python来消耗内存

我指的是某种意义上的东西：

我使用了这段代码来估计速度，并询问是否有类似的内存方法：

start_time = time.time()
xg_cl.fit(X_train, y_train, eval_metric="auc", early_stopping_rounds=300, eval_set=eval_set, verbose=True)
XgBoost_time_training = (time.time() - start_time)
print("--- %s seconds ---" % (time.time() - start_time))

PS:我需要一个可以在WINDOWS而不是linux上执行的代码。

Answer 1

您可以使用memory_profiler.profile，下面是一个示例

import numpy as np

from xgboost import XGBRegressor
from sklearn.model_selection import train_test_split

from memory_profiler import profile


precision = 10
fp = open('memory.log', 'w+')
@profile(precision=precision, stream=fp)
def xgb_test():

    X1 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    X2 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    Y = np.sin(X1) + 0.0001 * X2
    X = np.concatenate((X1, X2), axis = 1)

    x_train, x_val, y_train, y_val = train_test_split(X, Y, test_size = 0.2, random_state = 69)
    eval_set = [(x_train, y_train), (x_val, y_val)]
    XGBModel = XGBRegressor(max_depth = 10)
    XGBModel.fit(x_train,y_train,eval_set=eval_set,early_stopping_rounds=100,eval_metric="mae",verbose=2)

if __name__ == '__main__':
    xgb_test()

通过运行脚本，每一行(步骤)的内存消耗将被写入memory.log文件，如下所示：

Filename: test.py

Line #    Mem usage    Increment   Line Contents
================================================
    12 104.6992187500 MiB 104.6992187500 MiB   @profile(precision=precision, stream=fp)
    13                             def xgb_test():
    14
    15 104.6992187500 MiB   0.0000000000 MiB       X1 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    16 104.6992187500 MiB   0.0000000000 MiB       X2 = np.linspace(0, np.radians(360), 1000).reshape(-1, 1)
    17 104.7226562500 MiB   0.0234375000 MiB       Y = np.sin(X1) + 0.0001 * X2
    18 104.7382812500 MiB   0.0156250000 MiB       X = np.concatenate((X1, X2), axis = 1)
    19
    20 104.8320312500 MiB   0.0937500000 MiB       x_train, x_val, y_train, y_val = train_test_split(X, Y, test_size = 0.2, random_state = 69)
    21 104.8320312500 MiB   0.0000000000 MiB       eval_set = [(x_train, y_train), (x_val, y_val)]
    22 104.8320312500 MiB   0.0000000000 MiB       XGBModel = XGBRegressor(max_depth = 10)
    23 105.6718750000 MiB   0.8398437500 MiB       XGBModel.fit(x_train,y_train,eval_set=eval_set,early_stopping_rounds=100,eval_metric="mae",verbose=2)

其中，Increment列显示每个步骤的内存量。