MemoryError:使用Python的微软ML服务

Question

下面的脚本在小样本中运行良好

EXECUTE sp_execute_external_script
@language = N'Python',
@script = N'
print(df_training["flResp"].value_counts())',
@input_data_1 = N'SELECT * FROM tb_training_teste',
@input_data_1_name = N'df_training';

我测试了8419条记录，结果是可以的，如下所示：

Mensagem(ns) STDOUT do script externo:
0 4964
1 3452
9 3
Name: flResp, dtype: int64

但是，我的原始表有超过500,000条记录，我无法运行，因为下面的错误。有人能帮我找出哪里出了问题吗？以及如何修复它？

Error in execution. Check the output for more information.
MemoryError

SqlSatelliteCall error: Error in execution. Check the output for more information.
Mensagem(ns) STDOUT do script externo:
SqlSatelliteCall function failed. Please see the console output for more information.
Traceback (most recent call last):
File "C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\computecontext\RxInSqlServer.py", line 587, in rx_sql_satellite_call
rx_native_call("SqlSatelliteCall", params)
File "C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\PYTHON_SERVICES\lib\site-packages\revoscalepy\RxSerializable.py", line 358, in rx_native_call
ret = px_call(functionname, params)
RuntimeError: revoscalepy function failed.

Answer 1

我刚刚解决了这个问题，所以我会添加我的发现，以防将来它会对某人有所帮助。

SQL server似乎使用资源池来限制外部进程(如Python和R )可用的资源。

SELECT * FROM sys.resource_governor_external_resource_pools

默认情况下，将有一个max_memory_percent为20的池(名为default)。可以使用ALTER EXTERNAL RESOURCE POOL命令增加该值，例如：

ALTER EXTERNAL RESOURCE POOL [default]
WITH (
    MAX_MEMORY_PERCENT = 95
)
GO

ALTER RESOURCE GOVERNOR RECONFIGURE
GO

或者，可以在SQL Server Management Studio中对其进行更改：

​