Perdiodical data fetch应使用schedueler阻止其他线程

Question

我是python的新手。非常感谢您的帮助。那么让我们开始吧。

我正在运行一个FLASK应用程序作为rest api。对于每个请求，都会返回一个json。每秒可能有大约20个请求。在后台，我使用APscheduler每隔60秒从ldap获取一次实际数据。

scheduler = BackgroundScheduler()
scheduler.add_job(
    func=my_ldap.fetch_people_ldap,
    trigger=IntervalTrigger(seconds=60),
    id='fetching_data_job',
    name='Fetch data from ldap every 60 seconds')

scheduler.start()
atexit.register(lambda : scheduler.shutdown())

但实际上，当使用api调用命中datafetch时，应用程序由于内存问题而关闭，我认为这是因为我在访问ldap对象的同时，调度器调用了fetch_people_ldap函数。

我想通过阻塞处理api调用的线程来解决这个可怕的bug，直到成功退出ldap数据获取。但我不知道该怎么做。

有什么建议或解决方案吗？

这是我得到的错误日志:致命的Python错误:保存线程两次？

Thread 0x00007f95f3fff700 (most recent call first):
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 721 in result4
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 714 in result3
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 707 in result2
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 703 in result
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 796 in search_ext_s
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 802 in search_s
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 49 in fetch_people_ldap
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/apscheduler/executors/base.py", line 125 in run_job
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55 in run
  File "/usr/lib/python3.5/concurrent/futures/thread.py", line 66 in _worker
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Thread 0x00007f95f8910700 (most recent call first):
  File "/usr/lib/python3.5/threading.py", line 297 in wait
  File "/usr/lib/python3.5/threading.py", line 549 in wait
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/apscheduler/schedulers/blocking.py", line 28 in _main_loop
  File "/usr/lib/python3.5/threading.py", line 862 in run
  File "/usr/lib/python3.5/threading.py", line 914 in _bootstrap_inner
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

Current thread 0x00007f9605505700 (most recent call first):
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 791 in search_ext
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 795 in search_ext_s
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 802 in search_s
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 59 in check_node
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 118 in build_tree_recursive
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 124 in build_tree_recursive
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 124 in build_tree_recursive
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 129 in build_tree
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/app/app.py", line 83 in get_trainings_by_unit
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1598 in dispatch_request
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1612 in full_dispatch_request
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1982 in wsgi_app
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 1997 in __call__
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 197 in execute
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 209 in run_wsgi
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 267 in handle_one_request
  File "/usr/lib/python3.5/http/server.py", line 422 in handle
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 232 in handle
  File "/usr/lib/python3.5/socketserver.py", line 681 in __init__
  File "/usr/lib/python3.5/socketserver.py", line 354 in finish_request
  File "/usr/lib/python3.5/socketserver.py", line 341 in process_request
  File "/usr/lib/python3.5/socketserver.py", line 313 in _handle_request_noblock
  File "/usr/lib/python3.5/socketserver.py", line 234 in serve_forever
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 539 in serve_forever
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 702 in inner
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/werkzeug/serving.py", line 739 in run_simple
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 841 in run
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/app/app.py", line 92 in <module>

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

Answer 1

问题是您的API调用处理程序也调用LDAP来创建响应。如果你打算每秒收到20个请求，那么就会产生问题。预定的调用也没有多大意义，因为您什么时候使用检索到的数据？

我的方法是让计划的任务在后台更新一个公共数据存储。然后，您的API调用只需从该公共数据存储中读取数据，而无需接触LDAP。您是使用数据库还是只使用内存，这取决于您正在处理的数据的大小和复杂性。

主要的事情是将创建API响应与从LDAP获取数据解耦。

LDAP：我从您的评论中了解到，您认为您的只使用您自己的ldapCheck.py代码，但如果您查看完整的堆栈跟踪，它实际上是使用LDAP库进行调用：

API线程：

Current thread 0x00007f9605505700 (most recent call first):
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
...
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 124 in build_tree_recursive
...
File "/path/to/folder/tt_report_api/training_tool_report_api/sample/app/app.py", line 83 in get_trainings_by_unit
...
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/flask/app.py", line 841 in run

如果在计划的后台线程调用LDAP的同时发生这种情况，则会出现此错误。

Thread 0x00007f95f3fff700 (most recent call first):
  File "/path/to/folder/virtual_env_1/lib/python3.5/site-packages/pyldap-2.4.37-py3.5-linux-x86_64.egg/ldap/ldapobject.py", line 294 in _ldap_call
...
  File "/path/to/folder/tt_report_api/training_tool_report_api/sample/ldap_check/ldapCheck.py", line 49 in fetch_people_ldap
...
  File "/usr/lib/python3.5/threading.py", line 882 in _bootstrap

解决方案是不阻塞你的API线程。解决方案是确保ldapCheck.py", line 59 in check_node中的代码不会调用您的LDAP库，而是使用后台线程已经检索和存储的信息。