异步:等待执行器运行的长任务完成时的异常处理

Question

我有以下代码片段由python 3.10.5运行：

import time
import asyncio

async def main():
    loop = asyncio.get_running_loop()
    loop.run_in_executor(None, blocking)
    print(f"{time.ctime()} Hello!")
    await asyncio.sleep(1.0)
    print(f"{time.ctime()} Goodbye!")

def blocking():
    time.sleep(5.0)
    print(f"{time.ctime()} Hello from thread!")


try:
    asyncio.run(main())
except KeyboardInterrupt:
    print("Cancelled.")

当我让它运行时，由于python3.9中添加了shutdown_default_executor()方法，它允许通过将该任务包装在coroutine中来解决在executor中运行超过主事件循环的任务的问题。因此，我有以下输出：

Sun Sep 11 19:04:25 2022 Hello!
Sun Sep 11 19:04:26 2022 Goodbye!
Sun Sep 11 19:04:30 2022 Hello from thread!

接下来，当我在输出的第一行之后按Ctrl-C时，我得到：

Sun Sep 11 19:04:42 2022 Hello!
^CSun Sep 11 19:04:47 2022 Hello from thread!
Cancelled.

因此，它仍然能够处理这种情况。但是，当我在Goodbye!行之后(当主协同线已经完成，并且在执行器中等待任务完成)时，我得到了：

Sun Sep 11 19:04:49 2022 Hello!
Sun Sep 11 19:04:50 2022 Goodbye!
^CCancelled.
Sun Sep 11 19:04:54 2022 Hello from thread!
exception calling callback for <Future at 0x7f58475183d0 state=finished returned NoneType>
Traceback (most recent call last):
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 330, in _invoke_callbacks
    callback(self)
  File "/usr/lib/python3.10/asyncio/futures.py", line 398, in _call_set_state
    dest_loop.call_soon_threadsafe(_set_state, destination, source)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Exception in thread Thread-1 (_do_shutdown):
Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/base_events.py", line 576, in _do_shutdown
    self.call_soon_threadsafe(future.set_result, None)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 578, in _do_shutdown
    self.call_soon_threadsafe(future.set_exception, ex)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

问题是，为什么我在这里获得运行时错误，但在Hello!行之后碰到Hello!时却设法避免它(第二个例子)？如何优雅地处理这个运行时错误？

Answer 1

我想我已经想好了，但是请小心点--我对asyncio很陌生。

我的基础是一个Python3.10 asyncio.run代码：

def run(main, *, debug=None):

    # More code here
    # ...

    try:
        events.set_event_loop(loop)
        if debug is not None:
            loop.set_debug(debug)
        return loop.run_until_complete(main)
    finally:
        try:
            _cancel_all_tasks(loop)
            loop.run_until_complete(loop.shutdown_asyncgens())
            loop.run_until_complete(loop.shutdown_default_executor())
        finally:
            events.set_event_loop(None)
            loop.close()

你描述的行为是由那些嵌套的尝试-最后块来解释的。

如果KeyboardInterrupt是在loop.run_until_complete(main)期间发生的--在您的情况下是在Goodbye!之前--那么内部的尝试最终将被执行，这将正确地处理使用loop.shutdown_default_executor()的blocking。

另一方面，如果异常发生在Goodbye!之后，当前正在执行的代码是loop.shutdown_default_executor()，并且由于进一步清理会在没有等待任何东西的情况下关闭循环，包含blocking的Future将抛出RuntimeError('Event loop is closed')。

在那之后，Future似乎还在等待.除非你再次击中Ctrl-C。然后，

Exception ignored in: <module 'threading' from '/usr/lib/python3.10/threading.py'>
Exception in thread Thread-1 (_do_shutdown):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1537, in _shutdown
  File "/usr/lib/python3.10/asyncio/base_events.py", line 576, in _do_shutdown
    self.call_soon_threadsafe(future.set_result, None)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 795, in call_soon_threadsafe
    atexit_call()
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit
    self._check_closed()
  File "/usr/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
    t.join()
  File "/usr/lib/python3.10/threading.py", line 1096, in join
    self._wait_for_tstate_lock()
    raise RuntimeError('Event loop is closed')
  File "/usr/lib/python3.10/threading.py", line 1116, in _wait_for_tstate_lock
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    if lock.acquire(block, timeout):
KeyboardInterrupt:

抛出异常。

因此，似乎还有一个层--这一次是在线程级别--等待执行程序:D。

无论如何，我认为这是意料之中的--我们应该允许用户在不等待的情况下杀死程序(这可能需要很长时间)。

回到你的问题-如何优雅地处理它。我觉得你做不到。错误是在Future逻辑中抛出的，我们在stderr上看到的只是一个未使用结果的日志。但是仅仅因为我们不能处理它并不意味着用户必须看到它。

我不知道这是否是一个好的实践，但您可以将stderr重定向到null设备。或者找出是哪个伐木人在给他打补丁？对我有用的代码：

from contextlib import redirect_stderr
from os import devnull

fnull = open(devnull, 'w')
r = redirect_stderr(fnull)
r.__enter__()