Python多处理池某些进程在分叉时处于死锁状态，但在派生时运行。

Question

因此，我尝试尝试一些服务下载和调整图像大小(使用线程下载图像和进程来调整它们的大小)。我启动下载线程(使用将监视它们的管理器线程)，一旦图像在本地保存，我就将其路径添加到队列中。当下载所有图像时，管理器线程将向队列中添加一个毒丸。

同时，主线程监视队列，并在下载时从队列中获取路径，并从池中触发一个新的异步进程来调整映像的大小。

最后，当我试图加入池时，它有时会挂起来，似乎是一个僵局。这种情况并不是每次都会发生，但是IMG_URLS列表中的url越多，发生的频率就越高。如果出现这种死锁，日志会告诉我们，某些进程没有正确启动，或者由于“调整大小{file}”日志没有出现而立即处于死锁状态。

import logging
import multiprocessing as mp
import time
from queue import Queue
from threading import Thread


def resize_image(file):
    logging.info(f"resizing {file}")
    time.sleep(0.1)
    logging.info(f"done resizing {file}")


class Service(object):
    def __init__(self):
        self.img_queue = Queue()

    def download_image(self, url) -> None:
        logging.info(f"downloading image from URL {url}")
        time.sleep(1)
        file = f"local-{url}"
        self.img_queue.put(file)
        logging.info(f"image saved to {file}")

    def download_images(self, img_url_list: list):
        logging.info("beginning image downloads")

        threads = []
        for url in img_url_list:
            t = Thread(target=self.download_image, args=(url,))
            t.start()
            threads.append(t)

        for t in threads:
            t.join()
        logging.info("all images downloaded")
        self.img_queue.put(None)

    def resize_images(self):
        logging.info("beginning image resizing")
        with mp.Pool() as p:
            while True:
                file = self.img_queue.get()
                if file is None:
                    logging.info("got SENTINEL")
                    break
                logging.info(f"got {file}")
                p.apply_async(func=resize_image, args=(file,))
            p.close()
            p.join()
        logging.info("all images resized")

    def run(self, img_url_list):
        logging.info("START service")

        dl_manager_thread = Thread(target=self.download_images, args=(img_url_list,))
        dl_manager_thread.start()
        self.resize_images()

        logging.info(f"END service")


if __name__ == "__main__":
    FORMAT = "[%(threadName)s, %(asctime)s, %(levelname)s] %(message)s"
    logging.basicConfig(level=logging.DEBUG, format=FORMAT)

    IMG_URLS = list(range(8))

    service = Service()
    service.run(IMG_URLS)

当使用python 3.8.5 (Ubuntu20.04，Ryzen 2600)运行时。我得到以下信息：

[MainThread, 2020-11-30 19:58:01,257, INFO] START service
[Thread-1, 2020-11-30 19:58:01,257, INFO] beginning image downloads
[MainThread, 2020-11-30 19:58:01,257, INFO] beginning image resizing
[Thread-2, 2020-11-30 19:58:01,258, INFO] downloading image from URL 0
[Thread-3, 2020-11-30 19:58:01,258, INFO] downloading image from URL 1
[Thread-4, 2020-11-30 19:58:01,258, INFO] downloading image from URL 2
[Thread-5, 2020-11-30 19:58:01,259, INFO] downloading image from URL 3
[Thread-6, 2020-11-30 19:58:01,260, INFO] downloading image from URL 4
[Thread-7, 2020-11-30 19:58:01,260, INFO] downloading image from URL 5
[Thread-8, 2020-11-30 19:58:01,261, INFO] downloading image from URL 6
[Thread-9, 2020-11-30 19:58:01,262, INFO] downloading image from URL 7
[Thread-2, 2020-11-30 19:58:02,259, INFO] image saved to local-0
[MainThread, 2020-11-30 19:58:02,260, INFO] got local-0
[Thread-3, 2020-11-30 19:58:02,260, INFO] image saved to local-1
[Thread-4, 2020-11-30 19:58:02,260, INFO] image saved to local-2
[MainThread, 2020-11-30 19:58:02,261, INFO] got local-1
[MainThread, 2020-11-30 19:58:02,261, INFO] resizing local-0
[Thread-5, 2020-11-30 19:58:02,261, INFO] image saved to local-3
[Thread-6, 2020-11-30 19:58:02,261, INFO] image saved to local-4
[MainThread, 2020-11-30 19:58:02,261, INFO] got local-2
[MainThread, 2020-11-30 19:58:02,262, INFO] got local-3
[MainThread, 2020-11-30 19:58:02,262, INFO] resizing local-1
[Thread-7, 2020-11-30 19:58:02,262, INFO] image saved to local-5
[MainThread, 2020-11-30 19:58:02,262, INFO] got local-4
[MainThread, 2020-11-30 19:58:02,263, INFO] got local-5
[MainThread, 2020-11-30 19:58:02,263, INFO] resizing local-3
[Thread-8, 2020-11-30 19:58:02,263, INFO] image saved to local-6
[MainThread, 2020-11-30 19:58:02,263, INFO] resizing local-4
[MainThread, 2020-11-30 19:58:02,263, INFO] resizing local-5
[MainThread, 2020-11-30 19:58:02,263, INFO] got local-6
[MainThread, 2020-11-30 19:58:02,264, INFO] resizing local-6
[Thread-9, 2020-11-30 19:58:02,264, INFO] image saved to local-7
[MainThread, 2020-11-30 19:58:02,265, INFO] got local-7
[Thread-1, 2020-11-30 19:58:02,265, INFO] all images downloaded
[MainThread, 2020-11-30 19:58:02,265, INFO] got SENTINEL
[MainThread, 2020-11-30 19:58:02,265, INFO] resizing local-7
[MainThread, 2020-11-30 19:58:02,362, INFO] done resizing local-0
[MainThread, 2020-11-30 19:58:02,363, INFO] done resizing local-1
[MainThread, 2020-11-30 19:58:02,363, INFO] done resizing local-3
[MainThread, 2020-11-30 19:58:02,364, INFO] done resizing local-4
[MainThread, 2020-11-30 19:58:02,364, INFO] done resizing local-5
[MainThread, 2020-11-30 19:58:02,364, INFO] done resizing local-6
[MainThread, 2020-11-30 19:58:02,366, INFO] done resizing local-7

有时它会在这里吊起来。注意，没有调整本地-2日志的大小，因此进程没有启动，或者等待什么。

如果我改变池使用产卵而不是叉，它可以工作。我猜叉子在某些情况下复制了一些锁定，这就是问题所在，但我不清楚为什么在哪里。

with mp.get_context("spawn").Pool() as p:

有什么想法吗？

Answer 1

有时(当您运气不好时)当您的池正在旋转时，当您的下载线程向logging模块写入消息时，其中一个子进程将被“分叉”。logging模块使用一个受锁保护的队列来传递消息，因此当“叉”发生时，该锁可以在锁定状态下被复制。然后，当下载线程将其消息写入队列时，只释放主进程上的锁，因此您将有一个子进程等待该锁的副本将消息写入logging。该锁永远无法释放，因为下载程序线程没有被复制(叉不复制线程)。这就是所发生的死锁。这种类型的错误可以在某些方面进行修补，但这是“产卵”存在的原因之一。

此外，“派生”是所有架构所支持的唯一方法。它只是很容易使用一个库，碰巧是多线程下的引擎盖，而“叉子”只是不是真正的多线程友好。我对“叉服务器”知之甚少，以防你真的需要“叉”所提供的减少的开销。从理论上讲，这是一个更多的多线程安全。

分叉 父进程使用os.fork()对Python解释器进行分叉。子进程开始时实际上与父进程相同。父进程的所有资源都由子进程继承。注意，安全分叉多线程进程是有问题的。

下面是是一个更深入的讨论，其中有一些关于这个问题的参考资料，我用它作为我的主要资源。

Answer 2

只是一些额外的信息，以扩展伟大的答案，从亚伦。

这个python /增强似乎完全相同：https://bugs.python.org/issue6721

我在另一个问题中发现了同样的问题：记录多进程/多线程python脚本的死锁