如何在Django中使用uWSGI Spooler？

Question

在Django 3.2.3，Python3.7.9中，我尝试使用uWSGI Spooler而不是Celery来运行并发任务。我找到了一些资源，比如this、this和this，但都不起作用。在这段旅程中，我遇到了很多错误，我已经使用我在网上找到的解决方案修复了它们，现在我的情况是：

设置

uwsgi.ini

[uwsgi]

pythonpath          = /path/to/djproj
wsgi-file           = /path/to/djproj/wsgi.py
uid                 = myuid
module              = wsgi:application

master              = true
processes           = 1
threads             = 10
lazy-apps           = true
http                = 0.0.0.0:8080

vacuum              = true
log-format          = %(ltime) Worker: %(wid) %(status) %(method) %(uri) Size: %(size)
log-date            = %%Y %%m %%d %%H:%%M:%%S.000
# Let django handle most of the logging
disable-logging     = true
log-5xx             = true

harakiri            = 60
harakiri-verbose    = true

stats               = /tmp/djproj_stats.socket

# Spooling
spooler             = /path/to/tasks
spooler-harakiri    = 600
import              = djproj.tasks

tasks.py

import logging

logger = logging.getLogger(__name__)

try:
    from uwsgidecorators import spool

    logger.warning("Imported spool successfully.")
except Exception:
    logger.warning("Couldn't import spool.")

    def spool(func):
        def func_wrapper(**arguments):
            return func(arguments)

        return func_wrapper


@spool
def run_task(arguments):
    logger.warning("Running in spool.")
    from djproj.myapp.models import MyModel

    obj = MyModel.objects.get(id=arguments["obj_id"])
    obj.run()

djproj/myapp/models.py

# ...

def prepare_spooler_args(**kwargs):
    args = {}
    for name, value in kwargs.items():
        args[name.encode("utf-8")] = str(value).encode("utf-8")
    return args

class MyModel(models.Model):
    # ...
    def start_run_in_spooler(self):
        args = prepare_spooler_args(task_id=self.id)
        run_task(args)

结果

当我运行uwsgi --ini uwsgi.ini并访问触发此代码的端点时，我得到：

...
2021 09 20 12:56:20.000 - *** Stats server enabled on /tmp/djproj_stats.socket fd: 16 ***
2021 09 20 12:56:20.000 - spawned uWSGI http 1 (pid: 919)
2021 09 20 12:56:20.000 - [spooler /path/to/tasks pid: 917] managing request uwsgi_spoolfile_on_5a22c167ad32_826_2_189168444_1632124468_886229 ...
2021 09 20 12:56:20.000 - unable to find the spooler function, have you loaded it into the spooler process ?

我发现非常奇怪的是，网上关于如何做到这一点的资源如此之少。每次我寻找解决方案时，每个人都推荐芹菜，就像它是任何Python应用程序中并发的银弹一样，尽管Django + uWSGI是一个非常常见的组合，而且假脱机程序似乎是一个简单而轻量级的解决方案。如果任何人有任何关于如何让它工作的提示，那将是非常棒的。

Answer 1

在一位同事的帮助下，我们终于成功了。以下是必要的更改：

uwsgi.ini

import = djproj.tasks已更改为spooler-import = djproj.tasks。

tasks.py

必须将导入移出假脱机函数，并且需要初始化Django才能使导入工作。以下是代码的最终版本：

import logging
from functools import wraps

import django

logger = logging.getLogger(__name__)

try:
    from djproj.myapp.models import MyModel
except Exception:
    logger.warning(f"Django is not loaded yet! Setting up...")
    try:
        django.setup(set_prefix=False)
    except Exception:
        pass
    from djproj.myapp.models import MyModel

try:
    from uwsgidecorators import spool
except Exception:
    logger.warning("Couldn't import uwsgidecorators!")

    def spool(pass_arguments):
        def decorator(method):
            if callable(pass_arguments):
                method.gw_method = method.__name__
            else:
                method.gw_method = pass_arguments

            @wraps(method)
            def wrapper(*args, **kwargs):
                method(*args, **kwargs)

            return wrapper

        if callable(pass_arguments):
            return decorator(pass_arguments)
        return decorator


@spool(pass_arguments=True)
def run_task(task_id):
    task = MyModel.objects.get(id=task_id)
    task.run()

models.py和views.py

调用假脱机函数的model方法被删除。相反，我们只需在视图中直接调用假脱机函数：

class MyModelViewSet(viewsets.ModelViewSet):
    queryset = models.MyModel.objects.all()
    serializer_class = serializers.MyModelSerializer

    def create(self, request, *args, **kwargs):
        # ...

        run_task(task_id=task.id)

        # ...
        return Response(serializer.data, status=status_code, headers=headers)

希望这对任何面临类似问题的人都有帮助。