python-packaging:在需求和我的包之间共享build-backend

Question

我的python pyackage有几件必需品。其中两个需求是prophet (https://github.com/facebook/prophet)和george (https://github.com/dfm/george)。这些包的问题是，它们需要在安装之前安装numpy、pybind11和其他包，因为它们在构建过程中使用。但是，我想让用户在不事先安装几个包的情况下尽可能简单。

构建包所需的包可以在setup.py中使用setup_requires指定，也可以在pyproject.toml内的build-system中指定。我认为通过在我的pyproject.toml中指定外部包的那些构建需求，我将能够使用"pip install“实现单命令安装。然而，这并不起作用。我的建议是，对于每个必须构建轮子的包，都要使用单独的build-backend。因此，在构建第三方包时，构建需求并不存在。

我几乎找不到关于构建后端和构建需求的文档。你能给我一个提示吗？

我在这里有一个最小的工作示例:安装了docker的https://github.com/FelixKleineBoesing/pypackagingdemo，您可以通过以下命令开始构建过程：

docker build .

如果没有第10行，pip将无法成功构建george，因为缺少numpy和pybind11。

完整回溯：

Sending build context to Docker daemon  81.41kB
Step 1/8 : FROM python:3.7
 ---> 34a518642c76
Step 2/8 : RUN mkdir /buildbackend
 ---> Running in 38e398f63eed
Removing intermediate container 38e398f63eed
 ---> f88b7b3f4056
Step 3/8 : COPY ./abctest /buildbackend/abctest
 ---> 5976a8f6d708
Step 4/8 : COPY ./pyproject.toml /buildbackend/pyproject.toml
 ---> 73d4bb506910
Step 5/8 : COPY ./setup.py /buildbackend/setup.py
 ---> 0a377986a29d
Step 6/8 : COPY ./Manifest.in /buildbackend/Manifest.in
 ---> c30e903f8f84
Step 7/8 : RUN cd /buildbackend && pip3 install .
 ---> Running in 7925fc1a9a45
Processing /buildbackend
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting george>=0.3.1 (from abctest==0.0.1)
  Downloading https://files.pythonhosted.org/packages/27/19/9de575be629e3a41c3ca6b1e2c80e0ae90a2e831436c5f70cc8d72e37ab7/george-0.3.1.tar.gz (1.8MB)
Collecting numpy (from george>=0.3.1->abctest==0.0.1)
  Using cached https://files.pythonhosted.org/packages/63/0c/0261693cc3ad8e2b66e66dc2d2676a2cc17d3efb1c58a70db73754320e47/numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting scipy (from george>=0.3.1->abctest==0.0.1)
  Downloading https://files.pythonhosted.org/packages/dd/82/c1fe128f3526b128cfd185580ba40d01371c5d299fcf7f77968e22dfcc2e/scipy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (26.1MB)
Collecting pybind11 (from george>=0.3.1->abctest==0.0.1)
  Using cached https://files.pythonhosted.org/packages/4b/4d/ae1c4d8e8b139afa9682054dd42df3b0e3b5c1731287933021b9fd7e9cc4/pybind11-2.4.3-py2.py3-none-any.whl
Building wheels for collected packages: abctest
  Building wheel for abctest (PEP 517): started
  Building wheel for abctest (PEP 517): finished with status 'done'
  Stored in directory: /tmp/pip-ephem-wheel-cache-5ruh8xv_/wheels/e3/5f/a4/652949565a7089970182a50f7eec733873579b601c20f0cb97
Successfully built abctest
Building wheels for collected packages: george
  Building wheel for george (setup.py): started
  Building wheel for george (setup.py): finished with status 'error'
  ERROR: Complete output from command /usr/local/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-bijmj8ct/george/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-i9_120mu --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/george
  copying george/kernels.py -> build/lib.linux-x86_64-3.7/george
  copying george/utils.py -> build/lib.linux-x86_64-3.7/george
  copying george/gp.py -> build/lib.linux-x86_64-3.7/george
  copying george/modeling.py -> build/lib.linux-x86_64-3.7/george
  copying george/metrics.py -> build/lib.linux-x86_64-3.7/george
  copying george/__init__.py -> build/lib.linux-x86_64-3.7/george
  creating build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/basic.py -> build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/trivial.py -> build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/__init__.py -> build/lib.linux-x86_64-3.7/george/solvers
  copying george/solvers/hodlr.py -> build/lib.linux-x86_64-3.7/george/solvers
  running egg_info
  writing george.egg-info/PKG-INFO
  writing dependency_links to george.egg-info/dependency_links.txt
  writing requirements to george.egg-info/requires.txt
  writing top-level names to george.egg-info/top_level.txt
  reading manifest file 'george.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'george.egg-info/SOURCES.txt'
  copying george/kernel_interface.cpp -> build/lib.linux-x86_64-3.7/george
  creating build/lib.linux-x86_64-3.7/george/include
  copying george/include/.DS_Store -> build/lib.linux-x86_64-3.7/george/include
  creating build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/exceptions.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/george.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/hodlr.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/kernels.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/metrics.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/parser.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/include/george/subspace.h -> build/lib.linux-x86_64-3.7/george/include/george
  copying george/solvers/_hodlr.cpp -> build/lib.linux-x86_64-3.7/george/solvers
  running build_ext
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-bijmj8ct/george/setup.py", line 231, in <module>
      zip_safe=True,
    File "/usr/local/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 78, in run
      _build_ext.run(self)
    File "/usr/local/lib/python3.7/distutils/command/build_ext.py", line 340, in run
      self.build_extensions()
    File "/tmp/pip-install-bijmj8ct/george/setup.py", line 105, in build_extensions
      import numpy
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for george
  Running setup.py clean for george
Failed to build george
Installing collected packages: numpy, scipy, pybind11, george, abctest
  Running setup.py install for george: started

Answer 1

在我看来，george项目的打包有些问题，如果不先预安装numpy (就像您在Dockerfile中所做的那样)，就不能干净地构建。如果这个项目使用pyproject.toml (PEP517)的build-system特性，并将numpy添加到它的requires列表中，那么它就可以顺利构建了。

作为另一种选择，可以考虑预先构建自己的george项目轮子，并缓存/共享它们以供以后重用。在Docker的情况下，有很好的解决方案。