扩展setuptools扩展以在setup.py中使用setup.py？

Question

我正在编写一个连接C++库的Python扩展，并使用cmake来帮助构建过程。这意味着，现在，我知道如何捆绑它的唯一方法，我必须先用cmake编译它们，然后才能运行setup.py bdist_wheel。一定有更好的办法。

我想知道是否有可能(或者有人尝试过)调用CMake作为setup.py ext_modules构建过程的一部分？我猜想有一种方法可以创建某物的子类，但我不知道该在哪里查找。

我使用CMake是因为它为我提供了更多的控制权，可以用复杂的构建步骤构建c和c++库扩展，正如我所希望的那样。此外，我可以使用PYTHON_ADD_MODULE()命令在findPythonLibs.cmake中使用cmake直接构建Python扩展。我只希望这只是一步。

Answer 1

我想对此补充一下我自己的答案，作为霍弗尔所描述的一种补充。

谢谢您的回答，因为您的回答帮助我以同样的方式为我自己的存储库编写了一个安装脚本。

前言

写这个答案的主要动机是试图将缺失的部分“粘合在一起”。OP没有说明正在开发的C/ C++ Python模块的性质；我想先澄清一下，下面的步骤是用于创建多个.dll/ .so文件的C/ C++ cmake构建链，以及一个预编译的*.pyd/so文件，以及需要放置在脚本目录中的一些通用.py文件。

所有这些文件都是在运行cmake命令后直接实现的.有趣的。没有人建议以这种方式构建setup.py。

由于setup.py意味着您的脚本将成为包/库的一部分，需要构建的.dll文件必须通过库部分声明，其中包含源和列出的and，因此没有直观的方法告诉setuptools，在build_ext中发生的对cmake -b的一次调用所产生的库、脚本和数据文件都应该放在各自的位置。更糟糕的是，如果您想让setuptools跟踪这个模块并完全卸载，意味着用户可以卸载它，并在需要的情况下将所有跟踪从他们的系统中删除。

我为setup.py编写的模块是bpy，.pyd/ .so等效于将搅拌器构建为python模块，如下所述：

https://wiki.blender.org/wiki//User:Ideasman42/BlenderAsPyModule (更好的指令，但现在是死链接) http://www.gizmoplex.com/wordpress/compile-blender-as-python-module/ (可能更糟的指令，但似乎仍然在线)

您可以在这里查看我在github上的存储库：

https://github.com/TylerGubala/blenderpy

这就是我写这个答案的动机，希望能帮助其他人完成类似的事情，而不是抛弃他们的cmake构建链，或者更糟糕的是，必须维护两个独立的构建环境。如果这不是话题，我很抱歉。

那我该怎么做才能做到这一点呢？

1. 用我自己的类扩展setuptools.Extension类，它不包含源或libs属性的条目
2. 用自己的类扩展setuptools.commands.build_ext.build_ext类，该类具有执行必要构建步骤的自定义方法(git、svn、cmake、cmake - build )。
3. 扩展distutils.command.install_data.install_data类(讨厌，distutils.但是，似乎没有一个setuputils等价的类)，在setuptools的记录创建(install-files.txt)过程中标记构建的二进制库，以便

- The libraries will be recorded and will be uninstalled with `pip uninstall package_name`
- The command `py setup.py bdist_wheel` will work natively as well, and can be used to provide precompiled versions of your source code

​

1. 用我自己的类扩展setuptools.command.install_lib.install_lib类，这将确保构建的库从其生成的构建文件夹移到setuptools期望它们的文件夹中(在.dll上，它将把.dll文件放在bin/Release文件夹中，而不是setuptools所期望的位置)。
2. 用我自己的类扩展setuptools.command.install_scripts.install_scripts类，以便将脚本文件复制到正确的目录(Blender希望2.79或其他目录位于脚本位置)
3. 在执行构建步骤之后，将这些文件复制到一个已知的目录中，setuptools将该目录复制到我环境的steps目录中。此时，剩余的setuptools和distutils类可以负责编写已安装的文件. the记录，并且将完全可移动！

示例

这里有一个示例，或多或少来自我的存储库，但为更具体的事情进行了调整(您可以随时前往回购中心，自己查看)。

from distutils.command.install_data import install_data
from setuptools import find_packages, setup, Extension
from setuptools.command.build_ext import build_ext
from setuptools.command.install_lib import install_lib
from setuptools.command.install_scripts import install_scripts
import struct

BITS = struct.calcsize("P") * 8
PACKAGE_NAME = "example"

class CMakeExtension(Extension):
    """
    An extension to run the cmake build

    This simply overrides the base extension class so that setuptools
    doesn't try to build your sources for you
    """

    def __init__(self, name, sources=[]):

        super().__init__(name = name, sources = sources)

class InstallCMakeLibsData(install_data):
    """
    Just a wrapper to get the install data into the egg-info

    Listing the installed files in the egg-info guarantees that
    all of the package files will be uninstalled when the user
    uninstalls your package through pip
    """

    def run(self):
        """
        Outfiles are the libraries that were built using cmake
        """

        # There seems to be no other way to do this; I tried listing the
        # libraries during the execution of the InstallCMakeLibs.run() but
        # setuptools never tracked them, seems like setuptools wants to
        # track the libraries through package data more than anything...
        # help would be appriciated

        self.outfiles = self.distribution.data_files

class InstallCMakeLibs(install_lib):
    """
    Get the libraries from the parent distribution, use those as the outfiles

    Skip building anything; everything is already built, forward libraries to
    the installation step
    """

    def run(self):
        """
        Copy libraries from the bin directory and place them as appropriate
        """

        self.announce("Moving library files", level=3)

        # We have already built the libraries in the previous build_ext step

        self.skip_build = True

        bin_dir = self.distribution.bin_dir

        # Depending on the files that are generated from your cmake
        # build chain, you may need to change the below code, such that
        # your files are moved to the appropriate location when the installation
        # is run

        libs = [os.path.join(bin_dir, _lib) for _lib in 
                os.listdir(bin_dir) if 
                os.path.isfile(os.path.join(bin_dir, _lib)) and 
                os.path.splitext(_lib)[1] in [".dll", ".so"]
                and not (_lib.startswith("python") or _lib.startswith(PACKAGE_NAME))]

        for lib in libs:

            shutil.move(lib, os.path.join(self.build_dir,
                                          os.path.basename(lib)))

        # Mark the libs for installation, adding them to 
        # distribution.data_files seems to ensure that setuptools' record 
        # writer appends them to installed-files.txt in the package's egg-info
        #
        # Also tried adding the libraries to the distribution.libraries list, 
        # but that never seemed to add them to the installed-files.txt in the 
        # egg-info, and the online recommendation seems to be adding libraries 
        # into eager_resources in the call to setup(), which I think puts them 
        # in data_files anyways. 
        # 
        # What is the best way?

        # These are the additional installation files that should be
        # included in the package, but are resultant of the cmake build
        # step; depending on the files that are generated from your cmake
        # build chain, you may need to modify the below code

        self.distribution.data_files = [os.path.join(self.install_dir, 
                                                     os.path.basename(lib))
                                        for lib in libs]

        # Must be forced to run after adding the libs to data_files

        self.distribution.run_command("install_data")

        super().run()

class InstallCMakeScripts(install_scripts):
    """
    Install the scripts in the build dir
    """

    def run(self):
        """
        Copy the required directory to the build directory and super().run()
        """

        self.announce("Moving scripts files", level=3)

        # Scripts were already built in a previous step

        self.skip_build = True

        bin_dir = self.distribution.bin_dir

        scripts_dirs = [os.path.join(bin_dir, _dir) for _dir in
                        os.listdir(bin_dir) if
                        os.path.isdir(os.path.join(bin_dir, _dir))]

        for scripts_dir in scripts_dirs:

            shutil.move(scripts_dir,
                        os.path.join(self.build_dir,
                                     os.path.basename(scripts_dir)))

        # Mark the scripts for installation, adding them to 
        # distribution.scripts seems to ensure that the setuptools' record 
        # writer appends them to installed-files.txt in the package's egg-info

        self.distribution.scripts = scripts_dirs

        super().run()

class BuildCMakeExt(build_ext):
    """
    Builds using cmake instead of the python setuptools implicit build
    """

    def run(self):
        """
        Perform build_cmake before doing the 'normal' stuff
        """

        for extension in self.extensions:

            if extension.name == 'example_extension':

                self.build_cmake(extension)

        super().run()

    def build_cmake(self, extension: Extension):
        """
        The steps required to build the extension
        """

        self.announce("Preparing the build environment", level=3)

        build_dir = pathlib.Path(self.build_temp)

        extension_path = pathlib.Path(self.get_ext_fullpath(extension.name))

        os.makedirs(build_dir, exist_ok=True)
        os.makedirs(extension_path.parent.absolute(), exist_ok=True)

        # Now that the necessary directories are created, build

        self.announce("Configuring cmake project", level=3)

        # Change your cmake arguments below as necessary
        # Below is just an example set of arguments for building Blender as a Python module

        self.spawn(['cmake', '-H'+SOURCE_DIR, '-B'+self.build_temp,
                    '-DWITH_PLAYER=OFF', '-DWITH_PYTHON_INSTALL=OFF',
                    '-DWITH_PYTHON_MODULE=ON',
                    f"-DCMAKE_GENERATOR_PLATFORM=x"
                    f"{'86' if BITS == 32 else '64'}"])

        self.announce("Building binaries", level=3)

        self.spawn(["cmake", "--build", self.build_temp, "--target", "INSTALL",
                    "--config", "Release"])

        # Build finished, now copy the files into the copy directory
        # The copy directory is the parent directory of the extension (.pyd)

        self.announce("Moving built python module", level=3)

        bin_dir = os.path.join(build_dir, 'bin', 'Release')
        self.distribution.bin_dir = bin_dir

        pyd_path = [os.path.join(bin_dir, _pyd) for _pyd in
                    os.listdir(bin_dir) if
                    os.path.isfile(os.path.join(bin_dir, _pyd)) and
                    os.path.splitext(_pyd)[0].startswith(PACKAGE_NAME) and
                    os.path.splitext(_pyd)[1] in [".pyd", ".so"]][0]

        shutil.move(pyd_path, extension_path)

        # After build_ext is run, the following commands will run:
        # 
        # install_lib
        # install_scripts
        # 
        # These commands are subclassed above to avoid pitfalls that
        # setuptools tries to impose when installing these, as it usually
        # wants to build those libs and scripts as well or move them to a
        # different place. See comments above for additional information

setup(name='my_package',
      version='1.0.0a0',
      packages=find_packages(),
      ext_modules=[CMakeExtension(name="example_extension")],
      description='An example cmake extension module',
      long_description=open("./README.md", 'r').read(),
      long_description_content_type="text/markdown",
      keywords="test, cmake, extension",
      classifiers=["Intended Audience :: Developers",
                   "License :: OSI Approved :: "
                   "GNU Lesser General Public License v3 (LGPLv3)",
                   "Natural Language :: English",
                   "Programming Language :: C",
                   "Programming Language :: C++",
                   "Programming Language :: Python",
                   "Programming Language :: Python :: 3.6",
                   "Programming Language :: Python :: Implementation :: CPython"],
      license='GPL-3.0',
      cmdclass={
          'build_ext': BuildCMakeExt,
          'install_data': InstallCMakeLibsData,
          'install_lib': InstallCMakeLibs,
          'install_scripts': InstallCMakeScripts
          }
    )

一旦以这种方式编写了setup.py，构建python模块就像运行py setup.py一样简单，py setup.py将运行构建并生成外部文件。

建议您在缓慢的internet上为用户或不想从源构建的用户生产轮子。要做到这一点，您将需要安装wheel包(py -m pip install wheel)并通过执行py setup.py bdist_wheel生成一个轮式发行版，然后像其他软件包一样使用twine上传它。

Answer 2

我也遇到了同样的问题，我通过编写一个自定义的setuptools构建命令来解决这个问题，该命令复制了已构建的pyd。用CMake构建和用setuptools构建包似乎比用setuptools构建要干净一些。

这个包在皮皮上

代码在github上

希望这会有帮助。