TensorFlow如何通过array_ops.cc生成gen_array_ops.py？

Question

TensorFlow自动生成代码。我很好奇TF是如何通过array_ops.cc生成gen_array_ops.py的？

生成的python文件位于python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py

"""Python wrappers around TensorFlow ops.

This file is MACHINE GENERATED! Do not edit.
Original C++ source file: array_ops.cc
"""
...
...

Answer 1

Python代码生成是在构建时通过Bazel完成的。你可以在tensorflow/tensorflow.bzl中找到相关的定义，我将在这里发布标题：

# Generates a Python library target wrapping the ops registered in "deps".
#
# Args:
#   name: used as the name of the generated target and as a name component of
#     the intermediate files.
#   out: name of the python file created by this rule. If None, then
#     "ops/gen_{name}.py" is used.
#   hidden: Optional list of ops names to make private in the Python module.
#     It is invalid to specify both "hidden" and "op_whitelist".
#   visibility: passed to py_library.
#   deps: list of dependencies for the intermediate tool used to generate the
#     python target. NOTE these `deps` are not applied to the final python
#     library target itself.
#   require_shape_functions: leave this as False.
#   hidden_file: optional file that contains a list of op names to make private
#     in the generated Python module. Each op name should be on a line by
#     itself. Lines that start with characters that are invalid op name
#     starting characters are treated as comments and ignored.
#   generated_target_name: name of the generated target (overrides the
#     "name" arg)
#   op_whitelist: if not empty, only op names in this list will be wrapped. It
#     is invalid to specify both "hidden" and "op_whitelist".
#   cc_linkopts: Optional linkopts to be added to tf_cc_binary that contains the
#     specified ops.

def tf_gen_op_wrapper_py(
        name,
        out = None,
        hidden = None,
        visibility = None,
        deps = [],
        require_shape_functions = False,
        hidden_file = None,
        generated_target_name = None,
        op_whitelist = [],
        cc_linkopts = [],
        api_def_srcs = []):
    # ...

这是通过您可以在tensorflow/python/build_defs.bzl中找到的tf_gen_op_wrapper_private_py间接调用的。对于array_ops，您可以在tensorflow/python/BUILD中找到它

tf_gen_op_wrapper_private_py(
    name = "array_ops_gen",
    visibility = [
        "//learning/brain/python/ops:__pkg__",
        "//tensorflow/compiler/tests:__pkg__",
        "//tensorflow/contrib/quantization:__pkg__",
        "//tensorflow/python/kernel_tests:__pkg__",
    ],
)

这条规则是做什么的？它调用一个你可以在tensorflow/python/framework/python_op_gen_main.cc找到的源程序(这是主要的入口点，它使用其他相邻的源文件)。本质上，它是一个程序，它执行通过Python宏(在tensorflow/core/framework/op.h中定义)注册的操作，并相应地生成REGISTER_OP代码。我现在不能详细介绍，但是如果你想知道细节，你应该可以浏览代码。