Apache Airflow DockerOperator找不到docker模块

Question

因此，我正在尝试使用DockerOperator运行一个简单的dag。

我在我的conda环境中使用以下模块运行命令airflow webserver：

Package                Version
---------------------- -------------------
alembic                1.4.2
apache-airflow         1.10.10
apispec                1.3.3
argcomplete            1.11.1
attrs                  19.3.0
Babel                  2.8.0
cached-property        1.5.1
cattrs                 0.9.2
certifi                2020.4.5.1
chardet                3.0.4
click                  7.1.2
colorama               0.4.3
colorlog               4.0.2
configparser           3.5.3
croniter               0.3.32
defusedxml             0.6.0
dill                   0.3.1.1
dnspython              1.16.0
docker                 3.7.3
docker-pycreds         0.4.0
docutils               0.16
email-validator        1.1.1
Flask                  1.1.2
Flask-Admin            1.5.4
Flask-AppBuilder       2.3.4
Flask-Babel            1.0.0
Flask-Caching          1.3.3
Flask-JWT-Extended     3.24.1
Flask-Login            0.4.1
Flask-OpenID           1.2.5
Flask-SQLAlchemy       2.4.3
flask-swagger          0.2.13
Flask-WTF              0.14.3
funcsigs               1.0.2
future                 0.18.2
graphviz               0.14
gunicorn               19.10.0
idna                   2.9
iso8601                0.1.12
itsdangerous           1.1.0
Jinja2                 2.10.3
json-merge-patch       0.2
jsonschema             3.2.0
lazy-object-proxy      1.5.0
lockfile               0.12.2
Mako                   1.1.3
Markdown               2.6.11
MarkupSafe             1.1.1
marshmallow            2.21.0
marshmallow-enum       1.5.1
marshmallow-sqlalchemy 0.23.1
natsort                7.0.1
numpy                  1.18.5
pandas                 0.25.3
pendulum               1.4.4
pip                    20.0.2
prison                 0.1.3
psutil                 5.7.0
Pygments               2.6.1
PyJWT                  1.7.1
pyrsistent             0.16.0
python-daemon          2.1.2
python-dateutil        2.8.1
python-editor          1.0.4
python3-openid         3.1.0
pytz                   2020.1
pytzdata               2019.3
PyYAML                 5.3.1
requests               2.23.0
setproctitle           1.1.10
setuptools             47.1.1.post20200604
six                    1.15.0
SQLAlchemy             1.3.17
SQLAlchemy-JSONField   0.9.0
SQLAlchemy-Utils       0.36.6
tabulate               0.8.7
tenacity               4.12.0
termcolor              1.1.0
text-unidecode         1.2
thrift                 0.13.0
tzlocal                1.5.1
unicodecsv             0.14.1
urllib3                1.25.9
websocket-client       0.57.0
Werkzeug               0.16.1
wheel                  0.34.2
WTForms                2.3.1
zope.deprecation       4.4.0

执行此命令时，在尝试导入docker时，会在docker_operator.py文件中拾取一个错误。

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 236, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/ben.muller/airflow/dags/nba_dag.py", line 5, in <module>
    from airflow.operators.docker_operator import DockerOperator
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/docker_operator.py", line 25, in <module>
    from docker import APIClient, tls
ModuleNotFoundError: No module named 'docker'

我不明白这是怎么发生的，就好像我在我的终端中打开一个简单的python REPL并导入docker没有问题一样。

在本地运行"airflow console on localhost:8080“时也会出现此错误。

​

​

有谁知道如何克服这个问题吗？

Answer 1

坞站模块和Airflow安装在两个不同的环境中。坞站模块安装在虚拟环境中，Airflow安装在系统范围内(/usr/local/lib/python3.7/site-packages/airflow)。

在整个系统范围内安装坞站模块，或者在与坞站模块相同的虚拟环境中安装Airflow。