Docker-撰写和扩展Dokerfile:找不到外部提供者

Question

我已经部署了apache气流使用码头-合成使用扩展映像，以包括需求。我可以在气流UI中看到雪花提供商包已经安装(apache-气流提供者-雪花)，但是由于某些原因，我得到了一个DAG导入错误：

from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
ModuleNotFoundError: No module named 'airflow.providers.snowflake'

我已经登录到docker，使用了气流用户，并成功地导入了这个包。

Dockerfile：

FROM apache/airflow:2.3.0-python3.9


USER root
COPY requirements.txt .


RUN apt-get update && \
  apt-get install -y apt-utils

USER airflow
RUN pip3 install --no-cache-dir -r requirements.txt

见下文：

[philip@ip-10-5-51-10 airflow_2]$ docker ps
CONTAINER ID   IMAGE                               COMMAND                  CREATED             STATUS                    PORTS                                                 NAMES
0516db2653a0   airflow_2_airflow-webserver         "/usr/bin/dumb-init …"   About an hour ago   Up 52 minutes (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp             airflow_2_airflow-webserver_1
36142cc09a6b   airflow_2_airflow-scheduler         "/usr/bin/dumb-init …"   About an hour ago   Up 52 minutes (healthy)   8080/tcp                                              airflow_2_airflow-scheduler_1
c090e86e9279   airflow_2_airflow-triggerer         "/usr/bin/dumb-init …"   About an hour ago   Up 52 minutes (healthy)   8080/tcp                                              airflow_2_airflow-triggerer_1
ade8996d759f   airflow_2_airflow-worker            "/usr/bin/dumb-init …"   About an hour ago   Up 52 minutes (healthy)   8080/tcp                                              airflow_2_airflow-worker_1
8915e4fb4ffe   airflow_2_flower                    "/usr/bin/dumb-init …"   About an hour ago   Up 52 minutes (healthy)   0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp   airflow_2_flower_1
353f2fa5552f   postgres:13                         "docker-entrypoint.s…"   4 hours ago         Up 52 minutes (healthy)   5432/tcp                                              airflow_2_postgres_1
1bafa42ece77   redis:latest                        "docker-entrypoint.s…"   4 hours ago         Up 52 minutes (healthy)   6379/tcp                                              airflow_2_redis_1
408ad15538ce   denny/monitor-docker-slack:latest   "/monitor-docker-sla…"   12 months ago       Up 3 days                                                                       monitor-docker-slack
[philip@ip-10-5-51-10 airflow_2]$ docker exec -u 0 -it airflow_2_airflow-webserver_1 bash
root@0516db2653a0:/opt/airflow# su - airflow
airflow@0516db2653a0:~$ pip list
Package                                  Version
---------------------------------------- ------------
adal                                     1.2.7
alembic                                  1.7.7
amqp                                     5.1.1
anyio                                    3.5.0
apache-airflow                           2.3.0
apache-airflow-providers-amazon          3.3.0
apache-airflow-providers-celery          2.1.4
apache-airflow-providers-cncf-kubernetes 4.0.1
apache-airflow-providers-docker          2.6.0
apache-airflow-providers-elasticsearch   3.0.3
apache-airflow-providers-ftp             2.1.2
apache-airflow-providers-google          6.8.0
apache-airflow-providers-grpc            2.0.4
apache-airflow-providers-hashicorp       2.2.0
apache-airflow-providers-http            2.1.2
apache-airflow-providers-imap            2.2.3
apache-airflow-providers-microsoft-azure 3.8.0
apache-airflow-providers-mysql           2.2.3
apache-airflow-providers-odbc            2.0.4
apache-airflow-providers-postgres        4.1.0
apache-airflow-providers-redis           2.0.4
apache-airflow-providers-sendgrid        2.0.4
apache-airflow-providers-sftp            2.6.0
apache-airflow-providers-slack           4.2.3
apache-airflow-providers-snowflake       2.6.0
apache-airflow-providers-sqlite          2.1.3
apache-airflow-providers-ssh             2.4.3
apispec                                  3.3.2
argcomplete                              1.12.2
asn1crypto                               1.4.0
attrs                                    20.3.0
Authlib                                  1.0.1
azure-batch                              12.0.0
azure-common                             1.1.28
azure-core                               1.23.1
azure-cosmos                             4.2.0
azure-datalake-store                     0.0.52
azure-identity                           1.9.0
azure-keyvault-secrets                   4.4.0
azure-kusto-data                         0.0.45
azure-mgmt-containerinstance             1.5.0
azure-mgmt-core                          1.3.0
azure-mgmt-datafactory                   1.1.0
azure-mgmt-datalake-nspkg                3.0.1
azure-mgmt-datalake-store                0.5.0
azure-mgmt-nspkg                         3.0.2
azure-mgmt-resource                      21.0.0
azure-nspkg                              3.0.2
azure-storage-blob                       12.8.1
azure-storage-common                     2.1.0
azure-storage-file                       2.1.0
Babel                                    2.9.0
bcrypt                                   3.2.0
beautifulsoup4                           4.11.1
billiard                                 3.6.4.0
blinker                                  1.4
boto3                                    1.16.59
botocore                                 1.19.59
cached-property                          1.5.2
cachelib                                 0.6.0
cachetools                               4.2.2
catboost                                 0.25.1
cattrs                                   1.10.0
celery                                   5.2.6
certifi                                  2020.12.5
cffi                                     1.14.4
chardet                                  3.0.4
charset-normalizer                       2.0.12
click                                    8.1.2
click-didyoumean                         0.3.0
click-plugins                            1.1.1
click-repl                               0.2.0
clickclick                               20.10.2
cloudpickle                              2.0.0
colorama                                 0.4.4
colorlog                                 4.0.2
commonmark                               0.9.1
configparser                             3.5.3
connexion                                2.13.0
cron-descriptor                          1.2.24
croniter                                 0.3.37
cryptography                             3.3.1
cycler                                   0.11.0
dask                                     2022.2.0
db-dtypes                                1.0.0
decorator                                5.1.1
defusedxml                               0.6.0
Deprecated                               1.2.13
dill                                     0.3.3
distlib                                  0.3.4
distributed                              2022.2.0
dnspython                                2.1.0
docker                                   5.0.3
docutils                                 0.16
elasticsearch                            7.13.4
elasticsearch-dbapi                      0.2.9
elasticsearch-dsl                        7.4.0
email-validator                          1.1.2
eventlet                                 0.33.0
filelock                                 3.6.0
Flask                                    1.1.2
Flask-Admin                              1.5.4
Flask-AppBuilder                         3.4.5
Flask-Babel                              1.0.0
Flask-Caching                            1.10.1
Flask-JWT-Extended                       3.25.0
Flask-Login                              0.4.1
Flask-OpenID                             1.2.5
Flask-Session                            0.4.0
Flask-SQLAlchemy                         2.4.4
flask-swagger                            0.2.13
Flask-WTF                                0.14.3
flower                                   1.0.0
fonttools                                4.33.3
fsspec                                   2022.3.0
funcsigs                                 1.0.2
future                                   0.16.0
gevent                                   21.12.0
google-ads                               14.0.0
google-api-core                          2.7.2
google-api-python-client                 1.12.11
google-auth                              2.6.6
google-auth-httplib2                     0.1.0
google-auth-oauthlib                     0.5.1
google-cloud-aiplatform                  1.12.1
google-cloud-appengine-logging           1.1.1
google-cloud-audit-log                   0.2.0
google-cloud-automl                      2.7.2
google-cloud-bigquery                    2.34.3
google-cloud-bigquery-datatransfer       3.6.1
google-cloud-bigquery-storage            2.13.1
google-cloud-bigtable                    1.7.1
google-cloud-build                       3.8.1
google-cloud-container                   1.0.2
google-cloud-core                        2.3.0
google-cloud-datacatalog                 3.7.1
google-cloud-dataplex                    0.2.1
google-cloud-dataproc                    4.0.2
google-cloud-dataproc-metastore          1.5.0
google-cloud-dlp                         1.0.1
google-cloud-kms                         2.11.1
google-cloud-language                    1.3.1
google-cloud-logging                     3.0.0
google-cloud-memcache                    1.3.1
google-cloud-monitoring                  2.9.1
google-cloud-orchestration-airflow       1.3.1
google-cloud-os-login                    2.6.1
google-cloud-pubsub                      2.12.0
google-cloud-redis                       2.8.0
google-cloud-resource-manager            1.4.1
google-cloud-secret-manager              1.0.1
google-cloud-spanner                     1.19.2
google-cloud-speech                      1.3.3
google-cloud-storage                     1.44.0
google-cloud-tasks                       2.8.1
google-cloud-texttospeech                1.0.2
google-cloud-translate                   1.7.1
google-cloud-videointelligence           1.16.2
google-cloud-vision                      1.0.1
google-cloud-workflows                   1.6.1
google-crc32c                            1.3.0
google-resumable-media                   2.3.2
googleapis-common-protos                 1.56.0
graphviz                                 0.16
greenlet                                 1.1.2
grpc-google-iam-v1                       0.12.4
grpcio                                   1.44.0
grpcio-gcp                               0.2.2
grpcio-status                            1.44.0
gunicorn                                 20.1.0
h11                                      0.12.0
HeapDict                                 1.0.1
httpcore                                 0.14.7
httplib2                                 0.19.1
httpx                                    0.22.0
humanize                                 4.0.0
hvac                                     0.11.2
idna                                     2.10
importlib-metadata                       3.10.1
importlib-resources                      5.7.1
inflection                               0.5.1
iso8601                                  0.1.13
isodate                                  0.6.0
itsdangerous                             1.1.0
Jinja2                                   2.10.3
jmespath                                 0.10.0
joblib                                   1.0.0
json-merge-patch                         0.2
jsonpath-ng                              1.5.3
jsonschema                               3.2.0
kiwisolver                               1.4.2
kombu                                    5.2.4
kubernetes                               23.3.0
lazy-object-proxy                        1.4.3
ldap3                                    2.9.1
locket                                   1.0.0
lockfile                                 0.12.2
looker-sdk                               22.4.0
lxml                                     4.8.0
Mako                                     1.1.4
Markdown                                 3.3.4
MarkupSafe                               1.1.1
marshmallow                              3.15.0
marshmallow-enum                         1.5.1
marshmallow-oneofschema                  3.0.1
marshmallow-sqlalchemy                   0.26.1
matplotlib                               3.5.2
msal                                     1.17.0
msal-extensions                          0.3.1
msgpack                                  1.0.3
msrest                                   0.6.21
msrestazure                              0.6.4
mypy-boto3-rds                           1.22.0.post1
mypy-boto3-redshift-data                 1.22.0.post1
mysql-connector-python                   8.0.29
mysqlclient                              2.1.0
natsort                                  7.1.0
nox                                      2020.12.31
numpy                                    1.19.5
oauthlib                                 3.1.0
openapi-spec-validator                   0.2.9
oscrypto                                 1.2.1
packaging                                21.3
pandas                                   0.25.3
pandas-gbq                               0.17.4
paramiko                                 2.10.4
partd                                    1.2.0
pathspec                                 0.9.0
pendulum                                 2.1.2
Pillow                                   9.1.0
pip                                      22.0.4
platformdirs                             2.5.2
plotly                                   5.7.0
pluggy                                   1.0.0
ply                                      3.11
portalocker                              2.4.0
prison                                   0.2.1
prometheus-client                        0.14.1
prompt-toolkit                           3.0.29
proto-plus                               1.18.1
protobuf                                 3.20.1
psutil                                   5.8.0
psycopg2-binary                          2.9.3
py                                       1.11.0
pyarrow                                  6.0.1
pyasn1                                   0.4.8
pyasn1-modules                           0.2.8
pycparser                                2.20
pycryptodomex                            3.9.9
pydata-google-auth                       1.4.0
Pygments                                 2.7.4
PyJWT                                    1.7.1
PyNaCl                                   1.5.0
pyodbc                                   4.0.32
pyOpenSSL                                19.1.0
pyparsing                                2.4.7
pyrsistent                               0.17.3
pysftp                                   0.2.9
python-daemon                            2.3.0
python-dateutil                          2.8.1
python-editor                            1.0.4
python-http-client                       3.3.7
python-ldap                              3.4.0
python-nvd3                              0.15.0
python-slugify                           6.1.1
python3-openid                           3.2.0
pytz                                     2020.5
pytzdata                                 2020.1
PyYAML                                   5.4.1
redis                                    3.5.3
redshift-connector                       2.0.906
requests                                 2.27.1
requests-oauthlib                        1.3.1
rfc3986                                  1.5.0
rich                                     12.3.0
rsa                                      4.8
s3transfer                               0.3.4
scikit-learn                             0.24.1
scipy                                    1.7.3
scramp                                   1.4.1
sendgrid                                 6.9.7
setproctitle                             1.2.1
setuptools                               57.5.0
six                                      1.15.0
sklearn                                  0.0
slack-sdk                                3.15.2
slackclient                              1.0.7
sniffio                                  1.2.0
snowflake-connector-python               2.7.7
snowflake-sqlalchemy                     1.3.4
sortedcontainers                         2.4.0
soupsieve                                2.3.2.post1
SQLAlchemy                               1.4.9
sqlalchemy-bigquery                      1.4.3
SQLAlchemy-JSONField                     1.0.0
sqlalchemy-redshift                      0.8.9
SQLAlchemy-Utils                         0.38.2
sqlparse                                 0.4.1
sshtunnel                                0.4.0
starkbank-ecdsa                          2.0.3
statsd                                   3.3.0
swagger-ui-bundle                        0.0.8
tabulate                                 0.8.7
tblib                                    1.7.0
tenacity                                 8.0.1
termcolor                                1.1.0
text-unidecode                           1.3
threadpoolctl                            3.1.0
thrift                                   0.13.0
toolz                                    0.11.2
tornado                                  6.1
typing_extensions                        4.2.0
tzlocal                                  1.5.1
unicodecsv                               0.14.1
uritemplate                              3.0.1
urllib3                                  1.25.11
vine                                     5.0.0
virtualenv                               20.14.1
watchtower                               2.0.1
wcwidth                                  0.2.5
websocket-client                         0.59.0
Werkzeug                                 1.0.1
wheel                                    0.37.1
wrapt                                    1.14.0
WTForms                                  2.3.3
zict                                     2.1.0
zipp                                     3.4.0
zope.deprecation                         4.4.0
zope.event                               4.5.0
zope.interface                           5.4.0

Answer 1

嗨，我想这个提供程序还没有安装，您能按照一些步骤来确认是否安装了这个吗？

1. 第一次打开cmd并编写停靠程序ps

然后，

1. 复制webserver的集群id，这将类似于

。

在这张图片中，集群id将类似于从39d11开始(这将改变集群id将不同)

复制群集id后，可以将其写入cmd类型的/bin/bash ( docker exec -it -id-it)。

现在，当您在气流集群中时，转到气流 /dag文件夹，然后键入pip list，您可以看到没有提供程序安装，您必须再次在这里安装它，谢谢。