在Google的气流实例上安装apache- airflow -backport-providers-google模块

Question

我需要从Composer中执行数据融合管道，为此使用Data：

from airflow.providers.google.cloud.operators.datafusion import (
    CloudDataFusionCreateInstanceOperator,
    CloudDataFusionCreatePipelineOperator,
    CloudDataFusionDeleteInstanceOperator,
    CloudDataFusionDeletePipelineOperator,
    CloudDataFusionGetInstanceOperator,
    CloudDataFusionListPipelinesOperator,
    CloudDataFusionRestartInstanceOperator,
    CloudDataFusionStartPipelineOperator,
    CloudDataFusionStopPipelineOperator,
    CloudDataFusionUpdateInstanceOperator,
)

我遇到的问题是关于模块“apache-airflow-google”，在这个链接的支持下，我知道我需要使用这个模块：

1. 在气流实例中安装模块的参考( @Gonzalo Pérez Fernández回答)：https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/datafusion.html

当我试图在编写器(如PyPi包)上安装python依赖项时，我得到了以下错误：

UPDATE operation on this environment failed 7 minutes ago with the following error message:
Failed to install PyPI packages. 
apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0. Check the Cloud Build log at https://console.cloud.google.com/cloud-build/builds/a2ecf37a-4c47-4770-9489-6fb65e87d82f?project=341768372632 for details. For detailed instructions see https://cloud.google.com/composer/docs/troubleshooting-package-installation

原木桶是：

apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.
apache-airflow-backport-providers-google 2021.3.3 has requirement apache-airflow~=1.10, but you have apache-airflow 2.1.2+composer.
The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION  fail' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

是否有任何方法可以使用de模块"apache-airflow-backport-providers-google“，而不会在composer实例中出现缺乏灵活性的问题?或者，使用数据融合操作符而不需要更改或解析python中的包版本的最佳方法是什么？

作曲家图像版本: Composer -1.17.0-气流-2.1.2

谢谢。

Answer 1

没有必要在气流apache-airflow-backport-providers-google中安装2.0+。这个包实际上支持气流2操作员进入气流1.10.*。此外，在Composer版本的composer-1.17.0-airflow-2.1.2中，已经按照文档安装了apache-airflow-providers-google==5.0.0包。您应该能够使用您发布的代码片段导入数据融合操作符。

但是，如果不是这样的话，您可能应该在尝试重新安装apache-airflow-providers-google==5.0.0时处理日志中显示的冲突。

apache-airflow-providers-google 5.0.0 has requirement google-ads>=12.0.0, but you have google-ads 7.0.0.

您可以在您的google-ads=12.0.0依赖项中添加对PyPi的需求，并查看它是否有效。