我已经在AWS ECS上部署了airflow集群。Airflow Webserver、Scheduler和worker在同一主机上的不同容器下运行。
当我尝试使用Airflow When服务器获取日志时,我得到以下错误:
*** Log file does not exist: /usr/local/airflow/logs/master_populate_uc_crf_ap/populate_ss_ui_schema__country/2021-03-03T09:54:53.271590+00:00/1.log
*** Fetching from: http://ip-10-180-98-37.eu-west-1.compute.internal:8793/log/master_populate/populate_schema/2021-03-03T09:54:53.271590+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='ip-10-180-98-37.eu-west-1.compute.internal', port=8793): Max retries exceeded with url: /log/master_populate/populate_schema/2021-03-03T09:54:53.271590+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f52eae32e10>: Failed to establish a new connection: [Errno -2] Name or service not known'))如果我登录到运行所有三个容器(Webserver、scheduler和worker)的ec2,则可以成功运行以下命令:
curl http://ip-10-180-98-37.eu-west-1.compute.internal:8793/log/master_populate/populate_schema/2021-03-03T09:54:53.271590+00:00/1.log它确认日志正在生成,但Webserver无法从worker访问日志。
如果我登录到Webserver docker容器并运行相同的curl命令,则会发生以下错误:
curl: (6) Could not resolve host: ip-10-180-98-37.eu-west-1.compute.internal如果我将hostname(ip-10-180-98-37.eu-west-1.compute.internal)替换为本地主机,那么curl将在Webserver容器下成功运行。以下是airflow配置变量的值:
AIRFLOW__CORE__HOSTNAME_CALLABLE: socket:getfqdn
executor = SequentialExecutor网络模式为主机。敬请协助。
发布于 2021-03-08 19:13:42
这是因为您的容器无法访问日志文件。这应该与您的网络或安全组配置有关。我建议在您的ECS设置中检查这两个配置。
干杯:)
https://stackoverflow.com/questions/66455897
复制相似问题