气流错误-得到一个意想不到的关键字参数'min‘

Question

我试图运行一个非常简单的DAG测试，以获得GCP基本功能的诀窍，但是每次我触发DAG时，都会出现一个严重的错误，我似乎找不到任何关于如何解决它的信息。

错误是：

2020-03-18 22:20:56,627] {taskinstance.py:1059} ERROR - __init__() got an unexpected keyword argument 'min'@-@{"workflow": "notebook-test", "task-id": "notebook-test", "execution-date": "2020-03-18T22:20:41.232043+00:00"}
Traceback (most recent call last):
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 930, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/airflow/airflow/operators/python_operator.py", line 113, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/airflow/airflow/operators/python_operator.py", line 118, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/home/airflow/gcs/dags/test.py", line 44, in execute_nb
    parameters=params
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/execute.py", line 104, in execute_notebook
    **engine_kwargs
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
    return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/engines.py", line 341, in execute_notebook
    nb_man.notebook_start()
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/engines.py", line 69, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/engines.py", line 198, in notebook_start
    self.save()
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/engines.py", line 69, in wrapper
    return func(self, *args, **kwargs)
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/engines.py", line 139, in save
    write_ipynb(self.nb, self.output_path)
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/iorw.py", line 397, in write_ipynb
    papermill_io.write(nbformat.writes(nb), path)
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/iorw.py", line 128, in write
    return self.get_handler(path).write(buf, path)
  File "/opt/python3.6/lib/python3.6/site-packages/papermill/iorw.py", line 316, in write
    multiplier=self.RETRY_MULTIPLIER, min=self.RETRY_DELAY, max=self.RETRY_MAX_DELAY
TypeError: __init__() got an unexpected keyword argument 'min'

我的DAG代码是：

import airflow
import papermill as pm
from datetime import timedelta
from airflow import DAG
from airflow.operators.python_operator import PythonOperator


default_args = {
    'owner': 'airflow',
    'start_date': airflow.utils.dates.days_ago(1),
    'end_date': None,
    'retries': 0,    
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(
    dag_id="notebook-test",
    description="a test",
    default_args=default_args,
    catchup=True,
    schedule_interval=None,
    dagrun_timeout=(timedelta(seconds=30))
)

NB_PATH = "gs://BUCKET/data/"

params = {}


def execute_nb():
    input_nb = NB_PATH + "test.ipynb"
    output_nb = NB_PATH + "test_ran.ipynb"

    pm.execute_notebook(
        input_nb,
        output_nb,
        parameters=params
    )


op = PythonOperator(
    task_id="notebook-test",
    python_callable=execute_nb,
    dag=dag
)

op

我已经从尝试过的一个解决方案是更新坚韧的版本，但是将它添加到Composer环境的PyPi Packages选项卡中并不能解决任何问题。

任何帮助都将不胜感激，谢谢！

编辑：图像版本是composer-1.9.2-气流。

Answer 1

因此，这个问题与所提供的路径有关。

我必须添加import os和from pathlib import Path，然后使我的变量

NB_PATH = str(Path(os.path.abspath(__file__)).parents[1]) + "/data"。

这也要求我将jupyter添加为纸厂正确工作的PyPi依赖项，但它现在似乎正在工作！