使用云代理的和MS

Question

我正在使用GCP (Composer)上的气流构建一个Cloud (MS )到BigQuery集成。我在GKE集群中设置了一个云SQL代理，它运行良好，没有错误：

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: cloud-sql-proxy
  name: cloud-sql-proxy
  namespace: cloud-sql-to-bq
spec:
  replicas: 1
  selector:
    matchLabels:
      run: cloud-sql-proxy
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        run: cloud-sql-proxy
    spec:
      containers:
      - command:
        - /cloud_sql_proxy
        - -instances=[INSTANCE-NAME]=tcp:0.0.0.0:1433
        image: b.gcr.io/cloudsql-docker/gce-proxy:latest
        imagePullPolicy: IfNotPresent
        name: airflow-sqlproxy
        ports:
        - containerPort: 1433
          protocol: TCP
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      nodeSelector:
        cloud.google.com/gke-nodepool: default-pool
      restartPolicy: Always

我的DAG：

dag = DAG('mssql-export-demo', catchup=False, default_args=default_args)
cloud_storage_bucket_name = 'mssql-export-test'


export_customers = MsSqlToGoogleCloudStorageOperator(
    task_id='export_analysis',
    sql='SELECT * FROM vwAnalysis;',
    bucket=cloud_storage_bucket_name,
    filename='data/customers/export.json',
    schema_filename='schemas/export.json',
    mssql_conn_id='cloud_sql_proxy_conn',
    dag=dag
)

我还在气流中建立了一个连接，指向cloud_sql_proxy_conn.当我运行DAG时，会得到以下错误：

[2020-11-28 01:59:20,555] {taskinstance.py:1153} ERROR - Connection to the database failed for an unknown reason.
Traceback (most recent call last)
  File "src/pymssql.pyx", line 636, in pymssql.connec
  File "src/_mssql.pyx", line 1964, in _mssql.connec
  File "src/_mssql.pyx", line 683, in _mssql.MSSQLConnection.__init_
_mssql.MSSQLDriverException: Connection to the database failed for an unknown reason

没有其他错误消息，因此很难进行调试。有人有使用和Composer的经验来帮助我解决这个问题吗？

Answer 1

气流现在提供CloudSqlInstanceExportOperator，这意味着不需要在GKE中设置云SQL代理。