云函数遇到异常后无法继续运行

Question

所以我做了一个云函数，负责通过SFTP服务器读取文件，然后将它们推送到Google BigQuery。目录中一次有4-5个相关文件，有时会出错。为了弥补他们的错误，我在try-except块中读取它。它在本地工作，当脚本在Compute Engine实例上运行时(在我尝试将其移动到Cloud Functions之前，它被托管在那里。在Cloud Functions上，它在第一个文件遇到错误后停止运行。附件是日志截图以及。我已经尝试增加函数和分配给它的内存的时间限制，但是没有帮助。

整个脚本：

import pysftp
import json
import pandas as pd
from google.cloud import bigquery
def push_files():
  client = bigquery.Client()
  remote_path='/home/user/s/'
  cnopts = pysftp.CnOpts()
  cnopts.hostkeys = None    
  sftp=pysftp.Connection('host_ip',  private_key='private_key.pem' , username = 'user', cnopts=cnopts)
  files = [file for file in sftp.listdir(remote_path) if not file.__contains__('.ipynb')
        and not file.__contains__('sgp') and not file.__contains__('booking')
        and not file.__contains__('web') and not file.__contains__('SHIPPING')
        and not file.__contains__('addresses_v2') and not file.__contains__('fc_cust')]

  print(json.dumps(dict(
            severity="NOTICE",
            message=f"These are files to push {files}",)))

  for file in files:
    try:
      df = pd.read_csv(sftp.open(f"{remote_path}/{file}"), sep='\t', engine='python')
      table_id = f"sftp_data.{file}"
      job = client.load_table_from_dataframe(
          df, table_id
      )

      # Wait for the load job to complete.
      job.result()
      print(json.dumps(dict(
            severity="NOTICE",
            message=f"pushed {file}",)))
                        
    except Exception as e:
      print(json.dumps(dict(
            severity="NOTICE",
            message=f"could not push {file} because of {e.__str__()}",)))

  return "Pushed Files!"


def main(request):
  print('request: recieved')
  response = push_files()
  return f'Executed Request:{request} and {response}'

下面是云函数的日志：

​

Answer 1

您需要一个continue来恢复循环

for file in files:
    try:
      df = pd.read_csv(sftp.open(f"{remote_path}/{file}"), sep='\t', engine='python')
      table_id = f"sftp_data.{file}"
      job = client.load_table_from_dataframe(
          df, table_id
      )

      # Wait for the load job to complete.
      job.result()
      print(json.dumps(dict(
            severity="NOTICE",
            message=f"pushed {file}",)))
                        
    except Exception as e:
      print(json.dumps(dict(
            severity="NOTICE",
            message=f"could not push {file} because of {e.__str__()}",)))
      continue