GCP Cloud run (使用python)日志记录到云日志

Question

我正在尝试使用cloud run通过HTTPS (流)接收数百万条日志消息，并将它们发送到云日志。

但我发现有一些数据丢失，云日志中的消息数量少于云运行接收。

这是我尝试过的示例代码，

# unzip data
data = gzip.decompress(request.data)

# split by lines
logs = data.decode('UTF-8').split('\n')

# output the logs
log_cnt = 0
for log in logs:
    try:
        # output to jsonPayload
        print(json.dumps(json.loads(log_str))
        log_cnt += 1
    except Exception as e:
        logging.error(F"messsage: {str(e)}")

如果我比较云日志中的log_cnt和日志数量，log_cnt会更多。因此，有些印刷品没有完成传递消息。

我尝试使用日志API代替print()，但日志数量太多，无法使用日志API发送(12000次调用限制为一分钟)，导致延迟非常糟糕，无法稳定地处理请求。

我认为移动的实例数可能会导致这种情况，所以我测试了活动实例未更改时的情况，但仍然有3-5%的消息丢失。

我可以做些什么来将所有消息都发送到云日志中而不会造成任何损失？

(更新版)

数据行看起来像这样(大约1kb)

{"key1": "ABCDEFGHIJKLMN","key2": "ABCDEFGHIJKLMN","key3": "ABCDEFGHIJKLMN","key4": "ABCDEFGHIJKLMN","key5": "ABCDEFGHIJKLMN","key6": "ABCDEFGHIJKLMN","key7": "ABCDEFGHIJKLMN","key8": "ABCDEFGHIJKLMN","key9": "ABCDEFGHIJKLMN","key10": "ABCDEFGHIJKLMN","key11": "ABCDEFGHIJKLMN","key12": "ABCDEFGHIJKLMN","key13": "ABCDEFGHIJKLMN","key14": "ABCDEFGHIJKLMN","key15": "ABCDEFGHIJKLMN","key16": "ABCDEFGHIJKLMN","key17": "ABCDEFGHIJKLMN","key18": "ABCDEFGHIJKLMN","key19": "ABCDEFGHIJKLMN","key20": "ABCDEFGHIJKLMN","key21": "ABCDEFGHIJKLMN","key22": "ABCDEFGHIJKLMN","key23": "ABCDEFGHIJKLMN","key24": "ABCDEFGHIJKLMN","key26": "ABCDEFGHIJKLMN","key27": "ABCDEFGHIJKLMN","key28": "ABCDEFGHIJKLMN","key29": "ABCDEFGHIJKLMN","key30": "ABCDEFGHIJKLMN","key31": "ABCDEFGHIJKLMN","key32": "ABCDEFGHIJKLMN","key33": "ABCDEFGHIJKLMN","key34": "ABCDEFGHIJKLMN","key35": "ABCDEFGHIJKLMN"}

Answer 1

我建议您使用entries.write API，它允许您在同一时间内编写大量条目。在API资源管理器中查看我的测试。

​

​

它可以同时解决JSON格式和多次写入的问题。试一下这个API，如果对你更好，请告诉我。如果没有，我将删除此答案！