如何使用AWS S3 put对象绕过SlowDown错误

Question

我正在尝试访问一个文件，并使用boto在S3中更新它，但仍然得到减速错误，即使在请求之间暂停之后，根据下面的代码。我该如何解决这个问题呢？

body = b'Here we have some more data'
s3.put_object(Body=body,Bucket=bucket, Key=key)
time.sleep(10)
response = s3.get_object(Bucket=bucket, Key=key)
time.sleep(10)
print(response["Body"].read().decode('utf-8'))
currFile = response["Body"].read().decode('utf-8')
newFile = currFile + "\n" + "New Stuff!!!"
newFileB = newFile.encode('utf-8')
time.sleep(60)
s3.put_object(Body=newFileB,Bucket=bucket, Key=key)
time.sleep(10)
response = s3.get_object(Bucket=bucket, Key=key)
print(response["Body"].read().decode('utf-8'))

下面是错误：

Details
The area below shows the result returned by your function execution.
{
"errorMessage": "An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.",
"errorType": "ClientError",
"stackTrace": [
[
"/var/task/lambda_function.py",
43,
"lambda_handler",
"raise e"
],
[
"/var/task/lambda_function.py",
20,
"lambda_handler",
"s3.put_object(Body=body,Bucket=bucket, Key=key)"
],
[
"/var/runtime/botocore/client.py",
314,
"_api_call",
"return self._make_api_call(operation_name, kwargs)"
],
[
"/var/runtime/botocore/client.py",
612,
"_make_api_call",
"raise error_class(parsed_response, operation_name)"
]
]
}

Answer 1

我确实遇到了这个问题，我不确定为什么会发生这种情况，但这是生产代码必须一直处理的事情。解决方案就是不断尝试，并在相当长的一段时间内不放弃。在失败时，循环以1秒的延迟开始，然后在每个循环中将延迟增加1秒(delay_incr)，最后在最终放弃时，每个循环30秒的max_delay超时，这实际上是最大总计7.5分钟。当然，您可以旋转计时。到目前为止，这对我来说是成功的。

即使对于NAS文件服务器，我也必须做类似的事情，有时我必须等待一段时间才能读取文件。

def put_s3_core(bucket, key, strobj, content_type=None):  
    """ write strobj to s3 bucket, key
        content_type can be:
            binary/octet-stream (default)
            text/plain
            text/html
            text/csv
            image/png
            image/tiff
            application/pdf
            application/zip
            
    """
    delay = 1       # initial delay
    delay_incr = 1  # additional delay in each loop
    max_delay = 30  # max delay of one loop. Total delay is (max_delay**2)/2
    
    while delay < max_delay:   
        try:
            s3 = boto3.resource('s3')
            request_obj = s3.Object(bucket, key)
            if content_type:
                request_obj.put(Body=strobj, ContentType=content_type)        
            else:
                request_obj.put(Body=strobj)
            break               
                
        except ClientError:
            time.sleep(delay)
            delay += delay_incr
    else:
        raise
    ```