aws (ValidationException)误差

Question

我希望使用python的boto3模块对dynamodb进行批量写入，并且我得到了这个。这是我第一次使用aws或boto3。文档说，当存在空值和可能不正确的数据类型时，就会出现验证异常错误，但是我已经处理过所有这些，而且它似乎不起作用。

dynamodb一次只想有25个条目写入它吗？如果是的话，我如何控制这些批次呢？

我的请求：

client = boto3.client('dynamodb')
response = client.batch_write_item(RequestItems=batch_dict)

Batch_dict顶部：

{'scraper_exact_urls': [{'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},
 {'PutRequest': {'Item': {'Sku': {'S': 'T104P3'},
 'pps_id': {'N': '427285976'},
 'scraper_class_name': {'S': 'scraper_class_name'},
 'store_id': {'N': '1197386754'},
 'updated_by': {'S': 'user'},
 'updated_on': {'N': '1480714223'},
 'updated_url': {'S': 'http://www.blah.com'}}}},....

模式：

属性：“"scraper_class_name"=>\Aws\DynamoDb\Enum\Type::STRING，”=>\Aws\DynamoDb\Enum\Type：：NUMBER，"sku"=>\Aws\DynamoDb\Enum\Type::STRING，pps_id "store_id"=>\Aws\DynamoDb\Enum\Type::NUMBER，"updated_url"=>\Aws\DynamoDb\Enum\Type::STRING，"updated_by"=>\Aws\DynamoDb\Enum\Type::STRING，"updated_on"=>\Aws\DynamoDb\Enum\Type::NUMBER，字段："pps_id"，"scraper_class_name"，

错误：

ClientError: An error occurred (ValidationException) when calling the    BatchWriteItem operation: 1 validation error detected: Value .... Map value   must satisfy constraint: [Member must have length less than or equal to 25,   Member must have length greater than or equal to 1]

Answer 1

BatchWriteItem API一次处理25个项目。您可以使用来自non-copying batching question的以下代码对25个项块调用BatchWriteItem

def batch(iterable, n=1):
    l = len(iterable)
    for ndx in range(0, l, n):
        yield iterable[ndx:min(ndx + n, l)]

client = boto3.client('dynamodb')

for x in batch(batch_dict['scraper_exact_urls'], 25):
    subbatch_dict = {'scraper_exact_urls': x}
    response = client.batch_write_item(RequestItems=subbatch_dict)

Answer 2

以下是批次写入/删除许多项的Javascript版本，以防有人需要：

const batchWriteManyItems = async (tableName, itemObjs, chunkSize = 25) => {

        const buildParams = (table) => JSON.parse(`{"RequestItems": {"${table}": []}}`)

        const queryChunk = (arr, size) => {
            const tempArr = []
            for (let i = 0, len = arr.length; i < len; i += size) {
                tempArr.push(arr.slice(i, i + size));
            }

            return tempArr
        }

        await Promise.all(queryChunk(itemObjs, chunkSize).map(async chunk => {
            let itemParams = buildParams(tableName);
            itemParams.RequestItems[tableName] = chunk
            await dynamoDB.batchWriteItem(itemParams).promise()
        }))
    }

注意: itemObjs属性可以使用{PutRequest: {Item: ...}作为写项，{DeleteRequest: {Item: ...}可以删除项