Kinesis Firehose Putrecordbatch示例

Question

我正在寻找一个使用putrecordbatch发送多个记录到kinesis streams的示例。

我目前正在使用putrecord命令，通过以下方式将记录发送到kinesis streams。

aws firehose put-record --delivery-stream-name csvtoredshiftstreamingjson --record='Data="{\"productid\":1，\"productname\":\"phone\"，\"productprice\":\"2.30\"}"‘

以类似的方式请求帮助来编写putrecordbatch。

谢谢和问候，Srivignesh KN

Answer 1

来自aws firehose put-record-batch help

aws firehose put-record-batch \
        --delivery-stream-name csvtoredshiftstreamingjson \
        --records \
'"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"' \
'"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"' \
'"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"'

应该工作得很好，最多500条记录或命令行缓冲区的限制。

结果将如下所示：

{
  "FailedPutCount": 0, 
  "RequestResponses": [
    {
      "RecordId": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEuIFV0IGVuaW0gYWQgbWluaW0gdmVuaWFtLCBxdWlzIG5vc3RydWQgZXhlcmNpdGF0aW9uIHVsbGFtY28gbGFib3JpcyBuaXNpIHV0IGFsaXF1aXAgZXggZWEgY29tbW9kbyBjb25zZXF1YXQuIER1aXMgYXV0ZSBpcnVyZSBkb2xvciBpbiByZXByZWhlbmRlcml0IGluIHZvbHVwdGF0ZSB2ZWxpdCBlc3NlIGNpbGx1bSBkb2xvcmUgZXUgZnVnaWF0IG51bGxhIHBhcmlhdHVyLiBFeGNlcHRldXIgc2ludCBvY2NhZWNhdCBjdXBpZGF0YXQgbm9uIHByb2lkZW50LCBzdW50IGluIGN1bHBhIHF1aSBvZmZpY2lhIGRlc2VydW50IG1vbGxpdCBhbmltIGlkIGVzdCBsYWJvcnVtLg=="
    }, 
    {
      "RecordId": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEuIFV0IGVuaW0gYWQgbWluaW0gdmVuaWFtLCBxdWlzIG5vc3RydWQgZXhlcmNpdGF0aW9uIHVsbGFtY28gbGFib3JpcyBuaXNpIHV0IGFsaXF1aXAgZXggZWEgY29tbW9kbyBjb25zZXF1YXQuIER1aXMgYXV0ZSBpcnVyZSBkb2xvciBpbiByZXByZWhlbmRlcml0IGluIHZvbHVwdGF0ZSB2ZWxpdCBlc3NlIGNpbGx1bSBkb2xvcmUgZXUgZnVnaWF0IG51bGxhIHBhcmlhdHVyLiBFeGNlcHRldXIgc2ludCBvY2NhZWNhdCBjdXBpZGF0YXQgbm9uIHByb2lkZW50LCBzdW50IGluIGN1bHBhIHF1aSBvZmZpY2lhIGRlc2VydW50IG1vbGxpdCBhbmltIGlkIGVzdCBsYWJvcnVtLg=="
    }, 
    {
      "RecordId": "TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwgc2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduYSBhbGlxdWEuIFV0IGVuaW0gYWQgbWluaW0gdmVuaWFtLCBxdWlzIG5vc3RydWQgZXhlcmNpdGF0aW9uIHVsbGFtY28gbGFib3JpcyBuaXNpIHV0IGFsaXF1aXAgZXggZWEgY29tbW9kbyBjb25zZXF1YXQuIER1aXMgYXV0ZSBpcnVyZSBkb2xvciBpbiByZXByZWhlbmRlcml0IGluIHZvbHVwdGF0ZSB2ZWxpdCBlc3NlIGNpbGx1bSBkb2xvcmUgZXUgZnVnaWF0IG51bGxhIHBhcmlhdHVyLiBFeGNlcHRldXIgc2ludCBvY2NhZWNhdCBjdXBpZGF0YXQgbm9uIHByb2lkZW50LCBzdW50IGluIGN1bHBhIHF1aSBvZmZpY2lhIGRlc2VydW50IG1vbGxpdCBhbmltIGlkIGVzdCBsYWJvcnVtLg=="
    }
  ]
}

Answer 2

我能够通过以下方式解决这个问题。

aws firehose put-record-batch --delivery-stream-name S3TestingStream --records '"{\"productid\":1,\"productname\":\"phone\",\"productprice\":\"2.30\"}"' \
'"{\"productid\":2,\"productname\":\"bus\",\"productprice\":\"5.40\"}"' \
'"{\"productid\":3,\"productname\":\"car\",\"productprice\":\"3.20\"}"'

这为我提供了成功的putrecord之后的记录ids。

   "FailedPutCount": 0, 
"RequestResponses": [
    {
        "RecordId": "N3w6uvXwuxbd9sdXrNI3Kcptcwu5j/oco1wuIjq6vrbz+HAkDEw0N1Rah58pwx6h5dZozBScJ4EFFTQf0SxhH5A0Ie6LGdfJ0OuStB3zb512a2O/bmQretZWGopN3bvzMQO3EIZ3ksNaM63rCIbIT5fI6ZkYzX+rLu8QQtaMCS/881z6qwMGG7n3eJCtWK36uTs4M0X5hEpR1+j8B4hyjdXexu212f4j"
    }, 
    {
        "RecordId": "eWW4ZX9bwyIjVdJ/TQbb+Yh+vJFo/O/CdeT4MXby4T4DNydjoGgoPluQFJbwzFMV2Q6OPWXJblNgjRG3nGVGn73WPQC3AhqWQ3Bqc1ioa2E5yyYGEK1bbk57I4lFN9cdhkkC01POKwz9RszZLkJB38iZsIK4Gh2lp2p65LTPmBWdP0c8Hgd6WOJfrX8IHRhLhc00tuBBi6IM5jCdWRAyLhf45fGhPTee"
    }, 
    {
        "RecordId": "vf8NVYzfGQiCPX0ocRhdhqeTlobYv9FIOacHanqCrkbspHM71Zg36rS5YLu5clIvCSIs6cX6EnaStn5L6s8LgpGQNBBfgr5zk9ydRWNHHF3kkEvqZnPcHU64R4KpkvVhqDTVELqMsFxPfPFEP9Gikzc3d3IqN2EMc+T8fAj7g66nNZgQ3TF3y0SMeZpGyrezDYfLUzGuP/ZzbhCvMr2ypRW51SszGUBN"
    }
]

}