只允许单店kafka中的某些列连接

Question

我使用kafka将debezium收集的cdc数据发送到一个单一商店数据库，并使用这个kafka连接json：

{
"name": "my-connector",
"config": {
    "connector.class":"com.singlestore.kafka.SingleStoreSinkConnector",
    "tasks.max":"1",

    "transforms": "dropPrefix,unwrap",
    "transforms.dropPrefix.type": "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.dropPrefix.regex": "dbserver1.inventory.(.*)",
    "transforms.dropPrefix.replacement": "$1",
    
    "errors.tolerance": "all",
    "errors.log.enable": "true",
    "errors.log.include.messages": "true",

    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",

    "topics":"dbserver1.inventory.addresses",

    "connection.ddlEndpoint" : "memsql:3306",
    "connection.database" : "test",
    "connection.user" : "root",
    "connection.password": "password",


    "insert.mode": "upsert",
    "tableKey.primary.keyName" : "id",
    "fields.whitelist": "id,city",

    "auto.create": "true",
    "auto.evolve": "true",
    "transforms.unwrap.delete.handling.mode":"rewrite",
    "transforms.unwrap.add.fields": "ts_ms",

    "singlestore.metadata.allow": true,
    "singlestore.metadata.table": "kafka_connect_transaction_metadata"


 }
}

我希望singlestore数据库只接收和保存来自id列和city列的数据。但很明显

        "fields.whitelist": "id,city",

不像在jdbc接收器连接器中那样工作在这种kafka连接中。我该怎么处理呢？

Answer 1

又是我，看起来你应该可以使用作为你的CDC工具。它将允许您筛选表中的特定列，然后将插入/更新/删除插入到SingleStore中。https://docs.arcion.io/docs/references/filter-reference/