如何将Kafka从docker-run命令转换为docker-compose？

Question

我正在构建Kafka CDC，但根据文档，它运行许多docker-run命令。

我想把它全部放到一个docker-compose.yml中，但是我在最后一个命令时失败了，我不能转换成

以下是命令

docker run -d --name postgres \
           -p 5432:5432 \
           -e POSTGRES_USER=start_data_engineer \
           -e POSTGRES_PASSWORD=password debezium/postgres:12

docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.1
docker run -d --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.1

docker run -d --name connect -p 8083:8083 --link kafka:kafka \
                                          --link postgres:postgres \
                             -e BOOTSTRAP_SERVERS=kafka:9092 \
                             -e GROUP_ID=sde_group \
                             -e CONFIG_STORAGE_TOPIC=sde_storage_topic \
                             -e OFFSET_STORAGE_TOPIC=sde_offset_topic debezium/connect:1.1

这是我无法转换的行

docker run -it --rm --name consumer --link zookeeper:zookeeper \
                                    --link kafka:kafka debezium/kafka:1.1 \
               watch-topic -a bankserver1.bank.holding --max-messages 1 | grep '^{' | jq

这是到目前为止我的docker-compose.yml

version: '2'
services:
  zookeeper:
    image: debezium/zookeeper
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: debezium/kafka
    ports:
     - 9092:9092
    links:
     - zookeeper
    environment:
     - ZOOKEEPER_CONNECT=zookeeper:2181
  postgres:
    image: debezium/postgres:9.6
    ports:
     - "5432:5432"
    environment:
     - POSTGRES_USER=user
     - POSTGRES_PASSWORD=password
  connect:
    image: debezium/connect
    ports:
     - 8083:8083
     - 5005:5005
    links:
     - kafka
     - postgres
     - zookeeper
    environment:
     - BOOTSTRAP_SERVERS=kafka:9092
     - GROUP_ID=1
     - CONFIG_STORAGE_TOPIC=my_connect_configs
     - OFFSET_STORAGE_TOPIC=my_connect_offsets
     - STATUS_STORAGE_TOPIC=my_source_connect_statuses
  consumer:
    image: debezium/kafka:1.1
    links:
     - zookeeper
     - kafka
    command: watch-topic -a bankserver1.bank.holding --max-messages 1 | grep '^{' | jq

当我运行docker-compose up时，一切都运行正常。但是consumer总是会失败，并显示以下输出。

The ZOOKEEPER_CONNECT variable must be set, or the container must be linked to one that runs Zookeeper.
consumer_1   | WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations.
consumer_1   | The ZOOKEEPER_CONNECT variable must be set, or the container must be linked to one that runs Zookeeper.

-更新现在我只想阅读和拍摄。首先要确保它正常工作。

稍后，我会有一个源代码来处理这些阅读材料。

docker run -it --rm --name consumer --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.1 watch-topic -a bankserver1.bank.holding | grep --line-buffered '^{' | <your-file-path>/stream.py > my-output/holding_pivot.txt

Answer 1

以下内容将会起作用。

要点是

automatically.

• You
• 我不知道为什么，但是ZOOKEEPER_CONNECT和KAFKA_BROKER没有被设置，必须将命令分解成一个列表。
• 最后，命令还没有在容器内运行。

version: '2'
services:
  zookeeper:
    image: debezium/zookeeper
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: debezium/kafka
    ports:
     - 9092:9092
    environment:
     - ZOOKEEPER_CONNECT=zookeeper:2181
  postgres:
    image: debezium/postgres:9.6
    ports:
     - "5432:5432"
    environment:
     - POSTGRES_USER=user
     - POSTGRES_PASSWORD=password
  connect:
    image: debezium/connect
    ports:
     - 8083:8083
     - 5005:5005
    environment:
     - BOOTSTRAP_SERVERS=kafka:9092
     - GROUP_ID=1
     - CONFIG_STORAGE_TOPIC=my_connect_configs
     - OFFSET_STORAGE_TOPIC=my_connect_offsets
     - STATUS_STORAGE_TOPIC=my_source_connect_statuses
  consumer:
    image: debezium/kafka:1.1
    environment:
     - ZOOKEEPER_CONNECT=zookeeper:2181
     - KAFKA_BROKER=kafka:9092
    command: 
     - watch-topic 
     - -a 
     - bankserver1.bank.holding 
     - --max-messages 
     - "1"

Answer 2

消费者总是失败，并显示此输出。

如错误所示，您需要提供一个ZOOKEEPER_CONNECT。但是，您应该在那里使用entrypoint，而不是command。

在任何情况下，我都不知道Debezium容器是否有Python模块供您通过管道连接到stream.py，或者watch-topic做什么，但是您不需要另一个debezium/kafka容器，因为您可以执行到正在运行的容器中。

docker-compose exec kafka \
  bash -c "watch-topic -a bankserver1.bank.holding | grep --line-buffered '^{' | <your-file-path>/stream.py > my-output/holding_pivot.txt"