Docker中的Apache NiFi集群超过3个虚拟机

Question

我想在3个虚拟机的docker中创建一个NiFi集群。我找到了一个在一个节点上创建集群的docker-compose文件，并尝试编辑此文件。

我发现我需要zookeeper，但是每个实例需要一个zookeeper吗？我应该在docker中打开或映射哪些端口？

我找到的docker-compose文件：

version: "3"
services:
  zookeeper:
    hostname: zookeeper
    container_name: zookeeper
    image: zookeeper:3.6.1
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  nifi:
    image: apache/nifi:1.11.4
    ports:
      - 8080 # Unsecured HTTP Web Port
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min

我像这样更改了文件(在每个虚拟机上，ip都是正确的)

    version: "3"
    services:
  zookeeper:
    hostname: zookeeper
    container_name: zookeeper
    image: 'zookeeper:3.6.1'
    ports:
      - 2181
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  nifi:
    image: apache/nifi:1.11.4
    ports:
      - 8080 # Unsecured HTTP Web Port
      - 8082
      - 9001
    environment:
      - NIFI_WEB_HTTP_PORT=8080
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082
#      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ZK_CONNECT_STRING=192.168.2.10:2181,192.168.2.20:2181,192.168.2.30:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
      - NIFI_CLUSTER_ADDRESS=192.168.2.XX

在日志中，我发现了这条消息，但找不到任何解决方案

ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Answer 1

我发现我需要zookeeper，但是每个实例需要一个zookeeper吗？

不，你可以使用一个动物园管理员

以及我应该在docker中打开或映射哪些端口？

据我所知，你需要2888,3888,2181才能开通。但是只有2181与nifi 2888和3888通信，用于zookeeper集群通信。