群集模式路由网格不起作用，而是默认像主机模式一样工作。

Question

描述

群集模式的路由网格不起作用，相反，它的工作方式就像默认使用主机模式。

我们部署了一个由3个主节点和8个工作节点组成的集群，每个节点都位于使用Terraform和Ansible的云服务OpenStack的不同实例中。蜂群和路由网工作得很好，因为它停止工作，并开始像在主机模式下一样工作。我们没有改变任何东西，也没有做任何更新或部署新服务。我们试图重新启动群集并重新部署群集和所有服务，但是没有任何工作，我们无法使它再次在路由网格模式下工作。因此，我们决定销毁所有实例并从头开始(下面报告的问题)。我们做了一个干净的安装Ubuntu18.04 LTS，和码头，就像我们以前做的那样。然后我们设置一个主节点和两个工作人员(这次是手动的)并部署一个服务，但是群集仍然在主机模式下工作。

访问服务的唯一方法是通过正在运行的节点的IP地址，否则，就没有答案(超时)。我们试图使用管理器或其他工作实例的IP进行访问，但无法访问该服务。这就是为什么我们假设群在默认情况下使用主机模式，而不是入口网络和路由网格。

我们也尝试了不同的服务，如Mongo或Cassandra，但行为是一样的，蜂群看起来像使用主机模式工作。您只能通过使用运行服务的实例IP地址来访问服务。

对如何绕过主机和返回路由网有什么想法吗？我们只想通过使用管理器节点的IP地址来访问任何服务，这些管理节点被认为处于漏模式。

复制问题的步骤：

1. manager sudo docker swarm init --advertise-addr 158.39.201.14
2. worker-0 sudo docker swarm join --token SWMTKN-1-3np0cy0msnfurecckl4863hkftykuqkgeq998s1hix6jsoiarq-758o52hyma iyzv74w3u1yzltt 158.39.201.14:2377
3. worker-1 sudo docker swarm join --token SWMTKN-1-3np0cy0msnfurecckl4863hkftykuqkgeq998s1hix6jsoiarq-758o52hyma iyzv74w3u1yzltt 158.39.201.14:2377
4. manger sudo码头堆叠部署-c码头-Compose.yml nh

描述您收到的结果：

curl http://[worker-0-ip]:8089/bigdata 200 OK

curl http://[worker-1-ip]:8089/bigdata 失效超时

描述了您期望的结果：

curl http://[worker-0-ip]:8089/bigdata 200 OK

curl http://[worker-1-ip]:8089/bigdata 200 OK

您认为重要的附加信息(例如，仅偶尔发生问题)：

这个问题不是两天前发生的，而是突然开始发生的。我们没有做任何修改，也没有碰过服务器。

docker-compose.yml

version: '3.7'

networks:
  news-hunter:
    name: &network news-hunter

x-network: &network-base
  networks:
    - *network

services:
   blazegraph:
    <<: *network-base
    image: lyrasis/blazegraph:2.1.5
    ports:
      - published: 8089
        target: 8080
    deploy:
      placement:
        constraints:
          - node.role == worker 

管理器、worker-1和worker-2的IP表(都是相同的)：sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-INGRESS  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (2 references)
target     prot opt source               destination

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:8089
ACCEPT     tcp  --  anywhere             anywhere             state RELATED,ESTABLISHED tcp spt:8089
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

管理器端口：sudo netstat -tuplen

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        46731      14980/systemd-resol
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          17752      865/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      0          17757      865/sshd
tcp6       0      0 :::8089                 :::*                    LISTEN      0          306971     24992/dockerd
tcp6       0      0 :::2377                 :::*                    LISTEN      0          301970     24992/dockerd
tcp6       0      0 :::7946                 :::*                    LISTEN      0          301986     24992/dockerd
udp        0      0 127.0.0.53:53           0.0.0.0:*                           101        46730      14980/systemd-resol
udp        0      0 158.39.201.14:68        0.0.0.0:*                           100        46591      14964/systemd-netwo
udp        0      0 0.0.0.0:4789            0.0.0.0:*                           0          302125     -
udp6       0      0 fe80::f816:3eff:fef:546 :::*                                100        46586      14964/systemd-netwo
udp6       0      0 :::7946                 :::*                                0          301987     24992/dockerd

工作端口：sudo netstat -tuplen

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        44998      15283/systemd-resol
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          15724      1010/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      0          15726      1010/sshd
tcp6       0      0 :::8089                 :::*                    LISTEN      0          300227     25355/dockerd
tcp6       0      0 :::7946                 :::*                    LISTEN      0          283636     25355/dockerd
udp        0      0 0.0.0.0:4789            0.0.0.0:*                           0          285465     -
udp        0      0 127.0.0.53:53           0.0.0.0:*                           101        44997      15283/systemd-resol
udp        0      0 158.39.201.15:68        0.0.0.0:*                           100        233705     15247/systemd-netwo
udp6       0      0 :::7946                 :::*                                0          283637     25355/dockerd
udp6       0      0 fe80::f816:3eff:fee:546 :::*                                100        48229      15247/systemd-netwo

运行服务：sudo docker service ls

ID                  NAME                MODE                REPLICAS            IMAGE                      PORTS
m7eha88ff4wm        nh_blazegraph       replicated          1/1                 lyrasis/blazegraph:2.1.5   *:8089->8080/tcp

Stack：sudo docker stack ps nh

ID                  NAME                IMAGE                      NODE                DESIRED STATE       CURRENT STATE         ERROR               PORTS
tqkd9t4i03yt        nh_blazegraph.1     lyrasis/blazegraph:2.1.5   nh-worker-0         Running             Running 3 hours ago

docker version**:**的输出

Client: Docker Engine - Community
 Version:           19.03.6
 API version:       1.40
 Go version:        go1.12.16
 Git commit:        369ce74a3c
 Built:             Thu Feb 13 01:27:49 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.6
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.16
  Git commit:       369ce74a3c
  Built:            Thu Feb 13 01:26:21 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker info**:**的输出

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: hpcm67vxrmkm1wvlhfqbjevox
  Is Manager: true
  ClusterID: gnl96swlf7o3a976oarvjrazt
  Managers: 1
  Nodes: 3
  Default Address Pool: 10.0.0.0/8
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 158.39.201.14
  Manager Addresses:
   158.39.201.14:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-74-generic
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 3.852GiB
 Name: nh-manager-0
 ID: PHBO:E6UZ:RNJL:5LVU:OZXW:FM5M:GQVW:SCAQ:EEQW:7IIW:GARL:AUHI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

服务检查sudo docker service inspect --pretty nh_blazegraph

ID:             ef9s5lesysovh5x2653qc6dk9
Name:           nh_blazegraph
Labels:
 com.docker.stack.image=lyrasis/blazegraph:2.1.5
 com.docker.stack.namespace=nh
Service Mode:   Replicated
 Replicas:      1
Placement:
 Constraints:   [node.role == worker]
UpdateConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:   1
 On failure:    pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:         lyrasis/blazegraph:2.1.5@sha256:e9fb46c9d7b2fc23202945a3d71b99ad8df2d7a18dcbcccc04cfc4f791b569e9
Resources:
Networks: news-hunter
Endpoint Mode:  vip
Ports:
 PublishedPort = 8089
  Protocol = tcp
  TargetPort = 8080
  PublishMode = ingress

其他环境详细信息(AWS、VirtualBox、物理等)：

我们正在研究一个OpenStack IaaS云提供商。Out工作负载预计每分钟来自外部源的http请求超过1000次，节点之间的请求超过5000次。

交叉张贴：

https://forums.docker.com/t/swarm-mode-routing-mesh-not-working-instead-is-using-host-mode-by-default/89731 https://github.com/moby/moby/issues/40590

Answer 1

这表明vxlan的覆盖端口在集群中的节点之间被阻塞。vxlan使用的端口如下：

• TCP和UDP端口7946用于节点间通信
• UDP端口4789用于覆盖网络流量

来源：https://docs.docker.com/network/overlay/

显示的iptables表明，这不是在Linux主机本身内完成的(默认情况下，输入和输出策略配置为允许)，因此我将查看用于运行VM的网络策略和系统。例如，VMware NSX使用这些端口并因此阻塞了嵌入式VM。

Answer 2

当然，问题出现在UDP端口4789上。由于一些奇怪的原因，我们的基于IaaS的云提供商阻止了它。我们不知道原因。

但解决方案是更改用于容器入口网络的端口4789 UDP，方法是添加选项--data-path-port，如@BMitch在此related question中解释的那样

停靠群--数据路径-端口5789