pgpool2看门狗在试图使用坞-组合设置连接到对方时崩溃。

Question

我一直试图让pgpool2 +看门狗在本地的码头堆栈中运行。实际上，没有看门狗是非常容易的，而且我能够跟踪文档，并让pgpool在主从模式下执行负载平衡任务。

看门狗带来了麻烦。无论哪个pgpool容器首先启动，最终都会自行选择主节点，因为它首先初始化。好吧没问题。但是，初始化和尝试连接到主节点的任何其他节点都会被彻底拒绝，从而导致它们退出。

(请提前为所有日志和配置文件表示歉意；我将甜点限制在与看门狗相关的部分，但如果请求，我可以添加更多)。

pgpool-1 conf

use_watchdog = on
trusted_servers = 'app'
ping_path = '/bin'
wd_ipc_socket_dir = '/var/run/pgpool'
wd_monitoring_interfaces_list = ''
wd_hostname = '172.31.0.7'
wd_port = 9000
wd_authkey = ''

delegate_IP = ''
ifconfig_path = '/sbin'
if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0'
if_down_cmd = 'ifconfig eth0:0 down'
arping_path = '/usr/sbin'
arping_cmd = 'arping -U $_IP_$ -w 1'
clear_memqcache_on_escalation = on
wd_escalation_command = ''
wd_monitoring_interfaces_list = 'eth0'
wd_lifecheck_method = 'heartbeat'
wd_interval = 10
wd_heartbeat_port = 9694
wd_heartbeat_keepalive = 2
wd_heartbeat_deadtime = 30
heartbeat_destination0 = 'pgpool2'
heartbeat_destination_port0 = 9694
heartbeat_device0 = 'eth0'

wd_life_point = 3
wd_lifecheck_query = 'SELECT 1'
wd_lifecheck_dbname = 'template1'
wd_lifecheck_user = 'nobody'
wd_lifecheck_password = ''

other_pgpool_hostname0 = 'pgpool2'
other_pgpool_port0 = 5432
other_wd_port0 = 9000

pgpool-2 conf

use_watchdog = on
trusted_servers = 'srp'
ping_path = '/bin'
wd_ipc_socket_dir = '/var/run/pgpool'
wd_monitoring_interfaces_list = ''
wd_hostname = '172.31.0.9'
wd_port = 9000
wd_authkey = ''
delegate_IP = ''
ifconfig_path = '/sbin'
if_up_cmd = 'ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0'
if_down_cmd = 'ifconfig eth0:0 down'
arping_path = '/usr/sbin'           # arping command path
arping_cmd = 'arping -U $_IP_$ -w 1'
clear_memqcache_on_escalation = on
wd_escalation_command = ''
wd_monitoring_interfaces_list = 'eth0'
wd_lifecheck_method = 'heartbeat'
wd_interval = 10
wd_heartbeat_port = 9694
wd_heartbeat_keepalive = 2
wd_heartbeat_deadtime = 30
heartbeat_destination0 = 'pgpool1'
heartbeat_destination_port0 = 9694 
heartbeat_device0 = 'eth0'

wd_life_point = 3
wd_lifecheck_query = 'SELECT 1'
wd_lifecheck_dbname = 'template1'
wd_lifecheck_user = 'nobody'
wd_lifecheck_password = ''

other_pgpool_hostname0 = 'pgpool1'
other_pgpool_port0 = 5432
other_wd_port0 = 9000

docker-compose.yml

services:
...

  database:
    container_name: "database"
    image: postgres:9.5-custom
    environment:
      - POSTGRES_PASSWORD=password
      - PGDATA=/var/lib/postgresql/data/pgdata
    volumes:
      - '/var/lib/postgresql/data'
    expose:
      - '5432'
    ports:
      - '5431:5432'
  database-slave:
    container_name: "database-slave"
    image: postgres:9.5-custom
    environment:
      - POSTGRES_PASSWORD=password
      - PGDATA=/var/lib/postgresql/data/pgdata
      - REPLICATE_FROM=database
    volumes:
      - '/var/lib/postgresql/data'
    expose:
     - '5432'
    ports:
      - '5430:5432'
    depends_on:
      - database
  pgpool1:
    container_name: "pgpool1"
    image: pgpool:latest
    depends_on:
      - database
    environment:
      - PGPOOL_BACKENDS=0:database:5432,1:database-slave:5432
      - PGPOOL_SR_CHECK_PASSWORD=password
      - PGPOOL_POOL_PASSWD_USER=someuser
      - PGPOOL_POOL_PASSWD_PASSWORD=password
      - PGPOOL_WD_TRUSTED_SERVERS=app
      - PGPOOL_OTHER_PGPOOL_HOSTNAMES=0:pgpool2:9000
      - PGPOOL_WD_HEARTBEAT_DESTINATIONS=0:pgpool2:eth0
      - SLEEP=10
    expose:
      - '9000'
      - '5432'
      - '9694'

  pgpool2:
    container_name: "pgpool2"
    image: pgpool:latest
    depends_on:
      - database
    environment:
      - PGPOOL_BACKENDS=0:database:5432,1:database-slave:5432
      - PGPOOL_SR_CHECK_PASSWORD=password
      - PGPOOL_POOL_PASSWD_USER=someuser
      - PGPOOL_POOL_PASSWD_PASSWORD=password
      - PGPOOL_WD_TRUSTED_SERVERS=app
      - PGPOOL_OTHER_PGPOOL_HOSTNAMES=0:pgpool1:9000
      - PGPOOL_WD_HEARTBEAT_DESTINATIONS=0:pgpool1:eth0
      - SLEEP=15
    expose:
      - '9000'
      - '5432'
      - '9694'

pgpool-1 (主)输出：

2018-10-01 18:00:56: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:00:56: pid 21: DETAIL:  num_backends: 2 total_weight: 2.000000
2018-10-01 18:00:56: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:00:56: pid 21: DETAIL:  backend 0 weight: 1073741823.500000 flag: 0000
2018-10-01 18:00:56: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:00:56: pid 21: DETAIL:  backend 1 weight: 1073741823.500000 flag: 0000
2018-10-01 18:00:56: pid 21: DEBUG:  loading hba configuration
2018-10-01 18:00:56: pid 21: DETAIL:  loading file :"/etc/pgpool2/pool_hba.conf" for client authentication configuration file
2018-10-01 18:00:56: pid 21: LOG:  Backend status file /var/log/postgresql/pgpool_status does not exist
2018-10-01 18:00:56: pid 21: DEBUG:  pool_coninfo_size: num_init_children (32) * max_pool (4) * MAX_NUM_BACKENDS (128) * sizeof(ConnectionInfo) (136) = 2228224 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: DEBUG:  ProcessInfo: num_init_children (32) * sizeof(ProcessInfo) (32) = 1024 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: DEBUG:  Request info are: sizeof(POOL_REQUEST_INFO) 5304 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: DEBUG:  Recovery management area: sizeof(int) 4 bytes requested for shared memory
2018-10-01 18:00:56: pid 21: LOG:  waiting for watchdog to initialize
2018-10-01 18:00:56: pid 22: LOG:  setting the local watchdog node name to "172.31.0.8:5432 Linux 12affe97a14d"
2018-10-01 18:00:56: pid 22: LOG:  watchdog cluster is configured with 1 remote nodes
2018-10-01 18:00:56: pid 22: LOG:  watchdog remote node:0 on pgpool2:9000
2018-10-01 18:00:56: pid 22: DEBUG:  adding monitoring interface [0] name eth0 index 1950
2018-10-01 18:00:56: pid 22: INFO:  IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
2018-10-01 18:00:56: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:00:56: pid 22: DEBUG:  network interface tunl0 having flags 128
2018-10-01 18:00:56: pid 22: DEBUG:  network interface ip6tnl0 having flags 128
2018-10-01 18:00:56: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:00:56: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:00:56: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:00:56: pid 22: DEBUG:  network interface "eth0" is up and we can continue
2018-10-01 18:00:56: pid 22: LOG:  watchdog node state changed from [DEAD] to [LOADING]
2018-10-01 18:00:56: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = LOADING
2018-10-01 18:00:56: pid 22: DEBUG:  error in outbound connection to pgpool2:9000
2018-10-01 18:00:56: pid 22: DETAIL:  Connection refused
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = TIMEOUT Current State = LOADING
2018-10-01 18:01:01: pid 22: LOG:  watchdog node state changed from [LOADING] to [JOINING]
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = JOINING
2018-10-01 18:01:01: pid 22: DEBUG:  error in outbound connection to pgpool2:9000
2018-10-01 18:01:01: pid 22: DETAIL:  Connection refused
2018-10-01 18:01:01: pid 22: LOG:  new watchdog node connection is received from "172.31.0.9:27844"
2018-10-01 18:01:01: pid 22: DEBUG:  received watchdog packet type:A
2018-10-01 18:01:01: pid 22: DEBUG:  reading packet type A of length 242
2018-10-01 18:01:01: pid 22: DEBUG:  ADD NODE MESSAGE from hostname:"172.31.0.9" port:9000 pgpool_port:5432
2018-10-01 18:01:01: pid 22: NOTICE:  add node from hostname:"172.31.0.9" port:9000 pgpool_port:5432 rejected.
2018-10-01 18:01:01: pid 22: DETAIL:  verify the other watchdog node configurations
2018-10-01 18:01:01: pid 22: DEBUG:  sending packet, watchdog node:[�
                                                                     WaVU] command id:[4] type:[REJECT] state:[JOINING]
2018-10-01 18:01:01: pid 22: DEBUG:  sending watchdog packet to socket:7, type:[R], command ID:4, data Length:0

pgpool-2 (崩溃)输出：

2018-10-01 18:01:01: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:01:01: pid 21: DETAIL:  num_backends: 2 total_weight: 2.000000
2018-10-01 18:01:01: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:01:01: pid 21: DETAIL:  backend 0 weight: 1073741823.500000 flag: 0000
2018-10-01 18:01:01: pid 21: DEBUG:  initializing pool configuration
2018-10-01 18:01:01: pid 21: DETAIL:  backend 1 weight: 1073741823.500000 flag: 0000
2018-10-01 18:01:01: pid 21: DEBUG:  loading hba configuration
2018-10-01 18:01:01: pid 21: DETAIL:  loading file :"/etc/pgpool2/pool_hba.conf" for client authentication configuration file
2018-10-01 18:01:01: pid 21: LOG:  Backend status file /var/log/postgresql/pgpool_status does not exist
2018-10-01 18:01:01: pid 21: DEBUG:  pool_coninfo_size: num_init_children (32) * max_pool (4) * MAX_NUM_BACKENDS (128) * sizeof(ConnectionInfo) (136) = 2228224 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: DEBUG:  ProcessInfo: num_init_children (32) * sizeof(ProcessInfo) (32) = 1024 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: DEBUG:  Request info are: sizeof(POOL_REQUEST_INFO) 5304 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: DEBUG:  Recovery management area: sizeof(int) 4 bytes requested for shared memory
2018-10-01 18:01:01: pid 21: LOG:  waiting for watchdog to initialize
2018-10-01 18:01:01: pid 22: LOG:  setting the local watchdog node name to "172.31.0.9:5432 Linux b251a3a36756"
2018-10-01 18:01:01: pid 22: LOG:  watchdog cluster is configured with 1 remote nodes
2018-10-01 18:01:01: pid 22: LOG:  watchdog remote node:0 on pgpool1:9000
2018-10-01 18:01:01: pid 22: DEBUG:  adding monitoring interface [0] name eth0 index 1952
2018-10-01 18:01:01: pid 22: INFO:  IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
2018-10-01 18:01:01: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 22: DEBUG:  network interface tunl0 having flags 128
2018-10-01 18:01:01: pid 22: DEBUG:  network interface ip6tnl0 having flags 128
2018-10-01 18:01:01: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 22: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 22: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 22: DEBUG:  network interface "eth0" is up and we can continue
2018-10-01 18:01:01: pid 22: LOG:  watchdog node state changed from [DEAD] to [LOADING]
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = STATE CHANGED Current State = LOADING
2018-10-01 18:01:01: pid 22: LOG:  new outbound connection to pgpool1:9000
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = NEW OUTBOUND_CONNECTION Current State = LOADING
2018-10-01 18:01:01: pid 22: DEBUG:  sending packet, watchdog node:[] command id:[4] type:[ADD NODE] state:[LOADING]
2018-10-01 18:01:01: pid 22: DEBUG:  sending watchdog packet to socket:7, type:[A], command ID:4, data Length:242
2018-10-01 18:01:01: pid 22: DEBUG:  received watchdog packet type:R
2018-10-01 18:01:01: pid 22: DEBUG:  reading packet type R of length 0
2018-10-01 18:01:01: pid 22: DEBUG:  STATE MACHINE INVOKED WITH EVENT = PACKET RECEIVED Current State = LOADING
2018-10-01 18:01:01: pid 22: DEBUG:  received packet, watchdog node:[] command id:[4] type:[REJECT] state:[LOADING]
2018-10-01 18:01:01: pid 22: FATAL:  Add to watchdog cluster request is rejected by node "pgpool1:9000"
2018-10-01 18:01:01: pid 22: HINT:  check the watchdog configurations.
2018-10-01 18:01:01: pid 22: LOG:  Watchdog is shutting down
2018-10-01 18:01:01: pid 21: DEBUG:  reaper handler
2018-10-01 18:01:01: pid 21: DEBUG:  watchdog child process with pid: 22 exit with FATAL ERROR. pgpool-II will be shutdown
2018-10-01 18:01:01: pid 21: LOG:  watchdog child process with pid: 22 exits with status 768
2018-10-01 18:01:01: pid 21: FATAL:  watchdog child process exit with fatal error. exiting pgpool-II
2018-10-01 18:01:01: pid 23: LOG:  setting the local watchdog node name to "172.31.0.9:5432 Linux b251a3a36756"
2018-10-01 18:01:01: pid 23: LOG:  watchdog cluster is configured with 1 remote nodes
2018-10-01 18:01:01: pid 23: LOG:  watchdog remote node:0 on pgpool1:9000
2018-10-01 18:01:01: pid 23: DEBUG:  adding monitoring interface [0] name eth0 index 1952
2018-10-01 18:01:01: pid 23: INFO:  IPC socket path: "/var/run/pgpool/.s.PGPOOLWD_CMD.9000"
2018-10-01 18:01:01: pid 23: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 23: DEBUG:  network interface tunl0 having flags 128
2018-10-01 18:01:01: pid 23: DEBUG:  network interface ip6tnl0 having flags 128
2018-10-01 18:01:01: pid 23: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 23: DEBUG:  network interface lo having flags 65609
2018-10-01 18:01:01: pid 23: DEBUG:  network interface eth0 having flags 69699
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is active
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" link is up
2018-10-01 18:01:01: pid 23: DEBUG:  network interface "eth0" is up and we can continue

Pgpo-2在最后一个日志条目的点处退出。

我尝试了什么，我已经确认了什么

• 尝试以root方式运行pgpool，尝试以postgres的形式运行pgpool
• 尝试过wd_authkey，但没有尝试SSL
• 尝试使用docker privileged: true运行
• 尝试显式定义ports:，而不是仅仅公开它们，这样对接者网络就不需要将随机端口映射到指定的公开端口
• 尝试使用通配符listen_address，尝试将主机ip作为listen_address (此ip是从码头容器中派生的，因此我不确定这是否有效
• 尝试使用和不使用定义的wd_heartbeat设置
• 区分了两个进程的pgpool.conf，如果我将其设置为非通配符，则唯一不同的设置是other_wd_hosts、wd_hostname和listen_address。

在这一点上，我已经走开了几次，用一个清晰的头脑重新审视，我仍然不知道为什么我看到这种行为。我在这里漏掉的任何东西都会让主看门狗拒绝连接吗？

Answer 1

哦，天哪，在发这篇文章几分钟后，我就翻阅了一遍我经常检查的输出，我没有留下任何服务器名，帐号，或者类似的东西。我解决了问题。

tl;dr：wd_hostname和other_pgpool_hostname设置必须在不同的pgpool节点之间进行匹配。other_pgpool_hostname基本上是一个白名单设置。

我注意到这句话：

2018-10-01 18:01:01: pid 23: LOG: setting the local watchdog node name to "172.31.0.9:5432 Linux b251a3a36756"

我想知道将wd_hostname设置为IP地址是否有问题，因为other_pgpool_hostname设置没有被设置为IP，而是设置为docker-compose.yml服务名称(这是正确的)。

我更改了配置，将wd_hostname设置为匹配给定容器的docker-compose.yml服务名称，并神奇地将所有内容连接在一起。