etcd2不会通过CoreOS上的systemctl启动

Question

真希望有人能帮我。我把CoreOS etcd2成员的磁盘填满了。在重新启动之后，etcd2陷入了一种糟糕的境地。最后，我删除了/var/lib/etcd2 2/member数据目录，并按照下面的详细说明将机器移除并重新添加到集群中：https://coreos.com/etcd/docs/latest/runtime-configuration.html#remove-a-member

但是，虽然我可以手动运行etcd2，但是尝试用systemctl启动它是行不通的。这是This 2.服务：

[Unit]
Description=etcd2
Conflicts=etcd.service

[Service]
User=etcd
Type=notify
Environment=ETCD_DATA_DIR=/var/lib/etcd2
Environment=ETCD_NAME=%m
ExecStart=/usr/bin/etcd2
Restart=always
RestartSec=10s
LimitNOFILE=40000
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

# /run/systemd/system/etcd2.service.d/10-oem.conf
[Service]
Environment=ETCD_ELECTION_TIMEOUT=1200
# /run/systemd/system/etcd2.service.d/20-cloudinit.conf
[Service]
Environment="ETCD_ADVERTISE_CLIENT_URLS=http://172.31.9.22:2379,http://172.31.9.22:4001"
Environment="ETCD_DISCOVERY=https://discovery.etcd.io/567d080563e28e62cf886e48425f632b"
Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.31.9.22:2380"
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379"
Environment="ETCD_LISTEN_PEER_URLS=http://172.31.9.22:2380"
Environment="ETCD_DEBUG=true"

我添加了ETCD_DEBUG=true以尝试改进日志记录输出。说到这一点：

Feb 26 07:23:36 geo-coreos-database-02 systemd[1]: Starting etcd2...
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://172.31.9.22:2379,http://172.31.9.22:4001
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_DEBUG=true
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_DISCOVERY=https://discovery.etcd.io/567d080563e28e62cf886e48425f632b
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_ELECTION_TIMEOUT=1200
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.31.9.22:2380
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Feb 26 07:23:36 geo-coreos-database-02 etcd2[2939]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://172.31.9.22:2380
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: etcd2.service: Main process exited, code=exited, status=1/FAILURE
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: Failed to start etcd2.
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: etcd2.service: Unit entered failed state.
Feb 26 07:23:37 geo-coreos-database-02 systemd[1]: etcd2.service: Failed with result 'exit-code'.

不是很有帮助。但是，当我手动运行它时，将配置建立在runs 2服务的基础上，服务器在前台启动并运行，没有问题：

export ETCD_NAME="e0a8edc41f634fcf9451b5c68e3442bd"
export ETCD_DATA_DIR=/var/lib/etcd2 
export ETCD_ADVERTISE_CLIENT_URLS=http://172.31.9.22:2379,http://172.31.9.22:4001
export ETCD_DISCOVERY=https://discovery.etcd.io/567d080563e28e62cf886e48425f632b
export ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.31.9.22:2380
export ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
export ETCD_LISTEN_PEER_URLS=http://172.31.9.22:2380
export ETCD_DEBUG=true
etcd2 

这将启动服务器，正如我所期望的那样，通过调试，我甚至可以在服务器上运行etcdctl命令。对于如何进一步调试这一点，我完全不知道。我绝对不想为如此琐碎的事情创建一个新的集群，但这似乎总是使用etcd的方式，这是唯一对我们造成问题的东西，它们常常是这样的，晦涩难懂，很难修复。

Answer 1

当然，在我发问之后，我就会明白这一点。当我第一次手动运行这个命令时，我是以根用户的身份执行的。因此，数据目录属于root用户，而不是etcd用户。更改权限解决了问题。即使在调试模式下，它也不记录这一点，这仍然很糟糕.

Answer 2

我使用的是云配置，我必须将initial-cluster-state: existing添加到云配置中。没有这种情况，initial-cluster-state将默认为new，而且我重新添加的节点似乎正在恢复它的旧GUID，该GUID已被“永久删除”，并导致etcd2无法启动。

下面是我的云端配置中没有initial-cluster-state: existing的日志日志：

Jun 03 11:10:36 giscoreos2 systemd[1]: Stopped etcd2.
Jun 03 11:10:36 giscoreos2 systemd[1]: Starting etcd2...
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://giscoreos2.example.com:2379
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_DISCOVERY_SRV=example-corp.us
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://giscoreos2.example.com:2380
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://giscoreos2.example.com:2380
Jun 03 11:10:36 giscoreos2 etcd2[19970]: recognized and used environment variable ETCD_NAME=giscoreos2
Jun 03 11:10:36 giscoreos2 etcd2[19970]: etcd Version: 2.3.1
Jun 03 11:10:36 giscoreos2 etcd2[19970]: Git SHA: 2b67f52
Jun 03 11:10:36 giscoreos2 etcd2[19970]: Go Version: go1.5.3
Jun 03 11:10:36 giscoreos2 etcd2[19970]: Go OS/Arch: linux/amd64
Jun 03 11:10:36 giscoreos2 etcd2[19970]: setting maximum number of CPUs to 16, total number of available CPUs is 16
Jun 03 11:10:36 giscoreos2 etcd2[19970]: the server is already initialized as member before, starting as etcd member...
Jun 03 11:10:36 giscoreos2 etcd2[19970]: got bootstrap from DNS for etcd-server at http://giscoreos3.example.com:2380
Jun 03 11:10:36 giscoreos2 etcd2[19970]: got bootstrap from DNS for etcd-server at http://giscoreos1.example.com:2380
Jun 03 11:10:36 giscoreos2 etcd2[19970]: got bootstrap from DNS for etcd-server at http://giscoreos2.example.com:2380
Jun 03 11:10:36 giscoreos2 etcd2[19970]: listening for peers on http://giscoreos2.example.com:2380
Jun 03 11:10:36 giscoreos2 etcd2[19970]: listening for client requests on http://0.0.0.0:2379
Jun 03 11:10:36 giscoreos2 etcd2[19970]: name = giscoreos2
Jun 03 11:10:36 giscoreos2 etcd2[19970]: data dir = /var/lib/etcd2
Jun 03 11:10:36 giscoreos2 etcd2[19970]: member dir = /var/lib/etcd2/member
Jun 03 11:10:36 giscoreos2 etcd2[19970]: heartbeat = 100ms
Jun 03 11:10:36 giscoreos2 etcd2[19970]: election = 1000ms
Jun 03 11:10:36 giscoreos2 etcd2[19970]: snapshot count = 10000
Jun 03 11:10:36 giscoreos2 etcd2[19970]: advertise client URLs = http://giscoreos2.example.com:2379
Jun 03 11:10:36 giscoreos2 etcd2[19970]: restarting member d5f2eb850214f772 in cluster e1013a21e485d6ec at commit index 3
Jun 03 11:10:36 giscoreos2 etcd2[19970]: d5f2eb850214f772 became follower at term 242
Jun 03 11:10:36 giscoreos2 etcd2[19970]: newRaft d5f2eb850214f772 [peers: [], term: 242, commit: 3, applied: 0, lastindex: 3, lastterm: 1]
Jun 03 11:10:36 giscoreos2 etcd2[19970]: starting server... [version: 2.3.1, cluster version: to_be_decided]
Jun 03 11:10:36 giscoreos2 systemd[1]: Started etcd2.
Jun 03 11:10:36 giscoreos2 etcd2[19970]: failed to find member c7f5106228d2b8a7 in cluster e1013a21e485d6ec
Jun 03 11:10:36 giscoreos2 etcd2[19970]: failed to find member c7f5106228d2b8a7 in cluster e1013a21e485d6ec
Jun 03 11:10:36 giscoreos2 etcd2[19970]: added member 2350c0e41172376a [http://giscoreos1.example.com:2380] to cluster e1013a21e485d6ec
Jun 03 11:10:36 giscoreos2 systemd[1]: Starting Network fabric for containers...
Jun 03 11:10:36 giscoreos2 etcd2[19970]: the member has been permanently removed from the cluster
Jun 03 11:10:36 giscoreos2 etcd2[19970]: the data-dir used by this member must be removed.
Jun 03 11:10:36 giscoreos2 etcd2[19970]: streaming request ignored (ID mismatch got b584365ea9e04f4d want d5f2eb850214f772)
Jun 03 11:10:36 giscoreos2 etcd2[19970]: streaming request ignored (ID mismatch got b584365ea9e04f4d want d5f2eb850214f772)
Jun 03 11:10:36 giscoreos2 etcd2[19970]: added member c7f5106228d2b8a7 [http://giscoreos3.example.com:2380] to cluster e1013a21e485d6ec
Jun 03 11:10:36 giscoreos2 etcd2[19970]: added local member d5f2eb850214f772 [http://giscoreos2.example.com:2380] to cluster e1013a21e485d6ec
Jun 03 11:10:36 giscoreos2 etcd2[19970]: aborting publish because server is stopped

下面是我在云端配置中使用initial-cluster-state: existing的日志：

Jun 03 11:19:02 giscoreos2 systemd[1]: Starting etcd2...
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://giscoreos2.example.com:2379
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_DATA_DIR=/var/lib/etcd2
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_DISCOVERY_SRV=exammple-corp.us
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://giscoreos2.example.com:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=existing
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-1
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://giscoreos2.example.com:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: recognized and used environment variable ETCD_NAME=giscoreos2
Jun 03 11:19:02 giscoreos2 etcd2[20222]: etcd Version: 2.3.1
Jun 03 11:19:02 giscoreos2 etcd2[20222]: Git SHA: 2b67f52
Jun 03 11:19:02 giscoreos2 etcd2[20222]: Go Version: go1.5.3
Jun 03 11:19:02 giscoreos2 etcd2[20222]: Go OS/Arch: linux/amd64
Jun 03 11:19:02 giscoreos2 etcd2[20222]: setting maximum number of CPUs to 16, total number of available CPUs is 16
Jun 03 11:19:02 giscoreos2 etcd2[20222]: got bootstrap from DNS for etcd-server at http://giscoreos1.example.com:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: got bootstrap from DNS for etcd-server at http://giscoreos2.example.com:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: got bootstrap from DNS for etcd-server at http://giscoreos3.example.com:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: listening for peers on http://giscoreos2.example.com:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: listening for client requests on http://0.0.0.0:2379
Jun 03 11:19:02 giscoreos2 etcd2[20222]: resolving giscoreos1.example.com:2380 to 10.240.160.152:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: resolving giscoreos1.example.com:2380 to 10.240.160.152:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: resolving giscoreos2.example.com:2380 to 10.240.160.57:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: resolving giscoreos2.example.com:2380 to 10.240.160.57:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: resolving giscoreos3.example.com:2380 to 10.240.160.6:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: resolving giscoreos3.example.com:2380 to 10.240.160.6:2380
Jun 03 11:19:02 giscoreos2 etcd2[20222]: name = giscoreos2
Jun 03 11:19:02 giscoreos2 etcd2[20222]: data dir = /var/lib/etcd2
Jun 03 11:19:02 giscoreos2 etcd2[20222]: member dir = /var/lib/etcd2/member
Jun 03 11:19:02 giscoreos2 etcd2[20222]: heartbeat = 100ms
Jun 03 11:19:02 giscoreos2 etcd2[20222]: election = 1000ms
Jun 03 11:19:02 giscoreos2 etcd2[20222]: snapshot count = 10000
Jun 03 11:19:02 giscoreos2 etcd2[20222]: advertise client URLs = http://giscoreos2.example.com:2379
Jun 03 11:19:02 giscoreos2 etcd2[20222]: starting member b584365ea9e04f4d in cluster e1013a21e485d6ec
Jun 03 11:19:02 giscoreos2 etcd2[20222]: b584365ea9e04f4d became follower at term 0
Jun 03 11:19:02 giscoreos2 etcd2[20222]: newRaft b584365ea9e04f4d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
Jun 03 11:19:02 giscoreos2 etcd2[20222]: b584365ea9e04f4d became follower at term 1
Jun 03 11:19:02 giscoreos2 etcd2[20222]: the connection with 2350c0e41172376a became active
Jun 03 11:19:02 giscoreos2 etcd2[20222]: starting server... [version: 2.3.1, cluster version: to_be_decided]
Jun 03 11:19:02 giscoreos2 systemd[1]: Started etcd2.