部署Flanneld网络 Flanneld:用于解决容器之间网络互通,这里我们要配置TLS认证。 Docker1.12.5:docker的安装很简单,这里也不说了。 配置Flanneld 这里我们使用yum的方式部署Flanneld和docker # yum install flannel docker -y service配置文件/etc/systemd/system /flanneld.service [Unit] Description=Flanneld overlay address etcd agent After=network.target After=network-online.target 配置文件 # Flanneld configuration options # etcd url location. 注意:这两条语句只需要在其中一台机器上执行即可 启动flanneld 在各节点上启动flanneld # systemctl enable flanneld # systemctl start flanneld
继续查看日志,发现Flanneld续租有问题: ? 这里需要介绍一下Flanneld续租SubNet的机制。 ? 三、临时解决方式: 1)需要重启docker:删除flannel.1网桥,重启Flanneld和docker。 systemctl status flanneld ps -ef | grep dockerd systemctl stop flanneld ifconfig flannel.1 down ip link 重启异常机器上flanneld; 4. 若问题依旧可以设置定时重启flanneld进程重新续租。 3)设置etcd中对应的TTL时间为0,即不过期。 第二种是部署flannel server(运行在server模式的flannel),flanneld连接flannel server。
Flanneld部署 ---- 由于flanneld需要依赖etcd来保证集群IP分配不冲突的问题,所以首先要在etcd中设置 flannel节点所使用的IP段。 解压flanneld [root@master-01 ~]# tar xf flannel-v0.11.0-linux-amd64.tar.gz [root@master-01 ~]# mv flanneld 配置flanneld [root@master-01 ~]# cat /etc/kubernetes/flanneld FLANNEL_OPTIONS="--etcd-endpoints=https:/ 配置flanneld启动文件 [root@master-01 ~]# cat /usr/lib/systemd/system/flanneld.service[Unit]Description=Flanneld from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service
创建flannel配置目录 mkdir -p /etc/flanneld mkdir -p /var/lib/flanneld chown -R root:root /etc/flanneld chown EOF source /etc/flanneld/flanneld.conf ? EnvironmentFile=/etc/flanneld/flanneld.conf User=root ExecStartPre=source /etc/flanneld/flanneld.conf 修改etcd,flanneld,docker的unit文件: etcd,flanneld,docker三个系统服务有了依赖关系,flanneld依赖etcd,docker依赖flanneld。 flanneld systemctl status flanneld ?
分别拷贝到node节点上 因为我的hosts文件映射为 172.18.98.46 host1 172.18.98.47 host2 172.18.98.48 master scp flanneld mk-docker-opts.sh root@host1:/opt/kubernetes/bin/ scp flanneld mk-docker-opts.sh root@host2:/opt/kubernetes vim flanneld.service 内容如下 [Unit] Description=Flanneld overlay address etcd agent After=network-online.target ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS ExecStartPost=/opt/kubernetes/bin service flanneld start 成功启动,查看进程 # ps -ef | grep flanneld root 24305 1 0 14:28 ?
/ 拷贝flanneld密钥 ssh test-node-16 "mkdir -p /etc/flanneld/cert" ssh test-node-17 "mkdir -p /etc/flanneld /cert" scp /etc/flanneld/cert/flanneld*.pem test-node-16:/etc/flanneld/cert scp /etc/flanneld/cert/flanneld *.pem test-node-17:/etc/flanneld/cert 拷贝flannel启动文件 scp /etc/systemd/system/flanneld.service test-node -16:/etc/systemd/system/ scp /etc/systemd/system/flanneld.service test-node-17:/etc/systemd/system/ # flanneld" ssh test-node-17 "systemctl daemon-reload && systemctl enable flanneld && systemctl restart
http://192.168.118.77:4001 cluster-health 4、安装flannel(所有主机) yum install -y flannel cp /etc/sysconfig/flanneld /etc/sysconfig/flanneld.bak vi /etc/sysconfig/flanneld FLANNEL_ETCD_ENDPOINTS="http://192.168.118.77 :2379" systemctl enable flanneld.service systemctl start flanneld.service systemctl restart docker
/cert" 7 scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert 8 done 1.5 创建flanneld的systemd 提示:k8smaster01节点已创建创建flanneld的systemd,可直接分发至node节点。 /cert/flanneld.pem \ 6 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 7 get ${FLANNEL_ETCD_PREFIX /cert/flanneld.pem \ 6 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 7 ls ${FLANNEL_ETCD_PREFIX /cert/flanneld.pem \ 12 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 13 get ${FLANNEL_ETCD_PREFIX
/cert" 7 scp flanneld*.pem root@${master_ip}:/etc/flanneld/cert 8 done 1.5 写入集群 Pod 网段信息 \\ 17 -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\ 18 -etcd-endpoints=${ETCD_ENDPOINTS /cert/flanneld.pem \ 6 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 7 get ${FLANNEL_ETCD_PREFIX /cert/flanneld.pem \ 6 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 7 ls ${FLANNEL_ETCD_PREFIX /cert/flanneld.pem \ 13 --key-file=/etc/flanneld/cert/flanneld-key.pem \ 14 get ${FLANNEL_ETCD_PREFIX
1、创建证书签名请求 cat > flanneld-csr.json <<EOF { "CN": "flanneld", "hosts": [], "key": { "algo": | cfssljson -bare flanneld 创建证书存放目录: mkdir /opt/kubernetes/ssl/flannel 这里是复制到3master+3node上 cp flanneld \ -etcd-keyfile=/opt/kubernetes/ssl/flannel/flanneld-key.pem \ -etcd-endpoints=https://192.168.206.31 flanneld 使用系统缺省路由所在的接口和其它节点通信,对于有多个网络接口的机器(如,内网和公网),可以用 -iface=enpxx 选项值指定通信接口。 4、启动flannel并且设置开机自启动 systemctl daemon-reload systemctl enable flanneld systemctl start flanneld 5、查看flannel
mk-docker-opts.sh 192.168.0.125:/opt/kubernetes/bin/ scp flanneld mk-docker-opts.sh 192.168.0.126:/opt /kubernetes/bin/ flannel配置文件 [root@k8s-node02 bin]# vim /opt/kubernetes/cfg/flanneld FLANNEL_OPTIONS= [Unit] Description=Flanneld overlay address etcd agent After=network-online.target network.target Before 192.168.0.126:/opt/kubernetes/cfg [root@k8s-node01 ~]# scp /usr/lib/systemd/system/flanneld.service restart flanneld systemctl status flanneld 安装docker yum install -y yum-utils device-mapper-persistent-data
| cfssljson -bare flanneld [root@k8s-01 ~]# ls flanneld*pem ? /cert" scp flanneld*.pem root@${node_ip}:/etc/flanneld/cert done f、向etcd写入Pod网段信息 [root@k8s-01 work [root@k8s-01 work]# cat > flanneld.service << EOF [Unit] Description=Flanneld overlay address \\ -etcd-cafile=/etc/kubernetes/cert/ca.pem \\ -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\ /cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ ls ${FLANNEL_ETCD_PREFIX}/subnets
network/config '{"Network": "172.17.0.0/16"}' 2.配置flannel [root@k8s-master yaml]# vi /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. EnvironmentFile=-/etc/sysconfig/docker-network #ExecStart=/usr/bin/flanneld-start $FLANNEL_OPTIONS ExecStart=/usr/bin/flanneld -etcd-endpoints=http://10.10.200.224:2379 ExecStartPost=/usr/libexec # Flanneld configuration options # etcd url location.
/ 拷贝flanneld密钥 ssh test-node-16 "mkdir -p /etc/flanneld/cert" ssh test-node-17 "mkdir -p /etc/flanneld /cert" scp /etc/flanneld/cert/flanneld*.pem test-node-16:/etc/flanneld/cert scp /etc/flanneld/cert/flanneld *.pem test-node-17:/etc/flanneld/cert 拷贝flannel启动文件 scp /etc/systemd/system/flanneld.service test-node -16:/etc/systemd/system/ scp /etc/systemd/system/flanneld.service test-node-17:/etc/systemd/system/ # flanneld" ssh test-node-17 "systemctl daemon-reload && systemctl enable flanneld && systemctl restart
每个subnet都是从一个更大的IP池中划分的,flannel会在每个主机上运flanneld的agent,负责从池子中分配subnet。 查找路由表,外部容器ip的报文都会转发到flannel0虚拟网卡,这是一个P2P的虚拟网卡,然后报文就被转发到监听在另一端的flanneld。 flanneld通过etcd维护了各个节点之间的路由表,把原来的报文UDP封装一层,通过配置的iface发送出去。 报文通过主机之间的网络找到目标主机。 3 # Flanneld configuration options 4 # etcd url location. 3 root@docker01:~# systemctl enable flanneld.service 4 root@docker01:~# ps -ef | grep flanneld
对上图的简单解释: 1)数据从源容器中发出后,经由所在主机的docker0虚拟网卡转发到flannel0虚拟网卡,这是个P2P的虚拟网卡,flanneld服务监听在网卡的另外一端。 3)源主机的flanneld服务将原本的数据内容UDP封装后根据自己的路由表投递给目的节点的flanneld服务,数据到达以后被解包,然后直接进入目的节点的flannel0虚拟网卡, 然后被转发到目的主机的 /etc/sysconfig/flanneld.bak [root@node-1 ~]# vim /etc/sysconfig/flanneld # Flanneld configuration options [root@node-1 ~]# systemctl start flanneld.service [root@node-1 ~]# ps -ef|grep flannel root 9305 /etc/sysconfig/flanneld.bak [root@node-2 ~]# vim /etc/sysconfig/flanneld # Flanneld configuration options
下列所有操作需要在所有node节点上操作,并注意红色部分的修改 1.node节点不需要安装etcd来存储相关信息 yum -y install flannel kubernetes 2.修改flanneld 网络相关信息 [root@k8s-node1 ~]# vi /etc/sysconfig/flanneld # Flanneld configuration options FLANNEL_ETCD KUBELET_POD_INFRA_CONTAINER 对应到私库地址 , KUBELET_ARGS 配置的为dns解析信息 和master配置第4步 配置一致 5.启动服务 for SERVICES in kube-proxy kubelet flanneld
to /usr/lib/systemd/system/flanneld.service. ● flanneld.service - Flanneld overlay address etcd agent /bin/flanneld --ip-masq 3月 31 20:39:27 gysl-node2 flanneld[7508]: I0331 20:39:27.720837 7508 iptables.go to /usr/lib/systemd/system/flanneld.service. ● flanneld.service - Flanneld overlay address etcd agent /bin/flanneld --ip-masq 3月 31 20:39:39 gysl-node3 flanneld[7502]: I0331 20:39:39.088309 7502 iptables.go to /usr/lib/systemd/system/flanneld.service. ● flanneld.service - Flanneld overlay address etcd agent
用户态到内核: flanneld 对数据包进行 UDP 封装后,将数据再次注入 flannel0,交由内核网络栈根据路由转发。 进入 flanneld 用户态处理:flannel0 将数据包传递给 flanneld 进程,flanneld 根据目标 IP 查询 Etcd 获取 container-2 所在宿主机的公网 IP(例如 UDP 封装与发送:flanneld 将 IP 包封装进 UDP 包内,通过宿主机的物理接口(如 eth0)发送至目标宿主机的 8285 端口。 目标宿主机解封与注入:在 Node 2 上,flanneld 监听 8285 端口,接收到 UDP 包后解封,提取出原始 IP 包,并将其注入本地 flannel0。 第二次: 数据包从内核态经由 flannel0 进入 flanneld 用户态进行处理。第三次: flanneld 封装 UDP 包后,再将数据包注入内核,由物理接口输出。
\\ -etcd-cafile=/etc/kubernetes/cert/ca.pem \\ -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\ |grep Active" done 确保状态为 active (running),否则查看日志,确认原因"journalctl -u flanneld" 7) 检查分配给各 flanneld /cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/ /cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ ls ${FLANNEL_ETCD_PREFIX}/subnets /cert/flanneld.pem \ --key-file=/etc/flanneld/cert/flanneld-key.pem \ get ${FLANNEL_ETCD_PREFIX}/