将防火墙端口转发到k8s节点端口不工作

Question

我想配置端口转发80->32181，443->30598。32181和30598是NodePort of k8s入口控制器，我可以正确地建立连接：

$ curl http://localhost:32181
<html>
<head><title>404 Not Found</title></head>
<body>
...

$ curl https://localhost:30598 -k
<html>
<head><title>404 Not Found</title></head>
<body>
...

我所做的是：

$ cat /proc/sys/net/ipv4/ip_forward
1

$ firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: cockpit dhcpv6-client frp http https kube-apiserver kube-kubelet ssh
  ports:
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
    port=80:proto=tcp:toport=32181:toaddr=
    port=443:proto=tcp:toport=30598:toaddr=
  source-ports:
  icmp-blocks:
  rich rules:

但我不能通过80或443访问我的nginx

$ curl https://localhost:443 -k
curl: (7) Failed to connect to localhost port 443: Connection refused

更多信息：

centos: 8.2 4.18.0-348.2.1.el8_5.86_64 k8s: 1.22(带calico(v3.21.0)网络插件) firewalld: 0.9.3

和iptable的输出：

$ iptables -nvL -t nat --line-numbers
Chain PREROUTING (policy ACCEPT 51 packets, 2688 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       51  2688 cali-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:6gwbT8clXdHdC1b1 */
2       51  2688 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
3       51  2688 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 50 packets, 2648 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 1872 packets, 112K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:O3lYWMrLQYEMJtB5 */
2     1862  112K KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
3        0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0

Chain OUTPUT (policy ACCEPT 1922 packets, 116K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:tVnHkvAo15HuiPy0 */
2     1911  115K KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
3      758 45480 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

Chain KUBE-SERVICES (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-SVC-JD5MR3NA4I4DYORP  tcp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2        0     0 KUBE-SVC-Z6GDYMWE5TV2NNJN  tcp  --  *      *       0.0.0.0/0            10.110.193.197       /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
3        0     0 KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  *      *       0.0.0.0/0            10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
4        0     0 KUBE-SVC-EDNDUDH2C75GIR6O  tcp  --  *      *       0.0.0.0/0            10.97.201.174        /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
5        0     0 KUBE-SVC-EZYNCFY2F7N6OQA2  tcp  --  *      *       0.0.0.0/0            10.103.242.141       /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
6        0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
7        0     0 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
8        0     0 KUBE-SVC-CEZPIJSAUFW5MYPQ  tcp  --  *      *       0.0.0.0/0            10.97.166.112        /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
9        0     0 KUBE-SVC-H5K62VURUHBF7BRH  tcp  --  *      *       0.0.0.0/0            10.104.154.95        /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
10       0     0 KUBE-SVC-MOZMMOD3XZX35IET  tcp  --  *      *       0.0.0.0/0            10.96.73.22          /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
11       0     0 KUBE-SVC-CG5I4G2RS3ZVWGLK  tcp  --  *      *       0.0.0.0/0            10.97.201.174        /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
12    1165 69528 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     1859  112K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x4000/0x4000
2        3   180 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK xor 0x4000
3        3   180 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ random-fully

Chain KUBE-MARK-DROP (0 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-NODEPORTS (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        2   120 KUBE-SVC-EDNDUDH2C75GIR6O  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
2        1    60 KUBE-SVC-CG5I4G2RS3ZVWGLK  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181

Chain KUBE-MARK-MASQ (27 references)
num   pkts bytes target     prot opt in     out     source               destination
1        3   180 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-SEP-IPE5TMLTCUYK646X (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.147      0.0.0.0/0            /* kube-system/kube-dns:metrics */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.103.147:9153

Chain KUBE-SEP-3LZLTHU4JT3FAVZK (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.149      0.0.0.0/0            /* kube-system/kube-dns:metrics */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:192.168.103.149:9153

Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2        0     0 KUBE-SEP-IPE5TMLTCUYK646X  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ statistic mode random probability 0.50000000000
3        0     0 KUBE-SEP-3LZLTHU4JT3FAVZK  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */

Chain KUBE-SEP-ZOAMCQDU54EOM4EJ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.141      0.0.0.0/0            /* kubernetes-dashboard/dashboard-metrics-scraper */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/dashboard-metrics-scraper */ tcp to:192.168.103.141:8000

Chain KUBE-SVC-Z6GDYMWE5TV2NNJN (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.110.193.197       /* kubernetes-dashboard/dashboard-metrics-scraper cluster IP */ tcp dpt:8000
2        0     0 KUBE-SEP-ZOAMCQDU54EOM4EJ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/dashboard-metrics-scraper */

Chain KUBE-SEP-HYE2IFAO6PORQFJR (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.0.176        0.0.0.0/0            /* default/kubernetes:https */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ tcp to:192.168.0.176:6443

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
2        0     0 KUBE-SEP-HYE2IFAO6PORQFJR  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SEP-GJ4OJHBKIREWLMRS (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.146      0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */
2        2   120 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */ tcp to:192.168.103.146:443

Chain KUBE-SVC-EDNDUDH2C75GIR6O (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.97.201.174        /* ingress-nginx/ingress-nginx-controller:https cluster IP */ tcp dpt:443
2        2   120 KUBE-MARK-MASQ  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */ tcp dpt:30598
3        2   120 KUBE-SEP-GJ4OJHBKIREWLMRS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:https */

Chain KUBE-SEP-K2CVHZPTBE2YAD6P (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.146      0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */ tcp to:192.168.103.146:8443

Chain KUBE-SVC-EZYNCFY2F7N6OQA2 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.103.242.141       /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
2        0     0 KUBE-SEP-K2CVHZPTBE2YAD6P  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */

Chain KUBE-SEP-S6VTWHFP6KEYRW5L (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.147      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.147:53

Chain KUBE-SEP-SFGZMYIS2CE4JD3K (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.149      0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:192.168.103.149:53

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
2        0     0 KUBE-SEP-S6VTWHFP6KEYRW5L  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ statistic mode random probability 0.50000000000
3        0     0 KUBE-SEP-SFGZMYIS2CE4JD3K  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SEP-IJUMPPTQDLYXOX4B (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.147      0.0.0.0/0            /* kube-system/kube-dns:dns */
2        0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.103.147:53

Chain KUBE-SEP-C4W6TKYY5HHEG4RV (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.149      0.0.0.0/0            /* kube-system/kube-dns:dns */
2        0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:192.168.103.149:53

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  udp  --  *      *      !192.168.0.0/16       10.96.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
2        0     0 KUBE-SEP-IJUMPPTQDLYXOX4B  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ statistic mode random probability 0.50000000000
3        0     0 KUBE-SEP-C4W6TKYY5HHEG4RV  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain KUBE-SEP-GX372II3CQAGUHFM (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.145      0.0.0.0/0            /* kubernetes-dashboard/kubernetes-dashboard */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/kubernetes-dashboard */ tcp to:192.168.103.145:8443

Chain KUBE-SVC-CEZPIJSAUFW5MYPQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.97.166.112        /* kubernetes-dashboard/kubernetes-dashboard cluster IP */ tcp dpt:443
2        0     0 KUBE-SEP-GX372II3CQAGUHFM  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes-dashboard/kubernetes-dashboard */

Chain KUBE-SEP-I3RZS3REJP7POFLG (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.143      0.0.0.0/0            /* lens-metrics/kube-state-metrics:metrics */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/kube-state-metrics:metrics */ tcp to:192.168.103.143:8080

Chain KUBE-SVC-H5K62VURUHBF7BRH (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.104.154.95        /* lens-metrics/kube-state-metrics:metrics cluster IP */ tcp dpt:8080
2        0     0 KUBE-SEP-I3RZS3REJP7POFLG  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/kube-state-metrics:metrics */

Chain KUBE-SEP-ROTMHDCXAI3T7IOR (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.144      0.0.0.0/0            /* lens-metrics/prometheus:web */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/prometheus:web */ tcp to:192.168.103.144:9090

Chain KUBE-SVC-MOZMMOD3XZX35IET (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.96.73.22          /* lens-metrics/prometheus:web cluster IP */ tcp dpt:80
2        0     0 KUBE-SEP-ROTMHDCXAI3T7IOR  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* lens-metrics/prometheus:web */

Chain KUBE-SEP-OAYGOO6JHJEB65WC (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       192.168.103.146      0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */
2        1    60 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */ tcp to:192.168.103.146:80

Chain KUBE-SVC-CG5I4G2RS3ZVWGLK (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !192.168.0.0/16       10.97.201.174        /* ingress-nginx/ingress-nginx-controller:http cluster IP */ tcp dpt:80
2        1    60 KUBE-MARK-MASQ  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */ tcp dpt:32181
3        1    60 KUBE-SEP-OAYGOO6JHJEB65WC  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller:http */

Chain KUBE-PROXY-CANARY (0 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-nat-outgoing (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       49  3274 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully

Chain cali-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-fip-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Z-c7XtVd2Bq7s_hA */
2     1894  114K cali-nat-outgoing  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:nYKhEzDlr11Jccal */
3        0     0 MASQUERADE  all  --  *      tunl0   0.0.0.0/0            0.0.0.0/0            /* cali:SXWvdsbh4Mw7wOln */ ADDRTYPE match src-type !LOCAL limit-out ADDRTYPE match src-type LOCAL random-fully

Chain cali-PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       51  2688 cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:r6XmIziWUJsdOK6Z */

Chain cali-fip-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1     1894  114K cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:GBTAv2p5CwevEyJm */

Chain cali-fip-dnat (2 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain KUBE-KUBELET-CANARY (0 references)
num   pkts bytes target     prot opt in     out     source               destination

Answer 1

为了澄清，我张贴社区Wiki的答案。

此问题仅在转发到k8s服务NodePort时才存在。

为了解决这个问题，已经建立了一个外部Nginx作为TCP代理.。

这里可以找到关于外部NGINX的文档。

入口不直接支持TCP服务，因此需要进行一些额外的配置。您的NGINX入侵控制器可能是直接部署 (即带有Kubernetes规范文件)或通过官方赫尔姆图。根据部署方法的不同，TCP通过的配置将有所不同。