如何在service-node-port-range默认范围之外启动NodePort上的kubernetes服务？

Question

我一直在尝试在默认端口范围之外的NodePort上启动kubernetes-dashboard (以及最终的其他服务)，但几乎没有成功，下面是我的设置:云提供商: Azure (不是azure容器服务) OS: CentOS 7

以下是我尝试过的方法：

更新主机

$ yum update

安装kubeadm

$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
$ setenforce 0
$ yum install -y docker kubelet kubeadm kubectl kubernetes-cni
$ systemctl enable docker && systemctl start docker
$ systemctl enable kubelet && systemctl start kubelet

使用kubeadm启动集群

$ kubeadm init

允许在主节点上运行容器，因为我们只有一个节点集群

$ kubectl taint nodes --all dedicated-

安装pod网络

$ kubectl apply -f https://git.io/weave-kube

我们的kubernetes-dashboard部署(@ ~/kubernetes-dashboard.yaml

# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or     implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Configuration to deploy release version of the Dashboard UI.
#
# Example usage: kubectl create -f <this_file>

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubernetes-dashboard
  template:
    metadata:
      labels:
        app: kubernetes-dashboard
      # Comment the following annotation if Dashboard must not be deployed on master
      annotations:
        scheduler.alpha.kubernetes.io/tolerations: |
          [
            {
              "key": "dedicated",
              "operator": "Equal",
              "value": "master",
              "effect": "NoSchedule"
            }
          ]
    spec:
      containers:
      - name: kubernetes-dashboard
        image: gcr.io/google_containers/kubernetes-dashboard-amd64:v1.5.1
        imagePullPolicy: Always
        ports:
        - containerPort: 9090
          protocol: TCP
        args:
          # Uncomment the following line to manually specify Kubernetes API server Host
          # If not specified, Dashboard will attempt to auto discover the API server and connect
          # to it. Uncomment only if the default does not work.
          # - --apiserver-host=http://my-address:port
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 8880
    targetPort: 9090
    nodePort: 8880
  selector:
    app: kubernetes-dashboard

创建我们的部署

$ kubectl create -f ~/kubernetes-dashboard.yaml
deployment "kubernetes-dashboard" created
The Service "kubernetes-dashboard" is invalid: spec.ports[0].nodePort: Invalid value: 8880: provided port is not in the valid range. The range of valid ports is 30000-32767

我发现要更改有效端口的范围，可以在kube-apiserver上设置service-node- port -range选项，以允许不同的端口范围，所以我尝试这样做：

$ kubectl get po --namespace=kube-system
NAME                                    READY     STATUS    RESTARTS       AGE
dummy-2088944543-lr2zb                  1/1       Running   0              31m
etcd-test2-highr                        1/1       Running   0              31m
kube-apiserver-test2-highr              1/1       Running   0              31m
kube-controller-manager-test2-highr     1/1       Running   2              31m
kube-discovery-1769846148-wmbhb         1/1       Running   0              31m
kube-dns-2924299975-8vwjm               4/4       Running   0              31m
kube-proxy-0ls9c                        1/1       Running   0              31m
kube-scheduler-test2-highr              1/1       Running   2              31m
kubernetes-dashboard-3203831700-qrvdn   1/1       Running   0              22s
weave-net-m9rxh                         2/2       Running   0              31m

添加"--service-node-port-range=8880-8880“到kube-apiserver-test2-highr

$ kubectl edit po kube-apiserver-test2-highr --namespace=kube-system
{
  "kind": "Pod",
  "apiVersion": "v1",
  "metadata": {
    "name": "kube-apiserver",
    "namespace": "kube-system",
    "creationTimestamp": null,
    "labels": {
      "component": "kube-apiserver",
      "tier": "control-plane"
    }
  },
  "spec": {
    "volumes": [
      {
        "name": "k8s",
        "hostPath": {
          "path": "/etc/kubernetes"
        }
      },
      {
        "name": "certs",
        "hostPath": {
          "path": "/etc/ssl/certs"
        }
      },
      {
        "name": "pki",
        "hostPath": {
          "path": "/etc/pki"
        }
      }
    ],
    "containers": [
      {
        "name": "kube-apiserver",
        "image": "gcr.io/google_containers/kube-apiserver-amd64:v1.5.3",
        "command": [
          "kube-apiserver",
          "--insecure-bind-address=127.0.0.1",
          "--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota",
          "--service-cluster-ip-range=10.96.0.0/12",
          "--service-node-port-range=8880-8880",
          "--service-account-key-file=/etc/kubernetes/pki/apiserver-key.pem",
          "--client-ca-file=/etc/kubernetes/pki/ca.pem",
          "--tls-cert-file=/etc/kubernetes/pki/apiserver.pem",
          "--tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem",
          "--token-auth-file=/etc/kubernetes/pki/tokens.csv",
          "--secure-port=6443",
          "--allow-privileged",
          "--advertise-address=100.112.226.5",
          "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
          "--anonymous-auth=false",
          "--etcd-servers=http://127.0.0.1:2379"
        ],
        "resources": {
          "requests": {
            "cpu": "250m"
          }
        },
        "volumeMounts": [
          {
            "name": "k8s",
            "readOnly": true,
            "mountPath": "/etc/kubernetes/"
          },
          {
            "name": "certs",
            "mountPath": "/etc/ssl/certs"
          },
          {
            "name": "pki",
            "mountPath": "/etc/pki"
          }
        ],
        "livenessProbe": {
          "httpGet": {
            "path": "/healthz",
            "port": 8080,
            "host": "127.0.0.1"
          },
          "initialDelaySeconds": 15,
          "timeoutSeconds": 15,
          "failureThreshold": 8
        }
      }
    ],
    "hostNetwork": true
  },
  "status": {}

$ :wq

以下是截断的响应

# pods "kube-apiserver-test2-highr" was not valid:
# * spec: Forbidden: pod updates may not change fields other than `containers[*].image` or `spec.activeDeadlineSeconds`

所以我尝试了一种不同的方法，我编辑了kube-apiserver的部署文件，进行了上述相同的更改，并运行以下命令：

$ kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.json --namespace=kube-system

得到的回应是这样的：

The connection to the server localhost:8080 was refused - did you specify the right host or port?

那么现在我被卡住了，我该如何改变有效端口的范围呢？

Answer 1

您指定的--service-node-port-range=8880-8880错误。只需将其设置为一个端口，即可将其设置为一个范围。

第二个问题:您正在将服务设置为使用9090，但它不在范围内。

 ports:
  - port: 80
    targetPort: 9090
    nodePort: 9090

API Server也应该有一个部署，尝试在部署本身中编辑port-range并删除api服务器pod，以便通过新配置重新创建它。

Answer 2

服务节点端口范围设置为不常用端口是有原因的。为什么要在每个节点上发布它？你真的想这样吗？

另一种方法是在半随机的nodeport上公开它，然后在一个已知节点或一组节点上使用代理pod通过hostport访问它。

Answer 3

此问题：

The connection to the server localhost:8080 was refused - did you specify the right host or port?

是由我的端口范围(不包括8080 )引起的，kube-apiserver正在为其提供服务，因此我无法向kubectl发送任何更新。

我通过将端口范围更改为8080-8881并重新启动kubelet服务来修复它，如下所示：

$ service kubelet restart

现在一切都按预期进行了。