无法在伪造的jenkins内部运行kubectl

Question

我用码头安装了詹金斯-

docker network create jenkins

docker volume create jenkins-docker-certs
docker volume create jenkins-data

docker image pull docker:dind

docker image pull jenkinsci/blueocean

docker container run --name jenkins-docker \
  --restart unless-stopped \
  --detach \
  --privileged --network jenkins \
  --network-alias docker \
  --env DOCKER_TLS_CERTDIR=/certs \
  --volume jenkins-docker-certs:/certs/client \
  --volume jenkins-data:/var/jenkins_home \
  --publish 2376:2376\
  docker:dind

docker container run --name jenkins-blueocean \
  --restart unless-stopped \
  --detach \
  --network jenkins \
  --env DOCKER_HOST=tcp://docker:2376 \
  --env DOCKER_CERT_PATH=/certs/client \
  --env DOCKER_TLS_VERIFY=1 \
  --volume jenkins-data:/var/jenkins_home \
  --volume jenkins-docker-certs:/certs/client:ro \
  --publish 8080:8080 \
  --publish 50000:50000 \
jenkinsci/blueocean

现在我想在jenkins内部使用kubectl，所以我添加了库伯内特斯-克莱插件，并安装了kubectl作为这里。我有我的jenkins文件( kube config文件中的kubecreds)

pipeline {
    agent any
    stages {
        stage('Cloning Repo') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '${branch}']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'githubcreds', url: '<repo url>']]])
            }
        }
        stage('List pods') {
            steps {
                withKubeConfig([credentialsId: 'kubecreds']) {
                    sh 'curl -LO "https://storage.googleapis.com/kubernetes-release/release/v1.20.5/bin/linux/amd64/kubectl"'
                    sh 'chmod u+x ./kubectl'
                    sh './kubectl get pods -n stage'
                }
            }
        }
    }
}

但是运行这个jenkins文件会引发错误-

+ ./kubectl get pods -n stage
Unable to connect to the server: getting credentials: exec: executable aws not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at:
      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
[Pipeline] }
[kubernetes-cli] kubectl configuration cleaned up
[Pipeline] // withKubeConfig
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

我添加了aws sdk插件，但仍然是相同的错误。所以我想手动安装awscli然后尝试-

pipeline {
    agent any
    stages {
        stage('Cloning Repo') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '${branch}']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'githubcreds', url: '<repo url>']]])
            }
        }
        stage('List pods') {
            steps {
                withKubeConfig([credentialsId: 'kubecreds']) {
                    sh 'curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"'
                    sh 'unzip awscliv2.zip'
                    sh './aws/install --update -i . -b .'
                    sh './aws --version'
                    sh 'curl -LO "https://storage.googleapis.com/kubernetes-release/release/v1.20.5/bin/linux/amd64/kubectl"'
                    sh 'chmod u+x ./kubectl'
                    sh './kubectl get pods -n stage'
                }
            }
        }
    }
}

但出了差错-

+ ./aws/install --update -i . -b .
Found same AWS CLI version: ./v2/2.3.2. Skipping install.
[Pipeline] sh
+ ./aws --version
/var/jenkins_home/workspace/callbreak-deploy-job@tmp/durable-b3361486/script.sh: line 1: ./aws: Permission denied
[Pipeline] }
[kubernetes-cli] kubectl configuration cleaned up
[Pipeline] // withKubeConfig
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 126
Finished: FAILURE

关于我如何使它工作的任何想法.在詹金斯的工作中成功地运行kubectl？

谢谢

-编辑1

正如下面的答案所建议的，我试着使aws可执行-

pipeline {
    agent any
    stages {
        stage('Cloning Repo') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '${branch}']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'githubcreds', url: '<repo url>']]])
            }
        }
        stage('List pods') {
            steps {
                withKubeConfig([credentialsId: 'kubecreds']) {
                    sh 'rm awscliv2.zip'
                    sh 'rm -rf aws'
                    sh 'curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"'
                    sh 'unzip awscliv2.zip'
                    sh './aws/install --update -i . -b .'
                    sh 'chmod u+x ./aws'
                    sh './aws --version'
                    sh 'curl -LO "https://storage.googleapis.com/kubernetes-release/release/v1.20.5/bin/linux/amd64/kubectl"'
                    sh 'chmod u+x ./kubectl'
                    sh './kubectl get pods -n stage'
                }
            }
        }
    }
}

但还是一样的错误-

+ ./aws/install --update -i . -b .
Found same AWS CLI version: ./v2/2.3.2. Skipping install.
[Pipeline] sh
+ chmod u+x ./aws
[Pipeline] sh
+ ./aws --version
/var/jenkins_home/workspace/callbreak-deploy-job@tmp/durable-cf2a75a8/script.sh: line 1: ./aws: Permission denied
[Pipeline] }
[kubernetes-cli] kubectl configuration cleaned up
[Pipeline] // withKubeConfig
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 126
Finished: FAILURE

Answer 1

在创建了一个定制的jenkins停靠映像(已经安装了kubectl和 aws cli )并使用了aws插件之后，我终于能够在jenkins中运行kubectl。

我的文件

FROM jenkins/jenkins:2.303.2-jdk11
USER root
RUN apt-get update && apt-get install -y apt-transport-https \
       ca-certificates curl gnupg2 \
       software-properties-common
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-key fingerprint 0EBFCD88
RUN add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/debian \
       $(lsb_release -cs) stable"
RUN apt-get update && apt-get install -y docker-ce-cli
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
RUN chmod +x kubectl
RUN mv ./kubectl /usr/local/bin/kubectl
USER jenkins
RUN aws --version
RUN kubectl version --client
RUN jenkins-plugin-cli --plugins "blueocean:1.25.0 docker-workflow:1.26"

还有我的新jenkinsfile

pipeline {
    agent any
    stages {
        stage('Cloning Repo') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '${branch}']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'githubcreds', url: '<repo url>']]])
            }
        }
        stage('List pods') {
            steps {
                withAWS([credentials: 'awscreds']) {
                    sh 'aws eks --region ap-south-1 update-kubeconfig --name <name>'
                    sh 'kubectl apply -f deploy/stage/$service.yaml'
                }
            }
        }
    }
}

Answer 2

错误来自AWS，它有权限

你应该跑

chmod +x /usr/bin/aws*

或

sh 'chmod u+x ./aws'

或者二进制文件允许AWS cli在安装的任何地方执行。

因为你这么做是为了

sh 'chmod u+x ./kubectl'

Answer 3

您可以找到一个由我发布的工作示例，其中Jenkins运行在docker和docker中，AWS在容器中可用。您将在这里找到文件: docker-compose.yml和Dockerfile：

使用通过Docker安装的Jenkins运行AWS命令行接口:未找到命令？