deis构建器保持重启，活动探测失败

Question

我尝试删除pods，或重新缩放副本，或删除aws实例，但仍无法使deis构建器正常工作。它会在活性探测失败的情况下保持重启。下面是deis构建器的日志

$ kubectl describe pods/deis-builder-2995120344-mz2zg -n deis
Name:       deis-builder-2995120344-mz2zg
    Namespace:  deis
    Node:       ip-10-0-48-189.ec2.internal/10.0.48.189
    Start Time: Wed, 15 Mar 2017 22:29:03 -0400
    Labels:     app=deis-builder
            pod-template-hash=2995120344
    Status:     Running
    IP:     10.34.184.7
    Controllers:    ReplicaSet/deis-builder-2995120344
    Containers:
      deis-builder:
        Container ID:   docker://f2b7799712c347759832270716057b6ac3be68298eef3057c25727b66024c84a
        Image:      quay.io/deis/builder:v2.7.1
        Image ID:       docker-pullable://quay.io/deis/builder@sha256:3dab1dd4e6359d1588fee1b4f93ef9f5c70f268f17de5bed4bc13faa210ce5d0
        Ports:      2223/TCP, 8092/TCP
        State:      Waiting
          Reason:       CrashLoopBackOff
        Last State:     Terminated
          Reason:       Error
          Exit Code:    2
          Started:      Wed, 15 Mar 2017 22:37:37 -0400
          Finished:     Wed, 15 Mar 2017 22:38:15 -0400
        Ready:      False
        Restart Count:  7
        Liveness:       http-get http://:8092/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
        Readiness:      http-get http://:8092/readiness delay=30s timeout=1s period=10s #success=1 #failure=3
        Volume Mounts:
          /var/run/secrets/api/auth from builder-key-auth (ro)
          /var/run/secrets/deis/builder/ssh from builder-ssh-private-keys (ro)
          /var/run/secrets/deis/objectstore/creds from objectstore-creds (ro)
          /var/run/secrets/kubernetes.io/serviceaccount from deis-builder-token-qbqff (ro)
        Environment Variables:
          DEIS_REGISTRY_SERVICE_HOST:   127.0.0.1
          DEIS_REGISTRY_SERVICE_PORT:   5555
          HEALTH_SERVER_PORT:       8092
          EXTERNAL_PORT:            2223
          BUILDER_STORAGE:          s3
          DEIS_REGISTRY_LOCATION:       ecr
          DEIS_REGISTRY_SECRET_PREFIX:  private-registry
          GIT_LOCK_TIMEOUT:         10
          SLUGBUILDER_IMAGE_NAME:       <set to the key 'image' of config map 'slugbuilder-config'>
          SLUG_BUILDER_IMAGE_PULL_POLICY:   <set to the key 'pullpolicy' of config map 'slugbuilder-config'>
          DOCKERBUILDER_IMAGE_NAME:     <set to the key 'image' of config map 'dockerbuilder-config'>
          DOCKER_BUILDER_IMAGE_PULL_POLICY: <set to the key 'pullpolicy' of config map 'dockerbuilder-config'>
          DOCKERIMAGE:          1
          DEIS_DEBUG:           false
          POD_NAMESPACE:            deis (v1:metadata.namespace)
          DEIS_BUILDER_KEY:         <set to the key 'builder-key' in secret 'builder-key-auth'>
    Conditions:
      Type      Status
      Initialized   True
      Ready     False
      PodScheduled  True
    Volumes:
      builder-key-auth:
        Type:   Secret (a volume populated by a Secret)
        SecretName: builder-key-auth
      builder-ssh-private-keys:
        Type:   Secret (a volume populated by a Secret)
        SecretName: builder-ssh-private-keys
      objectstore-creds:
        Type:   Secret (a volume populated by a Secret)
        SecretName: objectstorage-keyfile
      deis-builder-token-qbqff:
        Type:   Secret (a volume populated by a Secret)
        SecretName: deis-builder-token-qbqff
    QoS Class:  BestEffort
    Tolerations:    <none>
    Events:
      FirstSeen LastSeen    Count   From                    SubObjectPath           Type        ReasonMessage
      --------- --------    -----   ----                    -------------           --------    -------------
      10m       10m     1   {default-scheduler }                            Normal      Scheduled   Successfully assigned deis-builder-2995120344-mz2zg to ip-10-0-48-189.ec2.internal
      10m       10m     1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      StartedStarted container with docker id 7eac3a357f61
      10m       10m     1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      CreatedCreated container with docker id 7eac3a357f61; Security:[seccomp=unconfined]
      9m        9m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      CreatedCreated container with docker id 8e730f2731ef; Security:[seccomp=unconfined]
      9m        9m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      StartedStarted container with docker id 8e730f2731ef
      9m        9m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      KillingKilling container with docker id 7eac3a357f61: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      9m        9m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      CreatedCreated container with docker id 5f4e695c595a; Security:[seccomp=unconfined]
      9m        9m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      KillingKilling container with docker id 8e730f2731ef: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      9m        9m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      StartedStarted container with docker id 5f4e695c595a
      8m        8m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      CreatedCreated container with docker id c87d762fc118; Security:[seccomp=unconfined]
      8m        8m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      StartedStarted container with docker id c87d762fc118
      8m        8m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      KillingKilling container with docker id 5f4e695c595a: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      7m        7m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      CreatedCreated container with docker id 416573d43fe4; Security:[seccomp=unconfined]
      7m        7m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      StartedStarted container with docker id 416573d43fe4
      7m        7m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      KillingKilling container with docker id c87d762fc118: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      7m        7m      1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal      KillingKilling container with docker id 416573d43fe4: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      7m        6m      4   {kubelet ip-10-0-48-189.ec2.internal}                   Warning     FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "deis-builder" with CrashLoopBackOff: "Back-off 40s restarting failed container=deis-builder pod=deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)"

      6m    6m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Created     Created container with docker id bf5b29729c27; Security:[seccomp=unconfined]
      6m    6m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Started     Started container with docker id bf5b29729c27
      9m    5m  4   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Warning Unhealthy   Readiness probe failed: Get http://10.34.184.7:8092/readiness: dial tcp 10.34.184.7:8092: getsockopt: connection refused
      9m    5m  4   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Warning Unhealthy   Liveness probe failed: Get http://10.34.184.7:8092/healthz: dial tcp 10.34.184.7:8092: getsockopt: connection refused
      5m    5m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Started     Started container with docker id e457328db858
      5m    5m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Created     Created container with docker id e457328db858; Security:[seccomp=unconfined]
      5m    5m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Killing     Killing container with docker id bf5b29729c27: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      5m    5m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Killing     Killing container with docker id e457328db858: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      5m    2m  13  {kubelet ip-10-0-48-189.ec2.internal}                   Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "deis-builder" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=deis-builder pod=deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)"

      2m    2m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Started     Started container with docker id f2b7799712c3
      10m   2m  8   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Pulled      Container image "quay.io/deis/builder:v2.7.1" already present on machine
      2m    2m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Created     Created container with docker id f2b7799712c3; Security:[seccomp=unconfined]
      10m   1m  6   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Warning Unhealthy   Liveness probe failed: Get http://10.34.184.7:8092/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
      1m    1m  1   {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Normal  Killing     Killing container with docker id f2b7799712c3: pod "deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)" container "deis-builder" is unhealthy, it will be killed and re-created.
      7m    9s  26  {kubelet ip-10-0-48-189.ec2.internal}   spec.containers{deis-builder}   Warning BackOff     Back-off restarting failed docker container
      1m    9s  9   {kubelet ip-10-0-48-189.ec2.internal}                   Warning FailedSync  Error syncing pod, skipping: failed to "StartContainer" for "deis-builder" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=deis-builder pod=deis-builder-2995120344-mz2zg_deis(52027ebf-09f0-11e7-8bbf-0a73a2cd36e4)"

Answer 1

对于deis的工作流版本，helm ls显示了什么？

当您运行下面的命令时，容器的日志输出中显示了什么吗？

kubectl --命名空间deis日志deis-builder-2995120344-mz2zg

该日志位将有助于任何人试图帮助您找出不健康的构建器。

Answer 2

我的解决方案是删除deis并重新部署它。