Kubernetes nginx入口+ oauth2外部身份验证超时

Question

我正在尝试使用oauth2_proxy保护服务的状态页面，使用Azure AD作为外部身份验证提供者。目前，如果我浏览应用程序的公共url (https://sub.domain.com/service/hangfire)，我会得到504网关超时，它应该会指示我进行身份验证。

我主要遵循这个指南作为参考：https://msazure.club/protect-kubernetes-webapps-with-azure-active-directory-aad-authentication/

如果我禁用指导身份验证的注释，我可以毫无问题地进入公共状态页面。如果我浏览到https://sub.domain.com/oauth2，我会得到一个提示，要求我的提供商进行身份验证，这是我所期望的。我不确定问题出在入口配置中，但我无法在网上找到任何与此类似的案例，无论是堆栈溢出还是其他情况。

在这种情况下，所有内容(oauth部署、服务和入口规则)都位于“dev”名称空间中，但实际的入口部署除外，它位于自己的名称空间中。我不认为这会有什么不同，但是SSL终止是由集群外部的网关处理的。

oauth2部署：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: oauth2-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - name: oauth2-proxy
        image: quay.io/pusher/oauth2_proxy:v3.2.0
        imagePullPolicy: IfNotPresent
        args:
        - --provider=azure
        - --email-domain=domain.com
        - --upstream=http://servicename
        - --http-address=0.0.0.0:4180
        - --azure-tenant=id
        - --client-id=id
        - --client-secret=number
        env:
         - name: OAUTH2_PROXY_COOKIE_SECRET
           value: secret
        ports:
         - containerPort: 4180
           protocol : TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
spec:
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    app: oauth2-proxy 

入口规则：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: service-ingress1
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-url: https://sub.domain.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: https://sub.domain.com/oauth2/start?rd=$https://sub.domain.com/service/hangfire"
spec:
  rules:
  - host: sub.domain.com       
    http:
      paths:
      - path: /service/hangfire
        backend:
          serviceName: service
          servicePort: 80                    
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: service-oauth2-proxy
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: sub.domain.com      
    http:
      paths:
      - path: /oauth2
        backend:
          serviceName: oauth2-proxy
          servicePort: 4180           

当我浏览到url时，我得到了504个错误，但我在入口pod中没有看到任何错误。

Answer 1

我最终在这里找到了解决方案：https://github.com/helm/charts/issues/5958

我不得不为auth-url使用内部服务地址，这是我在其他地方没有见过的。

nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.development.svc.cluster.local:4180/oauth2/auth

Answer 2

这是我对Azure AD的oAuth代理所做的事情：

  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"

我一直在使用这个oAuth代理：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: oauth2-proxy
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - env:
          - name: OAUTH2_PROXY_PROVIDER
            value: azure
          - name: OAUTH2_PROXY_AZURE_TENANT
            value: xxx
          - name: OAUTH2_PROXY_CLIENT_ID
            value: yyy
          - name: OAUTH2_PROXY_CLIENT_SECRET
            value: zzz
          - name: OAUTH2_PROXY_COOKIE_SECRET
            value: anyrandomstring
          - name: OAUTH2_PROXY_HTTP_ADDRESS
            value: "0.0.0.0:4180"
          - name: OAUTH2_PROXY_UPSTREAM
            value: "http://where_to_redirect_to:443"
        image: machinedata/oauth2_proxy:latest
        imagePullPolicy: IfNotPresent
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP

Answer 3

我的设置类似于4c74356b41

OAuth2-代理部署

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - args:
        - --azure-tenant=TENANT-GUID
        - --email-domain=company.com
        - --http-address=0.0.0.0:4180
        - --provider=azure
        - --upstream=file:///dev/null
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              key: client-id
              name: oauth2-proxy
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              key: client-secret
              name: oauth2-proxy
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              key: cookie-secret
              name: oauth2-proxy
        image: quay.io/pusher/oauth2_proxy:v3.1.0
        name: oauth2-proxy

OAuth2-代理服务

apiVersion: v1
kind: Service
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: oauth2-proxy
  type: ClusterIP

OAuth2-代理入口

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  rules:
  - host: myapp.hostname.net
    http:
      paths:
      - backend:
          serviceName: oauth2-proxy
          servicePort: 80
        path: /oauth2

OAuth2-代理配置

apiVersion: v1
kind: Secret
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
data:
# Values below are fake
  client-id: AAD_CLIENT_ID
  client-secret: AAD_CLIENT_SECRET
  cookie-secret: COOKIE_SECRET

使用AAD Ingress的应用程序

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$request_uri
    nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
  labels:
    app: myapp
  name: myapp
  namespace: monitoring
spec:
  rules:
  - host: myapp.hostname.net
    http:
      paths:
      - backend:
          serviceName: myapp
          servicePort: 80
        path: /
  tls:
  - hosts:
    - myapp.hostname.net

需要完成的另一个步骤是将重定向URI添加到AAD应用程序注册。导航到您在Azure门户中的AAD应用注册>身份验证>添加https://myapp.hostname.net/oauth2/callback到重定向URI>保存