码头组合中的分布式安全MinIO

Question

我在Docker中有一个相当复杂的系统。所有东西都在一个大的docker-compose文件中运行。以前，所有东西都运行在我的Docker群中的一个(管理器)节点上，所以我已经为我的域(使用certbot)生成了一个CERT，并且在我的撰写文件中使用了下面的MinIO服务：

  object_storage:
    image: minio/minio:RELEASE.2020-12-10T01-54-29Z
    ports:
      - 9000:9000
    environment:
      MINIO_ACCESS_KEY_FILE: object_storage_user
      MINIO_SECRET_KEY_FILE: object_storage_password
    command: server /data
    depends_on:
      - fluentd
    volumes:
      - object_storage_data:/data
      - ./certs/domain.crt:/root/.minio/certs/public.crt
      - ./certs/domain.key:/root/.minio/certs/private.key
    networks:
      - object_storage_net
    secrets:
      - object_storage_user
      - object_storage_password
    logging:
      driver: "fluentd"
      options:
        fluentd-address: ${SYSTEM_HOST}:24224
        tag: object-storage

上述实现如预期的那样工作！但是现在我有两个分开的服务器来运行MinIO。这些服务器作为工作节点连接到我的Docker Swarm。MinIO不应该在管理器节点上运行(只在两个分离的工作节点上运行)！

>>> docker node ls
ID                                          HOSTNAME    STATUS      AVAILABILITY   MANAGER STATUS   ENGINE VERSION
mcbkz9m5nzf7oa3fiqk0lf4qo *  manager         Ready           Active                    Leader                    20.10.1
dz4e3k70g8ik2z4bcx8u0ft9ao   minio_1          Ready           Active                                                   20.10.2
r0qpdn2guyy5773vo8vg2trzo    minio_2          Ready           Active                                                   20.10.2

我的当前MinIO实现在我的docker-compose文件中：

object_storage_1:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000:9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_1_1:/data1
     - object_storage_data_1_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_1
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

 object_storage_2:
   image: minio/minio:RELEASE.2020-12-10T01-54-29Z
   ports:
     - 9000
   environment:
     MINIO_ACCESS_KEY_FILE: object_storage_user
     MINIO_SECRET_KEY_FILE: object_storage_password
   command: server https://object_storage_{1...2}/data{1...2}
   depends_on:
     - fluentd
   volumes:
     - object_storage_data_2_1:/data1
     - object_storage_data_2_2:/data2
     - ./certs/domain.crt:/root/.minio/certs/public.crt
     - ./certs/domain.key:/root/.minio/certs/private.key
   networks:
     - object_storage_net
   secrets:
     - object_storage_user
     - object_storage_password
   deploy:
     restart_policy:
       condition: on-failure
     placement:
       constraints:
         - node.hostname == minio_2
   logging:
     driver: "fluentd"
     options:
       fluentd-address: ${SYSTEM_HOST}:24224
       tag: object-storage

如果我检查我的MinIO服务实例的日志，就会得到以下错误：

Unable to read 'format.json' from https://object_storage_1:9000/data1: Post "https://object_storage_1:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_1
Unable to read 'format.json' from https://object_storage_2:9000/data1: Post "https://object_storage_2:9000/minio/storage/data1/v22/readall?disk-id=&file-path=format.json&volume=.minio.sys": x509: certificate is valid for my_domain.app, not object_storage_2

但是我可以在9000端口上找到MinIO，只是有一个弹出错误：

​

​

我只想通过我的域(my_domain.app:9000)访问my_domain.app:9000。在本例中，MinIO不使用真正的服务器名，但是它使用“虚拟”Docker网络(例如：https://object_storage_2:9000)。

我的问题：

如何为“虚拟”码头网络(例如: object_storage_2)?

• Where )生成证书？
• 如何生成“虚拟”码头网络的证书(例如: object_storage_1或object_storage_1)？
• 是可以用我的生成(为我的域)证书解决的？

我对每一个提示和解决方案都敞开心扉！

Answer 1

我不得不将(域) CERT文件放到minio/certs/CAs文件夹中，而不是/root/.minio/certs文件夹。此外，如果没有将CERT复制到节点，则必须将CERT复制到工作节点(分离的服务器)，服务在worker节点上找不到它。

正确的volumes 参数如下所示：

volumes:
  - object_storage_data_1_1:/data1
  - object_storage_data_1_2:/data2
  - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt

我的几个MinIO服务的一个工作服务：

  object-storage-1:
    image: minio/minio:RELEASE.2021-08-17T20-53-08Z
    expose:
      - "9000"
      - "9001"
    environment:
      MINIO_ACCESS_KEY_FILE: object_storage_user
      MINIO_SECRET_KEY_FILE: object_storage_password
      MINIO_BROWSER_REDIRECT_URL: https://${SYSTEM_HOST}:9001
      MINIO_SERVER_URL: https://${SYSTEM_HOST}:9000
    command: server --console-address ":9001" http://object-storage-{1...4}/data{1...2}
    hostname: object-storage-1
    depends_on:
      - fluentd
    volumes:
      - object_storage_data_1_1:/data1
      - object_storage_data_1_2:/data2
      - ./certs/domain.crt:/root/.minio/certs/CAs/public.crt
    networks:
      - object_storage_net
    secrets:
      - object_storage_user
      - object_storage_password
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.hostname == minio1
    logging:
      driver: "fluentd"
      options:
        fluentd-address: ${SYSTEM_HOST}:24224
        tag: object-storage

和我必须创建一个配置：

upstream minio {
    server object-storage-1:9000;
    server object-storage-2:9000;
    server object-storage-3:9000;
    server object-storage-4:9000;
}

upstream console {
    ip_hash;
    server object-storage-1:9001;
    server object-storage-2:9001;
    server object-storage-3:9001;
    server object-storage-4:9001;
}

server {
    listen              9000 ssl;
    listen              [::]:9000 ssl;
    server_name         my.server.com;
    ssl_certificate     /ssl/domain.crt;
    ssl_certificate_key /ssl/domain.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;


    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio;

    }
}

server {
    listen              9001 ssl;
    listen              [::]:9001 ssl;
    server_name         my.server.com;
    ssl_certificate     /ssl/domain.crt;
    ssl_certificate_key /ssl/domain.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        # This is necessary to pass the correct IP to be hashed
        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://console;

    }
}