在码头运行CEPH

Question

我的DevOps技能很弱，对Docker不太了解。我需要和Ceph交Java朋友。首先，我试图在我的码头集装箱中运行CEPH。我看了Intellij，了解到并不是所有的容器都在运行。我的船坞组合看起来是这样的：

version: '2.1'

services:

  mon1:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "mon"
    environment:
      MON_IP: ${MON1_IP}
      CEPH_PUBLIC_NETWORK: ${MON1_CEPH_PUBLIC_NETWORK}
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    networks:
      - ceph-cluster-net

  mgr:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "mgr"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net
    ports:
      - ${DASHBOARD_PORT}:7000
    expose:
      - ${DASHBOARD_PORT}

  osd1:
    pid: host
    privileged: true
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "osd"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
      - /dev/:/dev/
    environment:
      OSD_DEVICE: ${OSD1_DEVICE}
      OSD_TYPE: disk
      OSD_FORCE_ZAP: 1
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net

  osd2:
    pid: host
    privileged: true
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "osd"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
      - /dev/:/dev/
    environment:
      OSD_DEVICE: ${OSD2_DEVICE}
      OSD_TYPE: disk
      OSD_FORCE_ZAP: 1
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net

  osd3:
    pid: host
    privileged: true
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "osd"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
      - /dev/:/dev/
    environment:
      OSD_DEVICE: ${OSD3_DEVICE}
      OSD_TYPE: disk
      OSD_FORCE_ZAP: 1
    depends_on:
      - mon1
    networks:
      - ceph-cluster-net

  rgw1:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "rgw"
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    depends_on:
      - osd1
      - osd2
      - osd3
    networks:
      - ceph-cluster-net
    ports:
      - ${RGW_PORT}:8080

  mds1:
    image: ceph/daemon:${CEPH_CONTAINER_VERSION}
    command: "mds"
    environment:
      CEPHFS_CREATE: 1
    volumes:
      - ${VOLUMES_PATH}/ceph:/etc/ceph
      - ${VOLUMES_PATH}/lib/:/var/lib/ceph/
    depends_on:
      - osd1
      - osd2
      - osd3
    networks:
      - ceph-cluster-net

networks:
  ceph-cluster-net:
    driver: bridge
    external: true

我有一个env示例文件：

CEPH_CONTAINER_VERSION=v3.1.0-stable-3.1-luminous-ubuntu-16.04-x86_64
VOLUMES_PATH=~/volumes_data/ceph
DASHBOARD_PORT=6000
RGW_PORT=6080
OSD1_DEVICE=/dev/sdb
OSD2_DEVICE=/dev/sdc
OSD3_DEVICE=/dev/sdd
MON1_IP=172.18.0.2
MON1_CEPH_PUBLIC_NETWORK=172.18.0.0/24

在运行docker-组合之前，我运行以下命令：

docker network create --driver bridge ceph-cluster-net
cp env-example .env
docker-compose up -d

不运行图像的日志如下所示：MON1:

importing contents of /var/lib/ceph/bootstrap-osd/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-mds/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-rgw/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /var/lib/ceph/bootstrap-rbd/ceph.keyring into /etc/ceph/ceph.mon.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring
2019-12-24 12:18:53.293858 7ff156f2a080 -1 unable to find any IP address in networks '172.18.0.0/24' interfaces ''

经理：

2019-12-24 12:18:54  /entrypoint.sh: static: does not generate config
2019-12-24 12:18:54  /entrypoint.sh: static: does not generate the admin key, so we can not get it.
2019-12-24 12:18:54  /entrypoint.sh: static: make it available with the help of your configuration management system.
2019-12-24 12:18:54  /entrypoint.sh: static: ceph-ansible is a good candidate to deploy a containerized version of Ceph.
2019-12-24 12:18:54  /entrypoint.sh: static: ceph-ansible will help you fetching the keys and push them on the right nodes.
2019-12-24 12:18:54  /entrypoint.sh: static: if you're interested, please visit: https://github.com/ceph/ceph-ansible
2019-12-24 12:23:54.568294 7f3373429700  0 monclient(hunting): authenticate timed out after 300
2019-12-24 12:23:54.568338 7f3373429700  0 librados: client.admin authentication error (110) Connection timed out
[errno 110] error connecting to the cluster

OSD1:

2019-12-24 12:18:50  /entrypoint.sh: static: does not generate config
2019-12-24 12:19:00  /entrypoint.sh: Timed out while trying to reach out to the Ceph Monitor(s).
2019-12-24 12:19:00  /entrypoint.sh: Make sure your Ceph monitors are up and running in quorum.
2019-12-24 12:19:00  /entrypoint.sh: Also verify the validity of client.bootstrap-osd keyring.

OSD2和OSD3:

2019-12-24 12:10:44  /entrypoint.sh: static: does not generate config
2019-12-24 12:10:44  /entrypoint.sh: ERROR- The device pointed by OSD_DEVICE (/dev/sdc) doesn't exist !
2019-12-24 12:18:48  /entrypoint.sh: static: does not generate config
2019-12-24 12:18:48  /entrypoint.sh: ERROR- The device pointed by OSD_DEVICE (/dev/sdc) doesn't exist !

RGW1:

2019-12-24 12:18:52  /entrypoint.sh: static: does not generate config
2019-12-24 12:19:02  /entrypoint.sh: Timed out while trying to reach out to the Ceph Monitor(s).
2019-12-24 12:19:02  /entrypoint.sh: Make sure your Ceph monitors are up and running in quorum.
2019-12-24 12:19:02  /entrypoint.sh: Also verify the validity of client.bootstrap-rgw keyring.

MDS1:

2019-12-24 12:18:53  /entrypoint.sh: static: does not generate config

在创建坞-撰写时，我使用了以下文章：https://github.com/VasiliyLiao/ceph-docker-compose

Answer 1

看着原木

unable to find any IP address in networks '172.18.0.0/24' interfaces ''

我先去查一下网络。也许更好的是，如果你对devops很软弱，那么尝试卡夫，只要做几个改变，你就可以让你的ceph做好准备。

Answer 2

我也有同样的问题。找到"mon1“配置应该有ipv4_address，并确保MON_IP与该ipv4_address相等。示例：

environment:
  MON_IP: 172.28.0.10
  CEPH_PUBLIC_NETWORK: 172.28.0.0/24
networks:
  ceph_network :
    ipv4_address: 172.28.0.10

我不确定这是解决这个问题的正确方法，但它对我有用。

Answer 3

您不需要使用docker组合，这里有ceph项目，您可以在docker容器中运行ceph服务和OSD服务，它之后的部署和维护非常简单。

https://github.com/ceph/ceph-ansible