druid kafka索引服务设置

Question

我按照文档进行了编辑：

druid-0.9.2/conf/druid/_common/common.runtime.properties

并添加了：

"druid-kafka-indexing-service"

连接到druid.extensions.loadList，并重新启动了所有的德鲁伊服务：middlemanager、overlord、coordinator、broker、historical

我跑了一下：

curl -X 'POST' -H 'Content-Type:application/json' -d @kafka_connect/script.json druid_server:8090/druid/indexer/v1/task

但得到的是：

{"error":"Could not resolve type id 'kafka' into a subtype of [simple type, class io.druid.indexing.common.task.Task]\n at [Source: HttpInputOverHTTP@4c467f1c; line: 1, column: 4]"}

输入的json有：

{
  "type": "kafka",
  "dataSchema": {
    "dataSource": "sensors-kafka",
    "parser": {
      "type": "string",
      "parseSpec": {
        "format": "json",
        "timestampSpec": {
          "column": "timestamp",
          "format": "auto"
        },
        "dimensionsSpec": {
          "dimensions": ["machine", "key"],
          "dimensionExclusions": [
            "timestamp",
            "value"
          ]
        }
      }
    },
    "metricsSpec": [
      {
        "name": "count",
        "type": "count"
      },
      {
        "name": "value_sum",
        "fieldName": "value",
        "type": "doubleSum"
      },
      {
        "name": "value_min",
        "fieldName": "value",
        "type": "doubleMin"
      },
      {
        "name": "value_max",
        "fieldName": "value",
        "type": "doubleMax"
      }
    ],
    "granularitySpec": {
      "type": "uniform",
      "segmentGranularity": "HOUR",
      "queryGranularity": "NONE"
    }
  },
  "tuningConfig": {
    "type": "kafka",
    "maxRowsPerSegment": 5000000
  },
  "ioConfig": {
    "topic": "sensor",
    "consumerProperties": {
      "bootstrap.servers": "kafka_server:2181"
    },
    "taskCount": 1,
    "replicas": 1,
    "taskDuration": "PT1H"
  }
}

知道我做错了什么吗？根据文档：http://druid.io/docs/0.9.2-rc3/development/extensions-core/kafka-ingestion.html，type为kafka

有没有办法检查扩展是否正确加载，或者我是否必须在每个组件的runtime.properties中指定扩展

Answer 1

我遇到了类似的问题，我通过修改"conf/druid/_common/common.runtime.properties“的文件，在druid.extensions.loadList中添加”druid kafka-indexing-service“来修复它，现在它的显示是这样的：

druid.extensions.loadList=["druid-parser-route", "mysql-metadata-storage", "druid-kafka-indexing-service"]

希望可以帮助任何人

Answer 2

supervisors json规范将在overlord /druid/indexer/v1/supervisor上的此端点上发送。

curl -X POST -H 'Content-Type: application/json' -d @kafka_connect/script.json http://druid_server:8090/druid/indexer/v1/supervisor

Answer 3

如果使用dockerized apache/druid，则需要设置

druid.extensions.loadList=“druid cached”，“druid datasketches”，“druid lookups-cached-global”，"postgresql-metadata-storage"，“druid kafka-indexing-service”

在文件/opt/druid/conf/druid/cluster/_common/common.runtime.properties中