Cygnus不会持久化数据，因为：“生成的命名空间名称太长，不能超过127”

Question

几天后，由于cygnus随机地坚持更新，我打破了我的头，我在日志中发现生成的名称空间的大小太长。

我在CentOS7上工作，我的实体使用标准类型: BikeHireDockingStation，错误提示生成的命名空间太长(127个字符)。它会生成167个：

sth_malaga.sth_/_urn:ngsi-ld:BikeHireDockingStation:10_BikeHireDockingStation.aggr.$_id.entityId_1__id.entityType_1__id.attrName_1__id.resolution_1__id.origin_1

但是即使我把类型改成自行车，大小也是124。

在这里您可以看到我在调用时获得的日志错误的一部分：$ docker container logs fiware-cygnus

time=2019-09-09T21:14:14.176Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | 
subsrv=N/A | comp=cygnus-ngsi | op=processRollbackedBatches | 
msg=com.telefonica.iot.cygnus.sinks.NGSISink[399] : CygnusPersistenceError. -, 
Command failed with error 67: 'namespace name generated from index name 
"sth_malaga.sth_/_urn:ngsi-ld:BikeHireDockingStation:10_BikeHireDockingStation.aggr.$_id.entityId_1__id.entityType_1__id.attrName_1__id.resolution_1__id.origin_1"
 is too long (127 byte max)' on server mongo-db:27017. 
The full response is { "ok" : 0.0, "errmsg" : "namespace name generated from index name 
\"sth_malaga.sth_/_urn:ngsi-ld:BikeHireDockingStation:10_BikeHireDockingStation.aggr.$_id.entityId_1__id.entityType_1__id.attrName_1__id.resolution_1__id.origin_1\" 
is too long (127 byte max)", "code" : 67, "codeName" : "CannotCreateIndex" }.
 Stack trace: 
[com.telefonica.iot.cygnus.sinks.NGSISTHSink$STHAggregator.persist(NGSISTHSink.java:374), 
com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistBatch(NGSISTHSink.java:108), 
com.telefonica.iot.cygnus.sinks.NGSISink.processRollbackedBatches(NGSISink.java:391), 
com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:373), 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67), 
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145), java.lang.Thread.run(Thread.java:748)]

有没有可能一个类型的最大大小是5？(对于5，名称空间的大小是126)

你能帮我解决这个问题吗？

我尝试过不同的场景：

fiware/orion:最新的fiware/cygnus-common:最新的mongo:3.6

下面的结果是：

time=2019-09-12T17:12:17.071Z | lvl=WARN | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-common | op=doPost | msg=org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet[186] : Received bad request from client. 
org.apache.flume.source.http.HTTPBadRequestException: Request has invalid JSON Syntax.
    at org.apache.flume.source.http.JSONHandler.getEvents(JSONHandler.java:119)
    at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:184)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:814)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_ARRAY but was BEGIN_OBJECT at line 1 column 2
    at com.google.gson.Gson.fromJson(Gson.java:806)
    at com.google.gson.Gson.fromJson(Gson.java:761)
    at org.apache.flume.source.http.JSONHandler.getEvents(JSONHandler.java:117)
    ... 16 more
Caused by: java.lang.IllegalStateException: Expected BEGIN_ARRAY but was BEGIN_OBJECT at line 1 column 2
    at com.google.gson.stream.JsonReader.expect(JsonReader.java:339)
    at com.google.gson.stream.JsonReader.beginArray(JsonReader.java:306)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:79)
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:60)
    at com.google.gson.Gson.fromJson(Gson.java:795)
    ... 18 more

配置为:fiware/orion:最新的fiware/cygnus-ngsi:1.13.0 mongo:3.6

结果是：

time=2019-09-12T17:22:15.466Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=processRollbackedBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[399] : CygnusPersistenceError. -, Command failed with error 67: 'namespace name generated from index name "sth_malaga.sth_/_EstacionBici:10_BikeHireDockingStation.aggr.$_id.entityId_1__id.entityType_1__id.attrName_1__id.resolution_1__id.origin_1" is too long (127 byte max)' on server mongo-db:27017. The full response is { "ok" : 0.0, "errmsg" : "namespace name generated from index name \"sth_malaga.sth_/_EstacionBici:10_BikeHireDockingStation.aggr.$_id.entityId_1__id.entityType_1__id.attrName_1__id.resolution_1__id.origin_1\" is too long (127 byte max)", "code" : 67, "codeName" : "CannotCreateIndex" }. Stack trace: [com.telefonica.iot.cygnus.sinks.NGSISTHSink$STHAggregator.persist(NGSISTHSink.java:374), com.telefonica.iot.cygnus.sinks.NGSISTHSink.persistBatch(NGSISTHSink.java:108), com.telefonica.iot.cygnus.sinks.NGSISink.processRollbackedBatches(NGSISink.java:391), com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:373), org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67), org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145), java.lang.Thread.run(Thread.java:748)]

最后配置:fiware/orion:最新的fiware/cygnus-ngsi:最新的mongo:3.6

结果是：

time=2019-09-12T17:25:48.943Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[492] : Batch accumulation time reached, the batch will be processed as it is
time=2019-09-12T17:25:49.007Z | lvl=DEBUG | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=run | msg=com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$PeriodicalNameMappingsReader[205] : [nmi] The configuration has not changed

但是它不会像这样创建分析mongo的sth_malaga数据库：$docker exec -it db-mongo bash

> show dbs
admin         0.000GB
config        0.000GB
local         0.000GB
orion         0.000GB
orion-malaga  0.000GB
> 

如你所见，我几乎要疯了。你能推荐最好的天鹅座，猎户座和芒果版本吗？

version: "3.5"
services:
  # Orion es el context broker
  orion:
    image: fiware/orion:latest
    hostname: orion
    container_name: fiware-orion
    depends_on:
      - mongo-db
    networks:
      - default
    expose:
      - "1026"
    ports:
      - "1026:1026"
    command: -dbhost mongo-db -logLevel DEBUG
    healthcheck:
      test: curl --fail -s http://orion:1026/version || exit 1

  # Configurando Cygnus para que almacene las actualizaciones que consultara STH-Comet
  cygnus:
    image: fiware/cygnus-ngsi:latest
    hostname: cygnus
    container_name: fiware-cygnus
    depends_on:
      - mongo-db-cygnus
    networks:
      - default
    expose:
      - "5050"
      - "5080"
    ports:
      - "5050:5050"
      - "5080:5080"
    environment:
      - "CYGNUS_MONGO_HOSTS=mongo-db-cygnus:27017" # servidor donde se hará la persistencia de datos
      - "CYGNUS_LOG_LEVEL=DEBUG" # Nivel de log para Cygnus
      - "CYGNUS_SERVICE_PORT=5050" # Puerto de Cynus en el que escucha las actualizaciones
      - "CYGNUS_API_PORT=5080" # Puerto de Cygnus para operacion
    healthcheck:
      test: curl --fail -s http://localhost:5080/v1/version || exit 1

  # STH-Comet consumira los datos almacenados en Mongo DB para el historico a corto plazo
  sth-comet:
    image: fiware/sth-comet:latest
    hostname: sth-comet
    container_name: fiware-sth-comet
    depends_on:
      - cygnus
      - mongo-db-cygnus
    networks:
      - default
    ports:
      - "8666:8666"
    environment:
      - STH_HOST=0.0.0.0
      - STH_PORT=8666
      - DB_PREFIX=sth_
      - DB_URI=mongo-db-cygnus:27017
      - LOGOPS_LEVEL=DEBUG
    healthcheck:
      test: curl --fail -s http://localhost:8666/version || exit 1

  # Database orion
  mongo-db:
    image: mongo:3.6
    hostname: mongo-db
    container_name: db-mongo
    expose:
      - "27017"
    ports:
      - "27017:27017"
    networks:
      - default
    command: --bind_ip_all --smallfiles
    volumes:
      - mongo-db:/data

# Database cygnus
  mongo-db-cygnus:
    image: mongo:latest
    hostname: mongo-db-cygnus
    container_name: db-mongo-cygnus
    expose:
      - "27018"
    ports:
      - "27018:27017"
    networks:
      - default
    command: --bind_ip_all 
    volumes:
      - mongo-db-cygnus:/data

networks:
  default:
    ipam:
      config:
        - subnet: 172.18.1.0/24

volumes:
  mongo-db: ~
  mongo-db-cygnus: ~

我已经试过了。但在本例中，在第二个数据库中，cygnus没有写入任何内容。如果我将cygnus版本更改为( image: fiware/cygnus-ngsi:1.14.0)，它才能像以前一样工作(它只更新一些实体)。这意味着使用两个版本的数据库不会带来任何改进。

Answer 1

这个问题的根本原因在天鹅座之外。它在 the index length limit that MongoDB prior to version 4.2 has中。

根据Cygnus的版本，它以不同的方式处理问题：

• 1.14.0之前的版本抛出异常，中断数据持久化操作，并在日志中打印丑陋的Java堆栈跟踪。我知道这是您的情况。
• 1.14.0及更高版本正确地处理了错误情况，因此不创建索引(在有关它的日志中打印警告跟踪)，但数据是持久的。因此，在这种情况下，天鹅座可以完成它的工作，尽管如果你有大量的数据，你可能会遇到访问数据的慢查询。

最好的解决方案是将MongoDB升级到4.2，这应该会完全消除这个问题。但在这种情况下，您应该考虑两件事：

• MongoDB 4.2还没有得到Cygnus的正式支持，尽管user reports是肯定的。
• 我不知道Orion Context Broker是否可以与MongoDB 4.2一起工作。我不知道任何正面或负面的报告，所以我的建议是您测试它:)。在最坏的情况下，您可以使用两个独立的MongoDB实例(4.2用于天鹅座，3.6用于猎户座)。