使用Logstash在Mongodb和ElasticSearch之间同步数据

Question

我正在尝试将我的MongoDB数据库与elasticsearch同步，并使用Kibana上的数据。我正在使用插件logstash-input-mongoDb和logstash-output-elasticsearch.

该问题出现在Logstash中的配置文件中，以连接到MongoDB。一旦安装了这两个插件，我就创建了一个配置文件，以提及输入源详细信息(MongoDB)和输出(ElasticSearch)。我在这里写过：

input {
       uri => 'mongodb://localhost:27017/t_data/data'
       placeholder_db_dir => '/opt/logstash-mongodb/'
       placeholder_db_name => 'logstash_sqlite.db'
       collection => 't_data'
       batch_size => 5000
}

filter {
}

output {
        stdout {
                codec => rubydebug
       }
       elasticsearch {
               action => "index"
               index => "mongo_t_data"
               hosts => ["localhost:9200"]
       }
}

然后在logstash中的bin目录中运行以下命令，其中将配置文件保存为mongodata.conf。

 \logstash-7.10.1\bin>logstash -f mongodata.conf

我得到以下错误：

 [2021-01-17T00:11:22,730][ERROR][logstash.agent           ] Failed to execute action 
 {:action=>LogStash::PipelineAction::Create/pipeline_id:main, 
 :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at 
 line 2, column 13 (byte 22) after input {\r\n        uri ", :backtrace=>["C:/PESU/logstash-7.10.1- 
 windows-x86_64/logstash-7.10.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", 
 "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", 
 "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "C:/PESU/logstash-7.10.1- 
 windows-x86_64/logstash-7.10.1/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", 
 "C:/PESU/logstash-7.10.1-windows-x86_64/logstash-7.10.1/logstash- 
 core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "C:/PESU/logstash-7.10.1-windows- 
 x86_64/logstash-7.10.1/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
 [2021-01-17T00:11:23,310][INFO ][logstash.agent           ] Successfully started Logstash API 
 endpoint {:port=>9600}
 [2021-01-17T00:11:28,026][INFO ][logstash.runner          ] Logstash shut down.
 [2021-01-17T00:11:28,058][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: 
 Logstash stopped processing because of an error: (SystemExit) exit

我在这里附上了截图：

​

我的目标是成功地将logstash连接到Mongodata和elasticsearch，以便在需要时能够同步数据和添加文档。我怀疑使用记事本创建.conf文件是个问题。

Answer 1

MongoDB，因为Version3.6有一个名为changesteams的特性，可以用于此任务，请在此处查看：https://medium.com/@montumodi/realtime-sync-from-mongodb-to-elasticsearch-using-change-streams-2559ad312011

Answer 2

您的输入配置错误。它缺少输入插件的类型。

应该是这样的：

input {
    mongodb {
        uri => 'mongodb://localhost:27017/t_data/data'
        placeholder_db_dir => '/opt/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 't_data'
        batch_size => 5000
    }
}

试着改变一下，看看它是否有效。