用Logstash对弹性搜索的Logary -简单配置

Question

我几乎一整天都在尝试使用Logary > Logstash > ElasticSearch为我的ElasticSearch应用程序设置日志记录。我已经尝试了所有我能想到的配置的变化，并且在文档中浏览了几个小时，但都没有结果。

这里怎么了？

 var logary = LogaryFactory.New("Logary Specs", with => with

                .Target<Logstash.Builder>("ls", conf => conf.Target
                        .Hostname("localhost")
                        .Port(1939)
                        .EventVersion(Logstash.EventVersion.One)
                        .Done())

                .Target<Debugger.Builder>("debugger")
            );

这里是logstash.conf：

input { 
    stdin {}

    tcp {
         codec => json_lines { charset => "UTF-8" }
         host  => localhost
         port  => 1939
         type  => "app"
     }
}

output {

   stdout {}
   elasticsearch {
      host => localhost
      protocol => "http"
   }
}

我尝试过不使用output.elasticsearch.protocol，就像在docs中使用主机=> "0.0.0.0“一样，以及两者之间的每一个组合。

我已经安装了9200和Kibana，我可以看到日志通过stdin/stdout显示，但不能从我使用Logary的应用程序日志中看到。还验证了Logary调试器目标是否工作。

我的管道怎么了？

Logstash 1.4.2

ES 1.3.4

Answer 1

您需要一些组件来调试它。

我为您编写了一个用于调试它的示例程序；这是可以工作的最简单的东西：

open System

open Logary
open Logary.Configuration
open Logary.Targets

[<EntryPoint>]
let main argv = 
  use logary =
    withLogary' "logstash-sample" (
      withTargets [
        Logstash.create (Logstash.LogstashConf.Create "10.0.0.120") "logstash"
        Debugger.create Debugger.empty "debugger"
      ] >>
      withRules [
        Rule.createForTarget "logstash"
        Rule.createForTarget "debugger"
      ] >>
      withInternalTargets Verbose [
        Debugger.create (Debugger.empty) "console"
      ]
    )

  let logger = logary.GetLogger "logstash-sample"
  Logger.debug logger "Hello World"

  Console.ReadKey true |> ignore
  0

这是它的预期调试器输出(不是控制台)。

'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\logstash_sample.exe', Symbols loaded.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Logary.dll', Symbols loaded.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\NodaTime.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\FSharp.Core\v4.0_4.3.0.0__b03f5f7f11d50a3a\FSharp.Core.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Newtonsoft.Json.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Intelliplan.JsonNet.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\Intelliplan.JsonNet.NodaTime.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'Y:\dev\ip\_scratch\logstash-sample\bin\Debug\FSharp.Actor.dll'
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
D 2014-10-18T10:25:42.4192736+00:00: supervisor status: Running [Logary.Registry.runRegistry]
I 2014-10-18T10:25:42.4192736+00:00: initing logstash target, connecting to 10.0.0.120:1936... [Logary.Targets.Logstash.logstashLoop]
D 2014-10-18T10:25:42.4383230+00:00: scheduler status: Running [Logary.Registry.runRegistry]
D 2014-10-18T10:25:42.4473276+00:00: registry status: Running [Logary.Registry.runRegistry]
D 2014-10-18T10:25:42.4977127+00:00: will poll metrics every 0:00:00:00.5 [Logary.Registry.registry]
D 2014-10-18T10:25:42.5335455+00:00: Hello World [logstash-sample]
I 2014-10-18T10:25:42.5515369+00:00: initing logstash target, connected [Logary.Targets.Logstash.logstashLoop]
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Numerics\v4.0_4.0.0.0__b77a5c561934e089\System.Numerics.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.ComponentModel.DataAnnotations\v4.0_4.0.0.0__31bf3856ad364e35\System.ComponentModel.DataAnnotations.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Runtime.Serialization\v4.0_4.0.0.0__b77a5c561934e089\System.Runtime.Serialization.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml.Linq\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.Linq.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
'logstash_sample.exe' (Managed (v4.0.30319)): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_64\System.Data\v4.0_4.0.0.0__b77a5c561934e089\System.Data.dll', Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
D 2014-10-18T10:25:42.7268749+00:00: polling metrics [Logary.Registry.registry]
D 2014-10-18T10:25:42.8571421+00:00: send-recv stream not open, opening [Logary.Targets.Logstash.logstashLoop]
I 2014-10-18T10:25:43.1096339+00:00: start shutdown [Logary.Configuration.Config.shutdown]
I 2014-10-18T10:25:43.1186538+00:00: flush start [Logary.Registry.registry]
I 2014-10-18T10:25:43.1667422+00:00: flush Ack [Logary.Registry.registry]
I 2014-10-18T10:25:43.1787861+00:00: shutdown metrics polling [Logary.Registry.registry]
I 2014-10-18T10:25:43.1798152+00:00: shutdown schedules [Logary.Registry.registry]
I 2014-10-18T10:25:43.1798152+00:00: shutdown targets [Logary.Registry.registry]
I 2014-10-18T10:25:43.2018104+00:00: shutdown Ack [Logary.Registry.registry]
I 2014-10-18T10:25:43.2018104+00:00: shutting down immediately [Logary.Registry.registry]
I 2014-10-18T10:25:43.2118258+00:00: stop shutdown [Logary.Configuration.Config.shutdown]
The program '[4820] logstash_sample.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).

此代码如下：

https://github.com/logary/logary/commit/0dca450677dc1ca691292d379a734c7051743c1c

您可以执行Install-Package Intelliplan.Logary -Pre，然后复制Logary.dll-dll文件。确保您有相同的输出。

如果您可以按照上面的输出成功地连接到logstash，您可以通过打印到它的日志来验证它是否正在接收事件或失败；

tail -f /var/log/logstash/logstash-agent.log

Answer 2

有几件事你可以试试。首先，再次检查您的elasticsearch集群是否实际上是绿色/黄色。我浪费了几个小时，因为我在我配置的elasticsearch索引模板中指定了更多的副本。

当然，请检查elasticsearch和logstash日志中的任何错误。

接下来，尝试使用

elasticsearch_http { host => "localhost" }

这使用的是http协议而不是内部协议，您可以使用与logstash捆绑的完全相同的弹性搜索版本。

最后，如果这不起作用，我建议在logstash目录中插入并修改实现插件的ruby脚本。它在工作时很好，但是当它不工作时，日志记录就会有些缺乏；这很容易用一些日志语句来修复。