如何使用Flume NG获取控制台采集的日志？

Question

我正在测试Flume NG (1.2.0)来收集日志。这是一个简单的测试，Flume收集日志文件flume_test.log并将收集到的日志作为sysout打印到控制台。conf/flume.conf是：

agent.sources = tail
agent.channels = memoryChannel
agent.sinks = loggerSink

agent.sources.tail.type = exec
agent.sources.tail.command = tail -f /Users/pj/work/flume_test.log
agent.sources.tail.channels = memoryChannel

agent.sinks.loggerSink.channel = memoryChannel
agent.sinks.loggerSink.type = logger

agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 100

我按如下方式运行了Flume：

$ $FLUME_HOME/bin/flume-ng agent --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/flume.conf --name agent1 -Dflume.root.logger=DEBUG,console

运行Flume后，控制台上的日志为：

Info: Sourcing environment configuration script /usr/local/lib/flume-ng/conf/flume-env.sh
+ exec /Library/Java/JavaVirtualMachines/jdk1.7.0_07.jdk/Contents/Home/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/usr/local/lib/flume-ng/conf:/usr/local/lib/flume-ng/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file /usr/local/lib/flume-ng/conf/flume.conf --name agent1
2012-09-12 18:23:52,049 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1
2012-09-12 18:23:52,052 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1
2012-09-12 18:23:52,054 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting
2012-09-12 18:23:52,056 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 9
2012-09-12 18:23:52,054 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting
2012-09-12 18:23:52,056 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] Node manager started
2012-09-12 18:23:52,057 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started
2012-09-12 18:23:52,058 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:/usr/local/lib/flume-ng/conf/flume.conf for changes
2012-09-12 18:23:52,058 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:/usr/local/lib/flume-ng/conf/flume.conf
2012-09-12 18:23:52,063 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: loggerSink Agent: agent
2012-09-12 18:23:52,063 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:loggerSink
2012-09-12 18:23:52,063 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created context for loggerSink: type
2012-09-12 18:23:52,063 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:loggerSink
2012-09-12 18:23:52,063 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration[agent]
SOURCES: {tail={ parameters:{command=tail -f /Users/pj/work/flume_test.log, channels=memoryChannel, type=exec} }}
CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }}
SINKS: {loggerSink={ parameters:{type=logger, channel=memoryChannel} }}

2012-09-12 18:23:52,068 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] Created channel memoryChannel
2012-09-12 18:23:52,082 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] Creating sink: loggerSink using LOGGER
2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] Post validation configuration for agent
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[agent]
SOURCES: {tail={ parameters:{command=tail -f /Users/pj/work/flume_test.log, channels=memoryChannel, type=exec} }}
CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }}
AgentConfiguration created with Configuration stubs for which full validation was performed[agent]
SINKS: {loggerSink=ComponentConfiguration[loggerSink]
  CONFIG: 
    CHANNEL:memoryChannel
}

2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)] Channels:memoryChannel

2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)] Sinks loggerSink

2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Sources tail

2012-09-12 18:23:52,085 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration  for agents: [agent]
2012-09-12 18:23:52,085 (conf-file-poller-0) [WARN - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:227)] No configuration found for this host:agent1

我认为Flume启动正常，所以我连续向flume_test.log添加了几行代码。但它不会将添加的行打印到控制台上的flume_test.log。

这个测试有什么问题？感谢您的评论和更正。

Answer 1

问题在于flume.conf (agent)中的代理名称与启动脚本中--name之后的代理名称(agent1)不匹配。

将name选项从--name agent1更改为--name agent后，问题就解决了。

谢谢我的同事莱尼。