使用SLF4j配置Apache Beam和Kafka客户端的日志记录

Question

我使用Apache光束vs Kafka客户端来处理我的数据，但是库Kafka客户端产生了很多消息，所以它们在堆栈跟踪中真的很混乱，我如何更改这个包的日志级别？

依赖项的版本：slf4jVersion=1.7.29、apacheBeamVersion=2.18.0

我已经向src/main/resources/添加了一个包含以下内容的文件log4j.properties，但它似乎不起作用

log4j.rootLogger=WARN, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
log4j.logger.org.apache.kafka.clients.consumer.internals=ERROR

build.gradle

dependencies {
    implementation "org.apache.beam:beam-sdks-java-core:${apacheBeamVersion}"
    implementation "org.apache.beam:beam-runners-google-cloud-dataflow-java:${apacheBeamVersion}"
    implementation "org.apache.beam:beam-sdks-java-io-google-cloud-platform:${apacheBeamVersion}"
    implementation "org.apache.beam:beam-sdks-java-io-kafka:${apacheBeamVersion}"
    runtimeOnly "org.slf4j:slf4j-jdk14:${slf4jVersion}"
    ...
}

我希望在日志中避免的日志消息：

INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Seeking to LATEST offset of partition custom-group-0
Mar 16, 2020 1:55:34 PM org.apache.kafka.clients.consumer.internals.SubscriptionState maybeSeekUnvalidated
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Resetting offset for partition custom-group-0 to offset 9347.
Mar 16, 2020 1:55:35 PM org.apache.kafka.clients.consumer.internals.SubscriptionState lambda$requestOffsetReset$3
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Seeking to LATEST offset of partition custom-group-0
Mar 16, 2020 1:55:35 PM org.apache.kafka.clients.consumer.internals.SubscriptionState maybeSeekUnvalidated
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Resetting offset for partition custom-group-0 to offset 9347.
Mar 16, 2020 1:55:36 PM org.apache.kafka.clients.consumer.internals.SubscriptionState lambda$requestOffsetReset$3
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Seeking to LATEST offset of partition custom-group-0
Mar 16, 2020 1:55:36 PM org.apache.kafka.clients.consumer.internals.SubscriptionState maybeSeekUnvalidated
INFO: [Consumer clientId=consumer-3, groupId=Reader-0_offset_consumer_1776081144_custom-group-group-local] Resetting offset for partition custom-group-0 to offset 9347.

Answer 1

您不应该使用slf4j-jdk14，因为它将java.util.logging作为实现，并且不会考虑log4j2配置。所以把这个去掉

runtimeOnly "org.slf4j:slf4j-jdk14:${slf4jVersion}"

为了使用log4j2配置，您需要一个log4j2实现。我用过

implementation "org.apache.logging.log4j:log4j-slf4j-impl:2.13.3"

如果您想像我一样使用yaml配置，只需添加以下两个依赖项

implementation "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.5"
implementation "com.fasterxml.jackson.core:jackson-databind:2.10.5"

在我的例子中，我在我的一个传递依赖中有另一个slf4j实现，我必须排除它才能使它工作。你要知道。