如何设置火花放电中第三方罐的日志级别

Question

我使用星火将数据从Hive写到data，使用这个jar：kinetica-spark-7.0.6.1-jar-with-dependencies.jar。但是，当我运行spark-submit时，jar中的记录器正在打印JDBC连接字符串，其凭据如下：

...
22/03/03 03:00:58 INFO spark.LoaderParams: Using JDBC connection string: jdbc:kinetica://10.xx.xx.xx:9191;UID=xxx;PWD=xxx
22/03/03 03:00:58 INFO spark.LoaderParams: Installing truststore to bypass certificate check.
22/03/03 03:00:58 INFO spark.LoaderParams: Using URL(s) http://10.xx.xx.xx:9191to create a GPUdb connection
22/03/03 03:00:59 INFO spark.LoaderParams: Connecting to http://10.xx.xx.xx:9191 as user <xxx>
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 47
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 42
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 53
22/03/03 03:00:59 INFO spark.ContextCleaner: Cleaned accumulator 52
...

它来自于以下代码：kinetica.spark.LoaderParams

我希望将这个spark.LoaderParams类的日志级别设置为更高的级别，这样连接字符串就不会显示在日志中的任何位置。

有什么办法我能做到吗？

下面是我的log4j.properties配置：

log4j.rootLogger=${root.logger}
root.logger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
shell.log.level=WARN
log4j.logger.org.eclipse.jetty=INFO
log4j.logger.org.spark-project.jetty=INFO
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
log4j.logger.org.apache.spark.repl.Main=${shell.log.level}
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=${shell.log.level}

log4j.logger.jobLogger=INFO, RollingAppenderU
log4j.appender.RollingAppenderU=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RollingAppenderU.DatePattern='.'yyyy-MM-dd
log4j.appender.RollingAppenderU.layout=org.apache.log4j.PatternLayout
log4j.appender.RollingAppenderU.layout.ConversionPattern=[%p] %d %c %M - %m%n
log4j.appender.fileAppender.MaxFileSize=1MB
log4j.appender.fileAppender.MaxBackupIndex=1

Answer 1

在我的配置中，我使用以下方法在WARN上记录WARN语句，以及从INFO的动能火花连接器记录其他所有内容

log4j.logger.com.kinetica.spark=INFO
log4j.logger.com.kinetica.spark.LoaderParams=WARN