文档建议反馈控制台
首页
学习
活动
专区
圈层
工具
MCP广场
文章/答案/技术大牛
发布
社区首页 >问答首页 >我正在为我的应用程序使用spark，但是我得到了不必要的日志。如何在spark java应用程序中禁用日志

问我正在为我的应用程序使用spark，但是我得到了不必要的日志。如何在spark java应用程序中禁用日志
EN

Stack Overflow用户
提问于 2017-05-06 12:49:29
回答 3查看 570关注 0票数 1

下面是我在控制台中获得的日志。

代码语言：javascript
复制
.spark.executor.Executor       : Finished task 185.0 in stage 189.0 (TID 4477). 11508 bytes result sent to driver
2017-05-06 10:00:18.767  INFO 3336 --- [er-event-loop-2] o.apache.spark.scheduler.TaskSetManager  : Starting task 188.0 in stage 189.0 (TID 4480, localhost, executor driver, partition 188, ANY, 6317 bytes)
2017-05-06 10:00:18.769  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.769  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.769  INFO 3336 --- [result-getter-1] o.apache.spark.scheduler.TaskSetManager  : Finished task 185.0 in stage 189.0 (TID 4477) in 75 ms on localhost (executor driver) (185/200)
2017-05-06 10:00:18.769  INFO 3336 --- [launch worker-5] org.apache.spark.executor.Executor       : Running task 188.0 in stage 189.0 (TID 4480)
2017-05-06 10:00:18.770  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.770  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.771  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 2 non-empty blocks out of 201 blocks
2017-05-06 10:00:18.771  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.773  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 3 non-empty blocks out of 401 blocks
2017-05-06 10:00:18.773  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.773  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.773  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.773  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.774  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 1 ms
2017-05-06 10:00:18.775  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 2 non-empty blocks out of 201 blocks
2017-05-06 10:00:18.775  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.777  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 3 non-empty blocks out of 401 blocks
2017-05-06 10:00:18.777  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.786  INFO 3336 --- [launch worker-6] org.apache.spark.executor.Executor       : Finished task 182.0 in stage 189.0 (TID 4474). 11508 bytes result sent to driver
2017-05-06 10:00:18.786  INFO 3336 --- [er-event-loop-1] o.apache.spark.scheduler.TaskSetManager  : Starting task 189.0 in stage 189.0 (TID 4481, localhost, executor driver, partition 189, ANY, 6317 bytes)
2017-05-06 10:00:18.787  INFO 3336 --- [result-getter-2] o.apache.spark.scheduler.TaskSetManager  : Finished task 182.0 in stage 189.0 (TID 4474) in 132 ms on localhost (executor driver) (186/200)
2017-05-06 10:00:18.787  INFO 3336 --- [launch worker-6] org.apache.spark.executor.Executor       : Running task 189.0 in stage 189.0 (TID 4481)
2017-05-06 10:00:18.790  INFO 3336 --- [launch worker-5] org.apache.spark.executor.Executor       : Finished task 188.0 in stage 189.0 (TID 4480). 11356 bytes result sent to driver
2017-05-06 10:00:18.790  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.790  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.791  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.791  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.792  INFO 3336 --- [er-event-loop-2] o.apache.spark.scheduler.TaskSetManager  : Starting task 190.0 in stage 189.0 (TID 4482, localhost, executor driver, partition 190, ANY, 6317 bytes)
2017-05-06 10:00:18.792  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 2 non-empty blocks out of 201 blocks
2017-05-06 10:00:18.792  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.794  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 2 non-empty blocks out of 401 blocks
2017-05-06 10:00:18.794  INFO 3336 --- [launch worker-6] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.796  INFO 3336 --- [launch worker-4] org.apache.spark.executor.Executor       : Finished task 187.0 in stage 189.0 (TID 4479). 11356 bytes result sent to driver
2017-05-06 10:00:18.798  INFO 3336 --- [er-event-loop-0] o.apache.spark.scheduler.TaskSetManager  : Starting task 191.0 in stage 189.0 (TID 4483, localhost, executor driver, partition 191, ANY, 6317 bytes)
2017-05-06 10:00:18.798  INFO 3336 --- [launch worker-5] org.apache.spark.executor.Executor       : Running task 190.0 in stage 189.0 (TID 4482)
2017-05-06 10:00:18.798  INFO 3336 --- [result-getter-3] o.apache.spark.scheduler.TaskSetManager  : Finished task 188.0 in stage 189.0 (TID 4480) in 31 ms on localhost (executor driver) (187/200)
2017-05-06 10:00:18.798  INFO 3336 --- [result-getter-3] o.apache.spark.scheduler.TaskSetManager  : Finished task 187.0 in stage 189.0 (TID 4479) in 35 ms on localhost (executor driver) (188/200)
2017-05-06 10:00:18.800  INFO 3336 --- [launch worker-4] org.apache.spark.executor.Executor       : Running task 191.0 in stage 189.0 (TID 4483)
2017-05-06 10:00:18.801  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.801  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.802  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.802  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.803  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.803  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.803  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 201 blocks
2017-05-06 10:00:18.803  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.804  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 1 blocks
2017-05-06 10:00:18.804  INFO 3336 --- [launch worker-4] o.a.s.s.ShuffleBlockFetcherIterator      : Started 0 remote fetches in 0 ms
2017-05-06 10:00:18.804  INFO 3336 --- [launch worker-5] o.a.s.s.ShuffleBlockFetcherIterator      : Getting 1 non-empty blocks out of 401 blocks

下面是我的POM文件。

代码语言：javascript
复制
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-rest</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web-services</artifactId>
    </dependency>
     <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        </dependency> 
    <dependency>
        <groupId>info.debatty</groupId>
        <artifactId>java-string-similarity</artifactId>
        <version>RELEASE</version>
    </dependency>



    <dependency>
        <groupId>com.univocity</groupId>
        <artifactId>univocity-parsers</artifactId>
        <version>2.3.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.11</artifactId>
        <version>2.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>commons-compiler</artifactId>
        <version>2.6.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>2.1.0</version>
    </dependency>
        <dependency>
        <groupId>com.oracle</groupId>
        <artifactId>ojdbc6</artifactId>
        <version>11.2.0.3</version>
    </dependency>


    <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-network-common_2.10</artifactId>
       <version>1.4.0</version>
      </dependency>
      <dependency>
       <groupId>org.codehaus.janino</groupId>
       <artifactId>commons-compiler</artifactId>
       <version>2.7.5</version>
      </dependency>


    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>
apache-spark
apache-spark-sql
apache-spark-mllib
EN

回答 3

Stack Overflow用户

发布于 2017-05-06 18:02:13

我认为您可以将日志级别更改为

代码语言：javascript
复制
sparkContext.setLogLevel("WARN")

您可以选择以下日志级别

代码语言：javascript
复制
ALL, DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN

如果打印的日志在spark-shell中，那么您可以从位于conf/log4j.properties的配置文件中更改日志级别(从conf/log4j.properties.template更改名称)，然后根据需要更改日志级别

代码语言：javascript
复制
log4j.rootCategory=INFO, console

并重新打开shell，您将看到较少的输出。

票数 2
EN

Stack Overflow用户

发布于 2017-05-06 22:21:00

在spark-default-conf中将此设置spark.history.fs.cleaner.enabled设置为true。默认情况下，这有助于在7天后从hdfs中清除日志，可以通过设置spark.history.fs.cleaner.maxAge进行更改

票数 0
EN

Stack Overflow用户

发布于 2017-05-07 00:28:21

我在scala中使用了以下代码: Logger.getLogger("org").setLevel(Level.OFF) Logger.getLogger("akka").setLevel(Level.WARN)

您可以在Java中尝试类似上面的内容。

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：

https://stackoverflow.com/questions/43816838

复制
相关文章

相似问题

领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 粤公网安备44030502008569号

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档