Mahout - TestForest无法计算最终分析(混淆矩阵、准确度、kappa等)

Question

我目前正在尝试使用Mahout中随机森林的部分实现来对数据进行分类。

虽然我能够使用一组固定的训练森林对特定数量的数据进行分类，但我无法使用更大的数据(大约两倍的大小)和相同的分类器来做到这一点。

事实上，在MR过程中完成的分类工作很好，并显示出成功。可悲的是，当计算分析时，它总是以OutOfMemoryException结束，这可能是由于到达的GCOverheadLimit而发生的。我还添加了选项：

-Dmapred.child.java.opts="-Xmx20g -XX:-UseGCOverheadLimit“

呼叫，但它没有帮助。

我还记得，当我使用mahout的早期版本(我认为是0.7)时，它可以使用testforest方法对几乎任意大的数据集进行分类，还可以输出分析度量，如混淆矩阵等。我很困惑为什么整个过程中最简单的步骤会导致这样的错误。

有什么方法可以轻松解决这个问题吗？

以下是其中一个日志：

15/05/25 13:58:26 INFO mapreduce.Job:  map 97% reduce 0%
15/05/25 13:58:46 INFO mapreduce.Job:  map 98% reduce 0%
15/05/25 13:59:43 INFO mapreduce.Job:  map 99% reduce 0%
15/05/25 14:01:20 INFO mapreduce.Job:  map 100% reduce 0%
15/05/25 14:02:11 INFO mapreduce.Job: Job job_1432549186261_0032 completed successfully
15/05/25 14:02:12 INFO mapreduce.Job: Counters: 33
    File System Counters
        FILE: Number of bytes read=2202834240
        FILE: Number of bytes written=3408230
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=580537741
        HDFS: Number of bytes written=343262060
        HDFS: Number of read operations=150
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=60
    Job Counters 
        Failed map tasks=9
        Launched map tasks=39
        Other local map tasks=19
        Data-local map tasks=17
        Rack-local map tasks=3
        Total time spent by all maps in occupied slots (ms)=3387270
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=3387270
        Total vcore-seconds taken by all map tasks=3387270
        Total megabyte-seconds taken by all map tasks=10405693440
    Map-Reduce Framework
        Map input records=16993025
        Map output records=16993045
        Input split bytes=4950
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=15935
        CPU time spent (ms)=1798740
        Physical memory (bytes) snapshot=27353509888
        Virtual memory (bytes) snapshot=102048583680
        Total committed heap usage (bytes)=58348666880
    File Input Format Counters 
        Bytes Read=580532791
    File Output Format Counters 
        Bytes Written=343262060
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at org.apache.mahout.classifier.df.mapreduce.Classifier.parseOutput(Classifier.java:169)
    at org.apache.mahout.classifier.df.mapreduce.Classifier.run(Classifier.java:130)
    at org.apache.mahout.classifier.df.mapreduce.TestForest.mapreduce(TestForest.java:188)
    at org.apache.mahout.classifier.df.mapreduce.TestForest.testForest(TestForest.java:174)
    at org.apache.mahout.classifier.df.mapreduce.TestForest.run(TestForest.java:146)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.mahout.classifier.df.mapreduce.TestForest.main(TestForest.java:315)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
    at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Answer 1

好吧，我找到了解决方案，尽管我不清楚为什么它会起作用：

正在添加...

export HADOOP_CLIENT_OPTS="-Xmx20192m"

..。在脚本中起到了作用。

使用MAHOUT_HEAPSIZE=40000或-Dmapred.child.java.opts无济于事。

我在这里找到了解决方案的灵感：https://community.cloudera.com/t5/Data-Science-and-Machine/Java-heap-size-running-mahout-clusterdump/td-p/7752

如果您遇到类似的问题，您可能有兴趣设置可以在mahout脚本中找到的变量，该脚本也可以在线获得：

https://svn.apache.org/repos/asf/mahout/trunk/bin/mahout