迭代MapReduce (Twister)运行异常

Question

我正在尝试运行我的MapReduce应用程序。

这是主要的课程：

public class GPFunctionFitting {
    public static void main(String[] args) {
        // ....
        GPFunctionFitting client;
        try {
            client = new GPFunctionFitting();
            client.runMapReduce(partitionFile, numMapTasks); // line 44
            // ...
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

其方法是：

public ExpressionTree runMapReduce(String partitionFile, int numMapTasks) throws Exception {

        JobConf jobConf = new JobConf("gp-function-fitting-map-reduce" + uuidGen.generateRandomBasedUUID());
        jobConf.setMapperClass(GPMapTask.class);
        jobConf.setReducerClass(GPReduceTask.class);
        jobConf.setCombinerClass(GPCombiner.class);
        jobConf.setNumMapTasks(numMapTasks);
        jobConf.setNumReduceTasks(1);

        TwisterModel driver = new TwisterDriver(jobConf);
        driver.configureMaps(partitionFile); // line 74
        TwisterMonitor monitor = driver.runMapReduce();
        monitor.monitorTillCompletion();
        ExpressionTree solution = ((GPCombiner) driver.getCurrentCombiner()).getResult();

        driver.close();

        return solution;
    }

这是来自eclipse控制台的日志：

JobID:
gp-function-fitting-map-reduce9f25c1da-ab8b-446b-b453-d60f4165b3a9
TransmissionManager: There is a pending/ongoing transfer on
niotcp:///127.0.0.1:3045 for a total of 16384 bytes
TransmissionManager: All transfers have been completed
CommunicationsService: Closing Link =    niotcp://127.0.0.1:3045 null 0 
[main] INFO  cgl.imr.client.TwisterDriver  - MapReduce computation
termintated gracefully.

cgl.imr.base.TwisterException: Could not send all the map executor
requests. First error is Did not receive a response from the worker.14
[Thread-0] DEBUG cgl.imr.client.ShutdownHook  - Shutting down
completed.
    at cgl.imr.client.TwisterDriver.configureMaps(TwisterDriver.java:360)
    at cgl.imr.client.TwisterDriver.configureMaps(TwisterDriver.java:381)
    at map_reduce.GPFunctionFitting.runMapReduce(GPFunctionFitting.java:74)
    at map_reduce.GPFunctionFitting.main(GPFunctionFitting.java:44)

最后，我运行了twister的终端的错误：

2588033 [Thread-4] ERROR cgl.imr.worker.DaemonWorker  - Could not
instantiate the Mapper. cgl.imr.base.TwisterException: Could not
instantiate the Mapper.     at
cgl.imr.worker.Mapper.<init   (Mapper.java:120)     at
cgl.imr.worker.DaemonWorker.handleMapperRequest(DaemonWorker.java:228)
    at cgl.imr.worker.DaemonWorker.onEvent(DaemonWorker.java:471)   
    at cgl.imr.pubsub.nb.NBPubSubService.onEvent(NBPubSubService.java:154)
    at cgl.narada.service.qos.impl.QosServiceImpl.routeToEventConsumer(QosServiceImpl.java:938)
    at cgl.narada.service.qos.impl.QosServiceImpl.manageEventRoutingToDestinations(QosServiceImpl.java:530)
    at cgl.narada.service.qos.impl.QosServiceImpl.processReceivedEvent(QosServiceImpl.java:424)
    at cgl.narada.service.qos.impl.CommunicationsService.processDataPackets(CommunicationsService.java:236)
    at cgl.narada.service.qos.impl.CommunicationsService.processData(CommunicationsService.java:214)
    at cgl.narada.service.qos.impl.CommunicationsService.handleData(CommunicationsService.java:142)
    at cgl.narada.transport.TransportHandlerImpl.dataReceived(TransportHandlerImpl.java:516)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:323)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.processBuffer(NIOTCPReceiverThread.java:329)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.readDataFromSocket(NIOTCPReceiverThread.java:247)
    at cgl.narada.transport.niotcp.NIOTCPReceiverThread.run(NIOTCPReceiverThread.java:196)
Caused by: java.lang.ClassNotFoundException: map_reduce.GPMapTask   
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)   
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)    
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)    
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)    
at java.lang.ClassLoader.findSystemClass(ClassLoader.java:1004)     
at cgl.imr.util.CustomClassLoader.findClass(CustomClassLoader.java:102)
at cgl.imr.util.CustomClassLoader.loadClass(CustomClassLoader.java:129)
at java.lang.Class.forName0(Native Method)  
at java.lang.Class.forName(Class.java:348)  
at cgl.imr.worker.Mapper.<init   (Mapper.java:116)  ... 24 more

类GPMapTask存在，实现了MapTask接口，并且与主类在同一个包中。分区文件是适当生成的，包含12行，等于请求的映射作业数。

至少有人能告诉我在哪里找问题吗？我一直试图解决“类未发现异常”，但我开始相信问题在于分区配置。

Answer 1

.class要么丢失，要么存在于多个位置。然后编译器就会混淆，因为类定义是模棱两可的。确保所有类都包括在内，准确地一次。检查类路径中包含的所有jars的内容。

删除bin文件夹，再次清理和生成项目。使用Twister，正在运行的应用程序的.jar文件必须出现在app/目录中。每当应用程序更改时(从IDE运行时)导出最新版本的应用程序。