无法计算RDD，同时使用stratio和spark从Aerospike读取

Question

我已经写了一个程序，它的目的是从航空公司读取并将其转换为spark中的RDD。

public void sparkTest () throws UnsupportedDataTypeException{

        log.debug("TESTING SPARK WITH AEROSPIKE");
        String host = "localhost";
        int port = 3000;

        String namespace = "mynamespace";
        String inputSet = "myset";


        AerospikeDeepJobConfig inputConfigCell = AerospikeConfigFactory.createAerospike().host(host).port(3000)
                .namespace(namespace)
                .set(inputSet)
                ;

        log.debug("Print inputConfigCell ......");
        log.debug(inputConfigCell.getNamespace());
        log.debug(inputConfigCell.getSet());
        log.debug(inputConfigCell.getAerospikePort());
        log.debug(inputConfigCell.getHost());

        JavaRDD inputRDDCell = sparkContext.createJavaRDD(inputConfigCell);
        log.debug("Print RDD .............");
        log.debug(inputRDDCell);
}

我知道在我的航空公司集合中有许多记录，但无法访问“inputRDDCell”的RDD性质。即使是命名空间、集合、端口主机的日志也是完全正确的。我正在尝试使用inputRDDCell.first()，但它给出了异常，但当我简单地打印RDD对象时，它给了我非常广泛的输出。

请指导我，如何正确地从它生成可用的和功能的RDDs。我使用这个链接作为指导：http://www.programcreek.com/java-api-examples/index.php?source_dir=deep-examples-master/deep-aerospike/src/main/java/com/stratio/deep/examples/java/factory/ReadingCellFromAerospike.java

我已经使用了RDD，JAVARDD所有的东西，但得到了相同的输出。

日志的输出为：

[2016-03-10 15:58:05.812] boot - 13535 DEBUG [main] --- PushAnalysisService: TESTING SPARK WITH AEROSPIKE
[2016-03-10 15:58:05.825] boot - 13535 DEBUG [main] --- PushAnalysisService: Print inputConfigCell ......
[2016-03-10 15:58:05.827] boot - 13535 DEBUG [main] --- PushAnalysisService: mynamespace
[2016-03-10 15:58:05.829] boot - 13535 DEBUG [main] --- PushAnalysisService: myset
[2016-03-10 15:58:05.831] boot - 13535 DEBUG [main] --- PushAnalysisService: 3000
[2016-03-10 15:58:05.832] boot - 13535 DEBUG [main] --- PushAnalysisService: localhost
[2016-03-10 15:58:06.025] boot - 13535  INFO [main] --- MemoryStore: ensureFreeSpace(552) called with curMem=0, maxMem=539724349
[2016-03-10 15:58:06.035] boot - 13535  INFO [main] --- MemoryStore: Block broadcast_0 stored as values in memory (estimated size 552.0 B, free 514.7 MB)
[2016-03-10 15:58:06.161] boot - 13535  INFO [main] --- MemoryStore: ensureFreeSpace(901) called with curMem=552, maxMem=539724349
[2016-03-10 15:58:06.165] boot - 13535  INFO [main] --- MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 901.0 B, free 514.7 MB)
[2016-03-10 15:58:06.196] boot - 13535  INFO [sparkDriver-akka.actor.default-dispatcher-5] --- BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:49368 (size: 901.0 B, free: 514.7 MB)
[2016-03-10 15:58:06.205] boot - 13535  INFO [main] --- SparkContext: Created broadcast 0 from broadcast at DeepRDD.java:65
[2016-03-10 15:58:06.294] boot - 13535 DEBUG [main] --- PushAnalysisService: Print RDD .............
[2016-03-10 15:58:06.302] boot - 13535 DEBUG [main] --- PushAnalysisService: DeepRDD[0] at RDD at DeepRDD.java:62

Answer 1

在社区项目sasha-polev/aerospark和由名为aerospike/aerospark的Aerospike支持的分支之间有一个显着的区别。

社区one相当不活跃，只提供基本的RDD支持。由Aerospike支持的版本支持RDD、DataFrames和SparkSQL。您应该尝试将现有代码与它一起使用。