Scala到Java8 MLeap的转换

Question

我想使用MLeap来部署Spark ML机器学习模型，并使用它们来实时预测。

创建者发布了Scala教程，但我需要支持Java8代码库。

如何在Java 8中实现以下代码：

val pipeline = SparkUtil.createPipelineModel(uid = "pipeline", Array(featureModel, rfModel))

val sbc = SparkBundleContext()
for(bf <- managed(BundleFile("jar:file:/tmp/mnist.model.rf.zip"))) {
        pipeline.writeBundle.save(bf)(sbc).get
      }

val bundle = (for(bundleFile <- managed(BundleFile("jar:file:/tmp/simple-spark-pipeline.zip"))) yield {
  bundleFile.loadMleapBundle().get
}).opt.get

Answer 1

如果您仅使用普通的Spark ML转换器，则可以使用SimpleSparkSerializer轻松地保存和加载模型。

保存：

new SimpleSparkSerializer().serializeToBundle(model, "jar:file:/tmp/model.zip", trainData);

正在加载：

Transformer model = new SimpleSparkSerializer().deserializeFromBundle("jar:file:/tmp/model.zip");

Answer 2

你可以跳过加载spark，它是笨重的类，直接通过运行时加载。

private static Transformer kMeansModel;
private static MleapContext mleapContext;
private static BundleBuilder bundleBuilder;

public MLeapLocalService() throws IOException {
    mleapContext = new ContextBuilder().createMleapContext();
    bundleBuilder = new BundleBuilder();
    Resource res = resourceLoader.getResource("classpath:aihello.com/aimodels/kmeans-model.zip");
    kMeansModel = bundleBuilder.load(res.getFile(), mleapContext).root();
}

然后，您可以通过以下方式进行预测：

    LeapFrameBuilder builder = new LeapFrameBuilder();
    List<StructField> fields = new ArrayList<StructField>();
    fields.add(builder.createField("docs", builder.createString()));
    StructType schema = builder.createSchema(fields);
    List<Row> rows = new ArrayList<Row>();
    rows.add(builder.createRow(docs));
    DefaultLeapFrame frame = builder.createFrame(schema, rows);
    DefaultLeapFrame returnFrame = kMeansModel.transform(frame).get();