故障加载PySpark ALS模型

Question

我试着加载一个用派斯喀斯特创建的模型。我用以下代码创建了模型：

import pandas as pd
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS
from pyspark.ml.tuning import TrainValidationSplit, ParamGridBuilder
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
sc = SparkContext('local')
spark = SparkSession(sc)

data = pd.read_csv('matrix-out-small.csv')
df = spark.createDataFrame(data)

(training, test) = df.randomSplit([0.8, 0.2])

als = ALS(userCol="CustomerID", itemCol="ProductID", ratingCol="Rating", coldStartStrategy="drop", nonnegative=True)

# Tune model using param grid builder
param_grid = ParamGridBuilder().addGrid(als.rank, [12, 13, 14]).addGrid(als.maxIter, [18, 19, 20]).addGrid(als.regParam, [.17, .18, .19]).build()

evaluator = RegressionEvaluator(metricName="rmse", labelCol="Rating", predictionCol="prediction")

tvs = TrainValidationSplit(estimator=als, estimatorParamMaps=param_grid, evaluator=evaluator)

# fit model to training data
model = tvs.fit(training)

# extract best
best_model = model.bestModel  

best_model.save("modelSaveOut")

这将创建一个名为“ModelSaveOut”的目录，其中包含“ItemFactors”、“元数据”和“userFactors”

当我尝试使用ALS.load加载模型时，我得到以下信息：

model = ALS.load("modelSaveOut")

py4j.protocol.Py4JJavaError:调用o26.load时出错。：：org.apache.spark.ml.recommendation.ALSModel.(java.lang.String)：java.lang.NoSuchMethodException

model = TrainValidationSplit.load("modelSaveOut")

py4j.protocol.Py4JJavaError:调用o26.load时出错。：java.lang.IllegalArgumentException:需求失败:错误加载元数据:预期类名java.lang.IllegalArgumentException但找到类名org.apache.spark.ml.recommendation.ALSModel

我似乎没有使用正确的对象/方法加载模型。是否可以保存“bestModel”，还是需要使用不同的方法保存整个模型？

Answer 1

如果您读取异常跟踪

但是找到了类名org.apache.spark.ml.recommendation.ALSModel

它会确切地告诉你该怎么做：

 from pyspark.ml.recommendation import ALS, ALSModel

 ALSModel.load("modelSaveOut")