Pyspark无法将float转换为Float :-/

Question

我有一个pyspark rdd：

proba_classe_0.take(2)
[0.38030685472943737, 0.34728188900913715]

我想在DF上进行转换：

from pyspark.sql.types import FloatType
fields = [ StructField('probabilite' , FloatType() ) ]
schema = StructType(fields)
df_proba_classe_1 = spark.createDataFrame(proba_classe_1, schema=schema)
df_proba_classe_1.count()

我得到了一个奇怪的错误：

TypeError: StructType can not accept object 0.6196931452705625 in type <class 'float'>

Answer 1

您必须映射rdd，因为rdd是字符串类型

rdd = sc\
.parallelize(['0.38030685472943737', '0.34728188900913715'])\
.map(lambda x: float(x))

df = spark\
.createDataFrame(rdd, FloatType()).toDF("id")

df.show()

+----------+
|        id|
+----------+
|0.38030684|
| 0.3472819|
+----------+

df.printSchema()

root
 |-- id: float (nullable = true)