如何合并PySpark数据中的单词列表？

Question

我有一个包含单词列表的数据，我需要将它们合并成一个句子。

Dataframe:

temp = spark.createDataFrame([
    (0, ['Julia', 'is', 'awesome']),
    (2, ['Data-science', 'is','cool']),
    (3, ['Machine','learning'])
], ["id", "words"])

# +---+------------------------+
# |id |words                   |
# +---+------------------------+
# |0  |[Julia, is, awesome]    |
# |2  |[Data-science, is, cool]|
# |3  |[Machine, learning]     |
# +---+------------------------+

temp.printSchema()
# root
#  |-- id: long (nullable = true)
#  |-- words: array (nullable = true)
#  |    |-- element: string (containsNull = true)

我在申请rdd。

rdd_df = temp.rdd.map(lambda x: [x['id'], ' '.join(x['words'])])
spark.createDataFrame(rdd_df, temp.schema).show(10, False)

# +---+---------------------------------------------------------+
# |id |words                                                    |
# +---+---------------------------------------------------------+
# |0  |[ ' J u l i a ' ,   ' i s ' ,   ' a w e s o m e ' ]      |
# |2  |[ ' D a t a - s c i e n c e ' ,   ' i s ' , ' c o o l ' ]|
# |3  |[ ' M a c h i n e ' , ' l e a r n i n g ' ]              |
# +---+---------------------------------------------------------+

但是上面的代码没有返回所需的输出。在不使用RDD的情况下，我们还可以应用其他解决方案吗？

期望输出：

+---+--------------------+
|id |words               |
+---+--------------------+
|0  |Julia is awesome    |
|1  |Data-science is cool|
|2  |Machine             |
+---+--------------------+

Answer 1

如果您有一个单词列表(一个字符串数组)，则可以使用array_join组合它们。

from pyspark.sql import functions as F
temp = spark.createDataFrame([
    (0, ['Julia', 'is', 'awesome']),
    (1, ['Data-science', 'is','cool']),
    (2, ['Machine','learning'])
], ["id", "words"])

temp = temp.withColumn('words', F.array_join('words', ' '))

temp.show()
# +---+--------------------+
# | id|               words|
# +---+--------------------+
# |  0|    Julia is awesome|
# |  1|Data-science is cool|
# |  2|    Machine learning|
# +---+--------------------+