我正在为PySpark代码而苦苦挣扎,尤其是,我想在一个不可迭代的对象col上调用一个函数。
from pyspark.sql.functions import col, lower, regexp_replace, split
from googletrans import Translator
def clean_text(c):
c = lower(c)
c = regexp_replace(c, r"^rt ", "")
c = regexp_replace(c, r"(https?\://)\S+", "")
c = regexp_replace(c, "[^a-zA-Z0-9\\s]", "") #removePunctuation
c = regexp_replace(c, r"\n", " ")
c = regexp_replace(c, r" ", " ")
c = regexp_replace(c, r" ", " ")
# c = translator.translate(c, dest='en', src='auto')
return c
clean_text_df = uncleanedText.select(clean_text(col("unCleanedCol")).alias("sentence"))
clean_text_df.printSchema()
clean_text_df.show(10)当我在c = translator.translate(c, dest='en', src='auto')中运行代码时,Spark显示的错误是TypeError: Column is not iterable。
我想做的是逐字翻译:
发自:
+--------------------+
| sentence|
+--------------------+
|ciao team there a...|
|dear itteam i urg...|
|buongiorno segnal...|
|hi team regarding...|
|hello please add ...|
|ciao vorrei effet...|
|buongiorno ho vis...|
+--------------------+至:
+--------------------+
| sentence|
+--------------------+
|hello team there ...|
|dear itteam i urg...|
|goodmorning segna...|
|hi team regarding...|
|hello please add ...|
|hello would effet...|
|goodmorning I see...|
+--------------------+DataFrame的模式是:
root
|-- sentence: string (nullable = true)有谁能帮帮我吗?
非常感谢
发布于 2020-04-10 21:05:13
PySpark只是为支持Apache Spark而编写的Python API。如果你想使用定制的python函数,你必须定义一个用户定义函数(udf)。
保持您的clean_text()函数不变(将translate行注释掉),并尝试执行以下操作:
from pyspark.sql.functions import udf
from pyspark.sql.Types import StringType
def translate(c):
return translator.translate(c, dest='en', src='auto')
translateUDF = udf(translate, StringType())
clean_text_df = uncleanedText.select(
translateUDF(clean_text(col("unCleanedCol"))).alias("sentence")
)原始clean_text中的其他函数(lower和regexp_replace)是内置的spark函数,并在pyspark.sql.Column上运行。
请注意,使用此udf会影响性能。请参阅:Spark functions vs UDF performance?
https://stackoverflow.com/questions/61140263
复制相似问题