如何在具有数组列的DataFrame上提高Spark SQL查询性能？

Question

假设我有两个数据帧df1和df2，它们有许多数组类型的列。某些列可以具有null值。

df1.createOrReplaceTempView("temp1")
df2.createOrReplaceTempView("temp2")

我使用LATERAL VIEW EXPLODE来展平数组类型的列。下面是一个示例Spark SQL语句，我用它来查询数据帧。由于LATERAL VIEW EXPLODE的原因，此查询似乎存在性能问题。

SELECT tab1.id AS country, tab2.city_lve AS city FROM
(SELECT id, language_lve, population_lve FROM temp1
LATERAL VIEW EXPLODE(language) EXPLODED_NAMES AS language_lve
LATERAL VIEW EXPLODE(population) EXPLODED_NAMES AS population_lve
WHERE language_lve IS NOT NULL AND population_lve IS NOT NULL)
AS tab1
JOIN
(SELECT id, city_lve, link_lve FROM temp2
LATERAL VIEW EXPLODE(city) EXPLODED_NAMES AS city_lve
LATERAL VIEW EXPLODE(link) EXPLODED_NAMES AS link_lve
WHERE city_lve IS NOT NULL AND link_lve IS NOT NULL)
AS tab2 ON (tab2.link_lve = tab1.language_lve)

是否有有效的Spark SQL语句来查询数据帧？

Answer 1

Explode将解决您的问题

下面是一个使用它的不同问题的示例

https://stackoverflow.com/a/44418598/1461187