需要帮助在pyspark中的for循环中添加dataframe

Question

我们在where子句中有需要应用于查询的条件列表

Conditions=[
  Condition-1
  Condition-2
  ...
  Condition-n
]

我们有这样的查询

for condition in Conditions:
    df = spark.sql("SELECT col1, col2 from table where" + condition)

但是我们想要一个包含所有条件结果的最终数据帧，如何做that...our要求是这样的。

Answer 1

如果你的条件是多个AND，你可以加入它们。

示例：

from pyspark.sql import SparkSession

if __name__ == "__main__":
    spark = SparkSession.builder.getOrCreate()
    data = [
        {"a": 1, "b": 2, "c": 3},
        {"a": 3, "b": 3, "c": 7},
        {"a": 2, "b": 3, "c": 5},
    ]
    conditions = [
        "a > 2",
        "b < 4",
        "c > 5",
    ]
    df = spark.createDataFrame(data)
    df.createOrReplaceTempView("table")
    df = spark.sql("SELECT a, b from table where {}".format(" AND ".join(conditions)))
    df.show()

结果：

+---+---+                                                                       
|  a|  b|
+---+---+
|  3|  3|
+---+---+