使用时间戳pyspark更新json列

Question

+----------------+
|application_name|    
+----------------+
|{"application_name": "DIMENSIONS_USER",
  "dq_test_name": "contra_cp_dimension_agentpresence_isBillable_should_be_set"
}
+----------------+--------------------+-----------+

在这个我需要更新所有的rows..can的日期列值有人帮助我是新手，无法找到任何工作的解决方案

+----------------+
|application_name|    
+----------------+
|{"application_name": "DIMENSIONS_USER",
  "dq_test_name": "contra_cp_dimension_agentpresence_isBillable_should_be_set",
"date":01/001/2020
}
+----------------+

Answer 1

您可以定义一个UDF，它将为json字符串添加一个新值。

下面是一个例子：

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
import json


def update_app_data(data):
    app_data = json.loads(data)
    app_data["date"] = "01/001/2020"
    return json.dumps(app_data)


if __name__ == "__main__":
    spark = SparkSession.builder.master("local").appName("Test").getOrCreate()
    application_data = {
        "application_name": "DIMENSIONS_USER",
        "dq_test_name": "contra_cp_dimension_agentpresence_isBillable_should_be_set",
    }
    data = [
        {
            "application_name": json.dumps(application_data),
        }
    ]
    df = spark.createDataFrame(data=data)
    update_app_data_udf = F.udf(lambda x: update_app_data(x))
    df = df.withColumn("application_name", update_app_data_udf(F.col("application_name")))

输入数据帧如下所示：

+---------------------------------------------------------------------------------------------------------------------+
|application_name                                                                                                     |
+---------------------------------------------------------------------------------------------------------------------+
|{"application_name": "DIMENSIONS_USER", "dq_test_name": "contra_cp_dimension_agentpresence_isBillable_should_be_set"}|
+---------------------------------------------------------------------------------------------------------------------+

输出：

+--------------------------------------------------------------------------------------------------------------------------------------------+
|application_name                                                                                                                            |
+--------------------------------------------------------------------------------------------------------------------------------------------+
|{"application_name": "DIMENSIONS_USER", "dq_test_name": "contra_cp_dimension_agentpresence_isBillable_should_be_set", "date": "01/001/2020"}|
+--------------------------------------------------------------------------------------------------------------------------------------------+