我有一个包含以下模式和数据的数据框架。
schema:[system_name: string, file_name: string, data_tablename: string, create_date: timestamp]df:
+-----------+----------------+------------------+--------------------------+
|system_name|file_name |data_tablename |create_date |
+-----------+----------------+------------------+--------------------------+
|ABC |abc_11132020 |dbo.refine_abc |2020-11-13 19:34:01.448957|
|ABC |abc_11132020 |dbo.refine_abc |2020-11-13 20:44:26.315801|
|ABC |abc_11162020_1 |dbo.refine_abc |2020-11-16 20:07:12.354104|
+-----------+----------------+------------------+--------------------------+我要过滤create_date = max(create_date)所在位置的df
max_ts = df.select(max("create_date")).collect()
print(max_ts)
[Row(max(create_date)=datetime.datetime(2020, 11, 16, 20, 7, 12, 354104))]
print(max_ts[0][0])
2020-11-16 20:07:12.354104
file_details=df.filter(df.create_date==max_ts[0][0]).collect()
print(file_details) --> returning empty list. I was expecting row having file name:abc_11162020_1.
[]发布于 2020-11-20 00:41:07
要获取列中的最大值,可以使用agg()
max_ts = df.agg({"create_date": "max"}).collect()[0][0]过滤器:
df.filter(df.create_date==max_ts).show(truncate=False)
+-----------+--------------+--------------+--------------------------+
|system_name|file_name |data_tablename|create_date |
+-----------+--------------+--------------+--------------------------+
|ABC |abc_11162020_1|dbo.refine_abc|2020-11-16 20:07:12.354104|
+-----------+--------------+--------------+--------------------------+发布于 2020-11-20 01:45:58
通过将时间戳转换为unix时间戳,能够对列进行过滤。
from pyspark.sql.functions import max, unix_timestamp
df=df.withColumn("create_date", unix_timestamp("create_date", "MM/dd/yyyy HH:mm:ss"))
max_ts = df.select(max("create_date")).collect()
file_details=df.filter(df.create_date==max_ts[0][0]).collect()https://stackoverflow.com/questions/64915309
复制相似问题