我使用spark从亚马逊S3加载json文件。我想删除基于数据帧的两列的重复,保留最新的(我有时间戳列)。最好的办法是什么?请注意,副本可能分布在分区之间。我是否可以删除保留上一张记录的副本而不进行洗牌?我在处理1 TB的数据。
我想通过这两列对数据框架进行分区,这样所有重复的记录都将被“一致散列”到同一个分区中,因此,下面的分区级别排序by拖放重复将消除只保留一个重复的所有重复。我不知道这是否可能。如有任何信息,将不胜感激。
发布于 2019-04-15 23:26:25
使用row_number()窗口函数可能更容易完成任务,在c1下面是时间戳列,c2、c3是用于划分数据的列:
from pyspark.sql import Window, functions as F
# create a win spec which is partitioned by c2, c3 and ordered by c1 in descending order
win = Window.partitionBy('c2', 'c3').orderBy(F.col('c1').desc())
# set rn with F.row_number() and filter the result by rn == 1
df_new = df.withColumn('rn', F.row_number().over(win)).where('rn = 1').drop('rn')
df_new.show()编辑:
如果您只需要复制和删除唯一的行,那么添加另一个字段:
from pyspark.sql import Window, functions as F
# create a win spec which is partitioned by c2, c3 and ordered by c1 in descending order
win = Window.partitionBy('c2', 'c3').orderBy(F.col('c1').desc())
# window to cover all rows in the same partition
win2 = Window.partitionBy('c2', 'c3') \
.rangeBetween(Window.unboundedPreceding, Window.unboundedFollowing)
# set new columns: rn, cnt and filter the result by rn == 1 and cnt > 1
df_new = df.withColumn('rn', F.row_number().over(win)) \
.withColumn('cnt', F.count('c1').over(win2)) \
.where('rn = 1 and cnt > 1') \
.drop('rn', 'cnt')
df_new.show()https://stackoverflow.com/questions/55660085
复制相似问题