每行中最频繁的单词

Question

我试图在标记化Dataframe的每一行中找到最常用的单词如下：

print(df.tokenized_sents)

['apple', 'inc.', 'aapl', 'reported', 'fourth', 'consecutive', 'quarter', 'record', 'revenue', 'profit', 'combination', 'higher', 'iphone', 'prices', 'strong', 'app-store', 'sales', 'propelled', 'technology', 'giant', 'best', 'year', 'ever', 'revenue', 'three', 'months', 'ended', 'sept.']

['brussels', 'apple', 'inc.', 'aapl', '-.', 'chief', 'executive', 'tim', 'cook', 'issued', 'tech', 'giants', 'strongest', 'call', 'yet', 'u.s.-wide', 'data-protection', 'regulation', 'saying', 'individuals', 'personal', 'information', 'been', 'weaponized', 'mr.', 'cooks', 'call', 'came', 'sharply', 'worded', 'speech', 'before', 'p…']

...

wrds = []
for i in range(0, len(df) ):

    wrds.append( Counter(df["tokenized_sents"][i]).most_common(5) )

但它报告了一份清单如下：

print(wrds)

[('revenue', 2), ('apple', 1), ('inc.', 1), ('aapl', 1), ('reported', 1)]
...

我想创建以下数据格式；

print(final_df)

KeyWords                                                                         
revenue, apple, inc., aapl, reported
...

注:最后数据的行不是列表，而是单个文本值，例如:收入、苹果公司、 not 、营收、苹果公司、苹果公司。

Answer 1

使用df.apply

Ex:

import pandas as pd
from collections import Counter
tokenized_sents = [['apple', 'inc.', 'aapl', 'reported', 'fourth', 'consecutive', 'quarter', 'record', 'revenue', 'profit', 'combination', 'higher', 'iphone', 'prices', 'strong', 'app-store', 'sales', 'propelled', 'technology', 'giant', 'best', 'year', 'ever', 'revenue', 'three', 'months', 'ended', 'sept.'], 
                   ['brussels', 'apple', 'inc.', 'aapl', '-.', 'chief', 'executive', 'tim', 'cook', 'issued', 'tech', 'giants', 'strongest', 'call', 'yet', 'u.s.-wide', 'data-protection', 'regulation', 'saying', 'individuals', 'personal', 'information', 'been', 'weaponized', 'mr.', 'cooks', 'call', 'came', 'sharply', 'worded', 'speech', 'before', 'p…']

]

df = pd.DataFrame({"tokenized_sents": tokenized_sents})
final_df = pd.DataFrame({"KeyWords" : df["tokenized_sents"].apply(lambda x: [k for k, v in Counter(x).most_common(5)])}) 
#or
#final_df = pd.DataFrame({"KeyWords" : df["tokenized_sents"].apply(lambda x: ", ".join(k for k, v in Counter(x).most_common(5)))})
print(final_df)

输出：

                               KeyWords
0  [revenue, apple, aapl, sales, ended]
1   [call, saying, apple, issued, aapl]

Answer 2

是这样的吗？使用 .apply()

# creating the dataframe
df = pd.DataFrame({"token": [['apple', 'inc.', 'aapl', 'reported', 'fourth', 'consecutive', 'quarter', 'record', 'revenue', 'profit', 'combination', 'higher', 'iphone', 'prices', 'strong', 'app-store', 'sales', 'propelled', 'technology', 'giant', 'best', 'year', 'ever', 'revenue', 'three', 'months', 'ended', 'sept.'], ['brussels', 'apple', 'inc.', 'aapl', '-.', 'chief', 'executive', 'tim', 'cook', 'issued', 'tech', 'giants', 'strongest', 'call', 'yet', 'u.s.-wide', 'data-protection', 'regulation', 'saying', 'individuals', 'personal', 'information', 'been', 'weaponized', 'mr.', 'cooks', 'call', 'came', 'sharply', 'worded', 'speech', 'before', 'p…']
]})
# fetching 5 most common words using .apply and assigning it to keywords column in dataframe
df["keywords"] = df.token.apply(lambda x: ', '.join(i[0] for i in Counter(x).most_common(5)))
df

输出：

    token   keywords
0   [apple, inc., aapl, reported, fourth, consecut...   revenue, apple, inc., aapl, reported
1   [brussels, apple, inc., aapl, -., chief, execu...   call, brussels, apple, inc., aapl

使用for 循环 .loc() & .itertuples()

df = pd.DataFrame({"token": [['apple', 'inc.', 'aapl', 'reported', 'fourth', 'consecutive', 'quarter', 'record', 'revenue', 'profit', 'combination', 'higher', 'iphone', 'prices', 'strong', 'app-store', 'sales', 'propelled', 'technology', 'giant', 'best', 'year', 'ever', 'revenue', 'three', 'months', 'ended', 'sept.'], ['brussels', 'apple', 'inc.', 'aapl', '-.', 'chief', 'executive', 'tim', 'cook', 'issued', 'tech', 'giants', 'strongest', 'call', 'yet', 'u.s.-wide', 'data-protection', 'regulation', 'saying', 'individuals', 'personal', 'information', 'been', 'weaponized', 'mr.', 'cooks', 'call', 'came', 'sharply', 'worded', 'speech', 'before', 'p…']
]})
df["Keyword"] = ""
for row in df.itertuples():
    xount = [i[0] for i in Counter(row.token).most_common(5)]
    df.loc[row.Index, "Keyword"] = ', '.join(i for i in xount)
df

输出：

    token   Keyword
0   [apple, inc., aapl, reported, fourth, consecut...   revenue, apple, inc., aapl, reported
1   [brussels, apple, inc., aapl, -., chief, execu...   call, brussels, apple, inc., aapl