Python中的并行处理efficient_apriori代码

Question

我有来自eshop的1200万数据。我想用efficient_apriori包来计算关联规则。The problem is that 12 millions observations are too many，所以计算占用了太多的时间。有没有办法提高算法的速度？我正在考虑一些并行处理或者将python代码编译成C语言。我尝试过PYPY，但是PYPY不支持pandas包。感谢您的任何帮助或想法。

如果你想看我的代码：

import pandas as pd

from efficient_apriori import apriori

orders = pd.read_csv("orders.csv", sep=";")

customer = orders.groupby("id_customer")["name"].agg(tuple).tolist()

itemsets, rules = apriori(
            customer, min_support=100/len(customer), min_confidence=0
        )

Answer 1

您是否可以使用这种方法并行运行此任务：

from multiprocessing import Pool

length_of_input_file=len(raw_data_min)
total_offset_count=4  # number of parallel process to run
offset=int(length_of_input_file/total_offset_count // 1)
dataNew1=customer[0:offset-1]
dataNew2=customer[offset:2*offset-1]
dataNew3=customer[2*offset:3*offset-1]
dataNew4=customer[3*offset:4*offset-1]

def calculate_frequent_itemset(fractional_data):
    """Function that calculated the frequent dataset parallely"""
    itemsets, rules = apriori(fractional_data, min_support=MIN_SUPPORT, 
    min_confidence=MIN_CONFIDENCE)
    return itemsets, rules
        
p=Pool()
frequent_itemsets=p.map(calculate_frequent_itemset,(dataNew1,dataNew2,dataNew3,dataNew4))
p.close()
p.join()

itemsets1, rules1 =frequent_itemsets[0]
itemsets2, rules2=frequent_itemsets[1]
itemsets3, rules3=frequent_itemsets[2]
itemsets4, rules4=frequent_itemsets[3]