如何使一个具有函数的python模块并行运行命令行工具(w/o使用if __name__ ==‘_main__’：所以它是重要的)？

Question

我想要制作一个python模块，它具有一个方便的函数，可以在Windows上使用Python3.7并行运行命令。(适用于az cli命令)

我想让一个功能：

• 很容易使用:只需将命令列表作为字符串传递，并让它们并行执行。
• 让我看看这些命令生成的输出。
• 在python库中使用的构建
• 在Windows和Linux上工作得同样好(使用fork()，而Windows没有fork()，所以有时候多处理代码将在Linux上工作，而不是Windows)。
• 为了更大的方便，可以制造成一个重要的模块。

这是令人惊讶的困难，我认为，也许它以前不可能在较早版本的python？(我看到了几个2-8岁的Q&因为它说你必须使用if __name__==__main__:来实现并行处理，但是我发现在制造一个重要的模块时，它并没有以一贯可预测的方式工作。)

def removeExtraLinesFromString(inputstring):
    stringtoreturn = ""
    for line in inputstring.split("\n"):
        if len(line.strip()) > 0: #Only add non empty lines to the stringtoreturn
            stringtoreturn = stringtoreturn + line
    return stringtoreturn


def runCmd(cmd): #string of a command passed in here
    from subprocess import run, PIPE
    stringtoreturn = str( run(cmd, shell=True, stdout=PIPE).stdout.decode('utf-8') )
    stringtoreturn = removeExtraLinesFromString(stringtoreturn)
    return stringtoreturn


def exampleOfParrallelCommands():
    if __name__ == '__main__': #I don't like this method, because it doesn't work when imported, refractoring attempts lead to infinite loops and unexpected behavior.
        from multiprocessing import Pool
        cmd = "python -c \"import time;time.sleep(5);print('5 seconds have passed')\""
        cmds = []
        for i in range(12):  #If this were running in series it'd take at least a minute to sleep 5 seconds 12 times
            cmds.append(cmd)
        with Pool(processes=len(cmds)) as pool:
            results = pool.map(runCmd, cmds) #results is a list of cmd output
        print(results[0])
        print(results[1])
        return results

当我尝试将它作为一个模块导入时，它没有工作(因为if语句，所以我尝试重写代码来移动if语句，我想我删除了它一次，导致我的计算机进入循环，直到我关闭程序。)另一次，我能够将模块导入到另一个python程序中，但是为了完成这一工作，我必须添加__name__ == '__main__'，这是非常直观的。

我几乎放弃了，但在搜索了大量python网站之后，我终于想出了如何在这个Q&A (Python: execute cat subprocess in parallel)中看到用户jfs的代码之后，我修改了他的代码，所以它最好适合于回答我的问题。

Answer 1

toolbox.py

def removeExtraLinesFromString(inputstring):
    stringtoreturn = ""
    for line in inputstring.split("\n"):
        if len(line.strip()) > 0: #Only add non empty lines to the stringtoreturn
            stringtoreturn = stringtoreturn + line
    return stringtoreturn


def runCmd(cmd): #string of a command passed in here
    from subprocess import run, PIPE
    stringtoreturn = str( run(cmd, shell=True, stdout=PIPE).stdout.decode('utf-8') )
    stringtoreturn = removeExtraLinesFromString(stringtoreturn)
    return stringtoreturn


def runParallelCmds(listofcommands): 
    from multiprocessing.dummy import Pool #thread pool
    from subprocess import Popen, PIPE, STDOUT
    listofprocesses = [Popen(listofcommands[i], shell=True,stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True) for i in range(len(listofcommands))] 
    #Python calls this list comprehension, it's a way of making a list
    def get_outputs(process): #MultiProcess Thread Pooling require you to map to a function, thus defining a function.
        return process.communicate()[0] #process is object of type subprocess.Popen
    outputs = Pool(len(listofcommands)).map(get_outputs, listofprocesses) #outputs is a list of bytes (which is a type of string)
    listofoutputstrings = []
    for i in range( len(listofcommands) ):
        outputasstring = removeExtraLinesFromString(  outputs[i].decode('utf-8')  ) #.decode('utf-8') converts bytes to string
        listofoutputstrings.append( outputasstring )
    return listofoutputstrings

main.py

from toolbox import runCmd #(cmd)
from toolbox import runParallelCmds #(listofcommands)

listofcommands = []
cmd = "ping -n 2 localhost"
listofcommands.append(cmd)
cmd = "python -c \"import time;time.sleep(5);print('5 seconds have passed')\""
for i in range(12):
    listofcommands.append(cmd) # If 12 processes each sleep 5 seconds, this taking less than 1 minute proves parrallel processing

outputs = runParallelCmds(listofcommands)
print(outputs[0])
print(outputs[1])

产出：

Pinging ::1有32个字节的数据:答复:1: time<1ms答复:1:1:数据包:发送= 2，接收= 2，丢失=0 (0%损失)，大约往返时间以毫秒计:最小= 0ms，最大值= 0ms，平均= 0ms 5秒过去了