多处理子过程中的无序

Question

在使用主持人脚本（参见下文）调用执行计算的内部脚本后，以线性方式运行一些计算后，我很难在使用多处理尝试时将其执行。似乎每个CPU核心都在运行此列表集（testRegister）并启动计算，即使其他核心已经提前执行此任务（在同一会话中）。我怎样才能防止这种混乱的行为呢？这是我第一次尝试通过Python调用多个处理器。

更正：初始帖子没有显示测试是一个字符串，包含调用“内部脚本”，其中参数m1和m2不同于固定参数arg1和arg2，仅属于此“内部脚本”。

#!/usr/bin/env python3
import os
import subprocess as sub
import sys
import multiprocessing
fileRegister = []
testRegister = []


def fileCollector():
    for file in os.listdir("."):
        if file.endswith(".xyz"):
            fileRegister.append(file)
    fileRegister.sort()
    return fileRegister


def testSetup():
    data = fileRegister
    while len(data) > 1:
        for entry in fileRegister[1:]:
            m0 = str(fileRegister[0])
            m1 = str(entry)
            test =  str("python foo.py ") + str(m1) + str(" ") + str(m2) +\
                    str(" --arg1 --arg2")  # formulate test condition
            testRegister.append(test)
            testRegister.sort()
        del data[0]
    return testRegister


def shortAnalysator():
    for entry in testRegister:
        print(str(entry))
        sub.call(entry, shell=True)
        del testRegister[0]


def polyAnalysator():
    # apparently each CPU core works as if the register were not shared
    # reference: https://docs.python.org/3.7/library/multiprocessing.html
    if __name__ == '__main__':
        jobs = []
        for i in range(3):   # safety marging to not consume all CPU
            p = multiprocessing.Process(target=shortAnalysator)
            jobs.append(p)
            p.start()


fileCollector()
testSetup()
shortAnalysator()       # proceeding expectably on one CPU (slow)
# polyAnalysator()        # causing irritation
sys.exit()```

Answer 1

你的polyAnalysator三次运行shortAnalysator。尝试按如下方式更改polyAnalysator，并添加f方法。这使用multiprocessing Pool：

from multiprocessing import Pool

def f(test):
    sub.call(test, shell=True)


def polyAnalysator():
    # apparently each CPU core works as if the register were not shared
    # reference: https://docs.python.org/3.7/library/multiprocessing.html
    with Pool(3) as p:
        p.map(f, testRegister)