如何设法运行具有所有处理器内核的单个处理器项目？

Question

import multiprocessing, time
mpCores=multiprocessing.cpu_count()
print("This PC has", mpCores,"cores. Currently only utilising 1, upgrades to follow")

tic = time.perf_counter()                                           # ** Start the timer                                    

exec(open("xxxx.py").read())    # xxxxx
exec(open("xxxxx.py").read())    # xxxxx
exec(open("xxxxxx.py").read())  # xxxxxxx
exec(open("xxxxxxx.py").read())  # xxxxxxxx

toc = time.perf_counter()                                           # ** Stop the timer
print("TOTAL RUN TIME: ", f"{toc - tic:0.2f} seconds")   # the 0.4f means 4 decimal places

如您所见，代码使用单个处理器核心。我的处理器是12核的。我怎样才能使用 12 个核心来运行这个进程？

我正在运行一个 python 脚本。该脚本调用不同的 python 脚本，但允许其在单个处理器上运行

Answer 1

如果您要读取 Python 文件并使用

exec

函数运行它，那么您需要使用多处理池来并行执行这些文件：

from concurrent.futures import ProcessPoolExecutor
import multiprocessing
import time


def run_python_file(file_name: str) -> None:
    """Execute Python file as a separate process."""

    with open(file_name) as f:
        return exec(f.read())


def main():
    "Execute multiple Python files in parallel."
    tic = time.perf_counter()

    file_names = [
        "xxxx.py",
        "xxxxx.py",
        "xxxxxx.py",
        "xxxxxxx.py"
    ]

    mp_cores =  multiprocessing.cpu_count()
    with ProcessPoolExecutor(max_workers=mp_cores) as executor:
        executor.map(run_python_file, file_names)

    toc = time.perf_counter()
    print("TOTAL RUN TIME: ", f"{toc - tic:0.2f} seconds")

if __name__ == '__main__':
    main()

更好的方法？

我会使用

subprocess.run

方法来执行每个 Python 文件。由于这已经创建了一个单独的子进程来执行 Python 文件，因此您只需要一个多线程池就能够并行运行这些文件：

from concurrent.futures import ThreadPoolExecutor
import multiprocessing
import subprocess
import sys
import time


def run_python_file(file_name: str) -> int:
    """Execute Python file as a separate process."""

    completed_process = subprocess.run([sys.executable, file_name])
    return completed_process.returncode


def main():
    "Execute multiple Python files in parallel."
    tic = time.perf_counter()

    file_names = [
        "xxxx.py",
        "xxxxx.py",
        "xxxxxx.py",
        "xxxxxxx.py"
    ]

    mp_cores =  multiprocessing.cpu_count()
    with ThreadPoolExecutor(max_workers=mp_cores) as executor:
        return_codes = list(executor.map(run_python_file, file_names))
        print(return_codes)

    toc = time.perf_counter()
    print("TOTAL RUN TIME: ", f"{toc - tic:0.2f} seconds")

if __name__ == '__main__':
    main()