从Python运行许多CLI进程,避免CPU过载

问题描述 投票:0回答:1

我正在创建一个 python 脚本来通过 CLI 启动许多进程。对某个目录中找到的每个文件进行一次处理。然后我只需循环文件并启动处理该文件的进程。

for path in pathlist:
    # Prepare cli call
    p = subprocess.Popen(cmd...)
    processes.append(p)

我还将所有进程添加到列表中,以在脚本末尾等待它们。由于可能有数百个文件,我不想让 CPU 超载并由于太多上下文切换而使速度变慢。另外,在某些时候,内存也会成为限制因素。

我如何控制上述“逻辑”不“淹没”CPU/操作系统并减慢速度?

python command-line-interface
1个回答
0
投票

这是代码:

# Set a limit for the number of concurrent processes
max_processes = 10

# Initialize a counter for running processes
current_processes = 0

for path in pathlist:
    # Check if the maximum number of processes is reached
    while current_processes >= max_processes:
        time.sleep(1)  # Wait for a while before checking again

    # Prepare cli call
    p = subprocess.Popen(cmd...)
    processes.append(p)
    current_processes += 1

# Wait for all processes to finish
for p in processes:
    p.wait()
© www.soinside.com 2019 - 2024. All rights reserved.