获取一个分离的子进程的returncode？

Question

我想为一个作业调度器写一个提交器，由于我不知道作业什么时候来，也不知道作业会运行多久，所以我使用多进程为每个作业生成一个带有subprocess和detect的进程。由于我不知道作业什么时候来，作业会运行多久，所以我使用多进程为每个作业生成一个带子进程的进程，然后分离，以便能够处理下一个作业。同时这个效果还不错。不过 我想在作业完成后得到returncode，这可能吗？我尝试了几个子进程的变化，但那些返回RC的子进程在作业运行时都会阻塞进程。

#!/usr/bin/python3
# coding=utf-8
import time
import multiprocessing
import subprocess

JobsList = []

def SubmitJob(jobname):
    """ Submit the next requested job """
    print(f"Starting job {jobname}...")
    JobDir ="/home/xxxxx/Jobs/"
    JobMem = "{}{}.sh".format(JobDir, jobname)
    SysoutFile = "./Sysout/{}.out".format(jobname)
    fh = open(SysoutFile, 'w')
    kwargs = {}
    kwargs.update(start_new_session=True)
    p = subprocess.Popen(JobMem, shell = False, stdout = fh, **kwargs)
    pid = p.pid
    print(f"Job {jobname} pid {pid} submitted...")

def PrepareSubmit():
    """ Create and start one process per job """
    jobs = []

    for Job in JobsList:
        process = multiprocessing.Process(target=SubmitJob, 
                                          args=(Job,))
        jobs.append(process)
        JobsList.remove(Job)

    for j in jobs:
        j.start()

    for j in jobs:
        j.join()

    print("All jobs submitted...")

def main():
    """ Check queue for new job requests """
    number_of_lines = 0
    jobs_list = []

    while 1:
        job_queue = open("/home/xxxxx/Development/Python/#Projects/Scheduler/jobs.que", 'r')
        lines = job_queue.readlines()

        if len(lines) > number_of_lines:
            jobs_list.append(lines[len(lines)-1])
            NewJob = lines[len(lines)-1][:-1]
            JobsList.append(NewJob)
            PrepareSubmit()
            number_of_lines = number_of_lines+1

        time.sleep(1)

if __name__ == "__main__":

    main()

main()中的while循环只是为了测试。

有谁能告诉我这是否可能以及如何实现？先谢谢你。

---

这是给我一个返回代码的代码，但在前一个作业完成之前不会发送作业。因此，如果我有一个长期运行的作业，它会延迟运行作业的过程，我称之为阻塞。

def Submit(job):
    """ Submit the next requested job """
    print(f"Starting job {job}...")
    JobDir ="/home/uwe/Jobs/"
    JobMem = "{}{}.sh".format(JobDir, job)
    SysoutFile = "./Sysout/{}.out".format(job)
    fh = open(SysoutFile, 'w')
    kwargs = {}
    kwargs.update(start_new_session=True)
    p = subprocess.Popen(JobMem, shell = False, stdout = fh, **kwargs)
    pid = p.pid

    while p.poll() == None:
        a = p.poll()
        print(a)
        time.sleep(1)
    else:
        rc = p.returncode
        print(f"PID: {pid} rc: {rc}")

def main(): 
    JobsList = ['JOB90501','JOB00001','JOB00002','JOB00003']

    for Job in JobsList:
        Submit(Job)

罗伊，这是你上次提示后我现在的代码。

def SubmitJob(jobname):
    """ Submit the next requested job """
    JobDir ="/home/uwe/Jobs/"
    JobMem = "{}{}.sh".format(JobDir, jobname)
    SysoutFile = "./Sysout/{}.out".format(jobname)
    fh = open(SysoutFile, 'w')
    kwargs = {}
    kwargs.update(start_new_session=True)
    p = subprocess.Popen(JobMem, shell = False, stdout = fh, **kwargs)
    ProcessList[p] = p.pid
    print(f"Started job {jobname} - PID: {p.pid}")

def main(): 
    c_JobsList = ['JOB00001','JOB00002','JOB00003']

    for Job in c_JobsList:
        SubmitJob(Job)

    for p, pid in ProcessList.items():
        RcFile = "./Sysout/{}.rc".format(pid)
        f = open(RcFile, 'w')
        while p.poll() == None:
            a = p.poll()
            time.sleep(1)
        else:
            rc = p.returncode
            f.writelines(str(rc))
            print(f"PID: {pid} rc: {rc}")

        f.close()

还有输出

Started job JOB00001 - PID: 5426
Started job JOB00002 - PID: 5427
Started job JOB00003 - PID: 5429
PID: 5426 rc: 0
PID: 5427 rc: 0
PID: 5429 rc: 8

Answer 1

编辑（以下为原答案，供以后参考）。

为此要使用的natuaram手段是 Popen.poll但显然在某些情况下，它并不奏效（见 https:/lists.gt.netpythonbugs633489。). 我想提出的解决方案是使用 Popen.wait 超时时间很短，如下面的代码示例。

import subprocess
import time

p = subprocess.Popen(["/bin/sleep", "3"])
print(f"Created process {p.pid}")

count = 0
while True: 
    try:
        ret = p.wait(.001) # wait for 1 ms
        print(f"Got a return code {ret}")
        break
    except subprocess.TimeoutExpired as e: 
        print("..", end = "")

    time.sleep(.5)
    print(f"Still waiting, count is {count}")
    count += 1

print ("Done!")

我得到的输出是：

Created process 30040
..Still waiting, count is 0
..Still waiting, count is 1
..Still waiting, count is 2
..Still waiting, count is 3
..Still waiting, count is 4
..Still waiting, count is 5
Got a return code 0
Done

Original idea - Popen.poll

你应该使用的方法是 Popen.poll (文件). 它返回进程的退出状态，或者是 None 如果它还在运行的话。

要使用它，你必须保留当你调用 subprocess.Popen后来 poll 在这些物体上。