从 Python 中获取 future 的结果

Question

我有以下代码，它执行一个进程并在进程完成时调用回调函数

import os
import subprocess
import tempfile


def callback(future):
    print(future.temp_file_name)
    try:
        with open(future.temp_file_name, 'r') as f:
            print(f"Output from {future.temp_file_name}:")
            print(f.read())
    except Exception as e:
        print(f"Error in callback: {e}")


def execute(args):
    from concurrent.futures import ProcessPoolExecutor as Pool

    args[0] = os.path.expanduser(args[0])
    with tempfile.NamedTemporaryFile(delete=False) as temp_file:
        temp_file_name = temp_file.name

        print('temp_file_name', temp_file_name)

        pool = Pool()
        with open(temp_file_name, 'w') as output_file:
            process = subprocess.Popen(args, stdout=output_file, stderr=output_file)
            future = pool.submit(process.wait)
            future.temp_file_name = temp_file_name
            future.add_done_callback(callback)

            print("Running task")


if __name__ == '__main__':

    execute(['~/testSpawn.sh', '1'])
    execute(['~/testSpawn.sh', '2'])
    execute(['~/testSpawn.sh', '3'])

但是如果我尝试在回调中打印出结果

def callback(future):
    print(future.temp_file_name, future.result())

我收到以下错误

TypeError: cannot pickle '_thread.lock' object
exception calling callback for <Future at 0x77d48bc55a90 state=finished raised TypeError>

如何在回调中获取结果？ 重点是我希望在流程完成时收到通知

更新

我修改了回调函数，使其具有以下内容：

 if future.exception() is not None:
            print('exception', future.exception())
        else:
            print('result', future.result())

这只是意味着它没有那么难看并且可以打印出来

exception cannot pickle '_thread.lock' object

但它告诉我错误不是打印出来

future.results()

，而是回调函数中存在固有错误

Answer 1

subprocess.Popen

对象不能跨进程边界传递。

使用

ThreadPoolExecutor

，而不是

ProcessPoolExecutor

。