一个Python函数在多核上运行时不能正常工作，但在单核上运行正常”语法正确

Question

此功能在

n_jobs!=1

时无法正常工作

from joblib import Parallel, delayed
import numpy as np 
import pandas as pd

def foo(n_jobs):

    result = {}

    x = np.sin(4 * np.pi * np.arange(0, 1, 0.001))
    y = np.sin(8 * np.pi * np.arange(0, 1, 0.001) + np.pi/2)

    x2yT = np.zeros(x.shape[0])
    y2xT = np.zeros(x.shape[0])
    
    def parallelize(ite):
        
        xi = x[ite] * 2
        yi = y[ite] + 1
        y2xT[ite] = xi
        x2yT[ite] = yi

    r = Parallel(n_jobs=n_jobs)(delayed(parallelize)(i) for i in np.arange(x.shape[0]))

    result[f'y2xT'] = y2xT
    result[f'x2yT'] = x2yT
    return pd.DataFrame(result)

以下代码片段使用具有单核的函数

foo

生成一个图：

r0 = foo( n_jobs = 1)
r0.plot()

结果图如下所示：

.

以下代码片段使用具有多个核心的函数

foo

生成绘图：

r0 = foo( n_jobs = -1)
r0.plot()

结果图如下所示：

我怎样才能确保该功能正确利用多核

Answer 1

根据 Frank 的评论，我尝试将

y2xT

和

x2yT

从

numpy.ndarray

转换为

numpy.memmap

，并且有效！

def foo(n_jobs):

    result = {}
        
    x = np.sin(4 * np.pi * np.arange(0, 1, 0.001))
    y = np.sin(8 * np.pi * np.arange(0, 1, 0.001) + np.pi/2)


    # x2yT = np.zeros(x.shape[0])
    # y2xT = np.zeros(x.shape[0])
    
    x2yT = np.memmap('x2yT.pkl', dtype=np.float32, shape=x.shape[0], mode='w+')
    y2xT = np.memmap('y2xT.pkl', dtype=np.float32, shape=x.shape[0], mode='w+')
    
    def parallelize(ite):
        
        xi = x[ite] * 2
        yi = y[ite] + 1
        y2xT[ite] = xi
        x2yT[ite] = yi

    r = Parallel(n_jobs=n_jobs)(delayed(parallelize)(i) for i in np.arange(x.shape[0]))

    result[f'y2xT'] = y2xT
    result[f'x2yT'] = x2yT


    return pd.DataFrame(result)

r0 = foo( n_jobs = -1)
r0.plot()