Python：Numba遍历多维数组

Question

我有一个迭代通过一维数组的函数，并检查值是否高于阈值以创建一个掩码。它非常快。但是我如何使用它来迭代不同列上具有不同阈值的多个列。到目前为止，我的方法对于尺寸为18531的1-D阵列需要12μs。如果我迭代两列，我尝试了不同的功能，但只有一个阈值。我怎么能用多个阈值做到这一点？此外，我调查了numpy浮动16或numpy浮动32它慢得多。这是为什么？

import numpy as np
import numba
import pandas as pd
#######Column approach
@numba.jit
def compute_expressionCol_Numba(col,threshold):
    n=len(col)
    result = np.empty(n,dtype='bool')
    for i in range(n):
        if col[i] < threshold:
            result[i]=1
        else:
            result[i]=0
    return result

def compute_expressionCol(col,threshold):
    result = compute_expressionCol_Numba(col.values,threshold)
    return result

##### Multiple column approach

def compute_expressionDF(df,threshold):
    for i in df:
        result = compute_expressionCol_Numba(df[i].values,threshold)
    return result

def make_mask(df, threshold):
    result = np.where(df < threshold, 1 , 0)
    return result

def lt(df, thresh):
    return (df.values<thresh).view('i1')

import numexpr as ne

def lt_numexpr(df, thresh):
    return ne.evaluate('a<thresh',{'a':df.values})

一些时间测试：

for i in [np.float16,np.float32,np.float64]:
    print(i)
    randomDF = pd.DataFrame(np.random.rand(19000,2).astype(i),columns=['col1','col2'])
    thresh = 50
    %timeit compute_expressionCol(randomDF['col1'],50)
    %timeit compute_expressionCol(randomDF['col2'],50)
    %timeit for i in randomDF[['col1','col2']]: compute_expressionCol(randomDF[i],50)
    %timeit (randomDF[['col1','col2']].values < 50).astype(int)
    %timeit (randomDF.values < 50).astype(int)
    %timeit make_mask(randomDF[['col1','col2']],50)
    %timeit randomDF[['col1','col2']]<50
    %timeit randomDF['col1']<50
    %timeit pd.eval('randomDF[["col1","col2"]]<50')
    %timeit lt(randomDF, thresh=50)
    %timeit lt_numexpr(randomDF, thresh=50)

结果：

<class 'numpy.float16'>
40 ms ± 1.05 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
40.6 ms ± 1.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
82.8 ms ± 1.37 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
1.39 ms ± 40.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
416 µs ± 14.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
2.88 ms ± 31.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.95 ms ± 97.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
400 µs ± 8.08 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
31.2 ms ± 1.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
413 µs ± 22.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
598 µs ± 22.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
<class 'numpy.float32'>
30.3 µs ± 2.31 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
35.2 µs ± 2.81 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
1.01 ms ± 67 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
973 µs ± 93.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
37.5 µs ± 4.76 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
3.11 ms ± 544 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.56 ms ± 129 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
224 µs ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
32.4 ms ± 3.75 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
70.8 µs ± 2.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
555 µs ± 18.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
<class 'numpy.float64'>
26.5 µs ± 2.09 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
27.2 µs ± 836 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
1.09 ms ± 62.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.01 ms ± 38.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
43.1 µs ± 2.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
2.78 ms ± 118 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.54 ms ± 35.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
249 µs ± 10.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
30.4 ms ± 1.07 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
71.8 µs ± 3.05 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
558 µs ± 21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Answer 1

我认为np.where可能就是你所追求的。您可以为其提供数据框或系列

import numpy as np

def make_mask(df, threshold):
    result = np.where(df < threshold, 1 , 0)
    return result