通过 pyx 扩展 numpy random 在将 python 对象转换为 bitgen_t 时会引发 Cython 错误

Question

我正在尝试使用 Numpy C-API 通过

BitGenerator

扩展随机数，如 文档中的 numpy 示例所示，并且我遇到了

PyObject

无法转换为

bitgen_t

错误。

我已将第一个代码复制到名为

extension_rand.pyx

的文件中，如下所示：

extension_rand.pyx

from libc.stdint cimport uint32_t
from cpython.pycapsule import PyCapsule_IsValid, PyCapsule_GetPointer

import numpy as np
cimport numpy as np
cimport cython

from numpy.random cimport bitgen_t
from numpy.random import PCG64

np.import_array()

@cython.boundscheck(False)
@cython.wraparound(False)
def uniforms(Py_ssize_t n):
    """
    Create an array of `n` uniformly distributed doubles.
    A 'real' distribution would want to process the values into
    some non-uniform distribution
    """
    cdef Py_ssize_t i
    cdef bitgen_t *rng
    cdef const char *capsule_name = "BitGenerator"
    cdef double[::1] random_values
    cdef np.ndarray randoms

    x = PCG64()
    capsule = x.capsule
    # Optional check that the capsule if from a BitGenerator
    if not PyCapsule_IsValid(capsule, capsule_name):
        raise ValueError("Invalid pointer to anon_func_state")
    # Cast the pointer
    rng = PyCapsule_GetPointer(capsule, capsule_name)
    random_values = np.empty(n, dtype='float64')
    with x.lock, nogil:
        for i in range(n):
            # Call the function
            random_values[i] = rng.next_double(rng.state)
    randoms = np.asarray(random_values)

    return randoms

设置.py

我的

setup.py

文件如下：

# setup.py

from distutils.core import setup
from Cython.Build import cythonize
from setuptools.extension import Extension
import numpy as np
import os

lib_path = os.path.join(np.get_include(), '..', '..', 'random', 'lib')


extending = Extension(name="extension_rand", 
    sources=['extension_rand.pyx'],
    include_dirs=[np.get_include()],
    library_dirs=[lib_path],
    libraries=['npyrandom']
    )


setup(
    ext_modules=cythonize(
        extending, language_level="3", annotate="True"
    )
)

编译命令

当我跑步时

python setup.py build_ext --inplace

我收到以下错误：

错误日志

Compiling extension_rand.pyx because it changed.
[1/1] Cythonizing extension_rand.pyx

Error compiling Cython file:
------------------------------------------------------------
...
    capsule = x.capsule
    # Optional check that the capsule if from a BitGenerator
    if not PyCapsule_IsValid(capsule, capsule_name):
        raise ValueError("Invalid pointer to anon_func_state")
    # Cast the pointer
    rng = PyCapsule_GetPointer(capsule, capsule_name)
                              ^
------------------------------------------------------------

extension_rand.pyx:33:30: Cannot convert Python object to 'bitgen_t *'

Error compiling Cython file:
------------------------------------------------------------
...
    capsule = x.capsule
    # Optional check that the capsule if from a BitGenerator
    if not PyCapsule_IsValid(capsule, capsule_name):
        raise ValueError("Invalid pointer to anon_func_state")
    # Cast the pointer
    rng = PyCapsule_GetPointer(capsule, capsule_name)
                              ^
------------------------------------------------------------

extension_rand.pyx:33:30: Storing unsafe C derivative of temporary Python reference
Traceback (most recent call last):
  File "/Users/sayantanghosh/work/cython_test/setup.py", line 21, in <module>
    ext_modules=cythonize(
                ^^^^^^^^^^
  File "/Users/sayantanghosh/anaconda3/envs/nonlin/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
    cythonize_one(*args)
  File "/Users/sayantanghosh/anaconda3/envs/nonlin/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: extension_rand.pyx

系统信息/运行环境

我的

import numpy as np; np.show_runtime()

的输出是

[{'numpy_version': '1.26.4',
  'python': '3.11.10 | packaged by conda-forge | (main, Sep 30 2024, 17:57:16) '
            '[Clang 17.0.6 ]',
  'uname': uname_result(system='Darwin', node='calypso.local', release='22.6.0', version='Darwin Kernel Version 22.6.0: Thu Sep  5 20:48:48 PDT 2024; root:xnu-8796.141.3.708.1~1/RELEASE_X86_64', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3', 'SSSE3'],
                      'found': ['SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2'],
                      'not_found': ['AVX512F',
                                    'AVX512CD',
                                    'AVX512_KNL',
                                    'AVX512_SKX',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'Haswell',
  'filepath': '/Users/sayantanghosh/anaconda3/envs/nonlin/lib/libopenblasp-r0.3.21.dylib',
  'internal_api': 'openblas',
  'num_threads': 2,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]

我的

setuptools

版本是

'75.1.0'

，而

cython

版本是

'3.0.10'

。 我不确定这是否是

cython

或

numpy

C-API 相关问题。

如果有任何帮助，我将不胜感激。

Answer 1

我认为有两个问题：

1. from cpython.pycapsule import PyCapsule_IsValid, PyCapsule_GetPointer

   需要是

   from cpython.pycapsule cimport PyCapsule_IsValid, PyCapsule_GetPointer

   。 （

   cimport

   而不是

   import

   ，因此类型在 Cython 编译时已知。否则，它假设函数是返回任意 Python 对象的任意 Python 函数）。

2. PyCapsule_GetPointer

   返回

   void*

   ，您需要将其转换为

   bitgen_t*

   （即

   rng = <bitgen_t*>PyCapsule_GetPointer(capsule, capsule_name)

   ）