张量流错误：cuFFT、cuDNN、cuBLAS 和“断言 '__n < this->size()' 失败”

Question

我刚刚开始使用 TF 和 Keras，发现我无法在我的计算机上运行它们。我首先在 jupyter 笔记本中注意到这个问题，然后在 python 文件中重新创建它。

要重现的代码（main.py）：

import os
os.environ["KERAS_BACKEND"] = "tensorflow"
import keras
from keras import layers
from keras import ops


print("pass")    
model = keras.Sequential()
model.add(layers.Input(shape=(28,)))
print("pass")

python main.py

输出：

2024-10-16 22:30:52.054073: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1729081852.071578   41750 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1729081852.076640   41750 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
pass
/usr/include/c++/14.1.1/bits/stl_vector.h:1130: constexpr std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = pybind11::object; _Alloc = std::allocator<pybind11::object>; reference = pybind11::object&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
fish: Job 1, 'python main.py' terminated by signal SIGABRT (Abort)

我使用 arch linux 并安装了 python-tensorflow-opt-cuda 包。我有一张 nvidia 卡和可用的 cuda。

nvcc --version

输出：

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Sep_12_02:18:05_PDT_2024
Cuda compilation tools, release 12.6, V12.6.77
Build cuda_12.6.r12.6/compiler.34841621_0

python -c "import tensorflow as tf; print(tf.__version__)"

输出：

2024-10-16 22:35:45.309836: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:477] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1729082145.328052   42393 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1729082145.333326   42393 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2.18.0-rc1

pacman -Q cuda

输出：

cuda 12.6.2-1

pacman -Q cudnn

输出：

cudnn 9.2.1.18-1

pacman -Q nvidia

输出：

nvidia-dkms 560.35.03-14

nvidia-dkms 560.35.03-14

pacman -Q blas

输出：

blas 3.12.0-5

pacman -Q python-tensorflow-opt-cuda

输出：

python-tensorflow-opt-cuda 2.18rc1-2

pacman -Q python

输出：

python 3.12.7-1

pacman -Q python-numpy

输出：

python-numpy 2.1.2-1

pacman -Q python-keras

输出：

python-keras 3.4.1-1

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

输出：

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1650        Off |   00000000:01:00.0 Off |                  N/A |
| N/A   43C    P8              1W /   50W |       8MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+

Kernel: 6.11.3-zen1-1-zen
CPU: AMD Ryzen 5 4600H with Radeon Graphics (12) @ 3.000GH
GPU: NVIDIA GeForce GTX 1650 Mobile / Max-Q

我也不明白为什么 archlinux 软件包仓库中的 tf 是 2.18 版本。现在不是不稳定吗

我尝试重新安装所有软件包，但尚未降级。我也尝试使用docker版本的tensorflow，但仍然遇到同样的错误。

Answer 1

发现问题了。

它是过时的 python-optree 包。好像0.13.0版本没有这个问题。

问题是：

>>> from optree import *
>>> tree_map(lambda x: x, ())
/usr/include/c++/14.1.1/bits/stl_vector.h:1130: constexpr std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = pybind11::object; _Alloc = std::allocator<pybind11::object>; reference = pybind11::object&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
fish: Job 1, 'python' terminated by signal SIGABRT (Abort)

空元组或数组引发了这个问题。

在0.13.0版本中：

>>> from optree import *
>>> tree_map(lambda x: x, ())
()

这不是拖曳，一切正常。