GET 无法找到执行此计算的引擎

Question

我正在尝试微调 Facebook 发布的“Segment Anything”模型。具体来说，我正在尝试调整模型的一个分支，该模型具有一些用于微调医学图像的代码，称为 MedSAM（在 here 中找到）。尝试运行代码时，出现以下错误：

Traceback (most recent call last):
  File "/mnt/beegfs/khans24/medsam_finetuning/minimal.py", line 30, in <module>
    embedding = sam_model.image_encoder(input_image)
  File "/home/khans24/beegfs/miniconda/envs/sam/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/beegfs/khans24/segment-anything/segment_anything/modeling/image_encoder.py", line 107, in forward
    x = self.patch_embed(x)
  File "/home/khans24/beegfs/miniconda/envs/sam/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/beegfs/khans24/segment-anything/segment_anything/modeling/image_encoder.py", line 392, in forward
    x = self.proj(x)
  File "/home/khans24/beegfs/miniconda/envs/sam/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/khans24/beegfs/miniconda/envs/sam/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/khans24/beegfs/miniconda/envs/sam/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: GET was unable to find an engine to execute this computation

我曾多次尝试通过切换不同的 CUDA 和 cuDNN 版本来解决此问题，但无济于事。这是我运行它的系统的规格：

• PyTorch 2.0（但也尝试过 1.13 和 1.7）
• 运行时 CUDA 11.0
• 运行时 cuDNN 8.2.1

torch.backends.cudnn.version()

的输出是

8500

运行以下代码我得到了积极的结果：

print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.backends.cudnn.enabled)
print(torch.backends.cudnn.version())

输出是

True, 4, True, 8500

我也在高性能计算集群上运行这个，操作系统是RedHat Linux。我还使用“模块”包从模块文件加载到 CUDA 工具包中。由于我没有 root 访问权限，因此我安装东西的能力有限。

这也是一个最小的可复制示例：

import torch
import numpy as np
from skimage import io, transform
from segment_anything import SamPredictor, sam_model_registry
from segment_anything.utils.transforms import ResizeLongestSide

# Set up the model and device
model_type = 'vit_b'
checkpoint = 'load/medsam_20230423_vit_b_0.0.1.pth'
device = 'cuda:0'

sam_model = sam_model_registry[model_type](checkpoint=checkpoint).to(device)

# Generate a random image
image_size = 256
random_image = np.random.randint(0, 256, (image_size, image_size, 3), dtype=np.uint8)

# Resize the random image
sam_transform = ResizeLongestSide(sam_model.image_encoder.img_size)
resized_image = sam_transform.apply_image(random_image)

# Convert the resized image to a PyTorch tensor
resized_image_tensor = torch.as_tensor(resized_image.transpose(2, 0, 1)).to(device)

# Preprocess the image tensor
input_image = sam_model.preprocess(resized_image_tensor[None, :, :, :])

# Compute the image embedding using the sam_model
with torch.no_grad():
    embedding = sam_model.image_encoder(input_image)
    print(embedding.shape)

任何帮助表示赞赏。