face_recognition 抛出 CUDA 异常，但仅在 Celery 任务中

Question

我有一个 face_recognition 任务，它是一个更大的 Django 4.1 站点的一部分，人脸识别任务在带有 redis 的 celery 中运行。在过去的几年里，它在旧版本的 Ubuntu、celery、django、redis、人脸识别上运行没有问题。我将应用程序移至升级后的服务器并升级了所有要求。如果我在升级后的服务器上不使用 celery 运行任务，它会使用 cnn 或 hog 模型查找人脸。当我运行相同的代码，但作为 celery 任务时，我得到一个异常，指出我的 CUDA 安装有问题。但是，我用示例程序测试了 CUDA 安装，测试全部通过，没有错误。

人脸识别版本：1.3.0， 人脸识别模型：版本 0.3.0 Python版本：Python 3.8.16 操作系统：Ubuntu 22.04.2 LTS

我在 Django 项目的 virtualenv 中运行了所有这些测试。注意，有一个后端mysql数据库，我附上的图像有document_id = 3443.

我用了“vanilla”face_recognition代码，hog和cnn都找到了人脸

>>> import face_recognition
>>> image = face_recognition.load_image_file("/home/mark/Documents/archive/do_not_change/documents/originals/photo/Priest_Lake_2001.png")
>>> face_recognition.face_locations(image, model='cnn')
[(278, 490, 325, 443), (317, 556, 357, 517), (175, 410, 232, 353), (336, 381, 393, 324), (427, 404, 474, 356)]
>>> face_recognition.face_locations(image, model='hog')
[(182, 412, 234, 360), (286, 487, 329, 443), (314, 561, 366, 510), (343, 383, 395, 331), (426, 411, 488, 349)]
face_recognition.face_encodings produce 5 arrays for both hog and cnn.

我在没有使用celery的情况下测试了find_faces任务，cnn和hog都找到了faces

>>> from biometric_identification.tasks import find_faces_task
>>> find_faces_task(3443, use_cuda=False)
It found all 5 faces in the photograph.
>>> find_faces_task(3443, use_cuda=True)
It found all 5 faces in the photograph.

我使用 celery 运行 find_faces 任务，得到了这个异常： 使用 CUDA

>>> from biometric_identification.tasks import find_faces_task
>>> find_faces_task.delay(3443, use_cuda=True)
it generated 5 exceptions like this:
[2023-03-24 17:18:17,484: ERROR/ForkPoolWorker-8] Hit an exception in find_faces_task Error while calling cudaGetDevice(&the_device_id) in file /tmp/pip-install-lwueab1p/dlib_294881559a49454d89cb1b07b2dcb694/dlib/cuda/gpu_data.cpp:204. code: 3, reason: initialization error
Traceback (most recent call last):
  File "/home/mark/python-projects/archive/biometric_identification/tasks.py", line 80, in find_faces_task
    face_locations = face_recognition.face_locations(image, model="cnn", number_of_times_to_upsample=0)
  File "/home/mark/.virtualenvs/archive/lib/python3.8/site-packages/face_recognition/api.py", line 119, in face_locations
    return [_trim_css_to_bounds(_rect_to_css(face.rect), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, "cnn")]
  File "/home/mark/.virtualenvs/archive/lib/python3.8/site-packages/face_recognition/api.py", line 103, in _raw_face_locations
    return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudaGetDevice(&the_device_id) in file /tmp/pip-install-lwueab1p/dlib_294881559a49454d89cb1b07b2dcb694/dlib/cuda/gpu_data.cpp:204. code: 3, reason: initialization error

不使用 CUDA 也会产生异常，但不是来自

face_recognition.face_locations

，而是来自

face_recognition.face_encodings

：

[2023-03-24 17:49:48,872: ERROR/ForkPoolWorker-8] Hit an exception in find_faces_task Error while calling cudaGetDevice(&the_device_id) in file /tmp/pip-install-lwueab1p/dlib_294881559a49454d89cb1b07b2dcb694/dlib/cuda/gpu_data.cpp:204. code: 3, reason: initialization error
Traceback (most recent call last):
  File "/home/mark/python-projects/archive/biometric_identification/tasks.py", line 92, in find_faces_task
    face_encodings = face_recognition.face_encodings(image, known_face_locations=face_locations)
  File "/home/mark/.virtualenvs/archive/lib/python3.8/site-packages/face_recognition/api.py", line 214, in face_encodings
    return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
  File "/home/mark/.virtualenvs/archive/lib/python3.8/site-packages/face_recognition/api.py", line 214, in <listcomp>
    return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
RuntimeError: Error while calling cudaGetDevice(&the_device_id) in file /tmp/pip-install-lwueab1p/dlib_294881559a49454d89cb1b07b2dcb694/dlib/cuda/gpu_data.cpp:204. code: 3, reason: initialization error

find_faces 任务：

@app.task(bind=True, base=utils.BaseTaskWithRetry)
def find_faces_task(self, document_id, use_cuda=settings.USE_CUDA):
    logger.debug("find_faces_task in tasks START")
    try:
        temp_file = None
        from memorabilia.models import TaskStatus, Document      
        args = "document_id=%s, use_cuda=%s" % (document_id, use_cuda)
        ts = TaskStatus(document_id_id=document_id, task_id=self.request.id, task_name='find_faces_task', task_args=args, task_status=TaskStatus.PENDING)
        ts.save()
        import time
        time_start = time.time()
        from memorabilia.models import Document
        from biometric_identification.models import Face
        if len(Face.objects.filter(document_id=document_id)) != 0:
            # This document has already been scanned, so need to remove it and rescan
            # Have to manually delete each object per django docs to insure the 
            # model delete method is run to update the metadata.
            logger.debug("Document %s has already been scanned" % document_id)
            faces = Face.objects.filter(document_id=document_id)
            for face in faces:
                face.delete()
                logger.debug("Deleted face=%s" % face.tag_value.value)
        document = Document.objects.get(document_id=document_id)
        image_file = document.get_default_image_file(settings.DEFAULT_DISPLAY_IMAGE)
        image_path = image_file.path
        logger.debug("document_id=%s, image_path=%s" % (document_id, image_path))
        time_start_looking = time.time()
        temp_file = open(image_path, 'rb')
        temp_image = Image.open(temp_file)
        logger.debug("temp_image.mode=%s" % temp_image.mode)
        width, height = temp_image.size
        image = face_recognition.load_image_file(temp_file)
        # Get the coordinates of each face
        if use_cuda:
            # With CUDA installed
            logger.debug("Using CUDA for face recognition")
            face_locations = face_recognition.face_locations(image, model="cnn", number_of_times_to_upsample=0) 
        else:
            # without CUDA installed
            logger.debug("NOT using CUDA for face recognition")
            #face_locations = face_recognition.face_locations(image, number_of_times_to_upsample=2)
            face_locations = face_recognition.face_locations(image, model="hog", number_of_times_to_upsample=2)
        if len(face_locations) == 0:
            ts.task_status = TaskStatus.WARNING
            ts.comment = "Found %s faces" % len(face_locations)
        else:
            time_find_faces = time.time()
            # Get the face encodings for each face in the picture    
            face_encodings = face_recognition.face_encodings(image, known_face_locations=face_locations) 
            logger.debug("Found %s face locations and %s encodings" % (len(face_locations), len(face_encodings)))
            time_face_encodings = time.time()
            # Save the faces found in the database
            for location, encoding in zip(face_locations, face_encodings):
                # Create the new Face object and load in the document, encoding, and location of a face found
                # Locations seem to be of the form (y,x)
                from memorabilia.models import MetaData, MetaDataValue
                tag_type_people = MetaDataValue.objects.filter(metadata_id=MetaData.objects.filter(name='Tag_types')[0].metadata_id, value='People')[0]
                tag_value_unknown = MetaDataValue.objects.filter(metadata_id=MetaData.objects.filter(name='Unknown')[0].metadata_id, value='Unknown')[0]
                new_face = Face(document=document, face_encoding=numpy_to_json(encoding), face_location=location, image_size={'width': width, "height":height}, tag_type=tag_type_people, tag_value=tag_value_unknown)         
                # save the newly found Face object
                new_face.save()
                logger.debug("Saved new_face %s" % new_face.face_file) 
            time_end = time.time()
            logger.debug("total time = {}".format(time_end - time_start))
            logger.debug("time to find faces = {}".format(time_find_faces - time_start_looking))
            logger.debug("time to find encodings = {}".format(time_face_encodings - time_find_faces))
            ts.task_status = TaskStatus.SUCCESS
            ts.comment = "Found %s faces" % len(face_encodings)
        return document_id
    except Exception as e:
        logger.exception("Hit an exception in find_faces_task %s" % str(e))
        ts.task_status = TaskStatus.ERROR
        ts.comment = "An exception while finding faces: %s" % repr(e)
        ts.save(update_fields=['task_status', 'comment'])
        raise Exception("Hit an exception in find_faces_task for document_id=%s and use_cuda=%s" %(document_id, use_cuda)) from e
    finally:
        logger.debug("Finally clause in find-faces_task")
        logger.debug("temp_file=%s" % temp_file)
        logger.debug("temp_image=%s" % temp_image)
        if temp_file:
            temp_file.close()
            logger.debug("closed temp_file=%s" % temp_file)
        if temp_image:
            temp_image.close()
            logger.debug("closed temp_image=%s" % temp_image)
        ts.save(update_fields=['task_status', 'comment'])
        logger.debug("find_faces_task END")