我的 Bullmq 工作未运行队列中所有作业的任务

Question

我正在运行后台作业，使用 puppeteer 生成 pdf，并使用收件人姓名和电子邮件通过电子邮件发送这些 pdf。以下是我的工人逻辑

const sendCertificatesWorker = new Worker(
queueName,
async job => {
const {
data: { id, name, email, link },
collectionId,
nameOfOrganization,
thumbnail,
raw
} = job.data;

    // Generate Certificate
    const filePath = await GenerateCertificatePdfs(job.data.data, raw);
    await job.updateProgress("certificate generated");
    
    //Send email
    const result = await sendCertificatesEmail({
      name,
      email,
      link,
      filePath,
      nameOfOrganization,
      thumbnail
    });
    
    if (result.error) {
      await job.updateProgress(
        `email not sent and the reason is: ${result.message}`
      );
    
      await Certificate.updateOne(
        { _id: id },
        { $set: { status: "not sent" } }
      );
    
      // Update DB
      await job.updateProgress(
        `cert with id ${id} updated in database as not sent`
      );
    } else {
      await job.updateProgress(`email sent`);
    
      await Certificate.updateOne({ _id: id }, { $set: { status: "sent" } });
    
      // Update DB
      await job.updateProgress(
        `cert with id ${id} updated in database as sent`
      );
    }
    
    // Update collection
    const collection = await Collection.findById(collectionId);
    
    if (collection) {
      await job.updateProgress("updating the collection..");
      await collection.save();
    }
    
    return { sent: result.error };

},
{ connection: workerConnection }
);

现在的问题是所有作业都添加到队列中，但并非所有作业都被执行。有些在完成任何工作任务之前退出工作人员。查看我服务器的日志

A job with ID 22 is waiting
A job with ID 23 is waiting
A job with ID 24 is waiting
Job 22 is now active; previous status was waiting
Job 23 is now active; previous status was waiting
job with 23 is currently in progress: certificate generated
job with 23 is currently in progress: email sent
job with 23 is currently in progress: cert with id 64afda0da236e94d7290bd47 updated in database as sent
job with 23 is currently in progress: updating the collection..
23 has completed and returned false
Job 24 is now active; previous status was waiting
23 has completed!
22 has completed and returned false
job with 24 is currently in progress: certificate generated
job with 24 is currently in progress: email sent
job with 24 is currently in progress: cert with id 64afda0da236e94d7290bd3f updated in database as sent
job with 24 is currently in progress: updating the collection..
24 has completed and returned false
24 has completed!

Job Id 22 exited without generating the pdf nor email it.

我有理由相信这是一个内存问题，所以我像这样设置了我的 Redis 实例

// Configure redis connection
const queueConnection = new IORedis({
host: "localhost", // Redis server hostname
port: 6379, // Redis server port
maxmemoryPolicy: "noeviction",
maxRetriesPerRequest: null,
enableOfflineQueue: false,
retryStrategy() {
const delay = Math.min(3 \* 50, 2000);
return delay;
}
});

const workerConnection = new IORedis({
host: "localhost", // Redis server hostname
port: 6379, // Redis server port
maxmemoryPolicy: "noeviction",
maxRetriesPerRequest: null,
enableOfflineQueue: true,
retryStrategy() {
const delay = Math.min(3 \* 50, 2000);
return delay;
}
});

但问题仍然存在。有人知道可能会发生什么吗？

Answer 1

我的系统中也遇到了同样的问题。作业正在添加到队列中，但尚未处理，但状态显示已完成。而且它只发生在我的一个环境中（分阶段）。

调试设置、网络条件、消费者运行的节点并尝试在本地复制问题后，没有任何效果，一切都已就位。

然而，最后我尝试了最不可能的解决方案。我尝试连接到本地相同的 staging env bullmq 队列，问题是replicated。

在暂存中更改队列后，一切对我来说一切正常。

我尝试调试redis服务器中的两个队列，但似乎没有什么异常。我得到的唯一逻辑是，队列可能包含一些停滞、等待或失败的作业，这些作业正在堵塞，因为队列长时间处于活动状态，并且有 4k+ 作业 ID。不过，我已经设置了

removeOnFail

和

removeOnComplete

，这不应该是确切的问题，仍在寻找真正的原因。