Azure QueueTrigger 函数与 CosmosDB 输出绑定失败

问题描述 投票:0回答:1

我有一个队列触发器 Azure 函数(如下所示),可以读取和写入 CosmosDB。我知道该函数本身正在工作,因为我确实看到一些记录被写入数据库。但我看到很多错误,许多排队的项目最终进入了有毒队列。查看那里的项目,格式是正确的,没有理由相信交易失败是因为数据格式错误。

队列可能包含最多 40k 条消息。随着这些消息中的每一个都被单独推送到数据库,我的问题是否可能在于如何建立与数据库的连接(即在类构造函数中?)...我有点抓住救命稻草。

观察实时日志输出,我看到它成功运行了一段时间,然后发生了一些事情。请参阅代码下方日志中的片段。就好像该函数扩展超出了数据库的处理能力并开始超时或其他什么情况。不幸的是,我对 CosmosDB 还很陌生,而且一般都在队列中工作,所以我不知道该去哪里寻找,而且我似乎无法在 StackOverflow 上找到解决我的特殊情况的现有解决方案。

namespace MatterFunctions
{
    public class CosmosDBQueueTrigger
    {
        private readonly ILogger<CosmosDBQueueTrigger> _logger;
        private string _endpointUri;
        private string _privateKey;
        private CosmosClient _client;

        public CosmosDBQueueTrigger(ILogger<CosmosDBQueueTrigger> logger)
        {
            _logger = logger;

            _endpointUri = "****************/";
            _privateKey = "*******************";

            _client = new CosmosClient(_endpointUri, _privateKey);
        }

        [Function(nameof(CosmosDBQueueTrigger))]
        public async Task Run(
        [QueueTrigger("processed-records", Connection = "AzureWebJobsStorage")] string myQueueItem,
        ILogger log)
        {

            // Parse the message from the queue
            var message = JsonSerializer.Deserialize<ClientMatterDTO>(myQueueItem);

            message!.firmId = "******";
            message.id = $"{message.ClientNumber}.{message.MatterNumber}";

            Database database = await _client.CreateDatabaseIfNotExistsAsync(
                id: "ClientDB");

            Container container = await database.CreateContainerIfNotExistsAsync(
                id: "matters",
                partitionKeyPath: "/firmId",
                throughput: 400
            );

            var itemCount = container.GetItemLinqQueryable<ClientMatterDTO>(true)
                .Count(item => item.id == $"{message.ClientNumber}.{message.MatterNumber}");

            if (itemCount == 0)
            {
                await container.CreateItemAsync(
                    item: message,
                    partitionKey: new PartitionKey("qslwm")
                );
            }
        }
    }
}

这是包含一些成功执行的日志输出,以及 20-30 秒后出现的失败:

2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=d39ea17e-7dca-485d-8c48-eafcbd61a65d)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: e480ad92-82a9-465d-ada6-0cb70ea7b5bc, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=db067251-686c-48f2-9523-1c76e72ffe95)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: ec0942a3-4c5e-4b28-b002-e2d5d4f08f5c, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=34d2562b-6164-4c23-be2b-fcb227069861)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: f6771598-aea3-41bf-b18f-4703564ca85a, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=0ad87c71-77ea-4145-99c4-ad3227c136bd)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: 13ff4bf5-4531-4276-930d-40ee6348c6f7, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=e9fa5bcf-f848-4dcb-9c85-e60d54d338bd)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: 4d4bdeaa-ce61-4b31-9412-852e3e262951, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=be650680-1612-47db-b1a9-0dfe6ea6c820)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: 4f1278c0-40bc-4f34-bc62-fed17e8a7a8d, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=03e6ded5-50ea-4d7c-b76b-2a3d021662fd)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: 524be9ea-ca23-4eee-afd5-c3642ba39c6c, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executing 'Functions.CosmosDBQueueTrigger' (Reason='New queue message detected on 'processed-records'.', Id=f67753c0-7139-44be-9344-2dfc6b0f4583)
2024-04-18T20:30:08Z   [Information]   Trigger Details: MessageId: 55c55570-a60c-4832-bac4-9ea4a22923b6, DequeueCount: 4, InsertedOn: 2024-04-18T17:23:36.000+00:00
2024-04-18T20:30:08Z   [Information]   Executed 'Functions.CosmosDBQueueTrigger' (Succeeded, Id=36cc6a91-128a-4022-b534-60d4b87f1971, Duration=4715ms)
2024-04-18T20:30:08Z   [Information]   Executed 'Functions.CosmosDBQueueTrigger' (Succeeded, Id=5c6902ea-09f4-4cbe-ae1d-ef3a71889799, Duration=4714ms)
2024-04-18T20:30:08Z   [Information]   Executed 'Functions.CosmosDBQueueTrigger' (Succeeded, Id=f288ec25-242a-4bf6-a108-e76c4c340a54, Duration=4713ms)
2024-04-18T20:30:08Z   [Information]   Executed 'Functions.CosmosDBQueueTrigger' (Succeeded, Id=e0110841-8e94-4152-9e35-a424cded1e35, Duration=4715ms)
2024-04-18T20:30:08Z   [Information]   Executed 'Functions.CosmosDBQueueTrigger' (Succeeded, Id=d6bb7bbb-1c52-4a3c-a201-20801438b7eb, Duration=4736ms)
2024-04-18T20:30:08Z   [Information]   Executed 'Functions.CosmosDBQueueTrigger' (Succeeded, Id=962edf9b-0cfd-4a81-a5bd-ad8f4f5f38f5, Duration=4738ms)
2024-04-18T20:30:09Z   [Error]   Function 'CosmosDBQueueTrigger', Invocation id '43ed6607-6ae0-4425-a014-4c6ff2437861': An exception was thrown by the invocation.
2024-04-18T20:30:09Z   [Error]   Function 'CosmosDBQueueTrigger', Invocation id 'af11707b-5495-455b-8a5e-06d4b41a0584': An exception was thrown by the invocation.
2024-04-18T20:30:09Z   [Error]   Executed 'Functions.CosmosDBQueueTrigger' (Failed, Id=af11707b-5495-455b-8a5e-06d4b41a0584, Duration=1195ms)
2024-04-18T20:30:09Z   [Error]   Executed 'Functions.CosmosDBQueueTrigger' (Failed, Id=43ed6607-6ae0-4425-a014-4c6ff2437861, Duration=1195ms)
2024-04-18T20:30:08Z   [Information]   Stopping JobHost
2024-04-18T20:30:08Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.Storage.Common.Listeners.QueueListener' for function 'CosmosDBQueueTrigger'
2024-04-18T20:30:11Z   [Information]   Task hub worker stopped. IsGracefulStop: False. Latency: 00:00:02.4124437. Extension GUID d418edb2-0c02-447e-912a-8fa1cc269c87. InstanceId: . Function: . HubName: MatterFunctions. AppName: MatterFunctions. SlotName: Production. ExtensionVersion: 2.9.6. SequenceNumber: 4.
2024-04-18T20:30:11Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'ProcessBatch'
2024-04-18T20:30:16Z   [Information]   Stopping JobHost
2024-04-18T20:30:16Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.Storage.Common.Listeners.QueueListener' for function 'CosmosDBQueueTrigger'
2024-04-18T20:30:16Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'OrchestratorFunction'
2024-04-18T20:30:16Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'OrchestratorFunction'
2024-04-18T20:30:16Z   [Information]   Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'ProcessBatch'
2024-04-18T20:30:16Z   [Information]   Stopping task hub worker. IsGracefulStop: False. Extension GUID 2c961204-449f-450c-8218-271df7ccf20e. InstanceId: . Function: . HubName: MatterFunctions. AppName: MatterFunctions. SlotName: Production. ExtensionVersion: 2.9.6. SequenceNumber: 3.
2024-04-18T20:30:16Z   [Information]   Task hub worker stopped. IsGracefulStop: False. Latency: 00:00:00.1383927. Extension GUID 2c961204-449f-450c-8218-271df7ccf20e. InstanceId: . Function: . HubName: MatterFunctions. AppName: MatterFunctions. SlotName: Production. ExtensionVersion: 2.9.6. SequenceNumber: 4.
2024-04-18T20:30:16Z   [Information]   Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskListener' for function 'ProcessBatch'

我无法找到包含有关此错误的更详细信息的日志。有人能指出我正确的方向吗?

azure-functions azure-cosmosdb azure-queues azure-storage-queues
1个回答
0
投票

Application Insights 中的

exception
表可能有详细信息。通常这就是异常发生的地方。

就代码而言:

  1. 请不要每次执行都执行CreateXXXXIfNotExists。这是最佳实践的一部分:https://learn.microsoft.com/azure/cosmos-db/nosql/best-practice-dotnet
  2. 不要执行计数查询,而是查看是否:
    1. 执行 ReadItemAsync 并捕获带有 404 状态代码的 CosmosException 是可能的(假设您知道 id 和分区键)
    2. 您可以执行 Create 并捕获 StatusCode 409(冲突)的 CosmosException,这意味着该文档已经存在。
© www.soinside.com 2019 - 2024. All rights reserved.