针对 SQL Azure 的间歇性异常:未知错误 258

问题描述 投票:0回答:1

例外情况

我们在一个(共八个)客户实例中遇到了生产问题,我们无法理解(因此无法解决)。对同一 API 的 250 次调用中,该问题发生的次数不到 10 次。例外情况如下:

An exception occurred while iterating over the results of a query for context type '"OurNamespace.OurDbContext2"'."
""System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Transactions.TransactionException: The operation is not valid for the state of the transaction.
 ---> System.Transactions.TransactionPromotionException: Failure while attempting to promote transaction.
 ---> Microsoft.Data.SqlClient.SqlException (0x80131904): There is already an open DataReader associated with this Command which must be closed first.
 ---> System.ComponentModel.Win32Exception (258): Unknown error 258
   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
   at Microsoft.Data.SqlClient.SqlDelegatedTransaction.Promote()
ClientConnectionId:09c2251f-ce5e-4f27-9c37-77ba27e69399
Error Number:−2,State:0,Class:11
ClientConnectionId before routing:f245b876-d483-4827-bdd9-ec446c664f64

完整的调用堆栈

at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)
   at Microsoft.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
   at Microsoft.Data.SqlClient.SqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.InitializeReader(Enumerator enumerator)
   at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
   at lambda_method1124(Closure , QueryContext )
   at OurNamespace.AspNetCore.Identity.EntityFrameworkCore.TenantSettingsStore`1.FindById(Guid tenantId)
   at OurNamespace.AspNetCore.Identity.Caching.DefaultCache`1.GetOrAdd(String key, TimeSpan duration, Func`1 get)
   at OurNamespace.AspNetCore.Identity.ConfigureMultitenantIdentityOptions.Configure(TenantIdentityOptions options)
   at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
   at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
   at System.Lazy`1.CreateValue()
   at Microsoft.Extensions.Options.OptionsManager`1.Get(String name)
   at OurNamespace.AspNetCore.Identity.ExtendedUserValidator`1.get_TenantIdentityOptions()
   at OurNamespace.AspNetCore.Identity.MultitenantUserValidator`1.ValidateTenant(MultitenantUserManager`1 manager, TUser user, ICollection`1 errors)
   at OurNamespace.AspNetCore.Identity.MultitenantUserValidator`1.ValidateAsync(UserManager`1 manager, TUser user)
   at Microsoft.AspNetCore.Identity.UserManager`1.ValidateUserAsync(TUser user)
   at Microsoft.AspNetCore.Identity.UserManager`1.CreateAsync(TUser user)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.UserViewModels.UserViewModelService`1.CreateUserAsync(TUser user, Boolean sendInvitation, String additionalInvitationQueryParameters)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.UserViewModels.UserViewModelService`1.CreateUserAsync(CreateUserViewModel model)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonViewModels.CreatePersonAndUserService`2.<>c__DisplayClass7_0.<<Create>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at IRM.AspNetCore.Identity.ResilientTransaction.<>c__DisplayClass3_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass33_0`2.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at OurNamespace.AspNetCore.Identity.ResilientTransaction.ExecuteAsync(Func`2 action, CancellationToken cancellationToken)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonViewModels.CreatePersonAndUserService`2.Create(CreatePersonViewModel model, String password, ExternalLoginInfo loginInfo, CancellationToken cancellationToken)
   at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonsController`2.Post(CreatePersonViewModel personViewModel, ICreatePersonAndUserService`2 creationService)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at OurNamespace.Serilog.AspNetCore.SerilogCallContextMiddleware.Invoke(HttpContext context, ICallContext callContext)
   at OurNamespace.AspNetCore.Identity.UI.ManageTenantMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at OurNamespace.AspNetCore.Authentication.Cookies.SameSiteCookieFixMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
   at OurNamespace.AspNetCore.ErrorHandling.ExceptionHandlerMiddleware.Awaited(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

我们认为会影响问题的情况

以下是一些我们认为可以解决问题的注意事项:

  • 问题所在的 API 是唯一一个针对两个 DbContext 实例工作的 API。
    • 因此,我们根据推荐的方法自行创建执行策略。
    • 我们使用 TransactionScope(也如上面的链接中所述)。
    • 对第一个 DbContext 实例的所有调用都会成功。
    • 两个 DbContext 使用相同的数据库/连接字符串。
  • 应用程序和数据库存在于不同的 Azure 订阅中(位于同一数据中心)。
    • 这是八个实例中客户选择自行托管数据库的唯一实例。

我们使用的版本:

  • .NET Core 6
  • Microsoft.EntityFrameworkCore(以及 SQL Server 的相关参考):6.0.20
  • Microsoft.Data.SqlClient:5.1.1

它是一个 Linux 应用服务。我们还使用 SQL Azure 作为分布式缓存。

发生这种情况的示例代码

问题似乎发生在第一次调用 OurDbContext2 上的数据库时,这可能是不同的操作,具体取决于缓存和其他逻辑(上面的调用堆栈只是一个示例)。我选择上面的调用堆栈,因为在这个(不常见)示例中,我们以非常简单的方法从选项中使用的数据库获取设置:

public TenantSettings FindById(Guid tenantId)
{
    using var scope = _serviceProvider.CreateScope();
    using var context = scope.ServiceProvider.GetService<TContext>()!;
            
    var dbSet = context.Set<TenantSettings>();
    var result = dbSet.AsNoTracking().SingleOrDefault(ts => ts.TenantId == tenantId) ?? new TenantSettings { TenantId = tenantId };
            
    return result;
}

该问题不涉及任何选项或此特定操作(也请忽略我们使用 _serviceProvider.CreateScope 的反模式这一事实,因为我们对此有特定的原因),因为它也在其他情况下发生。我选择上述操作,因为我相信它非常清楚地表明该问题与“已经有一个与此命令关联的打开的 DataReader 必须首先关闭”无关,因为我们在其本地创建了一个新的 DbContext 实例自己的 DI 范围。

这是发生这种情况的另一个例子:

protected virtual Task<List<TRole>> GetDefaultRolesForNewUserAsync(TUser user, CancellationToken cancellationToken = default)
{
    return Roles.Where(r => r.AddForNewUser).AsNoTracking().ToListAsync(cancellationToken);
}

我们尝试过的事情

我们增加了 SQL Azure 实例的容量(从 20 DTU 到 50 DTU),从而减少了发生的问题。但从指标来看,我们看不到该实例处于任何类型的负载下。例如,我们检查了会话、DTU 使用情况、CPU、失败连接和其他指标。

我们在连接字符串中启用了 MARS,但这根本不影响问题。

我们已经检查了所有代码,以确保在使用或迭代列表之前没有错过任何 ToList/FirstOrDefault 来实际获取对象(因为这是“已经有一个与此命令关联的打开的 DataReader”的常见原因)必须先关闭它。”)。

我们已经完成了应用服务性能和可用性故障排除中的许多报告。似乎没有任何 SNAT 端口耗尽的情况,我们也没有发现任何其他迹象表明应用程序服务可能存在问题。

我们最好的猜测是,这是 SQL Azure 实例的某种基础设施问题(我们的经验是“未知错误 258”经常出现),但我们不知道它可能是什么。应用程序负载不重(两个不同的应用程序服务实例,每个实例大约 7-8 rpm),并且没有流量峰值。

entity-framework azure-sql-database .net-6.0 sqlclient
1个回答
0
投票

出现此错误 208 的原因可能是在 Azure SQL 数据库上打开了大量会话/连接,或者当 Azure SQL 数据库达到服务层的会话/连接限制时。请开始监视 Azure SL 数据库上保持打开状态的会话/连接数量。

验证您没有达到服务层(DTUvCore 模型)的“最大并发登录数”、“最大并发外部连接数”和“最大并发会话数”限制。

© www.soinside.com 2019 - 2024. All rights reserved.