我们在一个(共八个)客户实例中遇到了生产问题,我们无法理解(因此无法解决)。对同一 API 的 250 次调用中,该问题发生的次数不到 10 次。例外情况如下:
An exception occurred while iterating over the results of a query for context type '"OurNamespace.OurDbContext2"'."
""System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.Transactions.TransactionException: The operation is not valid for the state of the transaction.
---> System.Transactions.TransactionPromotionException: Failure while attempting to promote transaction.
---> Microsoft.Data.SqlClient.SqlException (0x80131904): There is already an open DataReader associated with this Command which must be closed first.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at Microsoft.Data.SqlClient.SqlDelegatedTransaction.Promote()
ClientConnectionId:09c2251f-ce5e-4f27-9c37-77ba27e69399
Error Number:−2,State:0,Class:11
ClientConnectionId before routing:f245b876-d483-4827-bdd9-ec446c664f64
at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
at Microsoft.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)
at Microsoft.Data.ProviderBase.DbConnectionPool.PrepareConnection(DbConnection owningObject, DbConnectionInternal obj, Transaction transaction)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.Open()
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReader(RelationalCommandParameterObject parameterObject)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.InitializeReader(Enumerator enumerator)
at Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable`1.Enumerator.MoveNext()
at System.Linq.Enumerable.TryGetSingle[TSource](IEnumerable`1 source, Boolean& found)
at lambda_method1124(Closure , QueryContext )
at OurNamespace.AspNetCore.Identity.EntityFrameworkCore.TenantSettingsStore`1.FindById(Guid tenantId)
at OurNamespace.AspNetCore.Identity.Caching.DefaultCache`1.GetOrAdd(String key, TimeSpan duration, Func`1 get)
at OurNamespace.AspNetCore.Identity.ConfigureMultitenantIdentityOptions.Configure(TenantIdentityOptions options)
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at Microsoft.Extensions.Options.OptionsManager`1.Get(String name)
at OurNamespace.AspNetCore.Identity.ExtendedUserValidator`1.get_TenantIdentityOptions()
at OurNamespace.AspNetCore.Identity.MultitenantUserValidator`1.ValidateTenant(MultitenantUserManager`1 manager, TUser user, ICollection`1 errors)
at OurNamespace.AspNetCore.Identity.MultitenantUserValidator`1.ValidateAsync(UserManager`1 manager, TUser user)
at Microsoft.AspNetCore.Identity.UserManager`1.ValidateUserAsync(TUser user)
at Microsoft.AspNetCore.Identity.UserManager`1.CreateAsync(TUser user)
at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.UserViewModels.UserViewModelService`1.CreateUserAsync(TUser user, Boolean sendInvitation, String additionalInvitationQueryParameters)
at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.UserViewModels.UserViewModelService`1.CreateUserAsync(CreateUserViewModel model)
at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonViewModels.CreatePersonAndUserService`2.<>c__DisplayClass7_0.<<Create>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at IRM.AspNetCore.Identity.ResilientTransaction.<>c__DisplayClass3_0.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass33_0`2.<<ExecuteAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync[TState,TResult](Func`4 operation, Func`4 verifySucceeded, TState state, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
at OurNamespace.AspNetCore.Identity.ResilientTransaction.ExecuteAsync(Func`2 action, CancellationToken cancellationToken)
at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonViewModels.CreatePersonAndUserService`2.Create(CreatePersonViewModel model, String password, ExternalLoginInfo loginInfo, CancellationToken cancellationToken)
at OurNamespace.AspNetCore.Identity.UI.Areas.Identity.Controllers.API.PersonsController`2.Post(CreatePersonViewModel personViewModel, ICreatePersonAndUserService`2 creationService)
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at OurNamespace.Serilog.AspNetCore.SerilogCallContextMiddleware.Invoke(HttpContext context, ICallContext callContext)
at OurNamespace.AspNetCore.Identity.UI.ManageTenantMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at OurNamespace.AspNetCore.Authentication.Cookies.SameSiteCookieFixMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Localization.RequestLocalizationMiddleware.Invoke(HttpContext context)
at OurNamespace.AspNetCore.ErrorHandling.ExceptionHandlerMiddleware.Awaited(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
以下是一些我们认为可以解决问题的注意事项:
我们使用的版本:
它是一个 Linux 应用服务。我们还使用 SQL Azure 作为分布式缓存。
问题似乎发生在第一次调用 OurDbContext2 上的数据库时,这可能是不同的操作,具体取决于缓存和其他逻辑(上面的调用堆栈只是一个示例)。我选择上面的调用堆栈,因为在这个(不常见)示例中,我们以非常简单的方法从选项中使用的数据库获取设置:
public TenantSettings FindById(Guid tenantId)
{
using var scope = _serviceProvider.CreateScope();
using var context = scope.ServiceProvider.GetService<TContext>()!;
var dbSet = context.Set<TenantSettings>();
var result = dbSet.AsNoTracking().SingleOrDefault(ts => ts.TenantId == tenantId) ?? new TenantSettings { TenantId = tenantId };
return result;
}
该问题不涉及任何选项或此特定操作(也请忽略我们使用 _serviceProvider.CreateScope 的反模式这一事实,因为我们对此有特定的原因),因为它也在其他情况下发生。我选择上述操作,因为我相信它非常清楚地表明该问题与“已经有一个与此命令关联的打开的 DataReader 必须首先关闭”无关,因为我们在其本地创建了一个新的 DbContext 实例自己的 DI 范围。
这是发生这种情况的另一个例子:
protected virtual Task<List<TRole>> GetDefaultRolesForNewUserAsync(TUser user, CancellationToken cancellationToken = default)
{
return Roles.Where(r => r.AddForNewUser).AsNoTracking().ToListAsync(cancellationToken);
}
我们增加了 SQL Azure 实例的容量(从 20 DTU 到 50 DTU),从而减少了发生的问题。但从指标来看,我们看不到该实例处于任何类型的负载下。例如,我们检查了会话、DTU 使用情况、CPU、失败连接和其他指标。
我们在连接字符串中启用了 MARS,但这根本不影响问题。
我们已经检查了所有代码,以确保在使用或迭代列表之前没有错过任何 ToList/FirstOrDefault 来实际获取对象(因为这是“已经有一个与此命令关联的打开的 DataReader”的常见原因)必须先关闭它。”)。
我们已经完成了应用服务性能和可用性故障排除中的许多报告。似乎没有任何 SNAT 端口耗尽的情况,我们也没有发现任何其他迹象表明应用程序服务可能存在问题。
我们最好的猜测是,这是 SQL Azure 实例的某种基础设施问题(我们的经验是“未知错误 258”经常出现),但我们不知道它可能是什么。应用程序负载不重(两个不同的应用程序服务实例,每个实例大约 7-8 rpm),并且没有流量峰值。