.NET Core 应用程序从子进程读取 stdout 和 stderr 时出现死锁 - 如何使用 WinDBG 进行故障排除?

问题描述 投票:0回答:1

我怀疑我陷入了https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.process.standardoutput?view=net-9.0#remarks,特别是当家长时的情况进程读取 stdout 和 stderr。

我无法重现它,但我的同事可以,这使得调查变得有些困难。

死锁发生在下面方法中的

process.StandardOutput.ReadLineAsync
内部。

public async Task<ProcessRunnerResult> Run(string executablePath,
    IProcessRunner.Mode mode,
    string? args = null,
    string? workingDirectory = null,
    Action<ProcessStartInfo>? beforeInvoke = null,
    Action<string>? notifyStdOutLine = null,
    CancellationToken cancellationToken = default)
{
    var process = new Process
    {
        StartInfo = new ProcessStartInfo
        {
            FileName = executablePath,
            Arguments = args,
            RedirectStandardOutput = true,
            RedirectStandardError = true,
            CreateNoWindow = true,
            UseShellExecute = false,
            WorkingDirectory = workingDirectory
        }
    };

    string? stdErr = null;
    try
    {
        beforeInvoke?.Invoke(process.StartInfo);

        process.Start();

        string? line;
        while ((line = await process.StandardOutput.ReadLineAsync(cancellationToken)) != null)
        {
            notifyStdOutLine?.Invoke(line);
        }

        stdErr = await process.StandardError.ReadToEndAsync(cancellationToken);
    }
    catch (Exception ex)
    {
        if (mode == IProcessRunner.Mode.NeverThrow)
        {
            return new ProcessRunnerResult(-1, null, ExceptionDispatchInfo.Capture(ex));
        }
        throw new ProcessRunnerException(executablePath, args, -1, ex);
    }

    if (mode == IProcessRunner.Mode.RethrowAndIfNonZeroExitCode && process.ExitCode != 0)
    {
        throw new ProcessRunnerException(executablePath, args, process.ExitCode);
    }

    return new ProcessRunnerResult(process.ExitCode, stdErr);
}

就好像当我的进程等待获取标准输出时,子进程溢出了标准错误缓冲区并等待我的进程清除它,因此我们双方都陷入了死锁。

我的问题是如何使用 WinDBG 和使用

procdump -ma
创建的完整内存转储来解决此类死锁?

我在网上搜索时遇到的两个最常见的命令对我来说毫无用处:

  1. !sos.syncblk
    没有显示任何有用的东西
0:010> !syncblk
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
-----------------------------
Total           25
CCW             0
RCW             0
ComClassFactory 0
Free            1
  1. sosex 不适用于 .NET Core,因此没有
    !dlk
    命令

我跑了

!clrstack -all
并得到了这个:

0:010> !clrstack -all
OS Thread Id: 0x11978
        Child SP               IP Call Site
00000040CDD7E3B8 00007ffb771b0d24 [HelperMethodFrame_1OBJ: 00000040cdd7e3b8] System.Threading.Monitor.ObjWait(Int32, System.Object)
00000040CDD7E4E0 00007ffadca6d83e System.Threading.Monitor.Wait(System.Object, Int32) [/_/src/coreclr/System.Private.CoreLib/src/System/Threading/Monitor.CoreCLR.cs @ 156]
00000040CDD7E510 00007ffadca788ee System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ManualResetEventSlim.cs @ 561]
00000040CDD7E5B0 00007ffadca91a09 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3072]
00000040CDD7E630 00007ffadca917f6 System.Threading.Tasks.Task.InternalWaitCore(Int32, System.Threading.CancellationToken) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/Task.cs @ 3007]
00000040CDD7E6B0 00007ffadcae3815 System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task, System.Threading.Tasks.ConfigureAwaitOptions) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs @ 104]
00000040CDD7E6F0 00007ffadcbec570 System.Runtime.CompilerServices.TaskAwaiter`1[[System.Int32, System.Private.CoreLib]].GetResult() [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs @ 335]
00000040CDD7E720 00007ffa7e62718a Aida.Cli.Program.(System.String[])
OS Thread Id: 0x8388
        Child SP               IP Call Site
00000040CE7FF670 00007ffb771b0d24 [DebuggerU2MCatchHandlerFrame: 00000040ce7ff670] 
OS Thread Id: 0x6064
        Child SP               IP Call Site
00000040CEC7F2E8 00007ffb771b0d24 [HelperMethodFrame: 00000040cec7f2e8] System.Threading.WaitHandle.WaitOneCore(IntPtr, Int32)
00000040CEC7F3F0 00007ffa7fe835f1 System.Threading.WaitHandle.WaitOneNoCheck(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/WaitHandle.cs @ 131]
00000040CEC7F450 00007ffadcb7e8ee System.Diagnostics.Tracing.CounterGroup.PollForValues() [/_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Tracing/CounterGroup.cs @ 351]
00000040CEC7F730 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cec7f730] 
OS Thread Id: 0xf174
        Child SP               IP Call Site
00000040CD94F228 00007ffb771b0d24 [HelperMethodFrame: 00000040cd94f228] System.Threading.WaitHandle.WaitOneCore(IntPtr, Int32)
00000040CD94F330 00007ffa7fe835f1 System.Threading.WaitHandle.WaitOneNoCheck(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/WaitHandle.cs @ 131]
00000040CD94F390 00007ffadca89486 System.Threading.PortableThreadPool+GateThread.GateThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.GateThread.cs @ 48]
00000040CD94F700 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cd94f700] 
OS Thread Id: 0x3758
        Child SP               IP Call Site
00000040CD98F960 00007ffb771b3164 [InlinedCallFrame: 00000040cd98f960] 
00000040CD98F960 00007ffadca8b2c0 [InlinedCallFrame: 00000040cd98f960] 
00000040CD98F920 00007ffadca8b2c0 System.Threading.PortableThreadPool+IOCompletionPoller.Poll() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.IO.Windows.cs @ 188]
00000040CD98FC60 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cd98fc60] 
OS Thread Id: 0x147f0
        Child SP               IP Call Site
00000040CD9CF4D0 00007ffb771b3164 [InlinedCallFrame: 00000040cd9cf4d0] 
00000040CD9CF4D0 00007ffadca8b2c0 [InlinedCallFrame: 00000040cd9cf4d0] 
00000040CD9CF490 00007ffadca8b2c0 System.Threading.PortableThreadPool+IOCompletionPoller.Poll() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.IO.Windows.cs @ 188]
00000040CD9CF7D0 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cd9cf7d0] 
OS Thread Id: 0x11f9c
        Child SP               IP Call Site
OS Thread Id: 0x8084
        Child SP               IP Call Site
00000040CF3FF368 00007ffb771b0d24 [HelperMethodFrame_1OBJ: 00000040cf3ff368] System.Threading.Monitor.ObjWait(Int32, System.Object)
00000040CF3FF490 00007ffa7fa0657e System.Threading.Monitor.Wait(System.Object, Int32) [/_/src/coreclr/System.Private.CoreLib/src/System/Threading/Monitor.CoreCLR.cs @ 156]
00000040CF3FF4C0 00007ffa7e67a7ef Microsoft.Extensions.Logging.Console.ConsoleLoggerProcessor.TryDequeue(Microsoft.Extensions.Logging.Console.LogMessageEntry ByRef) [/_/src/libraries/Microsoft.Extensions.Logging.Console/src/ConsoleLoggerProcessor.cs @ 165]
00000040CF3FF550 00007ffa7e67a70d Microsoft.Extensions.Logging.Console.ConsoleLoggerProcessor.ProcessLogQueue() [/_/src/libraries/Microsoft.Extensions.Logging.Console/src/ConsoleLoggerProcessor.cs @ 104]
00000040CF3FF820 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cf3ff820] 
OS Thread Id: 0x1920
        Child SP               IP Call Site
00000040CF57F1C8 00007ffb771b0d24 [HelperMethodFrame: 00000040cf57f1c8] System.Threading.WaitHandle.WaitMultipleIgnoringSyncContext(IntPtr*, Int32, Boolean, Int32)
00000040CF57F2F0 00007ffadca715c9 System.Threading.WaitHandle.WaitMultiple(System.ReadOnlySpan`1<System.Threading.WaitHandle>, Boolean, Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/WaitHandle.cs @ 269]
00000040CF57F3C0 00007ffadca713be System.Threading.WaitHandle.WaitMultiple(System.Threading.WaitHandle[], Boolean, Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/WaitHandle.cs @ 236]
00000040CF57F400 00007ffadc402e29 Microsoft.Build.BackEnd.Logging.LoggingService.g__LoggingEventProc|141_0() [/_/src/Build/BackEnd/Components/Logging/LoggingService.cs @ 1351]
00000040CF57F680 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cf57f680] 
OS Thread Id: 0x144f4
        Child SP               IP Call Site
00000040CF87F4E8 00007ffb771b0294 [InlinedCallFrame: 00000040cf87f4e8] Interop+Kernel32.g____PInvoke|199_0(IntPtr, Byte*, Int32, Int32*, System.Threading.NativeOverlapped*)
00000040CF87F4E8 00007ffa8011ca76 [InlinedCallFrame: 00000040cf87f4e8] Interop+Kernel32.g____PInvoke|199_0(IntPtr, Byte*, Int32, Int32*, System.Threading.NativeOverlapped*)
00000040CF87F4B0 00007ffa8011ca76 Interop+Kernel32.ReadFile(System.Runtime.InteropServices.SafeHandle, Byte*, Int32, Int32 ByRef, System.Threading.NativeOverlapped*) [/_/src/coreclr/System.Private.CoreLib/Microsoft.Interop.LibraryImportGenerator/Microsoft.Interop.LibraryImportGenerator/LibraryImports.g.cs @ 6838]
00000040CF87F590 00007ffa8011c8d0 System.IO.RandomAccess.ReadAtOffset(Microsoft.Win32.SafeHandles.SafeFileHandle, System.Span`1, Int64) [/_/src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs @ 52]
00000040CF87F640 00007ffa8011c6a3 System.IO.RandomAccess+c.b__19_0(System.ValueTuple`4<Microsoft.Win32.SafeHandles.SafeFileHandle,System.Memory`1,Int64,System.IO.Strategies.OSFileStreamStrategy>) [/_/src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Windows.cs @ 257]
00000040CF87F6B0 00007ffa8011ffd3 System.Threading.AsyncOverSyncWithIoCancellation+d__8`2[[System.ValueTuple`4[[System.__Canon, System.Private.CoreLib],[System.Memory`1[[System.Byte, System.Private.CoreLib]], System.Private.CoreLib],[System.Int64, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]], System.Private.CoreLib],[System.Int32, System.Private.CoreLib]].MoveNext() [/_/src/libraries/Common/src/System/Threading/AsyncOverSyncWithIoCancellation.cs @ 135]
00000040CF87F760 00007ffa7fa158b0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 179]
00000040CF87F7D0 00007ffa7fa190fa System.Runtime.CompilerServices.TaskAwaiter+c.b__12_0(System.Action, System.Threading.Tasks.Task) [/_/src/libraries/System.Private.CoreLib/src/System/Runtime/CompilerServices/TaskAwaiter.cs @ 273]
00000040CF87F920 00007ffa801217b5 System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.Execute() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/TaskContinuation.cs @ 647]
00000040CF87FA40 00007ffa7fa1c183 System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs @ 994]
00000040CF87FAC0 00007ffadca8c723 System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.NonBrowser.cs @ 102]
00000040CF87FDF0 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cf87fdf0] 
OS Thread Id: 0xa890
        Child SP               IP Call Site
00000040CFA3F558 00007ffb771b0d24 [HelperMethodFrame: 00000040cfa3f558] System.Threading.WaitHandle.WaitOneCore(IntPtr, Int32)
00000040CFA3F660 00007ffa7fe835f1 System.Threading.WaitHandle.WaitOneNoCheck(Int32) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/WaitHandle.cs @ 131]
00000040CFA3F6C0 00007ffadca8230f System.Threading.TimerQueue.TimerThread() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/TimerQueue.Portable.cs @ 87]
00000040CFA3F980 00007ffade16dd63 [DebuggerU2MCatchHandlerFrame: 00000040cfa3f980] 
OS Thread Id: 0x38cc
        Child SP               IP Call Site
OS Thread Id: 0x133f8
        Child SP               IP Call Site
OS Thread Id: 0x3ac4
        Child SP               IP Call Site

异步调用堆栈如下所示:

0:010> !dumpasync
STACK 1
0000022fc5458ee0 00007ffa7e946a40 ( ) System.Threading.Tasks.ValueTask<System.Int32>+ValueTaskSourceAsTask
  0000022fc5458f88 00007ffa7e947e78 (0) Microsoft.Extensions.HotReload.PayloadTypeReaderWriter+<ReadNextPayloadTypeAsync>d__0 @ 7ffa7e628ce0
    0000022fc545b200 00007ffa7e94d268 (0) StartupHook+<ReceiveDeltas>d__4 @ 7ffa7e6284e0
      0000022fc545b328 00007ffa7e971378 (1) StartupHook+<>c__DisplayClass1_0+<<Initialize>b__0>d @ 7ffa7e627420
        0000022fc54596e0 00007ffa7e9249b8 ( ) System.Threading.Tasks.UnwrapPromise<System.Threading.Tasks.VoidTaskResult>

STACK 2
<< Awaiting: 0000022fc856bb78 00007ffa7f6f6ce8 System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable<System.Int32>+ConfiguredValueTaskAwaiter >>
  0000022fc856baf0 00007ffa7f9290b0 (1) System.IO.Strategies.BufferedFileStreamStrategy+<ReadFromNonSeekableAsync>d__36 @ 7ffa8011f780
    0000022fc856bc10 00007ffa7f6fe9f0 (1) System.IO.StreamReader+<ReadBufferAsync>d__72 @ 7ffa8011aa80
      0000022fc856bdb0 00007ffa7f929590 (0) System.IO.StreamReader+<ReadLineAsyncInternal>d__63 @ 7ffa801201c0
        0000022fc86154f0 00007ffa7f9299e0 (0) Aida.TestGen.CSharp.Tools.ProcessRunner+<Run>d__0 @ 7ffa7f6dbfc0
          0000022fc8615638 00007ffa7fbc6ca8 (0) Aida.TestGen.CSharp.Tools.TestCodeThruDotNet+<Test>d__4 @ 7ffa7fa13b60
            0000022fc8606ef0 00007ffa80209578 (2) Aida.Cli.UnitTestEngine+<BuildAndTest>d__33 @ 7ffa8010b910
              0000022fc85e3da0 00007ffa800e28b0 (2) Aida.Cli.UnitTestEngine+<RunTest>d__32 @ 7ffa800f09f0
                0000022fc85b5980 00007ffa800b8780 (1) Aida.Cli.UnitTestEngine+<Generate>d__18 @ 7ffa7fe92980
                  0000022fc5e47940 00007ffa7f7215b8 (4) Aida.Cli.UnitTestEngine+<<Run>g__InternalRun|14_0>d @ 7ffa7ee956a0
                    0000022fc5e47a38 00007ffa7f721978 (0) Aida.Cli.Program+<>c__DisplayClass2_0+<<Main>b__9>d @ 7ffa7e663ef0
                      0000022fc5e47b48 00007ffa7f721d00 (1) System.CommandLine.Invocation.AnonymousCommandHandler+<InvokeAsync>d__6 @ 7ffa7e663a00
                        0000022fc5e47c68 00007ffa7f722100 (0) System.CommandLine.Invocation.InvocationPipeline+<>c__DisplayClass4_0+<<BuildInvocationChain>b__0>d @ 7ffa7e6635d0
                          0000022fc5e47d88 00007ffa7f7273c8 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c__DisplayClass17_0+<<UseParseErrorReporting>b__0>d @ 7ffa7e662d80
                            0000022fc5e47ec0 00007ffa7f729488 (0) Aida.Cli.Program+<>c__DisplayClass2_0+<<Main>b__8>d @ 7ffa7e6476c0
                              0000022fc5e47fb8 00007ffa7f729848 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c__DisplayClass12_0+<<UseHelp>b__0>d @ 7ffa7e646eb0
                                0000022fc5e4aee0 00007ffa7f72c328 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c__DisplayClass22_0+<<UseVersionOption>b__0>d @ 7ffa7e6468f0
                                  0000022fc5e4b018 00007ffa7f72e150 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c__DisplayClass19_0+<<UseTypoCorrections>b__0>d @ 7ffa7e646240
                                    0000022fc5e4b138 00007ffa7f754bf8 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c+<<UseSuggestDirective>b__18_0>d @ 7ffa7e645da0
                                      0000022fc5e4b6a8 00007ffa7f7567a8 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c__DisplayClass16_0+<<UseParseDirective>b__0>d @ 7ffa7e645940
                                        0000022fc5e4b7c8 00007ffa7f757440 (1) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c+<<RegisterWithDotnetSuggest>b__5_0>d @ 7ffa7e644de0
                                          0000022fc5e4b8e0 00007ffa7f758448 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c__DisplayClass8_0+<<UseExceptionHandler>b__0>d @ 7ffa7e6445d0
                                            0000022fc5e4ba00 00007ffa7f7594f0 (0) System.CommandLine.Builder.CommandLineBuilderExtensions+<>c+<<CancelOnProcessTermination>b__1_0>d @ 7ffa7e643f40
                                              0000022fc5e4bb38 00007ffa7f759db0 (0) System.CommandLine.Invocation.InvocationPipeline+<<InvokeAsync>g__FullInvocationChainAsync|2_0>d @ 7ffa7e643580
                                                0000022fc5e4bc48 00007ffa7f75a5c8 (0) System.CommandLine.Parsing.ParseResultExtensions+<InvokeAsync>d__0 @ 7ffa7e642ef0
                                                  0000022fc5e4bd60 00007ffa7f75cba8 (0) System.CommandLine.Parsing.ParserExtensions+<InvokeAsync>d__3 @ 7ffa7e631a70
                                                    0000022fc5e55550 00007ffa7f75e800 (0) Aida.Cli.Program+<Main>d__2 @ 7ffa7e62ad40
                                                      0000022fc5e55648 00007ffa7e8cb880 () System.Threading.Tasks.Task+SetOnInvokeMres

所以我想我知道为什么会出现死锁,但是,我想学习如何从内存转储中推断出它,但是由于

!sos.syncblk
无用且
!sosex.dlk
不可用,我不知道如何继续。

c# multithreading .net-core deadlock windbg
1个回答
0
投票

您不需要调试它,这是一个众所周知的操作,主要文档字面上为您编写了代码:

  var p = new Process();  
  p.StartInfo.UseShellExecute = false;  
  p.StartInfo.RedirectStandardOutput = true;  
  string eOut = null;
  p.StartInfo.RedirectStandardError = true;
  p.ErrorDataReceived += new DataReceivedEventHandler((sender, e) => 
                             { eOut += e.Data; });
  p.StartInfo.FileName = "Write500Lines.exe";  
  p.Start();  

  // To avoid deadlocks, use an asynchronous read operation on at least one of the streams.  
  p.BeginErrorReadLine();
  string output = p.StandardOutput.ReadToEnd();  
  p.WaitForExit();

简而言之,您不能在两个流上使用同步读取,至少一个(最好是两个)都需要是异步读取。

最新问题
© www.soinside.com 2019 - 2025. All rights reserved.