应用程序冻结futex()调用

问题描述 投票:2回答:1

我有用Haskell编写的微服务。它使用Scotty。 LTS是13.20。操作系统:Linux 3.10.0-957.el7.x86_64,可在Kubernets下运行。这项服务可以正常工作约0.5年,但现在我遇到了几次神秘的冻结。我认为这不是回归的结果,因为未修改代码,但增加了服务负载。

症状是:

  • CPU使用率-确定
  • 内存消耗-确定
  • [strace报告在futex(...)调用上冻结:futex(0x349c9c4, FUTEX_WAIT_PRIVATE, 83, NULL
  • 很多看起来像这样的线程:
F S   UID   PID  SPID  PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
4 S     0     1     1     0  0  80   0 -   274 do_sig ?        00:00:00 myinit
4 S     0     6     6     1  0  80   0 -  2922 do_wai ?        00:00:00 rc.init
4 S     0    32    32     0  0  80   0 -  2955 do_wai pts/0    00:00:00 bash
4 S     0  4464  4464     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service
1 S     0  4464  4465     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4466     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4467     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4468     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4469     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4470     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4471     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4472     6  0  80   0 - 268854048 timerf ?    00:00:00 ghc_ticker
1 S     0  4464  4473     6  0  80   0 - 268854048 ep_pol ?    00:00:00 my-service:w
1 S     0  4464  4474     6  0  80   0 - 268854048 poll_s ?    00:00:00 my-service:w
1 S     0  4464  4475     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4476     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4477     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4478     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4479     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4480     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4481     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4482     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4483     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4484     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service
1 S     0  4464  4485     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service
1 S     0  4464  4486     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
1 S     0  4464  4487     6  0  80   0 - 268854048 futex_ ?    00:00:00 my-service:w
0 R     0  4511  4511    32  0  80   0 - 12405 -      pts/0    00:00:00 ps

使用gdb附加到PID之后:

[New LWP 4487]
[New LWP 4486]
[New LWP 4485]
[New LWP 4484]
[New LWP 4483]
[New LWP 4482]
[New LWP 4481]
[New LWP 4480]
[New LWP 4479]
[New LWP 4478]
[New LWP 4477]
[New LWP 4476]
[New LWP 4475]
[New LWP 4474]
[New LWP 4473]
[New LWP 4472]
[New LWP 4471]
[New LWP 4470]
[New LWP 4469]
[New LWP 4468]
[New LWP 4467]
[New LWP 4466]
[New LWP 4465]
....
(gdb) bt full
#0  0x00007fc03ec23965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x00000000015185e9 in waitCondition (pCond=pCond@entry=0x2f029c0, pMut=pMut@entry=0x2f029f0) at rts/posix/OSThreads.c:117
No locals.
#2  0x000000000150713b in waitForWorkerCapability (task=<optimized out>) at rts/Capability.c:651
        cap = <optimized out>
#3  yieldCapability (pCap=pCap@entry=0x7fffc6ae0a78, task=task@entry=0x2f029b0, gcAllowed=gcAllowed@entry=true) at rts/Capability.c:888
        cap = <optimized out>
#4  0x0000000001504d85 in scheduleYield (task=0x2f029b0, pcap=0x7fffc6ae0a70) at rts/Schedule.c:672
        cap = 0x2e7cff0
        didGcLast = <optimized out>
#5  schedule (initialCapability=initialCapability@entry=0x2edf1b0, task=task@entry=0x2f029b0) at rts/Schedule.c:292
        t = <optimized out>
        cap = 0x2e7cff0
        ret = <optimized out>
        prev_what_next = <optimized out>
        ready_to_gc = <optimized out>
#6  0x0000000001505bee in scheduleWaitThread (tso=0x4200823388, ret=ret@entry=0x0, pcap=pcap@entry=0x7fffc6ae0b08) at rts/Schedule.c:2533
        task = 0x2f029b0
        cap = 0x2edf1b0
#7  0x0000000001500584 in rts_evalLazyIO (cap=cap@entry=0x7fffc6ae0b08, p=p@entry=0x15a00d0, ret=ret@entry=0x0) at rts/RtsAPI.c:530
        tso = <optimized out>
#8  0x00000000015102be in hs_main (argc=2, argv=0x7fffc6ae0cf8, main_closure=0x15a00d0, rts_config=...) at rts/RtsMain.c:72
        cap = 0x2edf1b0
        exit_status = <optimized out>
        status = <optimized out>
#9  0x00000000004311b0 in main ()
No symbol table info available.

因此,我的问题是:如何修复,调查,尝试什么,进行检查会更好?我本来打算切换到新的LTS,但是我不确定这是问题的原因(我在网络论坛中发现旧LTS / GHC版本存在类似问题)...恕我直言,它看起来像RTS中的错误。

haskell freeze
1个回答
0
投票

通常,当无工作要做时,工人会等待waitForWorkerCapability中的条件。例如。所有haskell线程都在IO上被阻止,因此我们没有任何可运行的东西。该条件在giveCapabilityToTask中用信号表示,在同一文件的几个地方都被调用。

如果您确定应该做的工作,那么您可能在RTS中发现了一个错误。尝试提供一个重现此问题的最小示例。 (我知道,通常根本不可能。)

但是您的代码或依赖项中可能存在错误。您可以尝试检查gdb中的capabilities。 (它们上有一个global array)。您可能对running_taskrun_queue_hdsuspended_ccallsspare_workersreturning_tasks_hd字段感兴趣。我认为您应该没有正在运行的任务,并且所有功能都应该排空队列,并且所有工作程序都应在某些功能的spare_workers列表中。

((这只是我对问题的理解。我不是GHC RTS的专家,并且可能会胡说八道。)

© www.soinside.com 2019 - 2024. All rights reserved.