我有用Haskell编写的微服务。它使用Scotty。 LTS是13.20。操作系统:Linux 3.10.0-957.el7.x86_64,可在Kubernets下运行。这项服务可以正常工作约0.5年,但现在我遇到了几次神秘的冻结。我认为这不是回归的结果,因为未修改代码,但增加了服务负载。
症状是:
strace
报告在futex(...)调用上冻结:futex(0x349c9c4, FUTEX_WAIT_PRIVATE, 83, NULL
F S UID PID SPID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD
4 S 0 1 1 0 0 80 0 - 274 do_sig ? 00:00:00 myinit
4 S 0 6 6 1 0 80 0 - 2922 do_wai ? 00:00:00 rc.init
4 S 0 32 32 0 0 80 0 - 2955 do_wai pts/0 00:00:00 bash
4 S 0 4464 4464 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service
1 S 0 4464 4465 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4466 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4467 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4468 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4469 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4470 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4471 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4472 6 0 80 0 - 268854048 timerf ? 00:00:00 ghc_ticker
1 S 0 4464 4473 6 0 80 0 - 268854048 ep_pol ? 00:00:00 my-service:w
1 S 0 4464 4474 6 0 80 0 - 268854048 poll_s ? 00:00:00 my-service:w
1 S 0 4464 4475 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4476 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4477 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4478 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4479 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4480 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4481 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4482 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4483 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4484 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service
1 S 0 4464 4485 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service
1 S 0 4464 4486 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
1 S 0 4464 4487 6 0 80 0 - 268854048 futex_ ? 00:00:00 my-service:w
0 R 0 4511 4511 32 0 80 0 - 12405 - pts/0 00:00:00 ps
使用gdb
附加到PID之后:
[New LWP 4487]
[New LWP 4486]
[New LWP 4485]
[New LWP 4484]
[New LWP 4483]
[New LWP 4482]
[New LWP 4481]
[New LWP 4480]
[New LWP 4479]
[New LWP 4478]
[New LWP 4477]
[New LWP 4476]
[New LWP 4475]
[New LWP 4474]
[New LWP 4473]
[New LWP 4472]
[New LWP 4471]
[New LWP 4470]
[New LWP 4469]
[New LWP 4468]
[New LWP 4467]
[New LWP 4466]
[New LWP 4465]
....
(gdb) bt full
#0 0x00007fc03ec23965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00000000015185e9 in waitCondition (pCond=pCond@entry=0x2f029c0, pMut=pMut@entry=0x2f029f0) at rts/posix/OSThreads.c:117
No locals.
#2 0x000000000150713b in waitForWorkerCapability (task=<optimized out>) at rts/Capability.c:651
cap = <optimized out>
#3 yieldCapability (pCap=pCap@entry=0x7fffc6ae0a78, task=task@entry=0x2f029b0, gcAllowed=gcAllowed@entry=true) at rts/Capability.c:888
cap = <optimized out>
#4 0x0000000001504d85 in scheduleYield (task=0x2f029b0, pcap=0x7fffc6ae0a70) at rts/Schedule.c:672
cap = 0x2e7cff0
didGcLast = <optimized out>
#5 schedule (initialCapability=initialCapability@entry=0x2edf1b0, task=task@entry=0x2f029b0) at rts/Schedule.c:292
t = <optimized out>
cap = 0x2e7cff0
ret = <optimized out>
prev_what_next = <optimized out>
ready_to_gc = <optimized out>
#6 0x0000000001505bee in scheduleWaitThread (tso=0x4200823388, ret=ret@entry=0x0, pcap=pcap@entry=0x7fffc6ae0b08) at rts/Schedule.c:2533
task = 0x2f029b0
cap = 0x2edf1b0
#7 0x0000000001500584 in rts_evalLazyIO (cap=cap@entry=0x7fffc6ae0b08, p=p@entry=0x15a00d0, ret=ret@entry=0x0) at rts/RtsAPI.c:530
tso = <optimized out>
#8 0x00000000015102be in hs_main (argc=2, argv=0x7fffc6ae0cf8, main_closure=0x15a00d0, rts_config=...) at rts/RtsMain.c:72
cap = 0x2edf1b0
exit_status = <optimized out>
status = <optimized out>
#9 0x00000000004311b0 in main ()
No symbol table info available.
因此,我的问题是:如何修复,调查,尝试什么,进行检查会更好?我本来打算切换到新的LTS,但是我不确定这是问题的原因(我在网络论坛中发现旧LTS / GHC版本存在类似问题)...恕我直言,它看起来像RTS中的错误。
通常,当无工作要做时,工人会等待waitForWorkerCapability
中的条件。例如。所有haskell线程都在IO上被阻止,因此我们没有任何可运行的东西。该条件在giveCapabilityToTask中用信号表示,在同一文件的几个地方都被调用。
如果您确定应该做的工作,那么您可能在RTS中发现了一个错误。尝试提供一个重现此问题的最小示例。 (我知道,通常根本不可能。)
但是您的代码或依赖项中可能存在错误。您可以尝试检查gdb中的capabilities。 (它们上有一个global array)。您可能对running_task
,run_queue_hd
,suspended_ccalls
,spare_workers
,returning_tasks_hd
字段感兴趣。我认为您应该没有正在运行的任务,并且所有功能都应该排空队列,并且所有工作程序都应在某些功能的spare_workers
列表中。
((这只是我对问题的理解。我不是GHC RTS的专家,并且可能会胡说八道。)