Record an analysis of a.Net property management background service stuck

One ： background

1. Tell a story

In recent months, I have been often asked by my friends , Why not update this series , ha-ha , It did stop for a long time , The main thing is to lay the foundation , analysis dump The ability to use is not in flexibility windbg, But have a deep understanding of the underlying knowledge , such as ： assembly ,C, C++,Win32 Api, Virtual memory ,Windows User mode and kernel mode , This is the book I read this year to share with you .

Some time ago, a friend on wechat said that his program got stuck , All threads are no longer working , It sounds pretty scary , The screenshot is as follows ：

Next, go straight to WinDbg Analyze .

Two ：Windbg analysis

1. Stuck analysis

Since all threads of the program are no longer working , The probability should be this time GC Triggered , I once saw a dump in GC Creating  background thread  when , because  dllmain  The deadlock caused by  background thread  Unable to generate the raised deadlock problem .

So here's the idea , Next use  ~* k  Look at all the thread stacks , Is there a GarbageCollectGeneration  function , Because it is GC Trigger entry point , Sure enough ,46 Thread number triggered GC operation .

  46  Id: 396c.3198 Suspend: 0 Teb: 00007ff6`22646000 Unfrozen
 # Child-SP          RetAddr               Call Site
00 00000028`d420bc18 00007ffa`8b6b8b61     ntdll!NtWaitForSingleObject+0xa
01 00000028`d420bc20 00007ffa`8b6b7124     ntdll!RtlpWaitOnCriticalSection+0xe1
02 00000028`d420bcf0 00000001`8000a725     ntdll!RtlpEnterCriticalSectionContended+0xa4
03 00000028`d420bd30 00000001`80011773     WiseVectorHelperOne_X64+0xa725
04 00000028`d420bd90 00007ffa`888faf8f     WiseVectorHelperOne_X64+0x11773
05 00000028`d420d2d0 00007ffa`79db4d45     KERNELBASE!ResumeThread+0xf
06 00000028`d420d300 00007ffa`79db8bee     coreclr!Thread::ResumeThread+0x29 [d:\a\_work\1\s\src\vm\threadsuspend.cpp @ 466] 
07 00000028`d420d350 00007ffa`79e13905     coreclr!ThreadSuspend::SuspendRuntime+0x17a [d:\a\_work\1\s\src\vm\threadsuspend.cpp @ 4046] 
08 00000028`d420d420 00007ffa`79db61cf     coreclr!ThreadSuspend::SuspendEE+0x16d [d:\a\_work\1\s\src\vm\threadsuspend.cpp @ 6517] 
09 (Inline Function) --------`--------     coreclr!GCToEEInterface::SuspendEE+0x21 [d:\a\_work\1\s\src\vm\gcenv.ee.cpp @ 25] 
0a 00000028`d420d5c0 00007ffa`79e325be     coreclr!WKS::GCHeap::GarbageCollectGeneration+0xff [d:\a\_work\1\s\src\gc\gc.cpp @ 36545] 
0b (Inline Function) --------`--------     coreclr!WKS::gc_heap::trigger_gc_for_alloc+0x12 [d:\a\_work\1\s\src\gc\gc.cpp @ 13832] 
0c 00000028`d420d610 00007ffa`79e35118     coreclr!WKS::gc_heap::try_allocate_more_space+0x24e [d:\a\_work\1\s\src\gc\gc.cpp @ 13934] 
0d (Inline Function) --------`--------     coreclr!WKS::gc_heap::allocate_more_space+0x11 [d:\a\_work\1\s\src\gc\gc.cpp @ 14369] 
0e (Inline Function) --------`--------     coreclr!WKS::gc_heap::allocate+0x58 [d:\a\_work\1\s\src\gc\gc.cpp @ 14400] 
0f 00000028`d420d690 00007ffa`79dcda8e     coreclr!WKS::GCHeap::Alloc+0x88 [d:\a\_work\1\s\src\gc\gc.cpp @ 35827] 

From the thread stack , The process is probably ：C# Assign an object , Triggered GC, Then all managed threads are paused , Then one of the threads is restored , It should be that this thread does not stay in gc On the safety point , Restart is to make it pause at a safe point , stay coreclr It can also be seen from the source code .

And then we go into  WiseVectorHelperOne_X64  Class library , A strange one dll, Finally entered   A critical region CriticalSection , So-called   A critical region   It's a win32 function , Usage and our lock almost , Finally, I will stop at A critical region , In fact, the phenomenon is very clear here , All managed threads are suspended , It is also in line with what my friend said that the program is stuck , The next step is to analyze why the program does not exit   A critical region  ？

2. Why can't I return CriticalSection

To find the answer , It can be used  !locks  To observe the current situation   A critical region   The thread of , Output is as follows ：

0:000> !locks

CritSec +63218af0 at 0000002863218af0
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       3198
EntryCount         0
ContentionCount    16d
*** Locked

CritSec WiseVectorHelperOne_X64+6a9a8 at 000000018006a9a8
WaiterWoken        No
LockCount          1
RecursionCount     1
OwningThread       3090
EntryCount         0
ContentionCount    1
*** Locked

Scanned 64 critical sections

Based on experience , The first reaction should be   Critical zone deadlock   了 , Experience belongs to experience , Next, let's take a look at  3198  and  3090  What are they waiting for ？

3. Is it really a critical deadlock

First, use the command to switch to  3198  Threads , See what resources it is waiting for ？

0:038>  ~~[3198]s
ntdll!NtWaitForSingleObject+0xa:
00007ffa`8b710c8a c3              ret
0:046> kb
 # RetAddr               : Args to Child                                                           : Call Site
00 00007ffa`8b6b8b61     : 00000001`8006a9a8 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForSingleObject+0xa
01 00007ffa`8b6b7124     : 00000000`00000000 00000000`00000000 00000001`8006a9a8 00000000`00000000 : ntdll!RtlpWaitOnCriticalSection+0xe1
02 00000001`8000a725     : 00000028`00668230 00000000`00000000 00000028`7fc9d9b0 00000028`00668230 : ntdll!RtlpEnterCriticalSectionContended+0xa4
03 00000001`80011773     : 00000001`00000aa8 00000000`00000000 00000000`00000000 00000000`00000000 : WiseVectorHelperOne_X64+0xa725
04 00007ffa`888faf8f     : 00000000`00000aa8 00000028`d420d308 00000000`00000000 00000000`00000000 : WiseVectorHelperOne_X64+0x11773
05 00007ffa`79db4d45     : 00000000`00000000 00000000`00000000 00000028`04dec6e0 00000001`8000cc3a : KERNELBASE!ResumeThread+0xf
06 00007ffa`79db8bee     : 00000028`00668230 00000000`00000040 00000000`00000001 00000000`00000000 : coreclr!Thread::ResumeThread+0x29 [d:\a\_work\1\s\src\vm\threadsuspend.cpp @ 466] 
07 00007ffa`79e13905     : 00000000`00000003 00000000`00000001 00000000`00000001 00000000`00000000 : coreclr!ThreadSuspend::SuspendRuntime+0x17a [d:\a\_work\1\s\src\vm\threadsuspend.cpp @ 4046] 
08 00007ffa`79db61cf     : 00000000`00001e73 00000000`00001e01 00000028`7f9f6698 00000000`00000000 : coreclr!ThreadSuspend::SuspendEE+0x16d [d:\a\_work\1\s\src\vm\threadsuspend.cpp @ 6517] 
09 (Inline Function)     : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!GCToEEInterface::SuspendEE+0x21 [d:\a\_work\1\s\src\vm\gcenv.ee.cpp @ 25] 
0a 00007ffa`79e325be     : a2098c12`cdff0000 00007ffa`79e35118 00007ffa`7a28c668 00000000`00000000 : coreclr!WKS::GCHeap::GarbageCollectGeneration+0xff [d:\a\_work\1\s\src\gc\gc.cpp @ 36545] 
0b (Inline Function)     : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!WKS::gc_heap::trigger_gc_for_alloc+0x12 [d:\a\_work\1\s\src\gc\gc.cpp @ 13832] 
0c 00007ffa`79e35118     : 00000028`7fc9da08 00000028`12bba6d8 00000000`00000002 00007ffa`79dbfc9f : coreclr!WKS::gc_heap::try_allocate_more_space+0x24e [d:\a\_work\1\s\src\gc\gc.cpp @ 13934] 
0d (Inline Function)     : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!WKS::gc_heap::allocate_more_space+0x11 [d:\a\_work\1\s\src\gc\gc.cpp @ 14369] 
0e (Inline Function)     : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!WKS::gc_heap::allocate+0x58 [d:\a\_work\1\s\src\gc\gc.cpp @ 14400] 
0f 00007ffa`79dcda8e     : 00000000`00000000 00000028`d420daa0 00007ffa`1a908888 00000028`7fc9da08 : coreclr!WKS::GCHeap::Alloc+0x88 [d:\a\_work\1\s\src\gc\gc.cpp @ 35827] 
10 (Inline Function)     : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!Alloc+0x18b [d:\a\_work\1\s\src\vm\gchelpers.cpp @ 240] 
11 (Inline Function)     : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!AllocateObject+0x22d [d:\a\_work\1\s\src\vm\gchelpers.cpp @ 1209] 
12 00007ffa`1b3337e2     : 00007ffa`1a908888 00000028`84d75cc0 00000028`12bb9ce0 00000028`64df1360 : coreclr!JIT_New+0x31e [d:\a\_work\1\s\src\vm\jithelpers.cpp @ 2724] 
....

From the output information : NtWaitForSingleObject  Is waiting for  000000018006a9a8  Critical section resources , And this happens to be  !locks  Of  3090  Resources held by threads , The screenshot is as follows ：

Next, let's look at  3090  What the thread is doing .

0:038> ~~[3090]s
WiseVectorHelperOne_X64+0xcc3a:
00000001`8000cc3a 4889442408      mov     qword ptr [rsp+8],rax ss:00000028`04dec6e8=0000000000000000
0:038> k
 # Child-SP          RetAddr               Call Site
00 00000028`04dec6e0 00000001`8000f1cb     WiseVectorHelperOne_X64+0xcc3a
01 00000028`04dec710 00000001`8000a751     WiseVectorHelperOne_X64+0xf1cb
02 00000028`04dec7a0 00000001`80011773     WiseVectorHelperOne_X64+0xa751
03 00000028`04dec800 00007ffa`888faf8f     WiseVectorHelperOne_X64+0x11773
04 00000028`04dedd40 00007ffa`79e19796     KERNELBASE!ResumeThread+0xf
05 (Inline Function) --------`--------     coreclr!Thread::StartThread+0x15 [d:\a\_work\1\s\src\vm\threads.cpp @ 528] 
06 00000028`04dedd70 00007ffa`79eaacea     coreclr!ThreadNative::StartInner+0x35a [d:\a\_work\1\s\src\vm\comsynchronizable.cpp @ 501] 
07 00000028`04dee010 00007ffa`1b3afc02     coreclr!ThreadNative::Start+0x8a [d:\a\_work\1\s\src\vm\comsynchronizable.cpp @ 387] 
08 00000028`04dee160 00007ffa`1b3cb018     System_Private_CoreLib!System.Threading.Tasks.Task.ScheduleAndStart+0x102
09 00000028`04dee1b0 00007ffa`1b40005a     System_Private_CoreLib!System.Threading.Tasks.Task.InternalStartNew+0x78
0a 00000028`04dee230 00007ffa`1b41f181     System_Private_CoreLib!System.Threading.Tasks.TaskFactory.StartNew+0x5a
...

From the thread stack information , The hosting layer performs a  Task.Start  operation , And then through Win32 Api It generates a OS Threads , In preparation for scheduling  OS Threads   When , I met  WiseVectorHelperOne_X64 , Finally, I will wait here indefinitely ,tmd That's strange , You can see this function in both threads , What on earth is it ？

4. Research WiseVectorHelperOne_X64

This is strange dll, It seems that the comer is not good , On baidu Check it out .

I went to , It was originally called   Intelligent anti-virus software   Hijacked ... What is the specific reason for being hijacked , I don't want to study , Then I took the result and communicated with my friends , Try to stop it and see .

3、 ... and ： summary

Combine the characteristics of the two thread stacks , All the discoveries are Win32 Api Doing it  Thread::ResumeThread  Was hijacked by anti-virus software , Generally speaking clr Build internally OS Thread time , Will first  Suspended, And then again  Resume, Reference source code ：

BOOL Thread::CreateNewOSThread(SIZE_T sizeToCommitOrReserve, LPTHREAD_START_ROUTINE start, void* args)
{
    HANDLE h = NULL;
    DWORD dwCreationFlags = CREATE_SUSPENDED;

    dwCreationFlags |= STACK_SIZE_PARAM_IS_A_RESERVATION;

    h = ::CreateThread(NULL     /*=SECURITY_ATTRIBUTES*/,

                             sizeToCommitOrReserve,
                             start,
                             args,
                             dwCreationFlags,
                             &ourId);
}

The reason for the hijacking , I can only ask the manufacturer , All we can do is stop it , At last, my friend was too kind , Sent a big red envelope