Source code analysis of Tencent libco collaboration open source library (III) -- Exploring collaboration switching process assembly register saving and efficient collaboration environment

---

Full series summary blog links

---

tencent Libco Collaborative open source library Source code analysis Full series summary blog

---

Foreword

---

For CO process switching This part I have done it quite thoroughly
For the later call blocking function Assist the process to achieve active scheduling and give up cpu This part I'll think it over later Save it for your next blog post This article mainly writes two functions co_resume and co_yield The principle of

---

tencent Libco Collaborative open source library Source code analysis （ 3、 ... and ）---- Explore the coordination process switching process Assembly register save Save the collaborative process environment efficiently

---

1、 Explore co_resume The principle of startup coprocessor function

---

The number of lines of code is very short But it needs to be tapped

The first line gets the... Of the current thread env Environmental Science This is the same for all coroutines in each thread It was introduced before Not much
The second line Get the currently running stack Process management structure pointer For the first call co_resume Of What we get at this time is main Pointer to function
The third line Is the first call to all coroutines co_resumecStart Is zero If the previous collaboration has been started Then call co_resume Words This statement segment will no longer enter
We can take a closer look at what is going on inside this function To look down

void co_resume( stCoRoutine_t *co )
{
    
	stCoRoutineEnv_t *env = co->env;
	stCoRoutine_t *lpCurrRoutine = env->pCallStack[ env->iCallStackSize - 1 ];
	if( !co->cStart )
	{
    
		coctx_make( &co->ctx,(coctx_pfn_t)CoRoutineFunc,co,0 );
		co->cStart = 1;
	}
	env->pCallStack[ env->iCallStackSize++ ] = co;
	co_swap( lpCurrRoutine, co );
}

---

1、 hitting coctx_make function Tapping and unthreading

---

coctx_make We can look at the definition of this function

Here I choose 32 Bit operating system The version of the function called because 32 I am familiar with And later for 32 position The flow of function calls under the operating system They are also familiar with each other and 64 position It's relatively different But the principle is the same Just the calling process is different So, by default 32 position The function version under

We analyze it line by line In this function, we also give Definitions of related structures and functions Easy to read
first line Set up esp Location If you are not familiar with the underlying knowledge or do not know the assembly language For registers in function calls If you don't understand the function I suggest we make up the foundation again Let me explain here why we need to subtract sizeof(coctx_param_t) Here we are just making room for the following function parameters The specific function is given above CoRoutineFunc This parameter makes room for And assign the parameter here

The second line Align stack space with memory location I thought about It should be the end of clearing 4 Address of bit
Lines three through seven Doing two things First of all For the following assembly function coctx_swap Of ret function Arrange the returned function address The second thing Just for CoRoutineFunc parameter assignment

For function parameter passing I believe this diagram is the clearest parameter transfer flowchart I have ever seen Of course This is limited to 32 position 64 Bit is a little different But it doesn't make much difference I'm not sure So I won't say more 64 Yes.
Parameter passing is the later parameter The more advanced Push To stack Then finally push Function address
Here we use esp Addressing In fact, it is generally not in the form of esp Addressing But rather ebp Addressing But it's almost the same When generating assembly code There will be a code movl esp, ebp take esp The value of is assigned to ebp Fix the header of the current stack frame Then call parameters.

Above description It is also the assigned address to the following And the explanation of function address assignment If there is a certain compilation basis I believe think It's still easy to figure out why the assignment is like this

---

As for this part Let's talk about the last line Will esp Subtract two more from your address void* Well Here I also thought for a while It can be said that I have come up with a reason
As there is no coctx_swap Assembler function code It will be given later The normal situation should be after making room for parameters Right where you are Assignment function pointer ret that will do Why subtract two more void* Just assign values

In my submission This problem Maybe only I know more about assembly And it is really possible that only those who have a deep understanding of the operating system can understand
Is that we are c Language After calling assembly c Language Compile into assembly It should be first push After the parameter Again push Function return address But we are in the back coctx_swap in We have changed esp The address of But the operating system doesn't care As the assembly function is called The last sentence ret The end of Is it really over Certainly not. There will be another assembly line generated later It is to pass parameters before recycling And the stack space automatically allocated downward Now it needs to be recycled

But the operating system is not clear. We switched esp But there's no way He must perform a call to reclaim parameters Statements that occupy stack space So here we have subtracted two more down void* Back recycling It's actually recycled Our new one is malloc Own stack space

Maybe my thinking is wrong But I've been thinking for a long time. I can only think like this And I do feel like this

typedef void* (*coctx_pfn_t)( void* s, void* s2 );
struct coctx_param_t
{
    
	const void *s1;
	const void *s2;
};
struct coctx_t
{
    
#if defined(__i386__)
	void *regs[ 8 ];
#else
	void *regs[ 14 ];
#endif
	size_t ss_size;
	char *ss_sp;
};

static int CoRoutineFunc( stCoRoutine_t *co,void * )
{
    
	if( co->pfn )
	{
    
		co->pfn( co->arg );
	}
	co->cEnd = 1;

	stCoRoutineEnv_t *env = co->env;

	co_yield_env( env );

	return 0;
}

int coctx_make(coctx_t* ctx, coctx_pfn_t pfn, const void* s, const void* s1) {
    
  // make room for coctx_param
  char* sp = ctx->ss_sp + ctx->ss_size - sizeof(coctx_param_t);
  sp = (char*)((unsigned long)sp & -16L);

  coctx_param_t* param = (coctx_param_t*)sp;
  void** ret_addr = (void**)(sp - sizeof(void*) * 2);
  *ret_addr = (void*)pfn;
  param->s1 = s;
  param->s2 = s1;

  memset(ctx->regs, 0, sizeof(ctx->regs));

  ctx->regs[kESP] = (char*)(sp) - sizeof(void*) * 2;
  return 0;
}

---

2、 follow up a victory with hot pursuit to co_swap Make a clean sweep of the process switching

---

It's over coctx_make Here we come co_resume The last line of code co_swap The core of this function is to call the assembly function coctx_swap Let's see

In fact, for the code here If the shared stack is not used The only thing to focus on is a line of code Other code is related to the shared stack Let's skip
Assembly code is called here coctx_swap After this line of code runs The coordination process is switched When I come back next time It is necessary to wait co_yield Will return to the next line of code So let's see coctx_swap Well

void co_swap(stCoRoutine_t* curr, stCoRoutine_t* pending_co)
{
    
 	stCoRoutineEnv_t* env = co_get_curr_thread_env();

	//get curr stack sp
	char c;
	curr->stack_sp= &c;

	if (!pending_co->cIsShareStack)
	{
    
		env->pending_co = NULL;
		env->occupy_co = NULL;
	}
	else 
	{
    
		env->pending_co = pending_co;
		//get last occupy co on the same stack mem
		stCoRoutine_t* occupy_co = pending_co->stack_mem->occupy_co;
		//set pending co to occupy thest stack mem;
		pending_co->stack_mem->occupy_co = pending_co;

		env->occupy_co = occupy_co;
		if (occupy_co && occupy_co != pending_co)
		{
    
			save_stack_buffer(occupy_co);
		}
	}

	//swap context
	coctx_swap(&(curr->ctx),&(pending_co->ctx) );

	//stack buffer may be overwrite, so get again;
	stCoRoutineEnv_t* curr_env = co_get_curr_thread_env();
	stCoRoutine_t* update_occupy_co =  curr_env->occupy_co;
	stCoRoutine_t* update_pending_co = curr_env->pending_co;
	
	if (update_occupy_co && update_pending_co && update_occupy_co != update_pending_co)
	{
    
		//resume stack buffer
		if (update_pending_co->save_buffer && update_pending_co->save_size > 0)
		{
    
			memcpy(update_pending_co->stack_sp, update_pending_co->save_buffer, update_pending_co->save_size);
		}
	}
}

---

coctx_swap.S
If you haven't learned assembly Here's the suggestion You can go directly to the upper right corner
Because it is Look at some C/C++ The underlying code If you don't know about assembly It has nothing to do with background development So I'd better take a few days to supplement the foundation and have a look

It has been written in great detail About C Language Function call process
So for the above statement coctx_swap(&(curr->ctx),&(pending_co->ctx)) We can easily know at present The pointer position of the target function we are about to turn to Is in coctx_t in esp Where it is stored
And for parameters &(curr->ctx) It should be in esp + 4 The location of &(pending_co->ctx) Is in esp + 8 The location of

With this pre knowledge We can do a lot with the following assembly code

The first paragraph
The first line assembles Put our current cooperation process coctx_t Value Put in eax in
Lines 2 through 8 assemble Is to put our current 8 All registers are stored in our structure struct coctx_t in I gave it to you earlier struct coctx_t The definition of You can look for it

The second paragraph
The first line assembles Coordinate our goals coctx_t Value Put in eax in
Lines 2 through 8 assemble Put our structure struct coctx_t in Preset 8 A register （ In fact, only esp There is a value ） Stored in the current register
The last line Called ret Because of our esp It has been switched to the preset esp So at this time ret Then he jumped to The pre-set covariance function CoRoutineFunc

64 Bit assembly code is a little more complicated But the principle is similar Let's see 32 You can
Now let's go and have a look CoRoutineFunc What are you doing

.globl coctx_swap
#if !defined( __APPLE__ )
.type  coctx_swap, @function
#endif
coctx_swap:

#if defined(__i386__)
    movl 4(%esp), %eax
    movl %esp,  28(%eax)
    movl %ebp, 24(%eax)
    movl %esi, 20(%eax)
    movl %edi, 16(%eax)
    movl %edx, 12(%eax)
    movl %ecx, 8(%eax)
    movl %ebx, 4(%eax)


    movl 8(%esp), %eax
    movl 4(%eax), %ebx
    movl 8(%eax), %ecx
    movl 12(%eax), %edx
    movl 16(%eax), %edi
    movl 20(%eax), %esi
    movl 24(%eax), %ebp
    movl 28(%eax), %esp

	ret

---

3、 ordinary CoRoutineFunc end co_resume

---

Remember before for esp Teng's parameter space see ( stCoRoutine_t *co,void * ) And that's where it comes in Here we get the parameters It was before coctx_make Parameters stored in Why is it stored there Think about function parameter calls

then co->pfn(co->arg) Is the function call of the initial setting of the coroutine In our sample code yes readwrite_routine
There is nothing else left

}
static int CoRoutineFunc( stCoRoutine_t *co,void * )
{
    
	if( co->pfn )
	{
    
		co->pfn( co->arg );
	}
	co->cEnd = 1;

	stCoRoutineEnv_t *env = co->env;

	co_yield_env( env );

	return 0;
}

---

2、 Look again. co_yield_ct

---

In fact, it feels a little cut here Because the active release process Switch coroutines
Not just active calls co_yield_ct() And call some blocking functions （ Of course, it's just a simulated blocking function In fact, it calls the function version implemented by itself ） Has reached the awaited Will take the initiative yield

About calling blocking functions why yield Let's save it for the last article

Here's another look You can choose to give up CPU Function of co_yield_ct How did it happen
It is relatively easy and simple here Let's save the complicated ones for the last one

---

1、co_yield Series of functions

---

Here's a simple look co_yield Definition of function
It is found that these two functions are just wrapped in a layer of outer packaging

The essence is called co_yield_env Let's go to this function again

void co_yield_ct()
{
    

	co_yield_env( co_get_curr_thread_env() );
}
void co_yield( stCoRoutine_t *co )
{
    
	co_yield_env( co->env );
}

---

2、 Light hearted co_yield_env function

---

It is as expected Sure enough, it is still a very relaxing function
Only four lines It is the same as what we said before

Get the previous collaboration structure call co_swap Switch to the original collaboration Continue the previous cooperation process
It's that simple ~

void co_yield_env( stCoRoutineEnv_t *env )
{
    
	stCoRoutine_t *last = env->pCallStack[ env->iCallStackSize - 2 ];
	stCoRoutine_t *curr = env->pCallStack[ env->iCallStackSize - 1 ];
	env->iCallStackSize--;
	co_swap( curr, last);
}

---

Conclusion

---

Write another one later About calling blocking functions How to actively switch processes
To study See you next ~ ヾ(￣▽￣)Bye~Bye~