[go] how to control the maximum number of concurrent processes

Abstract

This article will explain in Go How to use channel To limit the maximum concurrency .
The author's level is limited , If you have any questions, please leave a message at the bottom of the article ！

introduce - Problems encountered during concurrent programming

I met a demand when moving bricks two days ago , It takes a lot of requests for information from another service . So I enabled multiple coroutines to send requests concurrently , After writing the self-test, I had a wave , It's OK to have a small request at first , When more than one request is made , Many requests begin to fail , Report the following error ：socket: too many open files.

This error is due to linux Number of files opened by a process in ( Handle ) We've reached the ceiling , That is to say, there are too many concurrent processes .

Let's talk about how to use... In code channel To limit the maximum concurrency .

2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:n err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files
2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:t err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files
2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:v err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files
2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:l err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files
2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:s err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files
2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:h err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files
2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:r err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files
2022-05-30T15:21:50.624+0800    ERROR   project_manage/organization_service.go:156      func:GetProjectSimpleList GetStaffBaseInfoApi engName:g err:Get "http://xxx": dial tcp x.xx.xx.xx:80: socket: too many open files

Use channel Limit concurrency

Use channel The principle of limiting concurrency is very simple , Take advantage of Go in channel When the buffer is full , Will block writing .

The basic logic is as follows ：

• Create a channel, Set the buffer to the maximum number of concurrency you want , such as 100.
• Every time a collaboration is created , Just go channel Write a value in , Express placeholder .
• At the end of a collaboration execution , From channel Read out a value in , Express Abdication .
• When the maximum number of concurrent processes currently executing at the same time reaches , namely channel The buffer of is full , At this time, the new cooperation process is to be executed , placeholder Will be blocked , Until another collaboration exits Abdication after , Will execute .

The principle of this method is simple , The code is simple to implement , Very elegant . But here's the thing , When the maximum concurrent number does not touch other limits （ For example, there are too many simultaneous requests , The number of handles opened by the process is limited ）, The maximum concurrent number is not more , The faster the program executes , We can write another article to further discuss this issue .

Specifically Go The code is as follows ：

func GetInfo(idList []int) error {
    
	var wg sync.WaitGroup
  
	//  Limit the maximum number of concurrent to 100
	limitGoroutine := make(chan int, 100)
  
  wg.Add(len(idList))
	for _, id := range idList {
    
    
    //  Every time you start a collaboration , Just go channel Write a single digit in 
		limitGoroutine <- 0
    
		go func() {
    
			//  Concurrent requests for information 
      //  Store in database 
			wg.Done()
      
      //  When the cooperation process exits after execution , From channel Read one digit in 
			<-limitGoroutine
      
		}()
	}
	wg.Wait()
	return nil
}