[half understood] zero value copy

Tradition IO

1. Application call read Method sends a request to the operating system to read data , At this time by User mode Switch to a Kernel mode
2. When the system receives a request to read data , utilize DMA The controller reads the data from the disk into the system cache ( In the figure 2.1)
3. And then CPU It will write the data in the system cache to the application cache ( chart 2.2), At this time by Kernel mode Switch to a User mode
4. The application then calls write Method to inform the system to write data , At this time by User mode Switch to a Kernel mode .
5. CPU Write the data in the application cache to the system cache ( chart 2.3)
6. And then it's DMA The controller then writes the data from the system cache to the network card cache ( chart 2.4),write Method returns , At this time by Kernel mode Switch to a User mode .

In ordinary IO Copy with 4 Context switching process and 4 Secondary copy process , In high concurrency scenarios, performance will be greatly affected .

DMA

Data needs to be read and written CPU Issue the corresponding command to complete , because CPU Faster than IO It's much faster , In the process of copying data CPU It is impossible to be in the process of waiting , But if you don't wait ,CPU I don't know IO When is it done , In order to coordinate high-speed CPU And low-speed IO The contradiction of , So it introduces DMA,DMA（Direct Memory Access） Direct memory access technology , It's used for memory and IO Data transmission of equipment . As big as CPU to DMA Give an order , then DMA Get to work , then CPU Do other work ,DMA Tell when you're done CPU, This reduces CPU The waiting time of .

Zero value copy

The observation chart can 2.1 and 2.2 I feel that the data has done a useless job in this process , Move from kernel mode to user mode , Then move from user mode to kernel mode , Therefore, it is proposed that Zero value copy , Zero value copy is not no copy , Instead, there is no longer a data copy process between user state and kernel state , His realization has mmap and sendfile Two ways .

mmap

mmap (member map) Memory mapping , After the data is read into the system cache , The mapping of user mode data in the system cache can complete the data transmission process .

1. First send a to the operating system mmap command ( Upper figure 1.1), At this time by User mode Switch to a Kernel mode .
2. System received mmap After the command , Will use DMA Read the data from the disk to the cache of memory ( Upper figure 1.2).
3. Returns a map of the data buffer mmap（ Upper figure 1.3）, from Kernel mode Switch to User mode .
4. The user is initiating a write The operation of （ Upper figure 1.4）, from User mode Switch to Kernel mode .
5. cpu Then copy the data in the memory cache to another cache ( Upper figure 1.5)
6. DMA Copy the data to the network card and return ( Upper figure 1.6), And by the Kernel mode Switch to User mode

mmap The operation of 4 Context switching and 3 The next data copy process , Less than an ordinary copy cpu A copy of the .

sendfile

mmap The copy method of must first obtain the mapping of the data area , Then a user initiates a command to write data , Write on the network card , If there is no need to do any processing to the data , Just send the data intact ,mmap Command and write Commands can be merged into one command , Just tell the operating system which data to send out , The merged command is sendfile.

1. The user sends directly to the operating system sendfile The order of , At this time, the user mode is switched to the kernel mode
2. After the system receives the command , from DMA Read the data from the disk to the system cache
3. Then from cpu Copy the read data to another cache
4. Again by DMA Write the data on the Internet
5. Then return , Switch from kernel mode to user mode

sendfile The process of 2 Sub context switching and 3 The next data copy process , The data read in the whole process is not visible to the user space , Adapt to static file server .

sendfile upgrade

stay sendfile Of cpu Copy data from cache from one place to another , It will also waste extra space , Can I write data to the network card only for the first time DMA Copied data , Copy directly to the network card ？ So right. sendfile Once again to upgrade , Introduced DMA Scatter/Gather Dispersed / Collection function .

1. User initiated sendfile command ( Upper figure 1.1), After the system receives , from User mode Switch to Kernel mode
2. call DMA utilize scatter Read data from disk to cache discrete storage （ Upper figure 1.2）
3. cpu Send the file descriptor and data length of the cache data to another cache （ Upper figure 1.3）
4. DMA When reading data from another cache, according to the file descriptor and data length , Use scatter/gather Read from the cache to the network card （ Upper figure 1.4）
5. return , And there are Kernel mode Switch to User mode .

This process is still invisible to user space , And it needs the support of hardware . Yes 2 Subcontext switching and 2 Secondary copy process , The performance is greatly improved .

scene

kafka and rocketmq Are used to mmap technology , Both middleware have a message persistence process , And message reading process .

mq It is used in the process of message persistence and reading mmap+write, and kafka The persistence process uses mmap+write, For reading messages sendfile.

summary

Why DMA？

DMA To solve the problem of high-speed CPU And low speed IO Contradiction of operation ,cpu After initiating a read instruction , from dma Take over the reading and copying of data ,cpu Deal with other events .

Tradition IO Compared with zero value copy

name	The process	Number of context switches	Number of copies
Tradition IO	Read command 、 Read from disk 、 Write to cache 、 Write to user cache 、 Write to cache , Write to the network card	4	4
mmap+write	mmap command , Read from disk , Write to cache 、 return mmap、 launch write command , Read from cache , Write to another cache , Write to the network card	4	3
sendfile	sendfile command , Read from disk , Write buffer , Write to another cache , Write network card	2	3
sendfile Upgraded version	sendfile command , Read from disk , Write buffer , Read from cache according to file descriptor , Write network card	2	2

---