Hardware factors such as CPU, memory, and graphics card also affect the performance of your deep learning model

Reference: https://posts.careerengine.us/p/5dcb5f53bb8d8844ea19cd0f

Data flow path

Assuming that we now have a batch of image sets on the hard disk, which are to be read into the memory and sent to the GPU for computing, the following process will generally be experienced:
The cpu issues a read command, finds the image data from the hard disk, and stores it inIn memory;
The cpu takes out a batch of data from memory, converts it into numpy array, and performs data preprocessing/enhancement operations, such as flipping, translation, color transformation, etc.Return to memory after processing.
Each of the cpu memory (hereinafter referred to as memory) and the gpu memory (hereinafter referred to as video memory) open up a buffer, and a batch of data in the memory is transferred to the video memory through the PCIe channel.
The GPU core obtains data from the video memory for parallel computing, and returns the calculation results to the video memory.
The calculated result will be returned from the video memory to the memory through the PCIe channel.

These 5 steps involve several links that affect data transfer speed:

HDD read speed;
PCIe transfer speed;
Memory read/write speed;
CPU frequency.
Let's talk about hard drives first.

After the hardware bottleneck is multi-threading at the software level