Reasoning speed of model

1. Four descriptions of model size

2. Amount of computation 、 The relationship between access quantity and reasoning speed

2.1 Calculate density

It can be seen from the above that both the amount of calculation and the amount of visits can affect the reasoning speed of the model , Then further define the calculation density as the calculated quantity divided by the visited quantity , as follows ：

Computational density is used to reflect that a program is more computationally intensive , Or is it more intensive .

• Computing intensive programs ：Compute bound
• Memory intensive procedures ：Memory bound

2.2 Roofline Model

A procedure for evaluating in Hardware The mathematical model of the upper bound of the achievable performance .

The horizontal axis represents the calculated density , stay 2.1 We describe that each model has a computational density , That is to say, the horizontal axis represents the model . The vertical axis represents the calculation speed .

This mathematical model represents , Run a model on this hardware （ Give a calculated density ） The best computing speed that can be achieved .

You can see that this is a piecewise function ：

• For memory intensive programs ： Calculation speed = Calculate density × bandwidth , That is, it has a great relationship with the bandwidth of the hardware .
• For computationally intensive programs , Calculation speed = Peak calculation speed , That is, the computing power of hardware （ Peak calculation speed ） It matters a lot .

2.3 Speed of reasoning

stay 2.2 The calculation method of calculation speed is given in the part of , Knowing the calculation speed, we can calculate the reasoning speed , as follows

The calculation formula for memory access density is ：

Amount of computation / Calculation speed = Amount of computation / ( Calculate density * bandwidth ) = Amount of computation /( Amount of computation * bandwidth / Visit the stock ) =  Visit the stock / bandwidth

Here we don't distinguish programs like the above , Instead, it becomes an operator , Because there may be memory intensive operators in a model , There are also computationally intensive operators , Therefore, the reasoning speed is not as strict as the above formula , It's more complicated .

• Memory intensive operators ：Concat、Eltwise Add、ReLU、MaxPooling……
• Computationally intensive operators ：Conv、DeConv、FC、MatMul、LSTM……