The use and introduction of pytorch 23 hook and the implementation of plug and play dropblock based on hook

Hook（ hook ）： It literally means to hook a function , Modify input data or other operations before its execution , Or modify the output or other operations after its execution . adopt hook Operation can lock a layer object （model Module in ） Life cycle of , Monitor its execution status and execution results . stay pytorch in , Provide a wealth of hook api, So we can monitor and modify tensor In the model forword The state of . In Ben Bowen , be based on hook The operation realizes plug and play Dropout operation , Support dorpblock、dropout2d And so on Dropout operation .

stay pytorch in layer The life cycle of can be simply understood as ： Definition -> initialization -> Forward propagation -> Back propagation -> The destruction . Forward propagation and back propagation are the most important parts of its execution process , Through the analysis of forward propagation and back propagation hook operation , You can get the execution status of data in the model （ Such as the implementation CAM） And modify it （ Such as gradient clipping ）.

Grad-CAM： Gradient based category response feature visualization . adopt hook Operation to obtain the corresponding layer Of forward In the process feature map Of output and backward In the process grad Of output, And then feature_map_output And backward_grad_output Multiply the corresponding elements in , Then realize the visualization of category response characteristics .

Gradient cut ： Yes layer Of backward In the process grad Of output The value of , So that it cannot be greater than a specific value, resulting in gradient explosion .

1、pytorch Medium hook Interface

pytorch in the light of Model、modules.module、ScriptModule and Tensor 4 Corresponding modules are provided hook Interface . The needle is for Tensor Only one is provided for back propagation hook, because tensor The forward propagation process is fully visible to users . Aim at Model、modules.module、ScriptModul