Training on the device with MIT | 256Kb memory

title ：MIT|On-Device Training Under 256KB Memory（256KB Training on the device under the memory ）

author ：Ji Lin, Ligeng Zhu,  Song Han etc.

brief introduction ： By fine tuning the pre training model , Training on the device enables the model to adapt to the data collected from the sensor . For IOT devices with few memory resources , Training memory consumption is prohibitive . The author proposes an algorithm - System collaborative design framework , just 256KB Memory can be used for device side training . Training on equipment faces two unique challenges ：(1) The quantization map of neural network is difficult to optimize due to the mixing of bit accuracy and the lack of normalization ; (2) Limited hardware resources （ Memory and computing ） Complete reverse calculation is not allowed . In order to cope with optimization difficulties , The authors propose quantitative perceptual extension to calibrate gradient scale and stabilize quantitative training . In order to reduce the memory consumption , The authors suggest that sparse updating skip the less important gradient calculation layers and sub tensors . Algorithm innovation is realized by lightweight training system , Prune backwardly computed graphics to support sparse updates and unload automatic runtime differentiation to compile time . The author's framework is the first practical device transmission solution , Learn visual recognition on micro IOT devices , Can't use the existing framework memory 1/100. The author's research enables IOT devices not only to perform reasoning , But also constantly adapt to new data , For lifelong learning on equipment .

Paper download ：https://arxiv.org/pdf/2206.15472v1.pdf