Lifeifei's team applied vit to the robot, increased the maximum speed of planning reasoning by 512 times, and also cued hekaiming's Mae

Yang Jing From the Aofei temple
qubits | official account QbitAI

Human predictive power +ViT, What kind of chemical reaction will occur ？

It will make the robot's action planning ability fast and accurate .

This is the latest research of lifeifei's team ——MaskViT, adopt MVM, Mask visual modeling Yes Transformer pretraining , Thus the video prediction model is established .

Results show ,MaskViT Not only can it generate 256*256 video , It can also increase the reasoning speed of robot action planning to the highest 512 times .

Let's see what kind of research this is ？

Find inspiration from human beings

Research in the field of neuroscience shows that , Human cognition 、 Perception is supported by a predictive mechanism .

This predictive model of the world , It can be used to simulate 、 Evaluate and select different possible actions .

For humans , This process is fast and accurate .

If you can give robots a similar ability to predict . Then they can quickly plan in a complex dynamic environment 、 Perform various tasks .

such as , Predictive control through visual models , Maybe it's just a way , But it also puts forward higher requirements for calculating power and accuracy .

therefore , Lifeifei's team thought of many recent developments ViT framework , And hekaiming MAE Based on MVM,Masked Visual Modeling This self supervised pre training token .

But the details need to be operated , There are still many technical challenges .

One side , The complexity of the global attention mechanism is proportional to the square of the input sequence length , The cost of video processing is too high .

On the other hand , There is an inconsistency between video prediction task and autoregressive mask visual pre training . In the actual test , The model must predict the complete future frame sequence from scratch , This leads to poor video prediction quality .

Based on this background , Li Feifei's team proposed MaskViT—— Visual modeling by masking Transformer pretraining , Thus the video prediction model is established .

There are two design decisions .

First , To improve memory and training efficiency , Two types of window attention are used ： Spatial attention and spatiotemporal attention .

secondly , Masked during training token The proportion is variable .

In the reasoning stage , Video is generated through iterative refinement , The mask rate is gradually reduced according to the mask scheduling function .

experimental result

The research team worked on three different data sets , And four different indicators MaskViT.

Results show , Compared with the previous advanced methods ,MaskViT Both showed better performance , It can generate a resolution of 256 × 256 In the video .

still BAIR Ablation experiments were carried out .

And then , The team also demonstrated the use of real robots MaskViT The effect of real-time planning .

Reasoning speed can be increased up to 512 times .

The researchers say , This work shows that , With minimal domain knowledge , A general framework for visual modeling using masks , Give powerful prediction models like agents .

But at the same time , It also has some limitations .

For example, flicker artifacts may occur during quantization of each frame , Especially in RoboNet In this kind of video with static background .

And if we want to expand the scale of video prediction , Still challenging , Especially when there is a lot of camera movement .

future , They will explore how to integrate this video prediction method into more complex planning algorithms .

It is worth mentioning that , In this year 5 month , Hekaiming's team has proposed a video version MAE, And found that the best masking rate is as high as 90%.

Thesis link ：
https://arxiv.org/abs/2206.11894
Project links ：
https://maskedvit.github.io/
Hekaiming's thesis ：
https://arxiv.org/abs/2205.09113