The paper manual becomes 3D animation in seconds, the latest research of Wu Jiajun of Stanford University, selected for ECCV 2022

羿阁 发自 凹非寺
量子位 | 公众号 QbitAI

Is there any space poor friend,Do not know how to do get lego instruction every time？

这回,Can move the lego instruction to the！

Tsinghua alumni yao class、Assistant professor at Stanford jia-jun wu,Can lead the team developed a paper instruction into3D动画的技术,The paper has selected and now2022年计算机视觉顶会ECCV.

看完效果图,有网友直呼：It is of great help to lego lovers of all ages are！

3DAnimation instruction

Although the lego instructions are written by professional designers,But for those with poor imagination,不得不说,还是3DAnimation is more sweet.

This step into look easy,In fact behind the two technical difficulties.

The first problem is how to get the paper2DImage projection into3D动画.

The research team to do,Is the task is decomposed into a series of can smoothly、Effective implementation of short steps,Through the establishment of a model,Converts images on the manual machine to explain the algorithm,In order to simplify the machine learning task.

正如上图所示,If you want to put the figurea转化为图c,Need to extract the instruction of each parts of the image position,In order to build the final finished product.

Research in the face of the second challenge is,The shape of lego bricks is so changeable.

Although many of the basic parts shape almost,But just like in the picture the guitar head,Lego have many flexible and complex parts.而且,These parts may produce different combination also greatly increase the difficulty of the interpretation of the machine：Every building steps to form a new unknown image.

为了解决这两个挑战,The team proposed a new framework based on machine learning：Manual execution plan network(manual-To-executable-Plan Network, MEPNet).

Its core idea is to two-dimensional point detection method based on neural network and2D-3DMatching algorithm combining,For the invisible3DThe object of high precision prediction.

MEPNetThe operation of the two stage.The first stage to do,Is the basic shape and new parts3D模型、The target shape2DImage as the input information,Forecast for each part of a set of2D关键点、Rotation Angle and the mask.

在第二阶段中,By looking for basic shape and the possible link between new parts,The first stage and then predict2DThe key to reverse projection3D图像中.

值得一提的是,This method during practice does not require anyground truth图像.

另外,MEPNetData set, performance is superior to other existing methods.Compared with the methods based on end-to-end learning,MEPNetTo keep the efficiency model based on machine learning,And can be better to generate the unknown3D对象上.

最值得注意的是,MEPNetAble to use synthetic data trained individually,Which is applied to the real life scenario.

目前,All code and data is open source,感兴趣的小伙伴可以关注一下.

作者介绍

This paper from the Stanford university jia-jun wu team.The author also include：Ruocheng Wang、Yunzhi Zhang,麻省理工大学的Jiayuan Mao以及Autodesk AI Lab的Chin-Yi Cheng.

吴佳俊,Now, an assistant professor at Stanford university,Affiliated with Stanford vision and learning laboratory (SVL)And the Stanford artificial intelligence laboratory (SAIL).At the Massachusetts institute of technology to complete doctorate,本科毕业于清华大学姚班,曾被誉为“Tsinghua university, one of god”.

论文第一作者Ruocheng Wang,Master degree in computer science at Stanford,Students is jia-jun wu door.本科毕业于浙江大学计算机专业,Also at the university of California, Los Angeles, andAdnan DarwicheThe professor worked for a period of time.

One More Thing

Although the whole paper in lego, for example,But the author also referred to in the paper,In fact this technology can also be applied to other types of assembly instructions.

好多“Hard to install a long time”Netizens are called on to launch the ikea version：

不过,在一片欢呼声中,Some netizens also puts forward the different sounds：

I don't know if it is a surprise or destroyed my play the fun of high.

对此,你怎么看？Do you like watching the instruction spell lego,Or did you play？

参考链接：
[1]https://cs.stanford.edu/~rcwang/projects/lego_manual/
[2]https://twitter.com/_akhaliq/status/1552118469214314496
[3]https://arxiv.org/abs/2207.12572
[4]https://jiajunwu.com/