Meta opens the project aria pilot dataset and will develop real-time 3D maps in the future

For training belongs to AR The visual assistant of glasses 、 promote AR Positioning ability ,Meta As early as 2020 It began to pass Project Aria The project collects first person video data , For training AR Computational vision algorithm of glasses .Meta Express ： Existing computer vision algorithms are mainly trained based on photos and videos from the third person perspective , Therefore, always perceive the surrounding environment from the perspective of bystanders . If you want robots 、AR Glasses perceive the world like people , Then you will need to use first person data to train , That is, the picture people see when performing various tasks .

After two years ,Meta In Singapore 、 The British 、 The United States and other places have collected a large number of first person video data . The project has 3000 Participate in data collection , Include Meta staff 、 contractor 、 Paid external participants, etc , Project partners include Carnegie Mellon University 、 National University of Singapore 、 BMW and so on . The data collection environment includes Meta The office 、 Approved private residence 、 In public places .

lately , The company will be shooting data in the United States for AI、ML Scientific researchers are open , To accelerate machine perception and AI Technology development .Meta Express ： Release Aria Pilot The purpose of the dataset is , Show external researchers a repeatable research benchmark , The aim is to promote first person computer vision 、 Scene perception AI/ML The development of algorithms .

Aria Pilot Data sets

According to qingting.com , This data set is called Project Aria Pilot, Which includes 159 A first person video , Cumulative duration 7 Hours , Shot separately in each 5 Locations . The video contains various scenes of daily life , Like washing dishes 、 Open door 、 Cooking 、 Use mobile phones in the living room 、 Play a game 、 Exercise and so on . besides , It also includes desktop interactive data captured by the multi view mobile capture system , It contains videos of people interacting with objects . Besides ,Aria Pilot The dataset also contains a number of all-weather videos shot by actors , Recorded Aria Glasses sense all day / The effect of capturing environmental data .

actually ,Meta Previously, it has also launched an open source first person video dataset Ego4D, The difference is that Ego4D Shoot with a head mounted camera , The equipment is not limited to GoPro, as well as Vuzix Blade etc. AR/ Photographic glasses . by comparison ,Project Aria Pilot Really based on Meta R & D glasses equipment shooting , Its shooting angle 、 Height will better meet the training Meta AR The need of glasses assistant .

It is worth noting that ,Aria Pilot Is an anonymous video data set , For privacy and security, face 、 The license plate and other key information are blurred .

Meta Based on the original video , Not only does it remove private information , Automatic and manual marking are also added / notes , Help machine learning /AI The model understands the runaway reference frame and context information of the scene .

To help AI Understand the posture of multiple users in the same reference frame / motion ,Aria Pilot The dataset also aligns the data captured in the same scene with this reference frame , The purpose is to allow the algorithm to share the context information of the scene . meanwhile ,Project Aria Pilot The dataset also aligns the timeline of video data , In other words, different devices can share video data captured at the same time , This is expected to promote multi person sharing AR effect .

Project Aria Sensor details

Meta It's open Aria Sensors equipped with glasses , Different sensors are responsible for collecting different data , These include ：

• ​ One 110°FOV Rolling shutter RGB camera ;
• Two 150°FOV Global shutter single shot （ be used for SLAM And gesture tracking ）;
• Two 80°FOV Global shutter single shot （ Equipped with IR The light source , For eye tracking ）;
• Two 1KHz IMU+ barometer + Magnetometer + Environmental sensors ;
• Seven 48KHz Space microphone ;
• One 1Hz GPS modular .

In addition, it is equipped with an eye tracking module , It is mainly used to correct the wearer's gaze , It can also accelerate the research of human object interaction .

Except for the images ,Project Aria Also collect voice data , therefore Meta stay Aria Pilot Voice to text notes have also been added to the dataset . Such data can train algorithms to predict dialog rotation , And multi person conversation transcription .

The use of data

In terms of application scenarios ,Aria Pilot Data sets can be applied to a variety of research fields , Including camera relocation 、 Scene reconstruction and other machine perception and AI technology , And these studies will become improvements AR The key of equipment . Details ,Project Aria By collecting first person video data , Can help Meta Build real time 3D Space map and other software , It also helps to promote AR Hardware iteration of glasses . Besides , It can also help developers 、 Researchers explore AR Potential use in the real world .

Meta Express ：Project Aria The goal is , adopt AR Glasses and other products , Integrate computing devices into daily life , Do not interfere with people's daily interaction 、 Task execution 、 Sports and so on . meanwhile , Further enhance the surrounding physical world , Make human-computer interaction design and experience more humanized .

meanwhile , First person video data is also expected to be used to build real-time 3D Map , To help the visual assistant locate objects , Perform tasks such as finding keys quickly .Meta Express ： In the future AR Devices need to be more perceptive , In order to play a real value . And in order to make AR The device knows her and others 、 The relationship between surrounding objects , Analyze the current situation , Will need to be based on the physical environment 3D Map .

according to the understanding of ,Magic Leap、HoloLens etc. AR The device passes real-time 3D Space scanning to dynamically update the scene structure / Layout , But this way not only consumes electricity , It is difficult to scale . If we develop a large-scale 3D Map , Will help speed up AR Glasses understand the speed of the surrounding environment .

Meta Real time development 3D Map （LiveMaps） Use computer vision to identify the environment and perform positioning display . utilize LiveMaps function ,AR Glasses will be able to effectively view 、 analysis 、 Understand the world around , Better service for users . Besides ,LiveMaps Will be updated in real time , help AR Glasses track street changes and other information .Project Aria One of the purposes of , It's a test LiveMaps The practical application effect of .

Meta Pointed out that ,AR Equipment and experience will deepen people and the environment 、 The connection between things , And provide more practical functions and information , Reduce the time of looking down at your mobile phone .Meta Plan to build a people-oriented AR The ecological system , Design transparency 、 meaningful 、 Considerate 、 Humanized Technology . Reference resources ：Meta