Now, the ear is going into the metauniverse

mention AR/VR Equipment what would you think ？ Pictures of cyberpunk style , Or the sci-fi feeling brought by the superposition of virtual reality ？

When our eyes are still focused on the visual interaction level , A revolution in the field of hearing has been quietly emerging .

Before we talk about this auditory revolution , Let's feel it first XR In the era of “ Near the border ”.

notes ： It's better to wear headphones

This is a domestic human-computer interaction product platform company Rokid Recently released a group of applications for AR Glasses 6DoF Space sound field technology Demo video .

Different from traditional two channel 、 Stereophonic audio experience ,6DoF Spatial sound field technology can simulate the change of spatial position between sound source and human ear in hybrid reality 、 The change of sound intensity and direction caused by the presence or absence of obstructions , So that AR Glasses provide users with a more immersive auditory experience .

What is? 6DoF Space sound field ？

6DoF The spatial sound field is actually the embodiment of sound in the three-dimensional field . But this is not simply to make the sound more three-dimensional through more channels , It is an audio spatialization process synchronized with video spatialization . So there are two essential elements ——3D Audio and Real time feedback of head movement .

First of all to see 6DoF The first essential element of space sound field ——3D Audio . Conventional 5.1 Sound track can show sound on a horizontal plane , Sound localization therefore has a forward and backward 、 The left and right dimensions , This is known as 2D Audio . When an audio has both up and down dimensions , This audio is 3D The audio is on .

△ chart ：3D Audio graphics （ The picture comes from the Internet ）

6DoF The second essential element of space sound field —— Real time feedback of head movement . In the real world , When our head turns or shifts , The absolute position of the sound source itself will not change , The relative direction between the sound source and the head will change .

For example ： There is a guitar playing music in front of you , If you turn to the right , The music will change to your left ; If you turn to the left , The sound of music will change relatively to your right . therefore , To achieve a more realistic auditory experience in mixed reality , It is necessary to accurately locate the spatial position between the sound source and the user's head , That is to realize the real-time tracking of the user's head movement .

6DoF The realization of space sound field needs a high degree of cooperation between software and hardware

To meet the 6DoF The two essential elements of space sound field technology are not easy , On a technical level , This needs to be Space engine （Space Engine） and Audio engine （Audio Engine） Highly integrated , And make full use of hardware resources .

The core work of the space engine is Virtual and real space fusion . The engine uses 3D reconstruction technology to build the map in advance , Establish a virtual world coordinate system , And add virtual objects , Set pose 、 shape 、 Material and other properties .

Runtime , Get the observer by processing the sensor data （ If wearing AR glasses , The observer is the position of the human head ） Real space pose and local map , Then we can get the pose transformation of real space and virtual space through map matching , You can unify the position and posture in the virtual world coordinate system .

Depending on the type and number of sensors , The space engine can obtain different types of degrees of freedom for the observer （Degrees of Freedom-DoF） Information , So as to provide the necessary spatial information for the audio engine .

For example, the degree of freedom of the head is divided into ： Having both displacement and rotation 6DoF、 Only rotating 3DoF、 A virtual space where people can't move , The corresponding audio can be divided into 6DoF Space sound field 、3DoF Space sound field 、 Surround sound . therefore ,6DoF Space sound field technology needs to obtain more complex human head degrees of freedom .

△ chart ：6DoF freedom （ The picture comes from the Internet ）

The core work of the audio engine is to analyze the audio signal and HRTFs（Head Related Tranfer Functions, Head dependent transfer function , It is called head transfer function for short ） Do convolution , Generate Binaural audio .HRTFs It's at the horizontal angle （azimuth）、 Pitch angle （elevation） And distance （distance） The convolution kernel set measured by the coordinate sampling of these three measurement dimensions , Its accuracy is 6DoF The dominant factor of spatial sound field presentation effect .

But now commercially available HRTFs The accuracy that the database can achieve is not completely comparable to the human ear's hearing ability , What is more challenging is that everyone has different ergonomic parameters and psychoacoustic systems , It even changes with age .

Accurately measure everyone's HRTFs The parameters are obviously unrealistic , How to do it at a low cost ⽣ Become personalized HRTFs？ Already realized 6DoF The space sound field technology is landing Rokid The technical team gave a solution , That is, at the end of consideration NPU/GPU And so on , Combined with deep learning technology , To produce more refined components .

△ chart ：XR Equipment applications 6DoF The space sound field needs a high degree of cooperation between software and hardware

Besides , To increase occlusion 、 Reflection 、 Reverberation and other effects , Give Way 6DoF The space sound field is more realistic , It also requires such things as geometric acoustics （Geometric Acoutstics） Ray tracing and wave acoustics （Wave Acoustics） Spherical harmonics of （Spherical Harmonics） Decomposition and other technologies . This has a very high demand on the computing power of the equipment , It will also bring greater power load to the equipment , Increase equipment costs and safety risks . So in practical application , It often needs to be in the order of the spherical harmonic function 、 Make a corresponding compromise and balance between voice quality and spatial accuracy .

In addition to the algorithmic level ,6DoF The application of space sound field technology should also consider the hardware form of the equipment . Many current audio algorithms are based on in ear or head mounted speakers , but AR Glasses are wearable devices that users will wear for a long time in the future , If the in ear design is adopted, it will not only seriously damage the user's hearing , It's against AR The mission of integrating physical and digital , therefore , While maintaining the open horn design , How to protect 6DoF The presentation effect and security of space sound field have become new challenges .

at present ,Rokid The approach taken by the technical team is , adopt Directional sound technology Research and use of , To solve privacy problems . meanwhile , In order to make 6DoF The sound effect of the space sound field is richer and fuller , Through the design of sound cavity structure 、 Repair of sound frequency 、 Enhance the sound quality by means of sound harmonics and reverberation according to human hearing , Reduce the loss of audio effects , Let users really feel “ Near the border ”.

A sound revolution , Is quietly rising

6DoF Space sound field technology in AR The application on the equipment is on the ground , Let us see the broad application space of sound in hybrid reality . adopt 6DoF Space sound field technology ,AR Glasses and other devices can get rid of the field angle （FOV） Limit , Let the user find the content outside the screen through sound , To achieve 360 Content presentation of degree range .

meanwhile , In addition to visual interaction ,6DoF The application of spatial sound field technology makes hearing a new interactive dimension . combination 6DoF Space sound field , Users can quickly and accurately locate the direction of the vocal object in hybrid reality , Clearly distinguish the received sound information , Feel the change of sound distance and position …… This will allow users to get closer to the real world experience in mixed reality , So as to further reduce the sense of separation between the digital world and the real world in hybrid reality .

6DoF The new auditory experience brought by the space sound field is impacting the traditional stereo sound that has dominated for more than half a century , However, the application and popularization of any new technology do not depend on a single team 、 The power of a company , This requires constantly lowering the entry threshold , Attract more industry forces to join .

Such as Rokid It means that 6DoF The space sound field is integrated into the new and upgraded version YodaOS-XR Operating system , As YodaOS-XR The basic capabilities of the operating system are available to industry developers . meanwhile ,Rokid It is also planned to promote more applications in AR Development of special sound effects of glasses , Such as surround and micro bass high fidelity sound effects , With efficient and easy to use SDK Let developers really get it and use it .

According to a source ,Rokid New and upgraded YodaOS-XR The operating system may be released in the second half of this year , Contains many natural interaction engines 、 Amicable UI Interface 、 Native XR Application and application development framework . At that time, developers will be able to focus on polishing high-quality content , Develop various imaginative applications and contents , such as XR game 、XR meeting 、XR social contact 、XR Cinemas, etc , Join hands with the vast number of users to enter the real AR The world .

XR The ultimate goal of the times is the perfect integration of the virtual world and the physical world , This fusion is mainly some ways of exchanging information between human beings and the outside world , Such as touch 、 auditory 、 Vision 、 The sense of smell 、 Simulation and enhancement of taste .

6DoF The application of space sound field and other technologies has broadened XR The imaginary boundary of the device , It also quietly set off a perceptual interaction revolution . We may be able to foresee , After sight and hearing , Tactile sensation 、 The sense of smell 、 Taste, etc “ Sensory experience ” Will also be XR Times have been redefined .

* This article is authorized by qubit to publish , Opinions are owned only by the author .

—  End  —

「 Smart car 」 Communication group recruitment ！

Welcome to smart cars 、 Self driving partners join the community , Communicate with industry celebrities 、 Compare notes , Don't miss the development of smart car industry & Technological progress .

ps. Please note your name when adding friends - company - Position oh ~