Netease Yunxin appeared at the giac global Internet architecture conference to decrypt the practice of the new generation of audio and video architecture in the meta universe scene

7 month 22 Japan ,2022 GIAC The global Internet architecture conference was held in Shenzhen .GIAC It's one of the largest technical conferences in China , Specifically for architects 、 People such as technical directors and high-end technical practitioners .

As the annual event of Internet technology , The conference gathered 100 Many senior technical experts from front-line Internet enterprises , Around the meta universe 、 Artificial intelligence 、Web3 And other hot technical topics , It brings all participants a journey of technological upgrading and harvest .

Raoshicheng, Netease cloud audio and video architect I was invited to attend this conference , And brought the theme of 《 The practice of the new generation of audio and video architecture in the meta universe scene 》 The share of , Comprehensive introduction The challenge of the metauniverse scene to the audio and video architecture as well as Netease Yunxin's breaking strategy and practice of the new generation of audio and video architecture .

Netease cloud audio and video architect Raoshicheng

Nuggets 「 Meta universe 」

Netease's technical reserves

As The first sharing guest of the special session on the theme of Yuan universe , Rao Shicheng first introduced the development of the meta universe .

“ Meta universe ” This concept was first born in 1992 American science fiction writer Neil · Stephenson's works 《 An avalanche 》, The book builds a system in which the real world and virtual space coexist with unrestrained imagination ;2018 year 《 No. 1 player 》 The release of the film made the public have a concrete understanding of the metauniverse ; here we are 2020 year ,Roblox The successful listing will completely detonate the meta universe , It has driven major global technology companies to enter the market , Based on its first mover technical advantages, Nuggets yuancosmos industry .

Such a grand meta universe , What underlying technical support is needed ？

Rao Shicheng concludes , The realization of metauniverse requires blockchain 、 The Internet of things 、 Network and computing 、 Artificial intelligence 、 Electronic games 、 Six technical supports of interaction design . and NetEase self 2011 From the year onwards , So we started AR、VR、 Virtual human 、 Ultra low delay audio and video 、 Blockchain 、 Cloud games and other meta universe related technology reserves and planning , It is one of the earliest domestic technology companies involved in the field of meta universe .

At the infrastructure level , Netease has owned the following  WE-CAN Global intelligent routing network 、AI Computing platform 、AR/VR、 Blockchain 、 Smart hardware And so on . Based on these core technologies , Precipitated Netease intelligent enterprise 、 Netease Fuxi 、 Netease planet 、 Yixian advanced technology and other service platforms , And a series of reliable meta universe solutions have been formed on the platform , In the game 、 social contact 、 Online retailers 、 Finance 、 education 、 Best practices have been carried out in various fields such as medical treatment .

Netease meta universe capability panorama

「 Meta universe 」

Two challenges to audio and video architecture

“ The metauniverse emphasizes immersion 、 Highly restored interactive experience , At the same time, the scene is more comprehensive 、 More complicated , This poses a challenge to the traditional audio and video architecture .” Rao Shicheng said .

First , Simple video call 、 Voice chat and real-time interaction are far from meeting the needs of the meta universe , Ultra low delay audio and video 、 Business data transmission 、 Real time extraction of face data 、 Expression and movement migration 、 Real time voice 3D Sound effect And other key technologies have become important elements of the metauniverse .

Drive innovation and optimization of key technologies , Break through the boundaries of the original audio and video related technologies , Providing users with the ultimate immersive experience has become a powerful challenge for the audio and video architecture .

secondly , The audio and video services in the metauniverse are complex and diverse . Many businesses have RTC、 Ultra low delay data transmission 、 Low delay live streaming and other requirements , Different needs SDK Need independent integration , Access methods are also different , This makes it difficult to access 、 Inefficient problem .

How to design a good SDK framework , While providing high availability and high stability , Integrate all businesses through technology combination , Provide users with an overall one-stop solution ？ This is the second biggest challenge for audio and video architecture .

A game breaker

Optimization Practice of Netease cloud audio and video architecture

Raoshicheng introduced , Facing the above two challenges , Netease Yunxin has built a new generation of audio and video engine NERtcEngine Integrate with the new generation of audio and video SDK NERtcSDK.

Next generation audio and video engine ：NERtcEngine

NERtcEngine Based on the actual business scenario of metauniverse, a data engine with ultra-low delay data transmission is specially designed and implemented , At the same time, more and more intelligent algorithms and AI Algorithm , The algorithm integration mode is specially reconstructed , Design and implement the basic algorithm component engine .

Netease Yunxin new generation audio and video engine NERtcEngine

In the design of ultra-low delay data transmission architecture , Netease Yunxin has designed an independent data transmission channel , While ensuring the stability and efficiency of the original audio and video data transmission channel , Make a separate transmission protocol for it 、 send out / Receive the control unit and QoS Adjustment strategy . Besides , Netease Yunxin also has a large-scale distributed real-time transmission network WE-CAN, It is the first transport layer base independent of business logic in the industry , And deploy nodes in hundreds of major countries and regions around the world , Comply with the data security policy of each region , Full coverage IM news 、 Real time audio and video 、 live broadcast 、 on demand 、 Whiteboard and other product scenarios . both , Ensure ultra-low delay data transmission .

In the design of algorithm component engine architecture , Netease Yunxin integrates all algorithms into a unified interface , The integration method is very friendly . At the same time, it supports plug-in integration , Each algorithm can be used as an independent unit in a plug-in way , Flexibly realize customized integration of algorithms , And it's more scalable , The subsequent new algorithm only needs to realize the corresponding integration API It can be quickly and easily integrated into the framework .

A new generation of audio and video fusion SDK：NERtcSDK

NERtcSDK It is the integration of Netease Yunxin's new generation of audio and video services SDK. In the business scenario of metauniverse , A wider range 、 Deeper integration can bring greater value to users .

In the past business , Different needs must be connected to different SDK To realize the corresponding function . And in the NERtcSDK In the design of fusion architecture , Users only need to dock 1 individual SDK Can be achieved at the same time RTC、 Virtual human 、 Low latency live 、 Ultra low delay data transmission and other services , This is right 3D Virtual human social 、 For complex integration scenarios such as virtual real-time exhibitions , It greatly improves the speed and efficiency of user access , Greatly reduce the access cost .

Netease Yunxin new generation of audio and video integration SDK NERtcSDK

Rao Shicheng also introduced Netease cloud letter integration SDK A representative integration scheme in —— Design of high connection scheme . Keep alive through application 、 Fusion call 、 Call assisted “ SanBanFu ”, It covers most of the called arrival scenarios , Increase the call arrival rate , So as to ensure the connection rate of the application .

From emptiness to reality

Acceleration boost 「 Meta universe 」 Technical landing

“ All technology accumulation needs to be realized in the real scene , To see the value .” After introducing the technical layout of Netease Yunxin in metauniverse , Rao Shicheng showed that Netease promotes the meta universe “ From emptiness to reality ” Several major cases .

Rao Shicheng believes , Metacosmic applications are more than just moving the real world into the virtual world , Not just from offline to online , It's helping to improve production efficiency 、 The improvement of social and people's livelihood has real value , And promote the accelerated development of relevant industries .

Remote sensing machinery helps the digital transformation of traditional industries , Poor network environment 、 In the harsh outdoor working environment, the excavator can also be remotely controlled by mobile phones for production work without going deep into the site , It not only reduces the labor intensity , It also fundamentally eliminates the potential safety hazards of personal casualties , Help improve production efficiency .

AI Virtual customer service is increasingly common in financial intelligence virtual customer service 、 Internet Medical remote consultation 、 News virtual anchor 、 Social entertainment virtual idol 、 E-commerce virtual live broadcast and other scenes . Compared with the common virtual human solutions in the market , Netease Yunxin provides a one-stop service SDK, Users only need to connect to one SDK You can have virtual people at the same time 、RTC、 Live broadcast and other capabilities . meanwhile , The first client and cloud dual rendering technology , To meet the actual needs of different customers and scenarios .

Netease Yaotai is the first yuan universe landing product in China , Different from the single presentation of traditional video conference , Yaotai has more virtual immersion , It can also better meet the practical needs of the real world . Netease once moved the venue of the global investor conference to Yaotai virtual world , From many countries around the world 200 Many investors , Through their own virtual image , Exchange the latest developments of Netease business . The whole interactive scene is based on Netease Yunxin WE-CAN Global intelligent routing network and converged communication capability .

  //  

In addition to the dry goods sharing in the special session of the metauniverse theme , Netease Yunxin has set up a special booth at the global Internet architecture Conference , Attracted a large number of on-site audience consultation and discussion .