Technology at home and abroad people "see" the future of audio and video technology

planning / Teresa

since COVID-19 Since the outbreak , Although it breaks people's work 、 The rhythm of life , But it also accelerates the development of audio and video technology to a certain extent . This accelerated development is more than a vertical iteration of Technology , It also launched a horizontal touch to different scenes . When many activities are physically restricted , Such as ： social contact 、 shopping 、 entertainment 、 Work, etc. , We can all do it online , Even more and more rely on this way . Of course, behind this is the support of technicians . Regarding this , We did a review , Whether at home or abroad , Let's see what specific technology development people are more optimistic about on the vertical track of audio and video technology . in addition , On the basis of these contents , We extend to this 8 month 5 Beginning of the day LiveVideoStackCon 2022 Audio and Video Technology Conference Shanghai Station , Let's see what bigwigs and content will appear .

The following is a collection of technologies at home and abroad. People start from different directions of audio and video technology , Let's talk about what technology prospects they are optimistic about .

Streaming technology

“ We talked about that before , The future will also reaffirm ： Video is subverting business . The medical field Medical devices that are relying on connected video 、 Remote surgical capability and equipped with AI Monitoring intelligent hospital . Whether at home or in the hospital , For medical care , Streaming media technology has become crucial .

stay The consumer sector , With the closing of physical stores , Digital shopping platforms are booming . Those physical stores that continue to attract customers use in store entertainment consumption technologies such as virtual fitting rooms to attract customers .

Then there are technology giants like Google , They have integrated video into all aspects of business strategy ： From the service （Youtube and Google Meet） To streaming media products （Chromebook、Pixel and Nest etc. ）. Even if it's Facebook（ Now renamed Meta） The whole strategy has also been transferred to AR and VR Such video technology .

meanwhile , Streaming technology Provide users with extensive accessibility , Give more power to creators . Blockchain 、 encryption 、Web3 Waiting is driving decentralization . next , New profit models that avoid big companies will develop .”

—— Excerpt from 《2022 Ten technological trends of overseas streaming media 》

“ technology + Industry scenario ” This aspect is also what we have been paying attention to . therefore , We are LiveVideoStackCon 2022 Shanghai railway station specially launched 「 Audio and video +」 project , Starting from different industry scenarios , Look at the penetration of audio and video technology .

For more details about this topic, please scan the code or long press the QR code

AI And video codec

“LiveVideoStack: You think based on AI Will our codec soon surpass traditional coders ？ Or will the two encoders coexist for a long time ？

Leonardo Chiariglione： Traditional data processing technology will still show its strength , But in many areas , It has come to the end of its development . And artificial intelligence , Just getting started .

A few days ago , I published an article （ See https://blog.chiariglione.org/the-governance-of-the-mpai-ecosystem/）.

In this paper, , I wrote ：“ If one AI After careful training, the machine , It can be found that some specific coding modes are more common than others , Then it will probably achieve better compression rate than the coding mode deduced by human according to the mechanism of probability theory .” AI It can explain new things according to experience like human beings , Therefore, the ability of machines to accumulate experience will inevitably increase with the increase of processing and storage capacity . However , Don't expect AI Video coding will surpass traditional video coding in the short term . Although technology has developed rapidly , But the investment in technology in the past is so huge , The deployment of new technologies will inevitably take a long time .

The exciting future is at hand .”

—— Excerpt from 《 dialogue MPEG founder Leonardo Chiariglione: MPEG The spirit will be MPAI Medium continuation 》

---

“LiveVideoStack: For the future development of audio and video technology , What other innovations might be expected ？ stay AI Combined with video coding technology , What breakthroughs are possible in the future ？

Zhang Hao ： I'm looking forward to AI The innovation of image and video coding technology . At present, it is more and more difficult to improve the compression rate based on the traditional video coding framework , We look forward to a new framework . at present AI Video coding is not up to the latest traditional coding standards （ such as VVC） The level of , However, new research results have been released in this direction recently , I believe the performance improvement will accelerate . If the future AI Coding has become a widely used scheme in the industry , The optimization of the encoder may require different technologies . Of course , In complete AI Before coding becomes a reality , There may be an intermediate state , For example, combining traditional architecture and AI Coding technology and standard of module . This requires coding engineers to understand both traditional coding , Also understand AI Knowledge about .

At present, end-to-end AI+ Video coding , Although the results continue to emerge , However, it may be difficult to significantly surpass in the short term VVC Performance of . But based on the traditional coding framework , Add some more efficient coding AI modular （ Like filtering 、 forecast ）, It is possible to improve the compression rate in the short term . Therefore, I am optimistic about the traditional coding framework +AI The technological progress of this idea .”

—— Excerpt from 《 Zhang Hao, Central South University ： I'm looking forward to AI The innovation of image and video coding technology 》

Whether it's AI Video coding , still AI+ Traditional video coding framework , In essence, it still hopes to obtain high-definition images and videos while encoding efficiently . At this Shanghai railway station Conference **「 Video codec performance optimization and implementation 」 project **, We will start with different coding methods , Such as ： Data coding for machine intelligence 、 Efficient coding of high-dimensional visual data based on video , To solve the problems in different video application scenarios .

For more details about this topic, please scan the code or long press the QR code

Audio

“LiveVideoStack: For students who want to engage in audio work in the future , What suggestions and comments do you have ？

Jing Wong ： Compared with computer vision 、 Communication network 、 The development direction of AI and other concepts , At present, there are not many researchers specializing in audio signal processing or audio related technology , But the actual job demand is still quite large , In particular, there is a relative lack of high-level researchers . in fact , People engaged in audio work need to give consideration to both signal processing and computer programming , It is also easy to transition to other technical positions . Audio field （ Broadly speaking, it includes speech and audio processing ） Current and computer technology 、 Communication network 、 Artificial intelligence 、 Virtual reality and even biomedicine are closely integrated , Compared with the tradition, it is only from the perspective of signal processing or computer programming , The solution of many practical application problems tends to need interdisciplinary foundation .

For students who want to engage in audio work in the future , In particular, I want to make a technological breakthrough in combining theory with practice , It is suggested to learn digital signal processing first 、 Information theory 、 computer programming 、 Communication network 、 Basic knowledge such as artificial intelligence , Then master various commonly used algorithms and typical application scenarios of audio signal processing , The choice of research topic or work content can carry out specific research according to the needs of actual application scenarios . At present, there are many open source codes and learning materials on the Internet , Especially based on AI Sound processing technology , It is suggested that students should add more programming practice while systematically learning basic knowledge , In order to better understand the algorithm ideas , At least one or two cases should be implemented independently . Students entering the graduate study stage need to pay more attention to the top conferences and publications in the field , Attend some academic or industry conferences to communicate with peers , Be good at finding problems in research , And use the knowledge learned to analyze and solve .”

—— Excerpt from 《 Talk to Wang Jing ： Audio talents need to be cultivated , High level researchers are particularly lacking 》

On the premise of having a theoretical basis , Practice is particularly important . Especially when facing different scenes , The same audio technology will play different roles . At this Shanghai railway station **「 Near the border — Audio immersion experience 」 project **, Three scenarios will be covered ： Movies 、 Teleconference and 3D Online interactive scene . And what technologies will be used in different scenarios ？ Wait for you to explore ~

For more details about this topic, please scan the code or long press the QR code

Transmission network

“RTC The technical field has its own characteristics , It is easy for technicians in this field to ignore paying attention to users' feelings and appeals . for example ： The feeling of streaming media on the user side is not sensitive , technical HEVC/AV1 Than AVC How many times the compression efficiency is improved , The user may feel that the mobile phone is hot , Does it consume electricity . Publicity is important , But technology should not ignore the user's feelings to talk about progressiveness .

Technology iteration is not a process of Digital Competition , No one with a high digital index will become a mainstream technology , The process of technology iteration is a convergence effect , Being able to fit a certain kind of large-scale application scenario often becomes the mainstream or standard , As practitioners, they should not focus on technical indicators , It is very difficult to beat the industry pioneers with higher technical indicators , Therefore, blind technological refinement in the inherent field is also a kind of complacency , Latecomers should try their best to find broader application scenarios of technology and form new trends .

Post epidemic Era RTC Become a serious area of involution , On the one hand, the terminal capacity has not been upgraded , On the other hand, the application scenario traffic brought about during the epidemic shows signs of fading , Giants run rampant , And the new scene has not yet appeared . But high resolution 、 High bit rate applications such as real-time virtual reality are just budding , Large bit rate will make UDP Agreement to kernel The burden is increasing , High bandwidth and low latency 、 The contradiction of large concurrency will be more acute in the new scene , A new generation RTC Architecture may appear TCP/UDP Twin pattern .”

—— Excerpt from 《 after 5 Generation spanning 25 Year of RTC Architecture evolution history 》

---

“LiveVideoStack：QUIC/HTTP3 More and more popular , Some even suggested that WebRTC adopt QUIC To transmit , What do you think RTP be based on QUIC transmission ？RTP OVER QUIC Is that a good idea ?

Ron Frederick： QUIC It's a very interesting agreement , Same as TCP comparison , It has many advantages , Especially when transmitting real-time data （QUIC Packets can be processed in any order ）. although HTTP/2 Added in a single TCP The ability to multiplex multiple streams over a connection , but TCP Force data to always be processed sequentially , This means that packet loss on one multiplexed stream will prevent data processing in all other streams .QUIC There is potential to solve this problem , It may even evolve into different retransmission strategies that support different streams , This is very beneficial for audio and video content （ Because after a certain delay , Incoming packets will no longer be useful ）. I'm looking forward to seeing the development of this work ！”

—— Excerpt from 《 dialogue RTP author Ron Frederick: I'm looking forward to it QUIC The development of 》

In the world of transmission networks , Pursuing extremely low delay has become a top priority for technologists . Especially in high-throughput media transmission 、 Real time session business 、 Ten thousand people scene , Regarding this , This time **「 Multimedia transmission network optimization 」 project ** Will be discussed in .

For more details about this topic, please scan the code or long press the QR code

Video content production

“AI Technological development in the field of video , It provides a new technical means for the reconstruction of video content . be based on AI The super-resolution technology of can realize standard definition to high definition （SD turn HD）、 Or HD to 4K even to the extent that 8K Resolution improvement , It can make up for a lot of image details ; Based on AI Inverse tone mapping of （Inverse Tone Mapping） Technology and color enhancement technology , Contrast can be achieved 、 Color saturation and other levels of improvement . The details of these improvements , Need to use HDR High dynamic range and wide color gamut of video .NTIRE 2021 For the first time HDR Video image generation technology competition .

Based on typical application scenarios , Intelligent video reproduction can be divided into two categories: intelligent image quality improvement and intelligent old film repair . Among them, intelligent old film repair can greatly improve the efficiency of traditional manual repair , And super points and HDR Then further improve and make up for the details , Adjust brightness and saturation , Try to get close to the truth 4K The level of .”

—— Excerpt from 《HDR Analysis of technology trends 》

Higher definition video content production end is what we pay attention to , Its consumer side is also our concern . This time **「 Video content production and consumption experience innovation 」 project ** in , According to different consumption scenarios , Such as ： Live events 、APP、 Video programs, etc , What problems will you encounter in the corresponding production links ？

For more details about this topic, please scan the code or long press the QR code

---

The agenda of the conference

---

The related audio and video technology topics mentioned above are all in this LiveVideoStackCon 2022 Audio and Video Technology Conference Shanghai Station , On 8 month 5-6 Japan a . When the , There will be more topics to show you . There are more topics waiting for you to unlock , For details, see the special page of the conference or click to read the original ：https://sh2022.livevideostack.cn/topics