Audio and video technology development weekly | 252

Once a week , Overview of audio and video technology in the field of dry goods .

Press release ：[email protected]

13 Fund for Web The popularity of HTML5 Video player
When video streaming media swept the communication world , In order to maintain and improve user growth , Content creators and streaming media service providers need to embed HTML5 Video player . In this paper , In the future, we will learn about the available HTML5 Video player .

Severe Tire Damage： The first rock band in the world to broadcast live on the Internet
1993 year 6 month 24 Japan ,Severe Tire Damage There was a live performance on the Internet （ be based on MBONE）, This performance is of symbolic significance to the development of Internet and audio and video technology .Severe Tire Damage It has also become the first band to perform live on the Internet .

The evolution of screen display technology
With the continuous development and update of hardware equipment and streaming media technology , Screen display technology is also evolving . today , Let us follow the footsteps of history , Let's review the important milestones in the development of screen display technology .

Introduction to the video broadcasting and control system of the opening ceremony of the Beijing Winter Olympics
The opening ceremony of the Beijing Winter Olympics uses a lot of scientific and technological means to present the effect , As one of the core control systems of the opening ceremony , Played an important role . This paper deals with 2022 The video broadcasting and control system of the 2008 Beijing Winter Olympics is briefly introduced .

Audio and video test -- Video characteristic test
Open an online stopwatch on your computer , After starting the timer . Two devices to be tested are fixed in front of the screen , After the call , After a period of stabilization , Pick up your phone and take pictures , Is the time delay , Take photos here 10 Time , Calculate the difference and then take the average , Is the delay .

Cross platform player development ( Two ) QT for Linux & FFmpeg Environment building
In the last article, we were 「MAC OS」 Under the platform QT and FFmpeg development environment , This article mainly introduces how to 「Linux」 Under the platform QT and FFmpeg development environment .

I'll take you through VideoEditor Video export process
In this issue, we will continue to discuss VideoEditor One of the highlights —— Export video . After all, we clip videos , Add all kinds of good-looking and interesting special effects and music , All for exporting videos , There are four important points in exporting video .

The journey of audio and video development （15） OpenGL ES particle system - fountain
Face a relatively large or untried project or content , Fear 、 Cowardice often appears , At this time, you should clarify your goals , Hold on to the main line , Use structured thinking , Disassembly process , Then gradually realize each link , Solve the problems in each link , This is also the upgrade process of fighting monsters , Now let's enjoy this process .

RTX 30 series GPU： adopt AV1 Decoding opens a new era of video content
NVIDIA announce RTX A series of support AV1 decode , Enable through hardware level AV1 decode , Can handle up to 8K Of HDR flow .AV1 Is more efficient than H.264 high 50%, This means that only half of the Internet bandwidth is required to transmit the same video quality , and AV1 And support 10 Bit code .

https://www.nvidia.com/en-us/geforce/news/rtx-30-series-av1-decoding/

AOM Eco development email Group 2022Q2
AOM union （Alliance for Open Media） Ecological Development Group Q2 Email release , The content includes recent industry information 、AV1 progress 、AV1 Resources, etc .

https://storage.googleapis.com/downloads.aomedia.org/assets/pdf/AOMedia%20Decoder%20-%20Q2%202022%20Non%20Members.pdf  

 
AV1 Film particle synthesis tool
Film particles exist in many films 、 In TV , Although its essence is noise , But as part of the creative content , We hope to be able to retain film particles in the coding process .AV1 The coding tool for film particle synthesis is provided in , And as AV1 Part of the standard presents .

AMD RDNA 3 Architecture support AV1 codec
AMD Shared about support Radeon RX 7000 Video card RDNA 3 New details of the architecture , Confirm support AV1 codecs 、DisplayPort 2.0 Interface 、5nm Process technology 、 senior GPU encapsulation 、 Graphics pipeline optimization 、 The next generation Infinity Cache, And compare RDNA 2 Improve 50% Above energy efficiency .

https://gadgettendency.com/even-more-incredible-gpu-frequencies-displayport-2-0-av1-and-more-amd-shared-details-about-the-rdna-3-architecture/

NVIDIA stay FFmpeg To realize AV1 VDPAU Hardware acceleration
NVIDIA by FFmpeg Multimedia library provides support , In order to use the latest generation NVIDIA RTX 30“Ampere”GPU Can pass VDPAU API To take advantage of AV1 GPU Accelerated video decoding .

https://www.phoronix.com/scan.php?page=news_item&px=NVIDIA-AV1-VDPAU-FFmpeg

Audio PCM / WAV Format,
PCM(Pulse Code Modulation) Also known as pulse code modulation ,PCM The sound data in is not compressed , It's an analog signal sampled 、 quantitative 、 Encoded into standard digital audio data .

How to build a human voice synthesis system with high performance
Speech synthesis technology is an important part of human-computer interaction , The ultimate goal is to achieve a synthetic effect comparable to that of a real person . High performance speech synthesis has gradually become a trend in the future . High performance speech has three remarkable characteristics ： The rhythm is natural 、 Rich emotional style and clear sound quality . therefore , We aim at these three characteristics , Explore the algorithm , Form the fifth generation speech synthesis technology of Dharma hall .

Squealing suppression solution
The public address system has been used since , Often accompanied by howling problems , It greatly affects the user's sense of experience . Whistling can mask normal speech , It doesn't sound good , And the howling frequency point energy is very high , In serious cases, it can even damage the sound reinforcement equipment in the meeting , So we need to suppress the howling .

For the first time, researchers have realized the control and modulation of sound waves on a chip
Sound waves are slower than electromagnetic waves of the same frequency , But in the world of high-speed computing and communication , This is not a bad thing . Now? , From Harvard University SEAS For the first time, the researchers demonstrated the control and modulation of sound waves by using the electric field on the chip .

8K HDR!｜ by Chromium Realization HEVC Hard solution - principle / Actual measurement guide
This paper gives a brief account of Web Status quo of decoding scheme , The author is Chromium Browser implementation & Improve the problems encountered in the hard solution process and the implementation principle , The test results are attached at the end of the paper , Precompiled version for reference , I hope it can be solved FrontEnd bitter HEVC A long-standing problem .

Video codec document 、 Software and open source IP
This video takes the loop filter module as an example , Introduce the development and learning ideas from documents to software and then to hardware , It mainly includes open source hardware IP、 Hardware simulation 、 be based on PYNQ Of XK264 Presentation scheme, etc .

MediaCodec 、x264、faac Realize audio and video coding and pass rtmp The protocol realizes streaming
Let's learn this article together Android Terminal rtmp There are several stages that streaming must go through ： Including acquisition 、 Handle 、 code 、 Push flow, etc , Let's see .

H.264 Introduction
H.264 Also known as MPEG-4 , It's a block oriented , Video coding standard based on motion compensation , Is the most commonly used video coding format on the market , The first purpose of this article is to summarize knowledge , Second, it is to give a reference to the students who have just started audio and video .

Android AVDemo（10）： Video unpacking , from MP4 figure out H.264/H.265 -- audio and video engineering examples
In this audio and video project example , We will collect... Through disassembly → code → encapsulation → decapsulation → decode → Render the process and implement Demo Let's introduce how to iOS/Android Audio and video development on platform . Here is Android Chapter 10 ：Android Video unpacking Demo.

Scene adaptive transcoding system with controllable image quality
B The station receives hundreds of thousands of video submissions every day , It will consume most of the bandwidth resources .B The station will re transcode the video , On the premise of maintaining the same image quality , Remove data redundancy , To increase the compression ratio , Reduce bit rate , Avoid the waste of bandwidth resources . In order to improve the performance of video transcoding ,B The station has developed a scene adaptive transcoding system with controllable image quality .

3GPP XR Research on relevant standards
3GPP It solves the problems related to mobile communication , Meet the new network requirements introduced by the rapid development of transmission content and interaction .3GPP The standards and specifications developed are based on Release Manage as version , On average, one version will be completed in one to two years , It has developed to Rel-18.

Audio and video communication protocol --RTSP agreement
RTSP As an application layer protocol , Provides an extensible framework , Make the control and on-demand of streaming media possible , It is mainly used to control the transmission of data with real-time characteristics , But it itself is not used to transmit streaming media data , And must rely on the underlying transport protocol ( Such as RTP/RTCP) To complete the transmission of streaming media data .

China has made positive contributions ,ITU-R Finish as scheduled 6G Research Report on future technology trends
2022 year 6 month 13 Japan -24 Japan , International Telecommunication Union radio communication department 5D The working group held the 41 meeting ,ITU-R WP5D Completed on schedule 《 Research Report on future technology trends 》 The writing of . In our country IMT-2030（6G） As China 6G Industry university research is the main promotion platform for research and cooperation of all parties , Contribute Chinese wisdom to report writing , And undertake the editor of important chapters of the report .

TCP Two problems of flow control
Two basic questions , It can be used as an interview topic ：1. TCP window scale What's the biggest ？ Why? ？2. TCP Does a single stream have an upper throughput limit ？ If there is , What is it? ？ without , Why? ？

https://zhuanlan.zhihu.com/p/533881330

6 A picture will show you how to understand TCP Why three handshakes ？
TCP Why three handshakes ？ We need to find out the problem , First of all, understand TCP How to ensure reliable transmission . Now let's take a look at .

Sound network self-developed transport layer protocol AUT Landing practice of Dev for Dev special column
For the new demands and challenges brought by real-time interactive applications to network transmission , The voice network layer and decouple the application layer business requirements and transmission strategies in real-time interaction , On 2019 Self developed internal private transport layer protocol AUT, Bring together various transmission control capabilities under heterogeneous networks , This article will introduce in detail AUT The design and evolution of transmission protocol .

---

Collection of award-winning questions

The technology boss is ready , topic of conversation C It's up to you
In the coming LiveVideoStackCon 2022 Audio and Video Technology Conference Shanghai Station 8 month 5-6 Japan , We set it up 【 Technology business strategy （ round table ）】 project , Now? , We specially plan “ Round table pre communication ” Activities , Formally solicit the round table discussion of the Conference , Welcome to ask questions . We will get feedback from the three round tables , choose 3 A little friend who asks wonderful questions （ Each round table 1 name ）, Send out LiveVideoStack A commemorative refrigerator sticker ！ Hurry up ~. Click here 「 Sign up for the meeting 」.

---

Design principle of image signal processing chip ——13 Sharpen the image
This series mainly introduces the design of each core algorithm module in image signal processor and related cutting-edge research , Based on a typical camera imaging system , The contents involved include all kinds of defect correction , To mosaic , Denoise ,3A Algorithm , Over score ,HDR, Themes such as style transfer . This paper introduces the image sharpening operation closely related to image sharpness .

Joint denoising and mosaic removal based on deep learning
This article introduces an article published in 2020 year CVPR Paper on joint denoising and mosaic removal based on deep learning . This paper first introduces the basic concepts of de mosaic and de-noising , Introduce this article again paper Main content , Finally, a brief summary .

CVPR 2022 | Self enhanced unpaired image defogging based on density and depth decomposition
In this paper , We propose a self enhanced image defogging framework , be called D4（Dehazing via Decomposing transmission map into Density and Depth）, Used for image defogging and fog generation . Our proposed framework does not simply estimate transmission images or clear images , Instead, it focuses on exploring the scattering coefficient and depth information in foggy images and clear images .

Multimix： A small amount of supervision from medical images , Interpretable multi task learning
In this paper , I will discuss a new kind of semi supervision , Multi task medical imaging methods , be called Multimix,Ayana Haque（ME）,Abdullah-Al-Zubaer Imran,Adam Wang、Demetri Terzopoulos. The paper was ISBI 2021 Included , And in 4 At the meeting in October .

Open source ISP processor (xkISP) Release
xkISP Is based on Xilinx Open source image signal processor of development tool (ISP), By Fudan University VIP Lab and Alibaba DAMO CTL Laboratory joint development . up to now ,xkISP Support processing arbitrary resolution 12 Bit original image data .

TGU： Open source neural network processor
This paper introduces the open source neural network processor in songqingzeng's Laboratory of Tianjin University of Technology TGU.TGU It is a general configurable convolutional neural network accelerator , Support CNN,Relu,LeakyRelu,MaxPool,concat Wait for more than ten neural network operators .

CVPR22 ｜CMT：CNN and Transformer The efficient combination of （ Open source ）
to the end CNN and Transformer Which is better? ？ Of course, it's best to work together . Researchers from Huawei Noah lab have proposed a new vision network architecture CMT, By simply combining traditional convolution and Transformer, The network performance obtained is better than that proposed by Google EfficientNet,ViT and MSRA Of Swin Transformer.

FFmpeg Command analysis -yuv encapsulation mp4
This series mainly analyzes various FFmpeg command How is it implemented in the code , With FFmpeg4.2 The source code shall prevail . This time, we will explain yuv data Code as H264, And then encapsulate it into MP4 In the format .

https://juejin.cn/post/7086893134172389407

Gu Gexin AI became angry ！ The longest word in the world can be drawn ：Pneumonoultramicroscopicsilicovolcanoconiosis
The latest one proposed by Google AI——Parti, It is mainly to model text generated images as sequences and between sequences . structurally , All its components have only three parts ： Encoder 、 Decoder and image marker , And they are all based on standards Transformer.

Quantum neural network for generating learning tasks 2022 The latest review
This paper summarizes the latest progress of quantum generative learning model from the perspective of machine learning . We interpret these quantum generative learning models as quantum extensions of classical generative learning models , Including quantum circuit born machine 、 Quantum generated countermeasure network 、 Quantum Boltzmann machine and quantum automatic encoder .

AI The algorithm realizes “ Peerless martial arts ”—— Action aftereffect ！
Now the film making technology is more and more superb , Wushu special effects are really cool , For example, the afterimage effect really gives people a feeling that martial arts are unfathomable , How does that cool afterimage effect come true ？ Today, let's use Baidu open source deep learning framework to achieve such a video effect .

The illustration ： Mathematical principle analysis of convolution neural network
This time, , We will deepen our understanding of how neural networks work in CNNs. Out of suggestion , This article will cover quite complex mathematical equations , If you're not used to linear algebra and differential , Please don't be discouraged . My goal is not to make you remember these formulas , It's about giving you an intuitive idea of what's going on . 

Share 5 Commonly used feature selection methods , Introduction to machine learning ！！
In many books related to machine learning , It's hard to find content about feature selection , Because the problem to be solved by feature selection is often regarded as a sub module of machine learning , Generally, it will not be discussed separately . This article will combine Scikit-learn The examples provided introduce several common feature selection methods , Their respective advantages, disadvantages and problems .

Reading a text is based on DL Application scenario of unmanned visual perception system
Computer vision based on deep learning , Applied to driverless visual perception system , It is mainly divided into four parts ： Dynamic object detection 、 Access space 、 Lane line detection 、 Static object detection , This article mainly from the demand 、 difficulty 、 Analyze each perception part from three aspects .

Xiaomi's intelligent driving scheme was accidentally exposed ：5 Millimeter wave radar +1 camera , Realization L2 Class a autopilot
An official propaganda news in mainland China , It seems that the intelligent driving scheme of Xiaomi's first car has been exposed ：5R1V—— namely 5 Millimeter wave radar 1 Camera scheme ,L2 Class a autonomous driving ability , Carried in 2024 On the first electric vehicle in . Rice noodles friends , For this plan ,Are you OK？

ALITA: Large scale incremental data set for autonomous driving
This article will share a large-scale incremental data set for autonomous driving , This data set can be used to evaluate the performance of actual scenarios . Data sets , And for data processing and local evaluation Python-API Is open source .

Five thousand words to clarify the basic automotive software and domestic status
What is the basic software of automobile ？ By definition , It is used to realize the decoupling of software and hardware of automobile system , It has nothing to do with user application functions , But a series of supporting software collections that provide automotive system services . Generally speaking, it is board level chip driver 、 On board operating system 、Hypervisor And middleware .

Point cloud registration , In autopilot “ Drop blood to recognize one's relatives ”
With lidar 、4D Millimeter wave radar has gradually become the top stream in the automotive field , Its output point cloud (Point Cloud) It has also become a major data format to describe the three-dimensional world of vehicles after pixels . Point cloud is actually a data set , The point cloud output by different types of sensors contains slightly different data .

Chat 7 A common smart car technology
Although the appearance of the concept car is too avantgarde , Technology is also too detached , However, the concept embodied in the concept car can sometimes be realized in the real car . Some of the techniques in the listing may seem incredible . A few years later , You won't be surprised by these innovations .

‍WebXR Current situation and future
W3C Immersive Web Co chair of the working group Ada Rose Cannon around “WebXR The present and the future ” Focus on using existing API What applications can be built and the new features to be launched .

Use OpenCV Tag based augmented reality
Tag based AR, Also known as image recognition AR, Use an object or reference mark as a reference to determine the position or direction of the camera . Location based AR By scanning image ArUco Mark such a mark to work .ArUco Tag detection triggers an enhanced experience to locate objects 、 Text 、 Video or animation for display on the device . In this case , We will write a simple code , With the help of ArUco Tag to enhance the image on the video stream .

AR And VR How technology affects the way brands interact with users ？
from VR Social interaction in the game to AR Personalized online shopping experience ,AR and VR In many ways, it has become the lifeline for many companies to establish flexibility and increase customer participation for the future . however , How does this multi billion dollar industry affect the way brands interact with customers ？ Let's look at various scenes , Show the high-value solutions that these technologies can provide .

Wireless multi person interaction based on edge computing VR Game system
Wireless multi person interactive virtual reality （VR） The game also has VR High computing load and unpredictable interaction of multiplayer interactive games , This brings great challenges to the design of wireless communication system . We propose a mobile edge based computing （MEC） Wireless multiplayer interaction VR Game transmission framework .

What can virtual reality be used for （ One ）
Virtual reality （Virtual Reality, abbreviation VR） Appearance , Make the human simulation level achieve a qualitative leap , Reach a new level . Do you know what virtual reality can be used for ？

---

Read recommendations

A picture to understand the investment in the field of virtual reality in the first half of the year
2021 From the year onwards , Yuancosmos has become a popular word of traffic all over the world , Drive a new round of investment boom in China's virtual reality industry , According to the 《 China electronic news 》 Incomplete statistics ,2022 year 1-6 month , China VR/AR/XR The investment and financing situation in the field of and meta universe is as follows .

Google Refute a rumor and give up TensorFlow, It is still alive ！
I don't know if it was before “TensorFlow Will die ” The rumor of has spread too far ,Google I sent an urgent document to advertise it a few days ago ,TensorFlow No, “ die ”, Now it is developing very well , meanwhile ,Google And I didn't give up on continuing development TensorFlow, In the future, it will work with JAX amen .

Rust, The best choice for programmers to start a business ？
Startups are often under pressure to choose programming languages , Especially when considering choosing a relatively small number of new languages . You need to consider not only the syntax of the programming language itself 、 performance , We also need to pay attention to its attraction to talents . However , The ultimate choice of this startup company is Rust. Let's take a look at their experience ！

Sixty years of deep learning
from 1958 Frank . The perceptron invented by Rosenblatt 、RNN、LeNet-5 To Transformers wait , The former people drove the deep learning forward step by step . This paper mainly focuses on computer vision , Take you back to every milestone of wisdom condensation .

Talk about network security in automatic driving
stay 《 Fast and furious 8》 There is such a fragment in , Hackers find loopholes in car chips , Control the cars parked on the roadside and in the garage . When I first saw this clip , I'm shocked , Although it is the effect after artistic rendering , But I can't help thinking , Will this scene really appear in real life ？

---

Activity recommendation

LiveVideoStackCon 2022 The recruitment of lecturers for Beijing railway station has started ！

11 month 4-5 Japan ,LiveVideoStackCon 2022 Beijing railway station will continue to explore the integration and development of audio and video technology in different scenes , Here, you can not only exchange technical experience with the industry leaders , You can also feel the leading companies in the multimedia ecosystem 、 Top players are aware of the current development trend of the industry 、 Bottleneck challenge , And an in-depth interpretation of future planning .LiveVideoStackCon It's everyone's stage , If you're on a team 、 In the company , Years of experience in a field or technology , And is keen on technical exchanges , Welcome to apply to become LiveVideoStackCon Lecturer of .

Click on 「 Read the original 」, You can sign up at the bottom of the web page , You can also view the rights and interests of lecturers and application conditions .

Or submit Speech content + Personal introduction To email ：[email protected]

We will review and inform you of the final results as soon as possible .