当前位置：网站首页>How to correctly evaluate video image quality

How to correctly evaluate video image quality

2022-07-05 09:45:00 【Baidu geek said】

Reading guide ： This article discusses the factors that affect the image quality 、 Why should we continue to improve the quality of video , Then it introduces the importance of image quality evaluation 、 Factors influencing the confidence of video quality evaluation , Finally, it introduces the self-developed image quality evaluation system holy mirror and its business implementation .
The full text 5506 word , Estimated reading time 14 minute .

One 、 introduction

As a video service , Regular needs ：

Compare different videos , To determine which video has a better picture quality , So as to bring a better experience to users .
Compare the video generated by different coding parameters , To determine which configurations produce the best video experience .
Compare the video quality of different products , In order to know each other and each other , And bring better user experience through comparison .

All in all , There's a question that needs to be answered ： Which video has the best picture quality ？ This article will discuss the factors that affect the image quality 、 Why should we constantly improve the video quality 、 The importance of image quality evaluation 、 How to correctly evaluate the video quality 、 Self developed image quality evaluation system —— Holy mirror to introduce how we use holy mirror to objectively 、 confidence 、 Efficient evaluation of video quality , And then enhance the user experience .

Two 、 What are the factors that affect the image quality

When we mention “ video ” when , Generally speaking, it means “ Digital video ”. Visual signals are recorded 、 Compress 、 Storage can produce what we call “ video ”. meanwhile , because HVS(HVS, human visual system, The human visual system ) Characteristics , In video recording 、 Compress 、 In the process of storage , We can make a trade-off between visual quality loss and video data compression , Without affecting the visual quality .

generally speaking , The main factors affecting video quality are as follows ：

The resolution of the ： The number of pixels in the image , Under certain dimensions , The higher the resolution , More pixels , The details displayed are more detailed .
Frame rate ： The number of images displayed in one second , The frame rate of a movie is usually 24fps, The frame rate of a standard TV is usually 30fps.
brightness ： The range of image illumination intensity that can be displayed , The range of brightness that the human eye can perceive is 10-3 Nita, ~ 106 Nita, .
A deep ： The number of colors that can be displayed per pixel , The deeper the bit is , The more colors you can display , This makes the gradient smoother 、 A more natural .
Color gamut ： A color gamut is a subset of a particular color , Used to represent the range of all colors that can be displayed . Color gamut is generally used CIE 1931 The area on the chromaticity diagram ,CIE 1931 The edge of the curve represents the range of colors in the visible spectrum .
Bit rate ： Encoding video per second requires bit The quantity is called the code rate （bitrate）. under certain conditions , The bit rate will affect the video quality , The lower the bit rate , The more compression , The poorer the picture quality is . Of course , The higher the bit rate, the better the picture quality , In many cases , Higher bit rates often result in a waste of bandwidth .

3、 ... and 、 Why should we improve the image quality

We often think that ： Improve many factors that affect video quality , Improve video quality , It will improve the user experience , In turn, it will improve the relevant user indicators of the product . therefore , In line with the principle of "beginning with the end" , We should constantly improve the video quality .

However , Is it true ？ Have we thought about the following questions ：

Is the increase or decrease of video viewing duration caused by the fluctuation of video quality ？
When the user is watching the video , Is video content more influential or is video quality more influential ？
Why? 《 Journey to the west 》 The picture quality of is rough , But it can be played repeatedly 30 many years , And create the highest 96% The miracle of ratings ？
When the resolution is improved 、 Frame rate and other factors , The picture quality has improved , But it also means the bandwidth consumption 、 Calculate the force 、 Electricity, etc. are bound to increase . Power consumption and image quality of mobile phones 、 Between playback fluency and picture quality , Will users choose image quality first ？
Improving video resolution and frame rate also means an increase in bandwidth costs , for example 4K Video ratio 1080P The cost of video bandwidth is high 3~4 times . that , For streaming media service providers , If you still have doubts about the above questions , Why should we improve the image quality ？

2020 year , The double-blind test conducted by Warner Bros. and Pixar found that , Most consumers can't tell 8K And 4K The difference between .

look , The need to improve the image quality is not strong . In that case , Why should we improve the video quality ？ Why is the whole video industry working hard to improve the quality of video ？

1. The resolution of the display device is constantly upgraded . for example ,HUAWEI Mate 40 Pro The screen resolution is 2K(2772x1344) standard . For television ,8K Television has also become a trend . If the resolution of the video itself still stays at 480P Time , So when viewing on a high-resolution screen , There will be an obvious mosaic effect .

2. Video capture equipment is constantly upgraded . for example ,iPhone 13 Pro Can support 4K/60fps Video shooting of , even Android Thousand yuan gear VIVO Y53S, The minimum shooting resolution of the video has also reached 720P. under these circumstances , If the resolution of the video consumer remains at a lower resolution （ for example 480P）, That will cause serious inconsistency between the shooting preview and playback viewing experience .

3. Video coding and decoding algorithms are constantly upgrading . How much hardware resources are there , How complex software products will be produced . The evolving codec algorithm provides the underlying support for higher image quality applications . When you have the technology to make airplanes , Will not be satisfied with just making bicycles .

4. Constantly enriching consumer applications and playing methods requires higher image quality . According to the 《5G Hi Tech video ——VR Video technology white paper (2020) 》（https://t.hk.uy/baZR）, be used for VR Video in video program production and exchange , It is recommended to use at least 7680x3840 sizes . Especially in the meta universe , To feel the color, sound, smell and touch , Too bad picture quality can not meet the requirements .

5. Increasing competitive products . here , The baseline level of video quality depends on the competitive products in the industry . also , Video quality has become an effective means of publicity , for example B Stations 4K/120fps video , Watermelon 4K HD repair technology …… This will virtually form a kind of user's expectation of video quality , When the image quality of the product is lower than this expectation , The user experience will be reduced . Of course , Higher image quality depends on higher performance devices and higher bandwidth . But we need to understand ： Can also get 100 branch , Ability can only reach 100 There are only 100 The gap between the scores is very large .

Four 、 Why is it so important to evaluate video quality

As mentioned earlier , Many reasons promote the continuous improvement of video quality . However , You can't improve without measurement , Without measurement, it is impossible to evaluate whether the image quality has been improved 、 To what extent . in addition , At work , It is necessary to determine the overall requirements of the streaming media system by evaluating the video image quality 、 Compare the services provided by competitive products 、 Determine end-to-end video quality, etc .

Video quality assessment is like a scale , It constitutes the cornerstone of video quality improvement and codec improvement .

4.1 Subjective assessment

The obvious way to measure video quality is to ask users for their opinions , This is what we call ： Subjective assessment . however , Subjective assessment has the following problems ：

Subjective assessment is a tedious process 、 Non automated processes , It is not possible to use subjective assessment to evaluate each video .
Different individuals perceive the same video quality differently , Therefore, the results of subjective evaluation will fluctuate with different evaluators .

4.2 Objective assessment

even so , Subjective assessment is still a valuable method , And it can provide data support for objective video quality evaluation algorithm , for example VMAF It is the result of subjective evaluation modeling using algorithm indicators . Using algorithms to evaluate video quality is called ： Objective assessment .

4.3 Lord 、 The relationship between customer evaluation

It should be noted that , In the field of video quality assessment , Although the algorithm can be fast 、 Efficient 、 Automatic video quality evaluation , But the algorithm is only the fitting of subjective evaluation results 、 forecast . in other words , Objective evaluation algorithm must keep perceptual consistency with subjective evaluation .

Because of that , In practice , Objective algorithms are generally used to measure the market data , The adjustment of video codec 、 Optimization of encoding and decoding parameters and other effect evaluation , Subjective assessment will still prevail .MSU Video Codecs Comparisons In the competition , The result of subjective scoring is also an important indicator , for example HEVC/AV1 2019 The result of the game . As an engineer , We always think about using algorithms 、 Model to solve all problems , However , We must realize that , Not all problems can be solved by algorithms 、 Model . Remember 《 Who moved my cheese 》 Medium “ Hem ” Do you ？

Video quality master 、 Objective evaluation algorithm is not introduced here , More detailed information is available 《 Digital video concept , Methods and measurement indicators 》（https://t.hk.uy/baZT） Video quality indicators .

5、 ... and 、 What are the factors that affect the evaluation confidence ？

Subjective assessment is the cornerstone of image quality improvement , How can we ensure the objectivity of the subjective evaluation results 、 Confidence is the core of video quality evaluation .

In order to ensure the confidence of subjective evaluation ,ITU-T The first 9 The research group （SG9, study group 9） Some suggestions are put forward as subjective evaluation methods , for example ITU-R BT.500.

At work , We found that ： Not in accordance with standards and specifications , The result of subjective assessment is certainly unbelievable . however , With standards and specifications , Perform subjective image quality evaluation according to standards and specifications , Is the result of subjective assessment believable ？

At work , We found that , Is not the case, . The standard provides recommendations , But the specific implementation is limited by various factors , These limited factors will affect the final result confidence .

A seemingly simple thing （ Just ask someone to give a score ）, In fact, it is not simple .

5.1 Questions about evaluation samples

Video samples themselves have a great impact on the results of subjective evaluation of video quality , During the actual evaluation , The sample related problems encountered are mainly as follows ：

1. The evaluation video takes a long time , The scene changes a lot . In this case , Users may not have the patience to watch the entire video , Or jump to watch the video , As a result, some information is ignored in the evaluation process, resulting in poor confidence in the final result .

2. There are products in the evaluated video logo. In the video logo It will bring additional information interference to users' subjective evaluation , And then interfere with the final evaluation results . During the actual evaluation , We also observed in the video logo It will really affect the subjective scoring of users .

3. The scene coverage of the evaluation video is low , Or it only covers some product scenarios . For example, only scenery is selected 、 People and other scenes , And sports are missing , Videos of knowledge and other scenes .

5.2 Problems with profiling tools

1. Video scaling error . In order to be able to compare two videos at the same time , generally speaking , The evaluation tool will play two videos to be evaluated at the same time to let users score subjectively . however , As shown in the figure below , When the video resolution is large （1080x1920）, This method will cause the video display area to zoom , Thus, the places with poor image quality in the video are hidden due to the scaling of the video display area .

2. Error caused by video overlap . In order to solve the last problem , The evaluation tool will add an overlap style as shown in the following figure to avoid the error caused by scaling , At the same time, we also hope that the overlap method can more directly compare the differences between videos .

however , In the actual evaluation , We found that this looks better because there are stages in the video picture , It will also lead to the distrust of subjective evaluation results . The user is in the evaluation process , You need to constantly drag the dividing line of the picture , To observe the difference of image quality in different areas of the video . And for the ROI For encoded video , Use a video area A And another video area B comparing , Itself is very unreasonable .

3. Error caused by video motion . Different from the picture , Video is a sequence of frames that change continuously in time . The motion and change of the video itself will hide the image quality of some areas of some frames in the video to a certain extent . That's what we often say HVS Motion masking effect . To avoid this error , Especially for the R & D personnel of codec algorithm , It is often hoped that the video can be compared frame by frame , To discover potential hidden risks . Although such as QuickTime、VLC And other players provide single frame playback capability , However, the evaluation tools are less capable of providing single frame comparison .

4. Ignore errors caused by differences in playback devices . In the work , We have found more than once that PC Evaluation tool to evaluate the playback effect of mobile devices . Using the tools under the guidance of standards and specifications, we get a result , But it ignores the difference of display effect under different devices , Eventually, the result that should have been believed becomes very unbelievable . Especially for HDR In terms of video , The difference of equipment will be more obvious .

5.3 The cognitive differences of the evaluators

1. The error caused by the evaluator's inertial cognition . For the comparative evaluation of the fixed sequence , We find that the evaluator always predicts the following results based on the results of the previous pairs of samples , Then score according to the predicted results , Instead of scoring according to your own subjective perception . This kind of inertial cognition of prediction will bring great interference and error to the evaluation .

2. The error caused by the different focus of the evaluator . Different evaluators pay different attention to the same picture, resulting in large fluctuations in the evaluation results, which often occurs in the actual evaluation work . For example, users from the background 、 prospects 、 brightness 、 When evaluating from different angles, such as beauty , The scores given are also different .

6、 ... and 、 Image quality evaluation tool —— Spiritual state

after 2 More years , Experience close to 500+ Subjective assessment ,1000+ Evaluate the sample ,1000+ Evaluators （ Experts and ordinary users ）, Wade through one pit after another , Use technology to solve one after another of the distrust factors encountered in the actual evaluation , Gradually built a more confident video quality evaluation system —— Spirit mirror .

The holy mirror is based on ITU standard , Relying on self-developed 10+ Patented technology , With the help of Baidu's powerful video basic technology , On the basis of continuous practice, it supports multiple terminals （PC,Android,iOS） Video quality evaluation service for evaluation .

At the same time, holy mirror also provides a one-stop service for video image quality evaluation , covers ： Full link image quality assessment from video production to video consumption , Picture quality 、 Cost and other comprehensive evaluation and analysis .

In terms of evaluation methods , The holy mirror supports PC、Android、iOS Horizontal screen on the device / Vertical screen , Experts / Ordinary users , Full screen / Non full screen, etc 24 Three evaluation modes , Some of the evaluation modes are as follows ：

1. Single screen evaluation

2. Overlap evaluation on the same screen （ Support at the same time DCR and CCR Two evaluation models , And to ensure confidence , Overlap and contrast the same screen by comparing the video with the same area , You can slide the video area to select videos from different areas for comparison ）

3. Expert evaluation

4. Same screen evaluation

besides , The holy mirror also provides ：

camera Mock Ability , To replace the collected data of the camera with the specified video
Objective assessment of image quality , for example VMAF,PSNR,SSIM etc. , about PSNR, The holy mirror also provides spatial visualization , For more objective evaluation

Video attribute computing power , for example SITI, chroma , saturation , brightness , Bit rate , Frame rate , Resolution, etc
Video processing & Detection capability , For example, static frame detection , Monochrome frame detection , Sound and picture synchronization detection , Abnormal video structure, etc

The mirror passes through layered thoughts —— Infrastructure layer 、 Holy mirror tool layer 、 Holy mirror service layer , The upper layer encapsulates the lower layer , While ensuring the flexibility of use, the service is more in line with the business requirements .

For business , Holy mirror provides the following closed-loop management of the whole process of video quality evaluation service ：

at present , Lingjing has served Baidu FEED, good-looking , live broadcast 、 Network disk 、 Video quality evaluation of Xiaodu, etc & testing , And play an important role . At the same time, holy mirror has also proved its confidence in practice .

Next , We will continue to improve the power of the holy mirror ：