Hisilicon 3559 universal platform construction: introduction to YUV format

Preface

   Want to build a universal adaptation platform , The second step in the long march is YUV The conversion of color space in the format , Only when you are familiar with this format and modification method can you make adjustments according to your needs

Definition

  YUV The model is based on a luminance (Y component ) And two shades (UV component ) To define the color space , There are many common formats for subdividing , Expand the detailed description in the storage mode section . For example, the default image format of the camera we use under the Hisilicon platform is NV21 Or call it YUV420SP Format
  YUV The principle is to change the brightness （Luma） And chromaticity （Chroma） Separate .
  “Y” Indicates brightness , That is, gray value .
  “U” Indicates the difference between the blue channel and the brightness .
  “V” Indicates the difference between the red channel and the brightness .
   among Y The signal component represents the brightness signal , It also contains more green channels , pure Y Component can display a complete black-and-white image .
  U、V The components represent blue respectively (blue)、 red (red) Component signal , Contains only chromaticity information , therefore YUV Also known as YCbCr, among ,Cb、Cr The meaning of is equal to U、V,C It can be understood as component perhaps color.
   also ,YUV Unlike RGB That requires three independent video signals to be transmitted at the same time , So use YUV Mode transfer takes up very little bandwidth .

Storage format

   There are two main types of formats ,packed and planer（ Later, we will talk about the derived form sandwiched in the middle ）
  packed That is, for example, there is a corresponding pixel Y0U0V0 Y1U1V1 Y2U2V2 Y3U3V3, First, store all pixels in a row Y, And then it stores all the pixels U, And then all the pixels V. Is equivalent to YUV Split into three planes (plane) Storage .
  planer Namely Y0Y1Y2Y3 U0U1U2U3 V0V1V2V3, Every pixel point Y,U,V It's a continuous alternate storage

   It looks like it's with RGB Change the soup without changing the dressing . Don't worry don't worry , Listen and analyze slowly ~

RGB Reciprocal transformation

   talk about rgb By the way, and RGB The conversion formula ,

   It can also be reflected in the code

Y      =  (0.257 * R) + (0.504 * G) + (0.098 * B) + 16
Cr = V =  (0.439 * R) - (0.368 * G) - (0.071 * B) + 128
Cb = U = -(0.148 * R) - (0.291 * G) + (0.439 * B) + 128

B = 1.164(Y - 16) + 2.018(U - 128)
G = 1.164(Y - 16) - 0.813(V - 128) - 0.391(U - 128)
R = 1.164(Y - 16) + 1.596(V - 128)

   Obviously , A single picture has a lot of floating-point operations , Fortunately, the Hisilicon platform has ready-made hardware acceleration , We don't need to turn every pixel .
   stay RGB In the color space , The three colors are equally important , So you need to use the same resolution for storage , Use... At most RGB565 This form reduces the accuracy of quantization , But the amount of data is still very large .
   Because the study found that the human eye is more sensitive to brightness than chromaticity . The brightness information and color information of the image are separated , And use different resolutions for storage , In this way, on the premise of little impact on subjective feelings , Image data can be stored more effectively

YUV Value range of

   And RGB The value range of each component of each pixel is 0-255 Different （ Each component accounts for 8bit）,YUV There are two value ranges ：
  1. With Rec.601 As a representative （ It also includes BT.709 / BT.2020） In the radio and television standards of ,Y The range of phi is zero 16-235,U、V The range of phi is zero 16-240.FFmpeg Referred to as “mpeg” Range .
  2. With JPEG Is represented by ,Y、U、V The range of values of 0-255.FFmpeg Referred to as “jpeg” Range .
In practice, the most common is the 1 Of a range of values YUV（ You can observe for yourself YUV The data of , It will be found that the luminance component is not taken as 0、255 Something like that ）. Many people have doubts in this place , Why did you remove “ Both sides ” What's the value of ？
reason ：
  • Very low and high values are not transmitted in radio and television systems , In fact, it is to prevent overload caused by signal change , So put this “ Both sides ” The value of “ Protective belt ”.

YUV Sampling format for

   The main sampling formats are YUV4:4:4、YUV4:2:2、YUV4:2:0 , among YUV4:2:0 Is the most commonly used sampling format .
   Sampling is taking values according to a certain interval . The proportion refers to Y、U、V Pixels represented , The ratio of the three .
   The picture below is YUV4:4:4,YUV4:2:2,YUV4:2:0 Sampling diagram of , The black dot represents the Y component , Use a white circle to indicate the UV component .

  • YUV 4:4:4 sampling , every last Y Corresponding to a group of UV component .
  • YUV 4:2:2 sampling , Every two Y Share a group UV component .
  • YUV 4:2:0 sampling , Every four Y Share a group UV component .

YUV storage

   See here , You will find yuv A quintessence of , Start from the storage mode with RGB Drifting away
   The following shows the common YUV How data is stored , And attached at the back is a sample of each pixel YUV Method of data . Start with YUV The number of ！

(1) YUYV Format （ Belong to YUV422）

   Two adjacent Y Share two of its neighbors Cb、Cr, For pixels Y’00、Y’01 for , Its Cb、Cr Values are Cb00、Cr00, Other pixel points YUV And so on .

(2) UYVY Format （ Belong to YUV422）

   And YUYV The difference is UV In different order , Restore the YUV The value method is the same as above .

(3) YUV422P（ Belong to YUV422）

  YUV422P It's a flat pattern , Every pixel of it YUV The value extraction method also follows YUV422 The most basic extraction method of format , That is the two one. Y Share one UV. such as , For pixels Y’00、Y’01 for , Its Cb、Cr Values are Cb00、Cr00.

(4) YUV420P Format （YVU420P）（ Belong to YUV420）

  YUV420P,Y,U,V All three components are in plane form , It is divided into YU12（ Under the Android platform, it is also commonly called I420） and YV12.I420 Format and YV12 The difference in format is U Flat and V The position of the plane is different . Be careful , Above picture ,Y’00、Y’01、Y’10、Y’11 share Cr00、Cb00, And so on . stay I420 In the format ,U The plane follows Y After the plane , Then it is V Plane （ namely ：YUV）; but YV12 On the contrary （ namely ：YVU）. Actually , Just take care to extract from the right place U and V value ,YU12 and YV12 Can use the same algorithm to process .

I420: YYYYYYYY UU VV

YV12: YYYYYYYY VV UU

   Obviously ,YV12 and I420 There's no big difference , Changed UV It's just order I420 Also called YU12,,IYUV

(5) YUV420sp It is divided into NV12、NV21（YVU420SP）, Also belong to YUV420.

   The difference between the two formats is UV The order of interleaving is different , It's a kind of two-plane Pattern , namely Y and UV Divided into two Plane, however UV（CbCr） For interleaved storage , Instead of being divided into three plane. Its extraction method is similar to the previous one , namely Y’00、Y’01、Y’10、Y’11 share Cr00、Cb00, And so on .
   Suppose a resolution is 8X4 Of YUV Images , Their format is as follows ：
  YUV420sp The format is as follows

NV12: YYYYYYYY UVUV

NV21: YYYYYYYY VUVU

   alike ,NV12 and NV21 It's also an exchange uv The order of

Size calculation

   With w*h The size of the image YUV420 Take the data ,
   Its storage format is ： The total size is (w * h * 3/2) byte ,
  Y component ：(w * h) Bytes
  U(Cb) component ：(w * h/4) Bytes
  V(Cr) component ：(w * h/4) Bytes

A special case

   Consider the alignment mechanism adopted by Hisilicon for hardware acceleration , The actual data may be like this , If we want to make changes, remember to consider padding The problem of （ The initial resolution is just right , The next article will mention ）