[camera Foundation (II)] camera driving principle and Development & v4l2 subsystem driving architecture

One 、 Import

Two 、 Camera data output format

1、USB、VS、MIPI Camera architecture

• MIPI The resolution supported by the interface camera 、 Higher frame rate , Low power .
• MIPI Than USB A lower cost ,USB Cameras are expensive .
• USB The workload of camera development is small , Generally, they are free from driving .

2、RAW RGB DATA
RAW RGB yes sensor Of bayer array Acquired data （ Each sensor obtains the corresponding color brightness ）, camera sensor After photoelectric conversion, the output data is RAW data（RAW RGB）. stay Sensor in , Each sensitive spot can only be sensitive to light RGB One of the colors . therefore , What is usually said 30 Megapixels or 130 Megapixels, etc , It means to have 30 Wanor 130 Ten thousand light spots , Each sensitive spot can only be sensitive to one color .
RGB： Traditional red green blue format . such as RGB565, Its 16bit The data format is 5bitR+6bitG+5BitB,G More than a , The reason is that human eyes are more sensitive to green .

3、RAW RGB And RGB The difference between

• RAW RGB Each pixel has only one color （R、G、B One of the ）
• RGB Each pixel has three colors , Each value is in 0-255 Between .
• from sensor Output data RAW DATA（RAW RGB）, after ISP The color interpolation transformation becomes RGB.

4、YUV
Y Represents the luminance component （Luma）： If only Y Words , The image will look like a black-and-white photo .
U（Cb） Represents the chromaticity component ： It's the blue part of the picture that removes the brightness （Y）.
V（Cr） Represents the chromaticity component ： It's the red part of the picture that removes the brightness （Y）.
①YUV The advantages of ：
And RGB comparison , His advantage is that it takes up less space
②YUV Sampling format for
There are three main ways of sampling ,YUV4：4：4,YUV4：2：2,YUV4：2：0
image quality ：YUV4：4：4>YUV4：2：2>YUV4：2：0

YUV4：4：4 sampling , every last Y Corresponding to a group of UV component . intend 4 The data in pixels has 4 individual Y,4 individual U,4 individual V.
YUV4：2：2 sampling , every last Y Corresponding to a group of UV component . intend 4 The data in pixels has 4 individual Y,2 individual U,2 individual V.
YUV4：2：0 sampling , every last Y Corresponding to a group of UV component . intend 4 The data in pixels has 4 individual Y,1 individual U,1 individual V.
③YUV The storage format of
YUV The storage format of is divided into packaging format （packet formats） And flat format （planar formats）.

• about planar Of YUV Format , First, store all pixels in a row Y, Subsequent storage U、V. The storage type is represented by the sampling method suffix plus P, Such as YUV422P、YUV420P.
• about packet Of YUV Format , Every pixel point Y,U,V It's a continuous alternate storage . The storage type is represented by the sampling method suffix plus SP, Such as YUV422SP、YUV420SP.
  
  One frame YUV420 Image ,u and v The size of the space occupied by the component is... Of the original number of pixels 1/4, The total size occupied is widthheigth（3/2）
  One frame YUV422 Image ,u and v The space occupied by the components is half of the original pixels , The total size occupied is widthheigth2
  5、MJPEG/H264 Coding format
  Some cameras sensor Support output MJPEG/H264 coded , Because it has built-in DSP processor , take YUV Output after data encoding MJPEG/H264 stream .
  problem ： When code is needed ？ What's the role ？
• Internet transmission , Reduce bandwidth .
• Video coding compatible , Unified video coding standard .

3、 ... and 、 Video coding principle

1、 The history of video coding

2、 Video coding principle
I frame ： Keyframes , Using intra frame compression technology .
P frame ： Forward reference frame , In compression , Only refer to the previously processed frames , Using inter frame compression technology .
B frame ： Two way reference frame , In compression , It refers to the previous frame , Also refer to the frame behind it , Using inter frame compression technology .
except I/P/B Out of frame , And image sequences GOP.
GOP： Two I Between the frames is an image sequence , There is only one frame in an image sequence . On the premise of constant bit rate ,GOP The bigger the value is. ,P、B The more frames there will be , Average each I、P、B The more bytes a frame takes , It's easier to get better image quality . Pictured ：

Four 、V4L2 Subsystem driven architecture

V4L2： yes linux The kernel driver framework of video devices in the kernel , It provides a unified interface for the upper layer to access the lower layer video devices .

V4L2 Drive the core ： It is mainly to build a standard video device driver framework in the kernel , Provide unified interface function for video operation .
platform V4L2 Device drivers ： stay V4L2 Within the framework of , According to the characteristics of the platform, the platform related V4L2 Drive part , Including registration video_device and v4l2_dev
Concrete sensor drive ： Main power on 、 Provide working clock 、 Video image clipping 、 flow IO Turn on, etc , Implement various device control methods for the upper layer to call and register v4l2_subdev.
UVC： It's a kind of usb Video device driver . Used to support usb Video device , Anyone who usb Interface cameras can support .

V4L2 The core source code of is located in drivers/media/v4l2-core, According to the function, it can be divided into four categories ：

• Character device module ： from v4l2-dev.c Realization , Main function application character main equipment number 、 register class And to provide video device register 、 Logout and other related functions .
• v4l2 Basic framework L: from v4l2-device.c、v3l2-subdev.c、v3l2-fh.c、v4l2-ctrls.c Wait for the file to build v4l2 Basic framework .
• videobuf management ： from videobuf2-core.c、videobuf2-dma-contig.c、videobuf2-dma-sg.c、videobuf2-memops.c、videobuf2-vmalloc.c、v3l2-mem2mem.c Wait for the file to be implemented , complete videobuffer The distribution of 、 Manage and log off .
• loctl frame ： from v4l2-ioctl.c File implementation , structure v4l2 ioctl Framework .

V4L2 Registration process ：

videobuf management ：

V4L2 ioctl frame
User space for V4L2 The operation of the equipment is basically ioctl To achieve ,V4L2 The equipment has a large number of operable functions （ Configuration register ）, therefore V4L2 Of ioctl It is also very huge .

V4L2 The driver mainly uses ioctl The command values are as follows ：

5、 ... and 、 Camera video capture process

Camera data stream ：

6、 ... and 、 Common typical problems

1、 The camera picture is stuck

• Insufficient frame rate
• usb Not enough bandwidth
• Video capture format setting error ：YUV、MJPEG or H264 Format
• Encoding and decoding problems
• CPU Take up high

2、 The camera image rotates 90 Degree or 180 Degree problem

3、 The camera picture cannot be opened