Camera coordinate system, world coordinate system, pixel coordinate system conversion, and Fov conversion of OPENGLDEFocal Length and Opengl

目的：Understand how cameras work,Rendered and camera-taken depth maps

Recent studies on camera parameters and openglParameter relationship for the rendered image.It will be encountered in the process of going from the camera coordinate system to the pixel coordinate systemproject 矩阵.

Theoretical camera transformation

Camera-based learning of theoretical knowledge.Normally we build with camera parametersprojectGenerally need to go through the following two processes：
相机坐标系->图像坐标系->像素坐标系
Suppose we get a vertex in the camera coordinate system$p(x_c,y_c,z_c)$,Convert to vertices on the image$p(x_i,y_i)$.It's converted to a matrix as $M_{proj}$
Below we will describe a few steps：

1. 相机坐标系到图像坐标系$M_{p2c}$
   相同的位置,Different cameras will get different pictures.It can be seen that this process is generally related to the parameters of the camera.Some parameters of the camera are generally in the camera's manual,Instructions for the corresponding styles can also be found online,Some even inside the configuration file（For example, the image camera camera parameters in the scanner are all available）.In order to understand the function of these parameters,We need to understand the principle of camera imaging（小孔成像）.
   
   The coordinate system in the image is $(O^{\prime }x,O^{\prime }y)$, 相机坐标系为$(O_c{x}_c,O_c{y}_c,O_c{z}_c)$ , 方便理解 ,Transform the viewable view.因为$(O^{\prime }x,O^{\prime }y)$ 和$(O_c{x}_c,O_c{y}_c,O_c{z}_c)$ 在$(O_c{x}_c,O_c{y}_c)$ 平行.So according to geometrical symmetry,Move the image coordinate system to the mirrored position,如下图.其中 $\angle O_{c}AB$为直角.The pictures are not very accurate.

因为相似三角形（1）：
$$△AB{O}_c\sim △oC{O}_c$$
得到下面公式:
$$\frac{O_{c}o}{O_c{Z}_c}=\frac{oC}{AB}=\frac{O_{c}C}{O_{c}B}(1)$$
Another way of expressing（因为$O_c$ 为原点,changed symbols）为下面：
$$\frac{O_{c}o}{O_c{Z}_c}=\frac{f}{Z_c}(2)$$
$$\frac{oC}{AB}=\frac{x}{X_c}(3)$$

因为相似三角形（2）：
$$△AB{O}_c\sim △oC{O}_c$$
得到下面公式：
$$\frac{O_{c}C}{O_{c}B}=\frac{Cp}{BP}(4)$$
Another way of expressing（因为$O_c$ 为原点,changed symbols）为下面：
$$\frac{Cp}{BP}=\frac{y}{Y_c}(5)$$
因为$(1)(2)(3)(4)(5)$公式得到：
$$\frac{f}{Z_c}=\frac{x}{X_c}=\frac{y}{Y_c}$$
It is further transformed into the formula as follows：
$$x=\frac{f\ast X_c}{Z_c}(6)$$
$$y=\frac{f\ast Y_c}{Z_c}(7)$$
把其中$(6),(7)$The formula written in matrix form is obtained as follows：
$$\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\\ 1\end{array}\right]$$
The resulting matrix equation$M_{p2c}$为下面：
$$M_{p2c}=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]$$
A shorthand formula can be obtained：
$$\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=M_{p2c}\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\\ 1\end{array}\right]$$
这时候获取的$\left[\begin{array}{c}x\\ y\\ z\end{array}\right]$for physical units.不是像素为单位,Therefore, it needs to be converted into pixels,It requires the specific physical length of each pixel.Camera is requiredsensor参数.The camera coordinates to pixel coordinates will be introduced below.

2. The image coordinate system is converted to the pixel coordinate system$M_{i2p}$
   The camera coordinate system needs to be converted to a pixel coordinate system,It involves two issues,
   1）Two coordinate system problems.in the image coordinate system,Usually the origin is at the center point.And our traditional pixel display coordinate system origin is the upper left corner of the image,This vector needs to be translated.
   2）Unit conversion problem：Need to know how many physical units each pixel has(一般是mm),The physical unit is millimeters.conversion for both,需要知道$d_x,d_y$Indicate how much each column and each row represent, respectivelymm,表示一列的宽度$1pixel=d_{x}mm$;一行的宽度$1pixel=d_{y}mm$,一般情况下$d_x=d_y$.因此$x$The pixels represented by the coordinates are $\frac{x}{dx}$,相应的$y$The pixels represented by the coordinates are $\frac{y}{dy}$
   The two coordinate systems are shown as follows：
   
   The following formula is obtained by transforming the translation coordinate system：
   $$u=\frac{x}{d_x}+u_0(8)$$
   $$v=\frac{y}{d_y}+v_0(9)$$
   把其中$(8),(9)$The formula written in matrix form is obtained as follows：
   $$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{d_x}& 0& u_0\\ \frac{1}{d_y}& 0& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$
   The matrix formula is obtained as ：
   $$M_{i2p}=\left[\begin{array}{ccc}\frac{1}{d_x}& 0& u_0\\ \frac{1}{d_y}& 0& v_0\\ 0& 0& 1\end{array}\right]$$

Summarize the above two transformations：
$$M_{proj}=\left[\begin{array}{ccc}\frac{1}{d_x}& 0& u_0\\ \frac{1}{d_y}& 0& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]$$
Finally, the formula projection matrix is ​​obtained as ：
$$M_{proj}=\left[\begin{array}{cccc}\frac{f}{d_x}& 0& u_0& 0\\ 0& \frac{f}{d_y}& v_0& 0\\ 0& 0& 1& 0\end{array}\right]=\left[\begin{array}{cccc}{f}_x& 0& u_0& 0\\ 0& f_x& v_0& 0\\ 0& 0& 1& 0\end{array}\right]$$
Understand relationships through formulas,得到$f_x=\frac{f}{d_x}$,它表示focal lengthHow many pixels are there（This is the pixel unit）,同理$f_y=\frac{f}{d_y}$Also indicates how many pixels there are,一般情况下$d_x=d_y$,且focal length为一个,上图中的$f$.

The summary is as follows：
$$M_{proj}=\left[\begin{array}{cccc}{f}_x& 0& u_0& 0\\ 0& f_x& v_0& 0\\ 0& 0& 1& 0\end{array}\right]$$

OpenglCoordinate rendering theory

基于相机的OPENGL理论知识.Normally we build with camera parametersproject需要fov,aspect ratio,near,far比较多.Among them we found that there are4parameters in the specific rendering,The parameters needed to be able to render the space accurately,and they are easy to understand,And cut a space,Used for normalization operations,Calculations that are easy to render.For details, please BaidulearnopenglAbout camera settings.It sets about the above4个参数.它的project有两种,Below I will describe how they work：

如果不转化为fov,aspect ratio,near,far,可以直接project矩阵,通过glFrustum完成.但是在学习opengl渲染中,通常使用的gluPerspective,The parameters it uses are fov,aspect ratio,near,far.我们可以直接在shader里面设置project Matrix to get renderedproject矩阵.It is also imitationgluPerspective生成project矩阵, 同时,It can also be done by converting to the above4个,由opengl的gluPerspective帮你生成project矩阵.这两种方法都可以.
1）The parameters of the camera are converted to fov,aspect ratio,near,far这四个参数.
First understand these four parameters：
1）Field of view (FOV), aspect ratio
Aspect ratio is the x/y ratio of the final displayed image
FOV：Indicates the angle of the open field of view,见下图.
2）near,far Indicates the distance of the far and near tangent planes from the origin,See the two parallel planes in Fig.

The relationship can be seen in the figure above：

If you need to render a picture consistent with the image, the picture is represented asimg,We set by imageaspect ratio和FOV
根据定义,ratio is the x/y ratio of the final displayed image,
The calculation formula is as follows：
$$aspect=\frac{img.cols}{img.rows}$$
Bring in the above formula,topcorresponds to halfy轴,That is, in the pixel coordinate systempixelwidth,为
$$top=\frac{pixelwidth}{2.0}$$
其中near对应于focal length（像素坐标系）It's in pixel coordinates.单位统一.
The latter is converted into the formula as follows：
$$FOV=2.0\ast atanf(\frac{pixelwidth}{2.0\ast focalLength})\ast \frac{\pi }{180.0}$$
对于near 和far The distance can be set as close and as far as possible.
Its matrix is ​​as follows：

设置后,The rendering is the same as the image captured by the camera.

The derivation process is added later.