This article explains in detail the difficult problems and solutions faced by 3D cameras

Click on the above “ Computer vision workshop ”, choice “ Star standard ”

The dry goods arrive at the first time

The author 丨 HawkWang

Source   Computational photography  

One . 3D Camera challenges

As you have in the last article 52. Time of flight technology of light (TOF series 2) Seen in China , Include TOF Including technology 3D Imaging technology has made great progress since its birth , And it has been used in many fields .

However ,TOF There are also some key challenges in the practical application of the technology . In this article, I will introduce these challenges and experts' solutions one by one .

image TOF Such active light 3D Imaging technology , The first challenge is the impact of ambient light when used outdoors . You can see , If 3D The light source of the camera is the only light source in the scene , Then the whole service conditions are very consistent with its working principle . But outside , The ambient light may be much stronger than the camera's own light source , Make the result wrong .

Let's take an example , This video uses a RGBD The camera pointed at a door , We can see in direct sunlight , The depth of some areas may be wrong or missing .

Another challenge is the influence of scattering media , Because these media will lead to multiple optical paths , Affect the judgment of distance .

A typical example is as follows , This is a video of the autonomous vehicle competition sponsored by Hyundai in Korea , A autonomous vehicle is driving in sunny weather , It's natural to turn the corner ：

And on another rainy and foggy day , When this car challenges the same curve again , But hit the curb ：

If the above problems are only caused by the uncertainty of the outdoor environment , But even in a controlled indoor environment ,3D The camera will also encounter challenges , For example, because of the geometry of the target , Light is reflected many times , It leads to the distortion of distance judgment

Or because the target material is changeable , This leads to multiple light paths , Affect the judgment of distance

from 1860 Year to now ,3D Camera as well as in speed 、 Analytic force 、 Accuracy and other aspects have been greatly improved , And the future 3D The camera will become more and more suitable for various scenes —— And this needs to start with solving these challenges .

Two . How to deal with the influence of ambient light

As you saw just now , Ambient light affects the judgment of distance

So an obvious way is to take an image that blocks its own light source , The difference between the two images is used to obtain the image illuminated only by its own light source ：

If you think so , So congratulations , You're wrong O(∩_∩)O

This is due to the wave particle duality of light —— When we think like that , Potentially we think of light as a continuous wave . But in fact, this time is a kind of particle , So it is a discrete random process that it arrives at the camera sensor and is captured

There may be an average of... Per unit time 3 Photons enter the sensor , But because of the nature of discrete stochastic processes , So that sometimes there may not be a single photon , Sometimes more photons enter the sensor

This uncertainty is caused by a noise called photon noise (Photon Noise) Is a random variable , therefore , The actual measured value is the real measured value + The sum of noise , Here, the random variable photon noise obeys Poisson distribution , Its standard deviation is just equal to the square root of its mean . Photon noise is a very basic property of light - No matter how good your camera is , Even the human eye - Every camera experiences photon noise .

When ambient light is present , Photon noise will be contributed by both light source and ambient light

Because the intensity of sunlight is much greater than the light source , Therefore, photon noise mainly comes from sunlight :

Therefore, the overall signal-to-noise ratio will be very low ：

Look at this picture , You can see that in almost all cases , The sun is brighter than 3D The light source of the camera is strong 2-5 An order of magnitude . This is the outdoors 3D Why imaging is so challenging

A classic solution is to reduce noise by averaging multiple shots ：

When the number of captured images increases , The overall signal-to-noise ratio is also slowly improving ：

Although this method is useful , But you need to increase the number of shots , This reduces the real-time performance of the system . There is another problem , The light emitted by the light source is evenly distributed on the scene , The light intensity received by each target point in the scene is reduced , Therefore, the signal-to-noise ratio is lower in a single shot ：

Then another means of improvement is to change the scattered light source to the focused light source , And divide the scene into N Parts of , Only a part of each shot , Still shooting N Images , Then the signal-to-noise ratio will be further improved

Compared with the method of simply average denoising , At this time, the improvement of signal-to-noise ratio is very obvious

If the spotlight is good , Why don't we focus all the light on one point ？ This is the method used by many commercial lidar systems . Um. , Although the signal will be very high , But we must illuminate the scene one point at a time . This scanning is a very slow process .

Of course , The other extreme is the averaging method , Where the light is scattered throughout the scene . As we can see , To improve the signal-to-noise ratio , It takes a lot of images , This is also very slow .

therefore , Some scholars have proposed , There must be some optimized distribution of light sources , Intuitively speaking , The optimal distribution should be small enough , So we don't have to do any averaging , It should be big enough , So we don't have to do too many scans .

University of Wisconsin Madison Mohit Gupta The professor put forward such an expression ：

The dispersion of the light source is determined by a system constant The control of the , And it is inversely proportional to the intensity of ambient light , This means that when used outdoors on sunny days , Because the sun is very strong , So use a more focused light source . In the night scene , The scattering area of the light source can be correspondingly increased , Reduce shooting time and improve real-time performance .

Gupta The professor designed such a prototype to solve the adaptive problem of light source dispersion , You can change the rotation speed of the polygon mirror , To change the size of the scattering surface ：

Here are some of their experimental results , It was a fine morning 11 Click on the captured image , The ambient light at this time is about 75000 Lux

Use three methods , With the same number of shots , We can see that the adaptive method mentioned above is the best

At noon 12 spot (90000 Lux)、 Afternoon 1 spot (94000 Lux) The reconstruction of Columbia University also confirmed this view , That is, the best way is to control the diffusion of light , Adapt it to the environment

We have mentioned O'Toole stay 2015 Year of Homogeneous Codes for Energy-Efficient Illumination and Imaging The benefits of centralized active lighting are shown in

He and his team designed such a system ： It contains a small laser projector with low power consumption , Its emission intensity is only 30 Lumen , And a pair of carefully calibrated roller shutter cameras . There is also a controller to synchronize the light source with the camera ：

Such a system makes it possible to obtain high-quality 3D reconstruction results under strong ambient light

The results of the computational photography Laboratory of Northwestern University MC3D: Motion Contrast 3D Laser Scanner Push this principle further . This system uses a special sensor , It is only sensitive to specific time-domain motion changes , Not sensitive to static ambient light , This makes it very close to an ideal 3D The camera , Because it has a higher signal-to-noise ratio , The following video compares it with Kinect The result of reconstructing the same moving object

3、 ... and . summary

up to now , You've seen something like TOF This kind of active lighting 3D Several problems faced by cameras in practical use . Special , You should understand why when used outdoors 3D The camera will face challenges , And facing this challenge, the most important thing is to adjust the light source distribution , Adjust the sensing range of the sensor , Improve signal-to-noise ratio .

that , How to solve other problems ？ This is what I will introduce in my next article , Coming soon

Four . Reference material

Most of this article is for reference Mohit Gupta Professor at 2015 year ICCV Tutorial Courseware in ：Core Challenges Faced by Time-of-Flight Cameras, The best part of this courseware is that every page has the teacher's speech , I would like to pay tribute to the teacher's open spirit ！ Our next few articles will also continue to follow his ideas TOF Camera challenges and solutions .

This article is only for academic sharing , If there is any infringement , Please contact to delete .

blockbuster ！ Computer vision workshop - Study Communication group Established

Scan the code to add a little assistant wechat , You can apply to join 3D Visual workshop - Academic paper writing and contribution   WeChat ac group , Aimed at Communication Summit 、 Top issue 、SCI、EI And so on .

meanwhile You can also apply to join our subdivided direction communication group , At present, there are mainly ORB-SLAM Series source code learning 、3D Vision 、CV& Deep learning 、SLAM、 Three dimensional reconstruction 、 Point cloud post processing 、 Autopilot 、CV introduction 、 Three dimensional measurement 、VR/AR、3D Face recognition 、 Medical imaging 、 defect detection 、 Pedestrian recognition 、 Target tracking 、 Visual products landing 、 The visual contest 、 License plate recognition 、 Hardware selection 、 Depth estimation 、 Academic exchange 、 Job exchange Wait for wechat group , Please scan the following micro signal clustering , remarks ：” Research direction + School / company + nickname “, for example ：”3D Vision  + Shanghai Jiaotong University + quietly “. Please note... According to the format , Otherwise, it will not pass . After successful addition, relevant wechat groups will be invited according to the research direction . Original contribution Please also contact .

▲ Long press and add wechat group or contribute

▲ The official account of long click attention

3D Vision goes from entry to mastery of knowledge ： in the light of 3D In the field of vision Video Course cheng （ 3D reconstruction series 、 3D point cloud series 、 Structured light series 、 Hand eye calibration 、 Camera calibration 、 laser / Vision SLAM、 Autopilot, etc ）、 Summary of knowledge points 、 Introduction advanced learning route 、 newest paper Share 、 Question answer Carry out deep cultivation in five aspects , There are also algorithm engineers from various large factories to provide technical guidance . meanwhile , The planet will be jointly released by well-known enterprises 3D Vision related algorithm development positions and project docking information , Create a set of technology and employment as one of the iron fans gathering area , near 4000 Planet members create better AI The world is making progress together , Knowledge planet portal ：

Study 3D Visual core technology , Scan to see the introduction ,3 Unconditional refund within days

  There are high quality tutorial materials in the circle 、 Can answer questions 、 Help you solve problems efficiently

Feel useful , Please give me a compliment ~