CVPR 2022 oral Dalian Institute of technology proposed SCI: a fast and powerful low light image enhancement method

Three models  easy.pt  medium.pt difficult.pt, Every model 43k,

Toward Fast, Flexible, and Robust Low-Light Image Enhancement

The paper ：https://arxiv.org/abs/2204.10137

Code ：https://github.com/vis-opt-group/SCI

This paper presents a new low illumination image enhancement scheme ： Self calibration illumination learning （SCI）. By constructing a weight sharing illumination learning process that introduces a self calibration module , The complicated design process of network structure is abandoned , It achieves the purpose of using only simple operations for enhancement . A lot of experimental results show that ,SCI In visual quality 、 Computational efficiency 、 Breakthroughs have been made in the application of downstream visual tasks （ See the picture 1）. The study has been CVPR 2022 Included as Oral.

chart 1 The results of this method are compared with those of other methods

Research background

Low illumination image enhancement is a classical task in image processing , It has received extensive attention in both academic and industrial circles .2018-2020 Three consecutive sessions of the UG2+Prize Challenge The competition will take low illumination face detection as the main competition unit , It has greatly promoted the academic research on low illumination image enhancement technology . A mobile phone manufacturer in 2019 The conference will focus on the dark light shooting ability , It has set off another wave of industry using deep learning technology to solve the problem of low illumination image enhancement .

Existing low light image enhancement techniques focus on building data-driven depth networks , Usually its network model is complex , Resulting in low computational efficiency 、 Reasoning is slow , And because of the dependence on the distribution of training data, its performance in unknown scenarios is not guaranteed . in general , The existing technology generally lacks practicality . To solve the above problems , This article focuses on learning strategies , Build a fast 、 Flexible and robust low illumination image enhancement scheme .

Methods of this paper

（1） Weight sharing illumination learning

according to Retinex theory , The low light observation image is equal to the dot product of the clear image and the light , namely

. In the design method based on this model , The estimation of illumination is usually regarded as the main optimization objective , After getting accurate illumination , Clear images can be obtained directly from the above relationship . Inspired by the stage by stage illumination optimization process of the existing work , In this paper, we construct a progressive lighting optimization process , Its basic units are as follows ：

among

And respectively mean t Phase residuals and illumination . Represents the illumination estimation network . It should be noted that this is independent of the number of stages , That is, at each stage, the illumination estimation network keeps the shared state of structure and parameters . Further understanding of this module will lead to , Under the mechanism of progressive optimization and parameter sharing , Each stage wants to get output close to the goal . let me put it another way , Is there a possibility , Be able to make the output of each stage as close to and consistent with the goal as possible , thus , The multi-stage cascade test becomes the single-stage test , Will greatly reduce the cost of reasoning . To achieve this goal , A self calibration module is introduced as follows .

chart 2 The algorithm flow chart of this paper

（2） Self calibration module

The purpose of this module is to analyze the relationship between each stage , Ensure that the outputs at different stages of the training process can converge to the same state . The formula expression of the self calibration module is as follows ：

among

It is the input for the next stage after calibration . in other words , The input of the second stage and later in the original illumination learning process has become the result of the above formula （ The overall calculation process is shown in the figure 2 Shown ）, That is, the basic unit of the lighting optimization process is reformulated as ：

actually , The self calibration module introduces physical laws （ namely Retinex theory ）, The input of each stage is gradually corrected to indirectly affect the output of each stage , Then the convergence between stages is realized . chart 3 The function of self calibration module is explored , You can find , The introduction of self calibration module makes the results of different stages converge to the same state quickly （ That is, the results of the three stages coincide ）.

chart 3 Whether to use the enhanced results of self calibration module in the test phase t-SNE Distribution comparison （ The number of stages is 3）

（3） Unsupervised loss function

In order to better train the proposed learning framework , This part designs an unsupervised loss function , To constrain the illumination estimation at each stage , The formula is as follows ：

The former term and the latter term represent the data fidelity term and the smooth regularization term respectively （ For a detailed description of each variable, see the paper ）.

experimental result

（1） quantitative analysis

surface 1 The famous MIT-Adobe FiveK Comparison of quantitative results on data sets , It can be seen that , The proposed method achieves optimal performance . It is worth noting that , Although the proposed method is unsupervised , But in the PSNR And SSIM The results of this kind of reference index are optimal , The reason lies in the Ground Truth It is decorated by experts , It also shows that the results generated by the proposed method are more in line with human visual habits .

surface 1 stay MIT-Adobe FiveK Comparison of quantitative results on data sets

（2） Visual contrast in real scenes

chart 4 It shows the comparison of enhancement results of two groups in difficult real scenes . It can be seen that , Compared with other methods , The proposed method has moderate brightness 、 Rich in details 、 Natural tone 、 With higher visual quality .

chart 4 Comparison of enhancement results in real scenes

（3） Downstream mission performance analysis

In order to further explore SCI The advantages of , This paper compares the performance of two downstream tasks, face detection in low light and semantic segmentation at night . In the task of low illumination face detection , It defines two kinds of and SCI Related versions , One is to SCI As a preprocessing, the brightness of the data is enhanced （ Other comparison methods adopt the same method ） And fine tune the detection network based on the data , The other is SCI Joint fine tuning with detection network （ Write it down as SCI+）. chart 5 The test results are shown in , It can be seen that , The method proposed in this paper has obvious advantages , It can detect more small targets .

chart 5 Low light face detection results comparison

chart 6 It shows the performance of semantic segmentation at night , It can be seen that ,SCI The numerical results with competition are obtained , At the same time, it is more accurate in classification , More clear edge depiction .

chart 6 Comparison of semantic segmentation results at night

Summary and prospect

What this article puts forward SCI It has made a breakthrough in image quality and reasoning speed , It provides a new perspective for low illumination image enhancement , That is, how to endow the network model with stronger depiction ability under limited resources , It is believed that this perspective can also inspire other related visual enhancement fields . future , The author will continue to explore how to design more effective learning tools to build lightweight 、 Robust 、 A low illumination image enhancement scheme for more challenging real scenes .