Current state of supervised and unsupervised depth completion methods

Related tags

Deep Learningbenchmarkmachine-learningawesomecomputer-visionvoidsupervised-learningunsupervised-learning3d-reconstruction3d-visionkittiself-supervised-learningdepth-completion
Overview

Awesome Depth Completion

Table of Contents

About Sparse-to-Dense Depth Completion

In the sparse-to-dense depth completion problem, one wants to infer the dense depth map of a 3-D scene given an RGB image and its corresponding sparse reconstruction in the form of a sparse depth map obtained either from computational methods such as SfM (Strcuture-from-Motion) or active sensors such as lidar or structured light sensors.

Example 1: VOID dataset (indoor VIO)

Input RGB image Sparse point cloud Output point cloud from KBNet

Example 2: KITTI dataset (outdoor lidar)

Input RGB image Output point cloud from ScaffNet

Current State of Depth Completion Methods

Here we compile both unsupervised/self-supervised (monocular and stereo) and supervised methods published in recent conferences and journals on the VOID (Wong et. al., 2020) and KITTI (Uhrig et. al., 2017) depth completion benchmarks. Our ranking considers all four metrics rather than just RMSE.

Quick Links

Unsupervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 39.80 95.86 21.16 49.72
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow 60.68 122.01 35.24 67.34
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow 85.05 169.79 48.92 104.02
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 151.86 222.36 74.59 112.36
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 178.85 243.84 80.12 107.69

Supervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 59.40 181.42 19.37 46.56

Unsupervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 256.76 1069.47 1.02 2.95
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow 280.76 1121.93 1.15 3.30
Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data ACCV 2020 PyTorch 280.42 1095.26 1.19 3.53
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow 299.41 1169.97 1.20 3.56
A Surface Geometry Model for LiDAR Depth Completion RA-L & ICRA 2021 Tensorflow 298.3 1239.84 1.21 3.76
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 343.46 1263.19 1.32 3.58
DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion ITSC 2019 PyTorch 429.93 1206.66 1.79 3.62
In Defense of Classical Image Processing: Fast Depth Completion on the CPU CRV 2018 Python 302.60 1288.46 1.29 3.78
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 350.32 1299.85 1.57 4.07
Semantically Guided Depth Upsampling GCPR 2016 N/A 605.47 2312.57 2.05 7.38

Supervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Non-Local Spatial Propagation Network for Depth Completion ECCV 2020 PyTorch 199.5 741.68 0.84 1.99
CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion AAAI 2020 N/A 209.28 743.69 0.90 2.07
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 203.96 832.94 0.85 2.10
Adaptive context-aware multi-modal network for depth completion TIP 2021 PyTorch 206.80 732.99 0.90 2.08
PENet: Towards Precise and Efficient Image Guided Depth Completion ICRA 2021 PyTorch 210.55 730.08 0.94 2.17
FCFR-Net: Feature Fusion based Coarse- to-Fine Residual Learning for Depth Completion AAAI 2021 N/A 217.15 735.81 0.98 2.20
Learning Guided Convolutional Network for Depth Completion TIP 2020 PyTorch 218.83 736.24 0.99 2.25
DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network ICRA 2021 N/A 214.13 755.41 0.96 2.25
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 PyTorch 220.41 762.19 0.98 2.30
Sparse and noisy LiDAR completion with RGB guidance and uncertainty MVA 2019 PyTorch 215.02 772.87 0.93 2.19
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 N/A 220.41 762.19 0.98 2.30
Learning Joint 2D-3D Representations for Depth Completion ICCV 2019 N/A 221.19 752.88 1.14 2.34
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image CVPR 2019 PyTorch 226.50 758.38 1.15 2.56
Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints ICCV 2019 N/A 235.17 777.05 1.13 2.42
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 233.34 809.09 1.06 2.57
Confidence propagation through cnns for guided sparse depth regression PAMI 2019 PyTorch 233.26 829.98 1.03 2.60
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 249.95 814.73 1.21 2.80
Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End CVPR 2020 PyTorch 251.77 960.05 1.05 3.37
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation 3DV 2019 N/A 234.81 917.64 0.95 2.17
Depth coefficients for depth completion CVPR 2019 N/A 252.21 988.38 1.13 2.87
Depth estimation via affinity learned with convolutional spatial propagation network ECCV 2018 N/A 279.46 1019.64 1.15 2.93
Learning morphological operators for depth completion ACIVS 2019 N/A 310.49 1045.45 1.57 3.84
Sparsity Invariant CNNs 3DV 2017 Tensorflow 416.14 1419.75 1.29 3.25
Deep Convolutional Compressed Sensing for LiDAR Depth Completion ACCV 2018 Tensorflow 439.48 1325.37 3.19 59.39
Owner
I am a post-doctoral researcher at the UCLA Vision Lab under the supervision of Professor Stefano Soatto.
GitHub Repository
Code for "Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency" paper

UNICORN 🦄 Webpage | Paper | BibTex PyTorch implementation of "Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency" pap

118 Jan 06, 2023
An ever-growing playground of notebooks showcasing CLIP's impressive zero-shot capabilities.

Playground for CLIP-like models Demo Colab Link GradCAM Visualization Naive Zero-shot Detection Smarter Zero-shot Detection Captcha Solver Changelog 2

Kevin Zakka 101 Dec 30, 2022
🔪 Elimination based Lightweight Neural Net with Pretrained Weights

ELimNet ELimNet: Eliminating Layers in a Neural Network Pretrained with Large Dataset for Downstream Task Removed top layers from pretrained Efficient

snoop2head 4 Jul 12, 2022
GNN-based Recommendation Benchmark

GRecX A Fair Benchmark for GNN-based Recommendation Homepage and Documentation Homepage: Documentation: Paper: GRecX: An Efficient and Unified Benchma

73 Oct 17, 2022
Complete-IoU (CIoU) Loss and Cluster-NMS for Object Detection and Instance Segmentation (YOLACT)

Complete-IoU Loss and Cluster-NMS for Improving Object Detection and Instance Segmentation. Our paper is accepted by IEEE Transactions on Cybernetics

290 Dec 25, 2022
A unified framework for machine learning with time series

Welcome to sktime A unified framework for machine learning with time series We provide specialized time series algorithms and scikit-learn compatible

The Alan Turing Institute 6k Jan 08, 2023
Iranian Cars Detection using Yolov5s, PyTorch

Iranian Cars Detection using Yolov5 Train 1- git clone https://github.com/ultralytics/yolov5 cd yolov5 pip install -r requirements.txt 2- Dataset ../

Nahid Ebrahimian 22 Dec 05, 2022
Deep Residual Networks with 1K Layers

Deep Residual Networks with 1K Layers By Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Microsoft Research Asia (MSRA). Table of Contents Introduc

Kaiming He 856 Jan 06, 2023
Official implementation of "Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets" (CVPR2021)

Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets This is the official implementation of "Towards Good Pract

Sanja Fidler's Lab 52 Nov 22, 2022
The official project of SimSwap (ACM MM 2020)

SimSwap: An Efficient Framework For High Fidelity Face Swapping Proceedings of the 28th ACM International Conference on Multimedia The official reposi

Six_God 2.6k Jan 08, 2023
MEDS: Enhancing Memory Error Detection for Large-Scale Applications

MEDS: Enhancing Memory Error Detection for Large-Scale Applications Prerequisites cmake and clang Build MEDS supporting compiler $ make Build Using Do

Secomp Lab at Purdue University 34 Dec 14, 2022
Computer Vision application in the web

Computer Vision application in the web Preview Usage Clone this repo git clone https://github.com/amineHY/WebApp-Computer-Vision-streamlit.git cd Web

Amine Hadj-Youcef. PhD 35 Dec 06, 2022
deep learning model that learns to code with drawing in the Processing language

sketchnet sketchnet - processing code generator can we teach a computer to draw pictures with code. We use Processing and java/jruby code paired with

41 Dec 12, 2022
Fast, accurate and reliable software for algebraic CT reconstruction

KCT CBCT Fast, accurate and reliable software for algebraic CT reconstruction. This set of software tools includes OpenCL implementation of modern CT

Vojtěch Kulvait 4 Dec 14, 2022
Pytorch implementation for "Implicit Semantic Response Alignment for Partial Domain Adaptation"

Implicit-Semantic-Response-Alignment Pytorch implementation for "Implicit Semantic Response Alignment for Partial Domain Adaptation" Prerequisites pyt

4 Dec 19, 2022
This implementation contains the application of GPlearn's symbolic transformer on a commodity futures sector of the financial market.

GPlearn_finiance_stock_futures_extension This implementation contains the application of GPlearn's symbolic transformer on a commodity futures sector

Chengwei <a href=[email protected]"> 189 Dec 25, 2022
Puzzle-CAM: Improved localization via matching partial and full features.

Puzzle-CAM The official implementation of "Puzzle-CAM: Improved localization via matching partial and full features".

Sanghyun Jo 150 Nov 14, 2022
Code repository for the paper: Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild (ICCV 2021)

Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild Akash Sengupta, Ignas Budvytis, Robert

Akash Sengupta 149 Dec 14, 2022
Static-test - A playground to play with ideas related to testing the comparability of the code

Static test playground ⚠️ The code is just an experiment. Compiles and runs on U

Igor Bogoslavskyi 4 Feb 18, 2022
PyTorch implementation of Value Iteration Networks (VIN): Clean, Simple and Modular. Visualization in Visdom.

VIN: Value Iteration Networks This is an implementation of Value Iteration Networks (VIN) in PyTorch to reproduce the results.(TensorFlow version) Key

Xingdong Zuo 215 Dec 07, 2022