The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

Last update: Nov 02, 2022

Related tags

Overview

MultiModal-Collaborative (MMC) Learning Framework for integrating RGB and Thermal spectral modalities

This is the official code for NeurIPS 2021 Machine Learning for Autonomous Driving Workshop Paper, "Does Thermal data make the detection systems more reliable?" by Shruthi Gowda, Elahe Arani and Bahram Zonooz.

Methodology

Architecture

Detection Head : SSD
Detection Backbone : Resnet (CNN-based) or DEiT (Transformer-based)

MMC framework

MMC framework has multiple versions

KD.ENABLE: True
KD.ENABLE_DML: True

1. MMC (Base Version) : Det Loss + DML Loss 
    KD.DISTILL_TYPE : KL, AT, L2, L2B
    KL (KL divergence), AT (Attention loss), L2 (L2 norm at head layer), L2B (L2 norm of backbone features)
   
2. MMC v1 (Reconstruction) : Det Loss + DML Loss + Recon Loss
    KD.AUX_RECON = True
    KD.AUX_RECON_MODE = "normal"

3. MMC v2 (Cross Reconstruction) : Det Loss + DML Loss + Cross Recon Loss
    KD.AUX_RECON = True
    KD.AUX_RECON_MODE = "cross"

We also try other techniques for comparison

Fusion
1. Input Fusion
    KD.CONCAT_INPUT
2. Feature Fusion
    KD.CONCAT_FEATURES
    CONCAT_LAYERS

Installation

You can prepare the environment using:

pip install -r requirements.txt

You can build the project using the following script:

./build {conda_env_name}

Datasets

Two datasets "FLIR" and "KAIST" are used in this repo

FLIR : https://www.flir.eu/oem/adas/adas-dataset-form/

KAIST : https://soonminhwang.github.io/rgbt-ped-detection/

Running

Train

There are 2 networks, one receiving RGB images and one receiving thermal images. Both require different config files.

python train.py --config-file <thermal-config-file> --teacher-config-file <rgb-config-file>

Test

For evaluation only one network is used - the first network (RGB or Teacher network)

python test.py --config-file <config-file> --ckpt <model_final.pth>

Model Checkpoints

Cite Our Work

License

This project is licensed under the terms of the MIT license.

The official PyTorch code for NeurIPS 2021 ML4AD Paper, "Does Thermal data make the detection systems more reliable?"

Related tags

Overview

MultiModal-Collaborative (MMC) Learning Framework for integrating RGB and Thermal spectral modalities

Methodology

Architecture

MMC framework

Installation

Datasets

Running

Train

Test

Model Checkpoints

Cite Our Work

License

Owner

NeurAI

Get a Grip! - A robotic system for remote clinical environments.

Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation.

GeneralOCR is open source Optical Character Recognition based on PyTorch.

Cervix ROI Segmentation Using U-NET

PyTorch implementation of Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation.

Multivariate Time Series Transformer, public version

The official repository for "Revealing unforeseen diagnostic image features with deep learning by detecting cardiovascular diseases from apical four-chamber ultrasounds"

This code is a near-infrared spectrum modeling method based on PCA and pls

Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue

Code for approximate graph reduction techniques for cardinality-based DSFM, from paper

Tensorflow port of a full NetVLAD network

ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection

Offline Reinforcement Learning with Implicit Q-Learning

PyTorch Kafka Dataset: A definition of a dataset to get training data from Kafka.

Code for the paper Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations (AKBC 2021).

Predicting Price of house by considering ,house age, Distance from public transport

TransMIL: Transformer based Correlated Multiple Instance Learning for Whole Slide Image Classification

A cross-lingual COVID-19 fake news dataset

A curated list of long-tailed recognition resources.

Planner_backend - Academic planner application designed for students and counselors.