Keep CALM and Improve Visual Feature Attribution

Last update: Dec 07, 2022

Related tags

Deep Learning calm

Overview

Keep CALM and Improve Visual Feature Attribution

Jae Myung Kim^1*, Junsuk Choe^1*, Zeynep Akata², Seong Joon Oh^1†
_{* Equal contribution} _† _{Corresponding author}

¹ _{NAVER AI LAB} ² _{University of Tübingen}

Abstract

The class activation mapping, or CAM, has been the cornerstone of feature attribution methods for multiple vision tasks. Its simplicity and effectiveness have led to wide applications in the explanation of visual predictions and weakly-supervised localization tasks. However, CAM has its own shortcomings. The computation of attribution maps relies on ad-hoc calibration steps that are not part of the training computational graph, making it difficult for us to understand the real meaning of the attribution values. In this paper, we improve CAM by explicitly incorporating a latent variable encoding the location of the cue for recognition in the formulation, thereby subsuming the attribution map into the training computational graph. The resulting model, class activation latent mapping, or CALM, is trained with the expectation-maximization algorithm. Our experiments show that CALM identifies discriminative attributes for image classifiers more accurately than CAM and other visual attribution baselines. CALM also shows performance improvements over prior arts on the weakly-supervised object localization benchmarks.

Dataset downloading

For ImageNet and CUB datasets, please follow the common procedure for downloading the datasets.
For ImageNetV2, CUBV2, and OpenImages30k, please follow the procedure introduced in wsol-evaluation page.

How to use models

You can train CALM models by

$ python main.py --experiment_name=experiment_name/ \
                 --architecture=resnet50 \
                 --attribution_method=CALM_EM \
                 --dataset=CUB \
                 --use_bn=True --large_feature_map=True

You can evaluate the models on two different metrics,

$ python eval_pixel_perturb.py --experiment_name=experiment_name/ \
                               --architecture=resnet50 \ 
                               --attribution_method=CALM_EM \
                               --dataset=CUB \
                               --use_bn=True --large_feature_map=True \
                               --use_load_checkpoint=True \
                               --load_checkpoint=checkpoint_name/ \
                               --score_map_process=jointll --norm_type=clipping &
                               
$ python eval_cue_location.py --experiment_name=experiment_name/ \ 
                              --architecture=resnet50 \
                              --attribution_method=CALM_EM \
                              --dataset=CUB \
                              --use_bn=True --large_feature_map=True \
                              --use_load_checkpoint=True \
                              --load_checkpoint=checkpoint_name/ \
                              --score_map_process=jointll --norm_type=clipping --threshold_type=log &

Pretrained weights

For those who wish to use pretrained CALM weights,

Model name	Dataset	cls. accuracy	weights
CALM_EM	CUB	71.8	link
CALM_EM	OpenImages	70.1	link
CALM_EM	ImageNet	70.4	link
CALM_ML	CUB	59.6	link
CALM_ML	OpenImages	70.9	link
CALM_ML	ImageNet	70.6	link

Explainability scores

Cue localization and Remove-and-classify results. More details about the metrics are in the paper.

Cue localization (the higher, the better)	Remove-and-classify (the lower, the better)

License

Copyright (c) 2021-present NAVER Corp.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

Keep CALM and Improve Visual Feature Attribution

Related tags

Overview

Keep CALM and Improve Visual Feature Attribution

Abstract

Dataset downloading

How to use models

Pretrained weights

Explainability scores

License

Owner

NAVER AI

🤖 A Python library for learning and evaluating knowledge graph embeddings

Library to enable Bayesian active learning in your research or labeling work.

This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge column damage detection

ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs

clDice - a Novel Topology-Preserving Loss Function for Tubular Structure Segmentation

for a paper about leveraging discourse markers for training new models

Graph Convolutional Networks in PyTorch

Simple PyTorch implementations of Badnets on MNIST and CIFAR10.

Event-forecasting - Event Forecasting Algorithms With Python

The final project of "Applying AI to 3D Medical Imaging Data" from "AI for Healthcare" nanodegree - Udacity.

MediaPipeで姿勢推定を行い、Tokyo2020オリンピック風のピクトグラムを表示するデモ

Exact Pareto Optimal solutions for preference based Multi-Objective Optimization

NitroFE is a Python feature engineering engine which provides a variety of modules designed to internally save past dependent values for providing continuous calculation.

This code is an implementation for Singing TTS.

(JMLR'19) A Python Toolbox for Scalable Outlier Detection (Anomaly Detection)

An efficient toolkit for Face Stylization based on the paper "AgileGAN: Stylizing Portraits by Inversion-Consistent Transfer Learning"

Kaggle Ultrasound Nerve Segmentation competition [Keras]

BC3407-Group-5-Project - BC3407 Group Project With Python

Automatic self-diagnosis program (python required)Automatic self-diagnosis program (python required)

pyhsmm - library for approximate unsupervised inference in Bayesian Hidden Markov Models (HMMs) and explicit-duration Hidden semi-Markov Models (HSMMs), focusing on the Bayesian Nonparametric extensions, the HDP-HMM and HDP-HSMM, mostly with weak-limit approximations.