Pytorch implementation of the unsupervised object discovery method LOST.

Last update: Dec 25, 2022

Related tags

Overview

LOST

Pytorch implementation of the unsupervised object discovery method LOST. More details can be found in the paper:

Localizing Objects with Self-Supervised Transformers and no Labels [arXiv]
by Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet and Jean Ponce

If you use the LOST code or framework in your research, please consider citing:

@article{LOST,
   title = {Localizing Objects with Self-Supervised Transformers and no Labels},
   author = {Oriane Sim\'eoni and Gilles Puy and Huy V. Vo and Simon Roburin and Spyros Gidaris and Andrei Bursuc and Patrick P\'erez and Renaud Marlet and Jean Ponce},
   journal = {arXiv preprint arXiv:2109.14279},
   month = {09},
   year = {2021}
}

Installation

Dependencies

This code was implemented with python 3.7, PyTorch 1.7.1 and CUDA 10.2. Please install PyTorch. In order to install the additionnal dependencies, please launch the following command:

pip install -r requirements.txt

Install DINO

This method is based on DINO paper. The framework can be installed using the following commands:

> __init__.py; cd ../; ">

git clone https://github.com/facebookresearch/dino.git
cd dino; 
touch __init__.py
echo -e "import sys\nfrom os.path import dirname, join\nsys.path.insert(0, join(dirname(__file__), '.'))" >> __init__.py; cd ../;

The code was made using the commit ba9edd1 of DINO repo (please rebase if breakage).

Apply LOST to one image

Following are scripts to apply LOST to an image defined via the image_path parameter and visualize the predictions (pred), the maps of the Figure 2 in the paper (fms) and the visulization of the seed expansion (seed_expansion). Box predictions are also stored in the output directory given by parameter output_dir.

python main_lost.py --image_path examples/VOC07_000236.jpg --visualize pred
python main_lost.py --image_path examples/VOC07_000236.jpg --visualize fms
python main_lost.py --image_path examples/VOC07_000236.jpg --visualize seed_expansion

Launching on datasets

Following are the different steps to reproduce the results of LOST presented in the paper.

PASCAL-VOC

Please download the PASCAL VOC07 and PASCAL VOC12 datasets (link) and put the data in the folder datasets. There should be the two subfolders: datasets/VOC2007 and datasets/VOC2012. In order to apply lost and compute corloc results (VOC07 61.9, VOC12 64.0), please launch:

python main_lost.py --dataset VOC07 --set trainval
python main_lost.py --dataset VOC12 --set trainval

COCO

Please download the COCO dataset and put the data in datasets/COCO. Results are provided given the 2014 annotations following previous works. The following command line allows you to get results on the subset of 20k images of the COCO dataset (corloc 50.7), following previous litterature. To be noted that the 20k images are a subset of the train set.

python main_lost.py --dataset COCO20k --set train

Different models

We have tested the method on different setups of the VIT model, corloc results are presented in the following table (more can be found in the paper).

arch	pre-training	dataset
		VOC07	VOC12	COCO20k
ViT-S/16	DINO	61.9	64.0	50.7
ViT-S/8	DINO	55.5	57.0	49.5
ViT-B/16	DINO	60.1	63.3	50.0
ResNet50	DINO	36.8	42.7	26.5
ResNet50	Imagenet	33.5	39.1	25.5

Previous results on the dataset VOC07 can be obtained by launching:

python main_lost.py --dataset VOC07 --set trainval #VIT-S/16
python main_lost.py --dataset VOC07 --set trainval --patch_size 8 #VIT-S/8
python main_lost.py --dataset VOC07 --set trainval --arch vit_base #VIT-B/16
python main_lost.py --dataset VOC07 --set trainval --arch resnet50 #Resnet50/DINO
python main_lost.py --dataset VOC07 --set trainval --arch resnet50_imagenet #Resnet50/imagenet

Comments

Is LOST designed to perform well with DINO features specifically?

I've replaced LOST's backbone (basically the dino weights) with the ones in CLIP, and it did not work well. But when switching back to dino weights, both ViT and ResNet50 backbone could generate good feature maps. Why would this happen?
question

opened by zengyuy 3
Error in evaluation with Detectron2

Hi @osimeoni,

Thank you for making the code available!

When evaluating Detectron2 on VOC12 with the obtained pseudolables. I obtain the following error: AttributeError: "int object has no attribute 'value'. It seems that the coco_style_file is not registered by 'register_coco_instances' (see image underneath). Any idea how this can be fixed? Thanks.

opened by MarcVisions 2
Class-aware detection

Do you plan on releasing code for class-aware detection (i.e., to produce the results in Table 3 of https://arxiv.org/pdf/2109.14279.pdf)? I don't believe I see any of the necessary code for assigning object categories to boxes, but please correct me if I'm wrong.

opened by gholste 2
Multi-object discovery

HI, I have a confusion about the interesting work. How to perform multi-target discovery in the figure 1 (middle) of your paper？ Any advice is greatly appreciated.
question

opened by rgbd-zml 1
Lost not performing well using DINO with fine-tuning

I’ve trained DINO’s model with my own Dataset, doing a finetuning on the ViT’s pre trained models of DINO. After a feel experiments I noticed that, every time that a epoch of the DINO’s finetune ran, the loss of the training reduce, however the IoU (the validation metric that we are using) of the bounding boxes generated by the LOST algorithm gets worse. Can anyone explain me why this is happening and how can I fix it?

opened by ericyoshida 1
corLoc evaluation

Hi @osimeoni . I am suspicious about the corLoc evaluation part in the code. The corLoc for each image is true whenever one of the ground truth objects is hit! https://github.com/valeoai/LOST/blob/2b678aca89c18aa79c56ec3f6d4a0b979a91608d/main_lost.py#L311 What about other objects? Is it right?

opened by Mirsadeghi 1

Pytorch implementation of the unsupervised object discovery method LOST.

Related tags

Overview

LOST

Installation

Dependencies

Install DINO

Apply LOST to one image

Launching on datasets

PASCAL-VOC

COCO

Different models

Comments

Is LOST designed to perform well with DINO features specifically?

Error in evaluation with Detectron2

Class-aware detection

Multi-object discovery

Lost not performing well using DINO with fine-tuning

corLoc evaluation

Releases(models)

models(Feb 28, 2022)

Owner

Valeo.ai

TensorRT examples (Jetson, Python/C++)(object detection)

FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation.

Point cloud processing tool library.

Torch-ngp - A pytorch implementation of the hash encoder proposed in instant-ngp

transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

Character-Input - Create a program that asks the user to enter their name and their age

CTF challenges and write-ups for MicroCTF 2021.

PyTorch implementation of DARDet: A Dense Anchor-free Rotated Object Detector in Aerial Images

Fully Convlutional Neural Networks for state-of-the-art time series classification

This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

This repository contains numerical implementation for the paper Intertemporal Pricing under Reference Effects: Integrating Reference Effects and Consumer Heterogeneity.

Our VMAgent is a platform for exploiting Reinforcement Learning (RL) on Virtual Machine (VM) scheduling tasks.

A Multi-modal Model Chinese Spell Checker Released on ACL2021.

Code for KDD'20 "An Efficient Neighborhood-based Interaction Model for Recommendation on Heterogeneous Graph"

Source code of article "Towards Toxic and Narcotic Medication Detection with Rotated Object Detector"

Fast, accurate and reliable software for algebraic CT reconstruction

A project studying the influence of communication in multi-objective normal-form games

Official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

Similarity-based Gray-box Adversarial Attack Against Deep Face Recognition