Localizing-Visual-Sounds-the-Hard-Way

Code and Dataset for "Localizing Visual Sounds the Hard Way".

The repo contains code and our pre-trained model.

Environment

Python 3.6.8
Pytorch 1.3.0

Flickr-SoundNet

We provide the pretrained model here.

To test the model, testing data and ground truth should be downloaded from learning to localize sound source.

Then run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --gt_path "path to ground truth" --testset "flickr"

VGG-Sound Source

We provide the pretrained model here.

To test the model, run

python test.py --data_path "path to downloaded data with structure below/" --summaries_dir "path to pretrained models" --testset "vggss"

(Note, some gt bounding boxes are updated recently, all results on VGG-SS cause a 2~3% difference on IoU.)

Both test data should be placed in the following structure.

data path
│
└───frames
│   │   image001.jpg
│   │   image002.jpg
│   │
└───audio
    │   audio011.wav
    │   audio012.wav

Citation

@InProceedings{Chen21,
              title        = "Localizing Visual Sounds the Hard Way",
              author       = "Honglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani, Andrea Vedaldi, Andrew Zisserman",
              booktitle    = "CVPR",
              year         = "2021"}

Localizing Visual Sounds the Hard Way

Related tags

Overview

Localizing-Visual-Sounds-the-Hard-Way

Environment

Flickr-SoundNet

VGG-Sound Source

Citation

Owner

Honglie Chen

Tutorial page of the Climate Hack, the greatest hackathon ever

Official PyTorch Implementation of Rank & Sort Loss [ICCV2021]

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

This repository stores the code to reproduce the results published in "TiWS-iForest: Isolation Forest in Weakly Supervised and Tiny ML scenarios"

HGCAE Pytorch implementation. CVPR2021 accepted.

Fully Adaptive Bayesian Algorithm for Data Analysis (FABADA) is a new approach of noise reduction methods. In this repository is shown the package developed for this new method based on \citepaper.

High performance distributed framework for training deep learning recommendation models based on PyTorch.

Predict bus arrival time using VertexAI and Nvidia's Jetson Nano

code for paper -- "Seamless Satellite-image Synthesis"

Source code of the paper PatchGraph: In-hand tactile tracking with learned surface normals.

Solving reinforcement learning tasks which require language and vision

Keras implementation of "One pixel attack for fooling deep neural networks" using differential evolution on Cifar10 and ImageNet

PyTorch wrapper for Taichi data-oriented class

This game was designed to encourage young people not to gamble on lotteries, as the probablity of correctly guessing the number is infinitesimal!

Real-world Anomaly Detection in Surveillance Videos- pytorch Re-implementation

A PyTorch-based Semi-Supervised Learning (SSL) Codebase for Pixel-wise (Pixel) Vision Tasks

GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks

Answering Open-Domain Questions of Varying Reasoning Steps from Text

CV backbones including GhostNet, TinyNet and TNT, developed by Huawei Noah's Ark Lab.

AntiFuzz: Impeding Fuzzing Audits of Binary Executables