Visual Question Answering in Pytorch

Overview

Visual Question Answering in pytorch

/!\ New version of pytorch for VQA available here: https://github.com/Cadene/block.bootstrap.pytorch

This repo was made by Remi Cadene (LIP6) and Hedi Ben-Younes (LIP6-Heuritech), two PhD Students working on VQA at UPMC-LIP6 and their professors Matthieu Cord (LIP6) and Nicolas Thome (LIP6-CNAM). We developed this code in the frame of a research paper called MUTAN: Multimodal Tucker Fusion for VQA which is (as far as we know) the current state-of-the-art on the VQA 1.0 dataset.

The goal of this repo is two folds:

  • to make it easier to reproduce our results,
  • to provide an efficient and modular code base to the community for further research on other VQA datasets.

If you have any questions about our code or model, don't hesitate to contact us or to submit any issues. Pull request are welcome!

News:

  • 16th january 2018: a pretrained vqa2 model and web demo
  • 18th july 2017: VQA2, VisualGenome, FBResnet152 (for pytorch) added v2.0 commit msg
  • 16th july 2017: paper accepted at ICCV2017
  • 30th may 2017: poster accepted at CVPR2017 (VQA Workshop)

Summary:

Introduction

What is the task about?

The task is about training models in a end-to-end fashion on a multimodal dataset made of triplets:

  • an image with no other information than the raw pixels,
  • a question about visual content(s) on the associated image,
  • a short answer to the question (one or a few words).

As you can see in the illustration bellow, two different triplets (but same image) of the VQA dataset are represented. The models need to learn rich multimodal representations to be able to give the right answers.

The VQA task is still on active research. However, when it will be solved, it could be very useful to improve human-to-machine interfaces (especially for the blinds).

Quick insight about our method

The VQA community developped an approach based on four learnable components:

  • a question model which can be a LSTM, GRU, or pretrained Skipthoughts,
  • an image model which can be a pretrained VGG16 or ResNet-152,
  • a fusion scheme which can be an element-wise sum, concatenation, MCB, MLB, or Mutan,
  • optionally, an attention scheme which may have several "glimpses".

One of our claim is that the multimodal fusion between the image and the question representations is a critical component. Thus, our proposed model uses a Tucker Decomposition of the correlation Tensor to model richer multimodal interactions in order to provide proper answers. Our best model is based on :

  • a pretrained Skipthoughts for the question model,
  • features from a pretrained Resnet-152 (with images of size 3x448x448) for the image model,
  • our proposed Mutan (based on a Tucker Decomposition) for the fusion scheme,
  • an attention scheme with two "glimpses".

Installation

Requirements

First install python 3 (we don't provide support for python 2). We advise you to install python 3 and pytorch with Anaconda:

conda create --name vqa python=3
source activate vqa
conda install pytorch torchvision cuda80 -c soumith

Then clone the repo (with the --recursive flag for submodules) and install the complementary requirements:

cd $HOME
git clone --recursive https://github.com/Cadene/vqa.pytorch.git 
cd vqa.pytorch
pip install -r requirements.txt

Submodules

Our code has two external dependencies:

Data

Data will be automaticaly downloaded and preprocessed when needed. Links to data are stored in vqa/datasets/vqa.py, vqa/datasets/coco.py and vqa/datasets/vgenome.py.

Reproducing results on VQA 1.0

Features

As we first developped on Lua/Torch7, we used the features of ResNet-152 pretrained with Torch7. We ported the pretrained resnet152 trained with Torch7 in pytorch in the v2.0 release. We will provide all the extracted features soon. Meanwhile, you can download the coco features as following:

mkdir -p data/coco/extract/arch,fbresnet152torch
cd data/coco/extract/arch,fbresnet152torch
wget https://data.lip6.fr/coco/trainset.hdf5
wget https://data.lip6.fr/coco/trainset.txt
wget https://data.lip6.fr/coco/valset.hdf5
wget https://data.lip6.fr/coco/valset.txt
wget https://data.lip6.fr/coco/testset.hdf5
wget https://data.lip6.fr/coco/testset.txt

/!\ There are currently 3 versions of ResNet152:

  • fbresnet152torch which is the torch7 model,
  • fbresnet152 which is the porting of the torch7 in pytorch,
  • resnet152 which is the pretrained model from torchvision (we've got lower results with it).

Pretrained VQA models

We currently provide three models trained with our old Torch7 code and ported to Pytorch:

  • MutanNoAtt trained on the VQA 1.0 trainset,
  • MLBAtt trained on the VQA 1.0 trainvalset and VisualGenome,
  • MutanAtt trained on the VQA 1.0 trainvalset and VisualGenome.
mkdir -p logs/vqa
cd logs/vqa
wget http://webia.lip6.fr/~cadene/Downloads/vqa.pytorch/logs/vqa/mutan_noatt_train.zip 
wget http://webia.lip6.fr/~cadene/Downloads/vqa.pytorch/logs/vqa/mlb_att_trainval.zip 
wget http://webia.lip6.fr/~cadene/Downloads/vqa.pytorch/logs/vqa/mutan_att_trainval.zip 

Even if we provide results files associated to our pretrained models, you can evaluate them once again on the valset, testset and testdevset using a single command:

python train.py -e --path_opt options/vqa/mutan_noatt_train.yaml --resume ckpt
python train.py -e --path_opt options/vqa/mlb_noatt_trainval.yaml --resume ckpt
python train.py -e --path_opt options/vqa/mutan_att_trainval.yaml --resume ckpt

To obtain test and testdev results on VQA 1.0, you will need to zip your result json file (name it as results.zip) and to submit it on the evaluation server.

Reproducing results on VQA 2.0

Features 2.0

You must download the coco dataset (and visual genome if needed) and then extract the features with a convolutional neural network.

Pretrained VQA models 2.0

We currently provide three models trained with our current pytorch code on VQA 2.0

  • MutanAtt trained on the trainset with the fbresnet152 features,
  • MutanAtt trained on thetrainvalset with the fbresnet152 features.
cd $VQAPYTORCH
mkdir -p logs/vqa2
cd logs/vqa2
wget http://data.lip6.fr/cadene/vqa.pytorch/vqa2/mutan_att_train.zip 
wget http://data.lip6.fr/cadene/vqa.pytorch/vqa2/mutan_att_trainval.zip 

Documentation

Architecture

.
├── options        # default options dir containing yaml files
├── logs           # experiments dir containing directories of logs (one by experiment)
├── data           # datasets directories
|   ├── coco       # images and features
|   ├── vqa        # raw, interim and processed data
|   ├── vgenome    # raw, interim, processed data + images and features
|   └── ...
├── vqa            # vqa package dir
|   ├── datasets   # datasets classes & functions dir (vqa, coco, vgenome, images, features, etc.)
|   ├── external   # submodules dir (VQA, skip-thoughts.torch, pretrained-models.pytorch)
|   ├── lib        # misc classes & func dir (engine, logger, dataloader, etc.)
|   └── models     # models classes & func dir (att, fusion, notatt, seq2vec, convnets)
|
├── train.py       # train & eval models
├── eval_res.py    # eval results files with OpenEnded metric
├── extract.py     # extract features from coco with CNNs
└── visu.py        # visualize logs and monitor training

Options

There are three kind of options:

  • options from the yaml options files stored in the options directory which are used as default (path to directory, logs, model, features, etc.)
  • options from the ArgumentParser in the train.py file which are set to None and can overwrite default options (learning rate, batch size, etc.)
  • options from the ArgumentParser in the train.py file which are set to default values (print frequency, number of threads, resume model, evaluate model, etc.)

You can easly add new options in your custom yaml file if needed. Also, if you want to grid search a parameter, you can add an ArgumentParser option and modify the dictionnary in train.py:L80.

Datasets

We currently provide four datasets:

  • COCOImages currently used to extract features, it comes with three datasets: trainset, valset and testset
  • VisualGenomeImages currently used to extract features, it comes with one split: trainset
  • VQA 1.0 comes with four datasets: trainset, valset, testset (including test-std and test-dev) and "trainvalset" (concatenation of trainset and valset)
  • VQA 2.0 same but twice bigger (however same images than VQA 1.0)

We plan to add:

Models

We currently provide four models:

  • MLBNoAtt: a strong baseline (BayesianGRU + Element-wise product)
  • MLBAtt: the previous state-of-the-art which adds an attention strategy
  • MutanNoAtt: our proof of concept (BayesianGRU + Mutan Fusion)
  • MutanAtt: the current state-of-the-art

We plan to add several other strategies in the futur.

Quick examples

Extract features from COCO

The needed images will be automaticaly downloaded to dir_data and the features will be extracted with a resnet152 by default.

There are three options for mode :

  • att: features will be of size 2048x14x14,
  • noatt: features will be of size 2048,
  • both: default option.

Beware, you will need some space on your SSD:

  • 32GB for the images,
  • 125GB for the train features,
  • 123GB for the test features,
  • 61GB for the val features.
python extract.py -h
python extract.py --dir_data data/coco --data_split train
python extract.py --dir_data data/coco --data_split val
python extract.py --dir_data data/coco --data_split test

Note: By default our code will share computations over all available GPUs. If you want to select only one or a few, use the following prefix:

CUDA_VISIBLE_DEVICES=0 python extract.py
CUDA_VISIBLE_DEVICES=1,2 python extract.py

Extract features from VisualGenome

Same here, but only train is available:

python extract.py --dataset vgenome --dir_data data/vgenome --data_split train

Train models on VQA 1.0

Display help message, selected options and run default. The needed data will be automaticaly downloaded and processed using the options in options/vqa/default.yaml.

python train.py -h
python train.py --help_opt
python train.py

Run a MutanNoAtt model with default options.

python train.py --path_opt options/vqa/mutan_noatt_train.yaml --dir_logs logs/vqa/mutan_noatt_train

Run a MutanAtt model on the trainset and evaluate on the valset after each epoch.

python train.py --vqa_trainsplit train --path_opt options/vqa/mutan_att_trainval.yaml 

Run a MutanAtt model on the trainset and valset (by default) and run throw the testset after each epoch (produce a results file that you can submit to the evaluation server).

python train.py --vqa_trainsplit trainval --path_opt options/vqa/mutan_att_trainval.yaml

Train models on VQA 2.0

See options of vqa2/mutan_att_trainval:

python train.py --path_opt options/vqa2/mutan_att_trainval.yaml

Train models on VQA (1.0 or 2.0) + VisualGenome

See options of vqa2/mutan_att_trainval_vg:

python train.py --path_opt options/vqa2/mutan_att_trainval_vg.yaml

Monitor training

Create a visualization of an experiment using plotly to monitor the training, just like the picture bellow (click the image to access the html/js file):

Note that you have to wait until the first open ended accuracy has finished processing and then the html file will be created and will pop out on your default browser. The html will be refreshed every 60 seconds. However, you will currently need to press F5 on your browser to see the change.

python visu.py --dir_logs logs/vqa/mutan_noatt

Create a visualization of multiple experiments to compare them or monitor them like the picture bellow (click the image to access the html/js file):

python visu.py --dir_logs logs/vqa/mutan_noatt,logs/vqa/mutan_att

Restart training

Restart the model from the last checkpoint.

python train.py --path_opt options/vqa/mutan_noatt.yaml --dir_logs logs/vqa/mutan_noatt --resume ckpt

Restart the model from the best checkpoint.

python train.py --path_opt options/vqa/mutan_noatt.yaml --dir_logs logs/vqa/mutan_noatt --resume best

Evaluate models on VQA

Evaluate the model from the best checkpoint. If your model has been trained on the training set only (vqa_trainsplit=train), the model will be evaluate on the valset and will run throw the testset. If it was trained on the trainset + valset (vqa_trainsplit=trainval), it will not be evaluate on the valset.

python train.py --vqa_trainsplit train --path_opt options/vqa/mutan_att.yaml --dir_logs logs/vqa/mutan_att --resume best -e

Web demo

You must set your local ip address and port in demo_server.py line 169 and your global ip address and port in demo_web/js/custom.js line 51. The port associated to the global ip address must redirect to your local ip address.

Launch your API:

CUDA_VISIBLE_DEVICES=0 python demo_server.py

Open demo_web/index.html on your browser to access the API with a human interface.

Citation

Please cite the arXiv paper if you use Mutan in your work:

@article{benyounescadene2017mutan,
  author = {Hedi Ben-Younes and 
    R{\'{e}}mi Cad{\`{e}}ne and
    Nicolas Thome and
    Matthieu Cord},
  title = {MUTAN: Multimodal Tucker Fusion for Visual Question Answering},
  journal = {ICCV},
  year = {2017},
  url = {http://arxiv.org/abs/1705.06676}
}

Acknowledgment

Special thanks to the authors of MLB for providing some Torch7 code, MCB for providing some Caffe code, and our professors and friends from LIP6 for the perfect working atmosphere.

Comments
  • Training one model on multiple GPUs simultaneously not working (possible deadlock?)

    Training one model on multiple GPUs simultaneously not working (possible deadlock?)

    Hello again,

    I was training MUTAN_att on VQA2+VG and I tried to run another process to train a second model (modified MUTAN) and both processes now seem to be stuck. I checked iotop and it seems disk reads also stopped. I verified that the modified MUTAN works when trained alone.

    I suspect that the dataloader process freaks out when another training process is launched. Is this possible? I assumed that the new training process would generate its own dataloader processes.

    BONUS: I can't seem to kill all these processes in a graceful manner, CTRL-C doesn't work. Only kill -9 PID works but that seems to create zombie processes!

    Any help is appreciated!

    opened by ahmedmagdiosman 12
  • loss decreasing is very slow

    loss decreasing is very slow

    I try to use a single lstm and a classifier to train a question-only model, but the loss decreasing is very slow and the val acc1 is under 30 even through 40 epochs

    opened by eustcPL 8
  • training MUTAN+Att using pytorch code achieve low accuracy

    training MUTAN+Att using pytorch code achieve low accuracy

    Hi, thank you so much for your code. Right now, I am trying to replicate your ICCV results with the pytorch implementation. Here is the setting 'batch_size': None, 'dir_logs': None, 'epochs': None, 'evaluate': False, 'help_opt': False, 'learning_rate': None, 'path_opt': 'options/vqa/mutan_att_trainval.yaml', 'print_freq': 10, 'resume': '', 'save_all_from': None, 'save_model': True, 'st_dropout': None, 'st_fixed_emb': None, 'st_type': None, 'start_epoch': 0, 'vqa_trainsplit': 'train', 'workers': 16}

    options

    {'coco': {'arch': 'fbresnet152torch', 'dir': 'data/coco', 'mode': 'att'}, 'logs': {'dir_logs': 'logs/vqa/mutan_att_trainval'}, 'model': {'arch': 'MutanAtt', 'attention': {'R': 5, 'activation_q': 'tanh', 'activation_v': 'tanh', 'dim_hq': 310, 'dim_hv': 310, 'dim_mm': 510, 'dropout_hv': 0, 'dropout_mm': 0.5, 'dropout_q': 0.5, 'dropout_v': 0.5, 'nb_glimpses': 2}, 'classif': {'dropout': 0.5}, 'dim_q': 2400, 'dim_v': 2048, 'fusion': {'R': 5, 'activation_q': 'tanh', 'activation_v': 'tanh', 'dim_hq': 310, 'dim_hv': 620, 'dim_mm': 510, 'dropout_hq': 0, 'dropout_hv': 0, 'dropout_q': 0.5, 'dropout_v': 0.5}, 'seq2vec': {'arch': 'skipthoughts', 'dir_st': 'data/skip-thoughts', 'dropout': 0.25, 'fixed_emb': False, 'type': 'BayesianUniSkip'}}, 'optim': {'batch_size': 128, 'epochs': 100, 'lr': 0.0001}, 'vqa': {'dataset': 'VQA', 'dir': 'data/vqa', 'maxlength': 26, 'minwcount': 0, 'nans': 2000, 'nlp': 'mcb', 'pad': 'right', 'samplingans': True, 'trainsplit': 'train'}} Warning: 399/930911 words are not in dictionary, thus set UNK Warning fusion.py: no visual embedding before fusion Warning fusion.py: no question embedding before fusion Warning fusion.py: no visual embedding before fusion Warning fusion.py: no question embedding before fusion Model has 37840812 parameters

    Here is the result after 100 epoch Epoch: [99][1740/1760] Time 0.403 (0.412) Data 0.000 (0.007) Loss 0.8993 (0.9064) [email protected] 71.094 (73.912) [email protected] 94.531 (94.830) Epoch: [99][1750/1760] Time 0.387 (0.412) Data 0.000 (0.007) Loss 0.8277 (0.9061) [email protected] 71.875 (73.915) [email protected] 95.312 (94.833) Val: [900/950] Time 0.138 (0.188) Loss 3.1201 (2.8397) [email protected] 49.219 (52.236) [email protected] 75.000 (78.115) Val: [910/950] Time 0.189 (0.187) Loss 2.4805 (2.8372) [email protected] 58.594 (52.240) [email protected] 80.469 (78.139) Val: [920/950] Time 0.210 (0.187) Loss 2.8639 (2.8388) [email protected] 53.125 (52.226) [email protected] 77.344 (78.137) Val: [930/950] Time 0.179 (0.187) Loss 2.1427 (2.8388) [email protected] 59.375 (52.227) [email protected] 82.031 (78.137) Val: [940/950] Time 0.151 (0.187) Loss 3.1772 (2.8367) [email protected] 50.781 (52.263) [email protected] 72.656 (78.163)

    opened by gaopeng-eugene 8
  • iteration over a 0-d tensor when trying to run the demo

    iteration over a 0-d tensor when trying to run the demo

    When trying to run the demo, model(visual, question) fails, yielding:

    ...
    ~/phd/vqa.pytorch/vqa/external/skip-thoughts.torch/pytorch/skipthoughts.py in _process_lengths(self, input)
        132     def _process_lengths(self, input):
        133         max_length = input.size(1)
    --> 134         lengths = list(max_length - input.data.eq(0).sum(1).squeeze())
        135         return lengths
        136 
    
    ~/python3_env/lib/python3.5/site-packages/torch/tensor.py in __iter__(self)
        358         # map will interleave them.)
        359         if self.dim() == 0:
    --> 360             raise TypeError('iteration over a 0-d tensor')
        361         return iter(imap(lambda i: self[i], range(self.size(0))))
        362 
    
    TypeError: iteration over a 0-d tensor
    

    If I change line 134 to: lengths = [max_length - input.data.eq(0).sum(1).squeeze())] I don't get the error anymore, but I only get nonsensical results with the pretrained model, for different images, I don't know if this is related or not.

    Using torch 0.4.

    opened by marcotcr 7
  • Slow data loading?

    Slow data loading?

    First of all, thank you for open-sourcing your code! It is very useful for my research.

    I notice that data loading is quite slow, i.e., out of the total time for a batch (10s, 13s, 5s), data-loading takes about 80% of the time (8s, 11s, 4s) for most batches. Consequently I can only get about 4 epochs in per a day. I use 4 workers and the default options in the repository. Do you have any recommendations on how to speed that part up?

    Thanks, David

    opened by davidgolub 4
  • Why don't you apply a softmax function before the final prediction?

    Why don't you apply a softmax function before the final prediction?

    HI, thank you for great work, I have a little question. As a classification task,we usually apply a softmax function to convert the output of a model into a probabilistic vector, each entry of which represents the probability of the input that belonging to the corresponding category. However, it seems that in your code the output of the Mutan model (the output of the second multimodel fusion followed by only a linear transformation without a softmax) is directly fed into the loss function. Is there any special consideration?

    https://github.com/Cadene/vqa.pytorch/blob/be1b6113130cda123d14c83b24c9a04acc3900d0/vqa/models/att.py#L152

    opened by zwx8981 2
  • 'OpenEnded_mscoco_val2014_model_accuracy.json' is not created

    'OpenEnded_mscoco_val2014_model_accuracy.json' is not created

    Hi : The 'OpenEnded_mscoco_val2014_model_accuracy.json' is needed in line 26 in 'visu.py', but I can not find the function to generate it in your 'train.py'. When the 'trainsplit = train' , it can not generete the 'OpenEnded_mscoco_val2014_model_accuracy.json' . And only 'OpenEnded_mscoco_val2014_model_result.json' was cteated. I want to know what is wrong with it?

    Thanks

    opened by zizhu1234 2
  • Reproduce the problem using fusion scheme of concatenation

    Reproduce the problem using fusion scheme of concatenation

    Hi Cadene,

    Thank you so much for sharing the codes.

    I am trying to reproduce vqa problem using two basic fusion scheme element-wise sum and concatenation.

    What I tried is to set x_mm = torch.cat((x_q, x_v),0) in fusion.py for concat case compared to x_mm = torch.mul(x_q, x_v) of MLBFusion

    Am I right?

    Thanks in advance.

    opened by vuhoangminh 2
  • I do not have a demo server

    I do not have a demo server

    Warning: 565/930911 words are not in dictionary, thus set UNK Warning fusion.py: no visual embedding before fusion Warning fusion.py: no question embedding before fusion Warning fusion.py: no visual embedding before fusion Warning fusion.py: no question embedding before fusion Warning train.py: no optim checkpoint found at 'logs/vqa/mutan_noatt_train/best_optim.pth.tar' This is an error.

    I didn't have enough hard disk capacity, so i deleted the raw data Does this matter?

    opened by youngjaean 2
  • Extracting features from Visual Genome gives error

    Extracting features from Visual Genome gives error

    Hello,

    When extracting features from vgenome I get the following error.

    OSError: cannot identify image file 'data/vgenome/raw/images/2335991.jpg'

    It seems like this is a corrupt image and there are quite a few in the vgenome dataset. Are these invalid images handled manually or there is a built-in support.

    Thanks in advance.

    opened by mrfarazi 2
  • can you report a single model MUTAN + Att performance?

    can you report a single model MUTAN + Att performance?

    In your paper, you report ensemble model. Because you are the state-of-the-art VQA method right now, can you report a single model MUTAN+Att performance? It is easier to compare with.

    opened by gaopeng-eugene 2
  • default vqa2/mutan_att_train example (also for demo) is messed

    default vqa2/mutan_att_train example (also for demo) is messed

    Model from http://data.lip6.fr/cadene/vqa.pytorch/vqa2/mutan_att_train.zip gives extremely confusing results, I suggest everyone trying another model, e.g.  http://webia.lip6.fr/~cadene/Downloads/vqa.pytorch/logs/vqa/mutan_att_trainval.zip which is on vqa (yaml is in vqa.pytorch/options/vqa/).

    opened by dearkafka 0
  • What should be the size of the input_question in engine.py ?

    What should be the size of the input_question in engine.py ?

    Hi, thank you so much for providing your code! I want to check the output shapes at every stage of the model. I plan to do that by passing in some random tensors of the specific shape required model as I really cannot download and extract the entire VQA dataset just for testing.

    Can anyone please let me know the shape of input_question in the engine.py file here (with and without attention) before passing it as input to the model and how to create a random tensor of that specific shape. I tried using

    word_to_ix = {"hello": 0, "world": 1}
    lookup_tensor = torch.tensor([word_to_ix["world"]], dtype=torch.long).cuda()
    
    input_img = torch.randn(1,2048,14,14)
    input_img = x1.long()
    input_img = Variable(torch.LongTensor(input_img)).cuda()
    
    out = model(input_img,lookup_tensor)
    

    But I get this error

    ensor([1], device='cuda:0')
    Traceback (most recent call last):
      File "test.py", line 83, in <module>
        out = model(x1,lookup_tensor)
      File "/home/sarvani/anaconda3/envs/MMTOD_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/sarvani/Desktop/SaiCharan/misc/vqa.pytorch/vqa/models/att.py", line 160, in forward
        x_q_vec = self.seq2vec(input_q)
      File "/home/sarvani/anaconda3/envs/MMTOD_env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/sarvani/Desktop/SaiCharan/misc/vqa.pytorch/vqa/models/seq2vec.py", line 62, in forward
        lengths = process_lengths(input)
      File "/home/sarvani/Desktop/SaiCharan/misc/vqa.pytorch/vqa/models/seq2vec.py", line 12, in process_lengths
        max_length = input.size(1)
    IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
    
    

    Any help would be appreciated. Thanks!

    opened by jiteshm17 2
  • sysntax error in line 80

    sysntax error in line 80

    runfile('F:/project/vqa.pytorch-master/demo_server.py', wdir='F:/project/vqa.pytorch-master') File "F:\project\vqa.pytorch-master\demo_server.py", line 80 visual_data = visual_data.cuda(async=True) ^ SyntaxError: invalid syntax

    opened by ghost 1
  • THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THC/generic/THCStorage.c line=82 error=2 : out of memory

    THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THC/generic/THCStorage.c line=82 error=2 : out of memory

    Hi,

    i'm trying to run demo_server.py and i get this

    THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THC/generic/THCStorage.c line=82 error=2 : out of memory
    Segmentation fault (core dumped)
    

    What should I do?

    Regards, Huy

    opened by huy99ls01 0
  • Feature extraction: CUDA out of memory

    Feature extraction: CUDA out of memory

    Hi,

    I am trying to extract features for coco dir - however it always gives me RuntimeError: CUDA out of memory. I have tried using single GPU by setting CUDA_VISIBLE_DEVICES=0- still ti gives me the same Runtime error even when the GPU is free.

    What should I do?

    Regards, Vedika

    opened by AgarwalVedika 1
  • How to process the multiple choice answer

    How to process the multiple choice answer

    Hi, I am confused that how to use the multiple choice answer in the multiple-choice task when training and evaluate the model?

    Can we process the multiple choice answer the same as the open-ended task?

    Thanks.

    opened by WangWenshan 1
Releases(v2.0)
  • v2.0(Jul 18, 2017)

    Factory

    • vqa models, convnets and vqa datasets can be created via factories

    VQA 2.0

    • VQA2(AbstractVQA) added

    VisualGenome

    • VisualGenome(AbstractVQADataset) added for merging with VQA datasets
    • VisualGenomeImages(AbstractImagesDataset) added to extract features
    • extract.py now allows to extract VisualGenome features

    Variable features size

    • extract.py now allows to extract from images of size != 448 via cli arg --size
    • FeaturesDataset now have an optional opt['size'] parameter

    FBResNet152

    • convnets.py provides support for external pretrained-models as well as ResNets from torchvision
    • especially FBResNet152 is the porting of fbresnet152torch from torch7 used until now
    Source code(tar.gz)
    Source code(zip)
Owner
Remi
I'm working hard to create the first computer overlord.
Remi
An Unsupervised Graph-based Toolbox for Fraud Detection

An Unsupervised Graph-based Toolbox for Fraud Detection Introduction: UGFraud is an unsupervised graph-based fraud detection toolbox that integrates s

SafeGraph 99 Dec 11, 2022
Multi-angle c(q)uestion answering

Macaw Introduction Macaw (Multi-angle c(q)uestion answering) is a ready-to-use model capable of general question answering, showing robustness outside

AI2 430 Jan 04, 2023
Transformers based fully on MLPs

Awesome MLP-based Transformers papers An up-to-date list of Transformers based fully on MLPs without attention! Why this repo? After transformers and

Fawaz Sammani 35 Dec 30, 2022
Try out deep learning models online on Google Colab

Try out deep learning models online on Google Colab

Erdene-Ochir Tuguldur 1.5k Dec 27, 2022
Neural Magic Eye: Learning to See and Understand the Scene Behind an Autostereogram, arXiv:2012.15692.

Neural Magic Eye Preprint | Project Page | Colab Runtime Official PyTorch implementation of the preprint paper "NeuralMagicEye: Learning to See and Un

Zhengxia Zou 56 Jul 15, 2022
constructing maps of intellectual influence from publication data

Influencemap Project @ ANU Influence in the academic communities has been an area of interest for researchers. This can be seen in the popularity of a

CS Metrics 13 Jun 18, 2022
Evolution Strategies in PyTorch

Evolution Strategies This is a PyTorch implementation of Evolution Strategies. Requirements Python 3.5, PyTorch = 0.2.0, numpy, gym, universe, cv2 Wh

Andrew Gambardella 333 Nov 14, 2022
Some pre-commit hooks for OpenMMLab projects

pre-commit-hooks Some pre-commit hooks for OpenMMLab projects. Using pre-commit-hooks with pre-commit Add this to your .pre-commit-config.yaml - rep

OpenMMLab 16 Nov 29, 2022
Supporting code for the paper "Dangers of Bayesian Model Averaging under Covariate Shift"

Dangers of Bayesian Model Averaging under Covariate Shift This repository contains the code to reproduce the experiments in the paper Dangers of Bayes

Pavel Izmailov 25 Sep 21, 2022
imbalanced-DL: Deep Imbalanced Learning in Python

imbalanced-DL: Deep Imbalanced Learning in Python Overview imbalanced-DL (imported as imbalanceddl) is a Python package designed to make deep imbalanc

NTUCSIE CLLab 19 Dec 28, 2022
Repository for the paper "Exploring the Sensory Spaces of English Perceptual Verbs in Natural Language Data"

Sensory Spaces of English Perceptual Verbs This repository contains the code and collocational data described in the paper "Exploring the Sensory Spac

David Peng 0 Sep 07, 2021
Robot Reinforcement Learning on the Constraint Manifold

Implementation of "Robot Reinforcement Learning on the Constraint Manifold"

31 Dec 05, 2022
Deep Learning Tutorial for Kaggle Ultrasound Nerve Segmentation competition, using Keras

Deep Learning Tutorial for Kaggle Ultrasound Nerve Segmentation competition, using Keras This tutorial shows how to use Keras library to build deep ne

Marko Jocić 922 Dec 19, 2022
Project of 'TBEFN: A Two-branch Exposure-fusion Network for Low-light Image Enhancement '

TBEFN: A Two-branch Exposure-fusion Network for Low-light Image Enhancement Codes for TMM20 paper "TBEFN: A Two-branch Exposure-fusion Network for Low

KUN LU 31 Nov 06, 2022
TyXe: Pyro-based BNNs for Pytorch users

TyXe: Pyro-based BNNs for Pytorch users TyXe aims to simplify the process of turning Pytorch neural networks into Bayesian neural networks by leveragi

87 Jan 03, 2023
Real-time face detection and emotion/gender classification using fer2013/imdb datasets with a keras CNN model and openCV.

Real-time face detection and emotion/gender classification using fer2013/imdb datasets with a keras CNN model and openCV.

Octavio Arriaga 5.3k Dec 30, 2022
[CVPR'20] TTSR: Learning Texture Transformer Network for Image Super-Resolution

TTSR Official PyTorch implementation of the paper Learning Texture Transformer Network for Image Super-Resolution accepted in CVPR 2020. Contents Intr

Multimedia Research 689 Dec 28, 2022
Face Recognize System on camera AI OAK1

FRS on OAK1 Face Recognize System on camera OAK1 This project contains our work that deploy on camera OAK1 Features Anti-Spoofing Face detection Face

Tran Anh Tuan 6 Aug 08, 2022
Style transfer between images was performed using the VGG19 model

Style transfer between images was performed using the VGG19 model. The necessary codes, libraries and all other information of this project are available below

Onur yılmaz 2 May 09, 2022