Data augmentation needs little introduction I recon. It counters overfitting and makes your model generalize better, yielding better validation accuracies; or alternatively, allows you to use smaller datasets with similar performance.

In the Zoo that's the internet, I see many implementations of different augmentations, of which few are proper and nicely portable. A part from Digits yielding a great UI; ease of use; and deep learning turn-key solution, I strongly feel we can expand to the functional side as well to make this a deep learning killer-app.

For torch, I have made an implementation during lua preprocessing from frontend to backend to enable Digits to do so. In #330 there was already an attempt for augmentation, which happened on the dataset-creation side; something I am strongly against. Resizing and cropping I would consider a transformation, while I consider augmenting the data in its container an augmentation. I think therefore it's fine to resize during dataset loading (and squashing/filling/etc), but I would probably leave it at that.

Anyway, I set up a more dynamic structure to pass around these options on the torch side; instead of adding a dozen of arguments to each function, I am just adding a table.

Implements the following (screenshot):

I have iterated through many augmentation types but these were the most useful. Almost done, now running elaborate tests.

Progress

The code is already functional, though see progress below. See code, shoot!

Features

• [x] Make UI data transforms only visible for the Torch framework (invisible for Caffe)
• [x] ~~Implement UI option for normalization (scales the [0 255] to [0 1])~~
• [x] Data Augmentation UI
• [x] Flips (mirrors)
• [x] Quadrilateral rotations
• [x] Arbitrary rotations
• [x] Arbitrary scales
• [x] Augmenting in HSV space
• [x] Augmenting with noise (Thoughts?)
• [x] [Travis] Tests
• [x] Use Data Augmentation Template: data_augmentation.html

Testing

• [x] No augmentation
• [x] Flips (mirrors)
• [x] Quadrilateral rotations
• [x] Arbitrary rotations
• [x] Arbitrary scales
• [x] Arbitrary rotations & arbitrary scales
• [x] Augmenting in HSV space
• [x] Augmenting with noise
• [x] All Augmentations & benchmark speed; identify bottlenecks
• [x] Verify models reporting a slower learning/less overfitting trade-off : more generalization.

I am trying to run 50-layer residual network with 4 K40m GPUs and it's very slow (same batch_size 16 as running on single GPU), take 6 hours for 1 epoch. However, If I run it on 1 GPU the speed is normal.

System: CentOS, digits v3, nvcaffe-0.14

BTW, I tried use Googlenet and it was ok on 4 GPUs.

Any suggestion or potential issue?

Hi,

I'm running on Ubuntu 14.4 LTS.

ERROR: Expected caffe suffix "-nv". libcaffe.so does not match. Are you building from the NVIDIA/caffe fork?

[email protected]:~/digits$ pip install -r requirements.txt
You are using pip version 7.0.3, however version 7.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Requirement already satisfied (use --upgrade to upgrade): Pillow>=2.3.0 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): scipy>=0.13.3 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 3))
Collecting protobuf>=2.5.0 (from -r requirements.txt (line 4))
  Downloading protobuf-2.6.1.tar.gz (188kB)
    100% |████████████████████████████████| 188kB 2.3MB/s 
Collecting pydot>=1.0.2 (from -r requirements.txt (line 5))
  Downloading pydot-1.0.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): requests>=2.2.1 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): gevent>=1.0 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): Flask>=0.10.1 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 9))
Collecting Flask-WTF>=0.11 (from -r requirements.txt (line 10))
  Downloading Flask_WTF-0.12-py2-none-any.whl
Collecting Flask-SocketIO (from -r requirements.txt (line 11))
  Downloading Flask-SocketIO-0.6.0.tar.gz
Collecting lmdb (from -r requirements.txt (line 12))
  Downloading lmdb-0.86.tar.gz (144kB)
    100% |████████████████████████████████| 147kB 2.9MB/s 
Requirement already satisfied (use --upgrade to upgrade): nose>=1.3.1 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 13))
Requirement already satisfied (use --upgrade to upgrade): mock>=1.0.1 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 14))
Requirement already satisfied (use --upgrade to upgrade): beautifulsoup4>=4.2.1 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 15))
Requirement already satisfied (use --upgrade to upgrade): selenium>=2.25.0 in /home/ubuntu/anaconda/lib/python2.7/site-packages (from -r requirements.txt (line 16))
Collecting gunicorn (from -r requirements.txt (line 17))
  Downloading gunicorn-19.3.0-py2.py3-none-any.whl (110kB)
    100% |████████████████████████████████| 110kB 3.8MB/s 
Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/ubuntu/anaconda/lib/python2.7/site-packages/setuptools-17.1.1-py2.7.egg (from protobuf>=2.5.0->-r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): pyparsing in /home/ubuntu/anaconda/lib/python2.7/site-packages (from pydot>=1.0.2->-r requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): Werkzeug in /home/ubuntu/anaconda/lib/python2.7/site-packages (from Flask-WTF>=0.11->-r requirements.txt (line 10))
Collecting WTForms (from Flask-WTF>=0.11->-r requirements.txt (line 10))
  Downloading WTForms-2.0.2-py27-none-any.whl (128kB)
    100% |████████████████████████████████| 131kB 3.3MB/s 
Collecting gevent-socketio>=0.3.6 (from Flask-SocketIO->-r requirements.txt (line 11))
  Downloading gevent_socketio-0.3.6-py27-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): gevent-websocket in /home/ubuntu/anaconda/lib/python2.7/site-packages (from gevent-socketio>=0.3.6->Flask-SocketIO->-r requirements.txt (line 11))
Installing collected packages: protobuf, pydot, WTForms, Flask-WTF, gevent-socketio, Flask-SocketIO, lmdb, gunicorn
  Running setup.py install for protobuf
  Running setup.py install for pydot
  Running setup.py install for Flask-SocketIO
  Running setup.py install for lmdb
Successfully installed Flask-SocketIO-0.6.0 Flask-WTF-0.12 WTForms-2.0.2 gevent-socketio-0.3.6 gunicorn-19.3.0 lmdb-0.86 protobuf-2.6.1 pydot-1.0.2
[email protected]:~/digits$ sudo apt-get install graphviz
Reading package lists... Done
Building dependency tree       
Reading state information... Done
graphviz is already the newest version.
The following packages were automatically installed and are no longer required:
  linux-headers-3.13.0-49 linux-headers-3.13.0-49-generic
  linux-image-3.13.0-49-generic linux-image-extra-3.13.0-49-generic
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 267 not upgraded.
[email protected]:~/digits$ ./digits-devserver
  ___ ___ ___ ___ _____ ___
 |   \_ _/ __|_ _|_   _/ __|
 | |) | | (_ || |  | | \__ \
 |___/___\___|___| |_| |___/

Welcome to the DIGITS config module.

Where is caffe installed?
    (enter "SYS" if installed system-wide)
    [default is SYS]
(q to quit) >>> SYS
ERROR: Expected caffe suffix "-nv". libcaffe.so does not match. Are you building from the NVIDIA/caffe fork?

(q to quit) >>> 

See #17

Adds a new kind of job for performance evaluation of trained classifiers. It is now possible to visualize :

• accuracy / recall curve
• confusion matrix

Accuracy and the confusion matrix are computed against a chosen snapshot of a training task, and against both the validation set and testing set (if it exists). An "evaluate performance" button has been added on the training view. This is currently the only way to run an evaluation job. The results are stored in the job directory in the form of two pickle files.

Accuracy / recall curve

Confusion matrix

I chose a very simple representation of the confusion matrix (not in the form of a matrix !), because it is more adapted to datasets with lots of classes. For each class, the top 10 most represented classes are displayed, with their respective %.

Related jobs

I added a "Related jobs" section on each job show view. It displays the jobs which depends on the current job. For example, models trained on a specific dataset, evaluations ran on a specific model.

Let me know what you think, critiques and comments are more than welcome.

On my machine the image serving, e.g. of the mean.jpg does not work. The browser (tested IE and Chrome) cannot interpret the image probably due to the missing content type. The send_file function takes care of that all.

Closes #224

TODO before merge

• [x] Create models from HDF5 datasets using HDF5Data layers
• [x] Expose backend and compression information in REST API
• [x] Shard HDF5 files into acceptable dataset sizes - https://github.com/BVLC/caffe/issues/2953#issuecomment-137274066

TODO after merge

• Allow non-image data (see #197)
• Analyze prebuilt HDF5 datasets in "generic" path

@crohkohl, @danst18, I'm breaking the discussion in #203 out into a new issue.

Here's the situation as I understand it. Please correct me if any of this is wrong.

| map_size | Linux | OSX & Windows | | --- | --- | --- | | lower than size of dataset | LMDB runs out of memory | ? | | higher than system memory | No problem | LMDB can't allocate enough memory |

On Linux, you can just set it as high as you like and never see a problem. But that strategy blows up on other platforms.

Should [map_size] be made configurable? https://github.com/NVIDIA/DIGITS/pull/203#issuecomment-128859465

This is a sufficient but lazy solution. I would like to understand whether this can be avoided programmatically somehow before making a decision. My googling skills are failing me.

I want to install digits on my debian jessie.
When I build caffe(NVIDIA's fork), I got errors complaining that hdf5.h could not be found.

I'm sure I had installed libhdf5-serial-dev and libhdf5-dev, and I found the header file in /usr/include/hdf5/serial and its libs in /usr/lib/x86_64-linux-gnu.

So, what's wrong? Some one help me?

The build error message show below:

(venv)➜  caffe  make all --jobs=4
CXX src/caffe/layer_factory.cpp
CXX src/caffe/util/insert_splits.cpp
CXX src/caffe/util/db.cpp
CXX src/caffe/util/upgrade_proto.cpp
In file included from src/caffe/util/upgrade_proto.cpp:10:0:
./include/caffe/util/io.hpp:8:18: fatal error: hdf5.h: no such file or directory
 #include "hdf5.h"
                  ^
compilation terminated.
Makefile:512: recipe for target '.build_release/src/caffe/util/upgrade_proto.o' failed
make: *** [.build_release/src/caffe/util/upgrade_proto.o] Error 1
make: *** 正在等待未完成的任务....
In file included from ./include/caffe/common_layers.hpp:10:0,
                 from ./include/caffe/vision_layers.hpp:10,
                 from src/caffe/layer_factory.cpp:6:
./include/caffe/data_layers.hpp:9:18: fatal error: hdf5.h: no such file or directory
 #include "hdf5.h"
                  ^
compilation terminated.
Makefile:512: recipe for target '.build_release/src/caffe/layer_factory.o' failed
make: *** [.build_release/src/caffe/layer_factory.o] Error 1

I can't figure out why my model training mAP (val) doesn't get above zero. I'm trying to use the same approach and the SpaceNet_DetectNet_Train_Val.prototxt from this article.

My label files 000n.txt look like this: p 0.0 0 0.0 0 0 24 118 0 0 0 0 0 0 0 0

My images are 1280x1280, and I'm using these custom classes: dontcare,p

Where am I going wrong?

Ubuntu 14.04LTS Clean install nvidia dpkg install

$ sudo apt-get install cuda
$ sudo apt-get install digits

$ gedit .bashrc
add to endline next.

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

$ sudo reboot

$ nvidia-smi
Tue May 31 13:32:37 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.93     Driver Version: 352.93         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     Off  | 0000:01:00.0      On |                  N/A |
| 20%   37C    P8    10W / 160W |    289MiB /  4095MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 960     Off  | 0000:02:00.0     Off |                  N/A |
| 20%   43C    P8     9W / 160W |     13MiB /  4095MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

----digits run and create Dataset----

MNIST Image Size28x28 Image Type GRAYSCALE

run Image Classification Model

select Caffe and LeNet

run, and rize next error

ERROR: Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM

I am getting this error when trying to run training with my custom network.

status == CUDNN_STATUS_SUCCESS (8 vs. 0) CUDNN_STATUS_EXECUTION_FAILED

I found this post that refers to this error: https://github.com/BVLC/caffe/issues/1700#issuecomment-133476490

But it doesn't specify where or how to fix it. Also I am not sure if the issues are related or something completely different. Let me mention that this custom framework works perfectly fine when I run it in my local caffe install, and I can also see all the nodes if I hit the visualize button. It starts training and fails after the first epoch.

File "/opt/digits/digits/extensions/data/imageSegmentation/data.py", line 225, in split_image_list random.shuffle(self.random_indices) File "/usr/lib/python3.8/random.py", line 307, in shuffle x[i], x[j] = x[j], x[i] TypeError: 'range' object does not support item assignment

I'm Sorry, I'm trying to install Sunnybrook for the segmentation example on the docker container, as I want to run it over the TensorFlow backend (not Coffe). I tried to repeat the install procedure from inside the container doing docker exec -it XXXXX bash, being XXX the container ID, and later downloading the plugin from https://github.com/NVIDIA/DIGITS/tree/master/plugins/data and later doing the install proccedure, but it not works. Is there any official way to do this? I did pip install --ignore-installed setuptools (no error appears)

Installing collected packages: setuptools Successfully installed setuptools-44.1.1

git clone https://github.com/NVIDIA/DIGITS.git I went to /DIGITS/plugins/data/sunnybrook via "cd" finally I run pip install . No error appear, but after restarting docker, when trying to create a Sunny dataset it fails (See in the following post the error, I've posted appart, for clarity)

Can you help please? Kind regards

Apologies in advance since I'm new to all this but I'm confused regarding which version of DIGITS to install. I'm beginning a fresh install of the latest Ubuntu version and as of now, after hours of scouring the internet, I have found DIGITS versions that work standalone, versions that work in Docker, then there's the official DIGITS github page which has DIGITS upto version 6 and on the NGC, there's DIGITS 20.03???

What is going on I'm so confused. I was excited to get DIGITS up and running on my local machine just as soon as I had completed the Nvidia DLI's course and now I'm just stumped as to where to start. Would also like to know how different is DIGITS running for Tensorflow from the Caffe DIGITS.

Please help.

We have trained and deployed a custom model on the nano using a caffe detectnet model. We trained in digits, and it works well when conducting inference in DIGITS, but it will not show bounding boxes when running on the nano. Is there a patch for this issue?

So I am about to train with digits as specify in Hello AI Wold an then

this is the run code

inception_5b/relu_pool_proj ← inception_5b/pool_proj inception_5b/relu_pool_proj → inception_5b/pool_proj (in-place) Setting up inception_5b/relu_pool_proj TRAIN Top shape for layer 158 ‘inception_5b/relu_pool_proj’ 5 128 40 40 (1024000) Creating layer ‘inception_5b/output’ of type ‘Concat’ Layer’s types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT Created Layer inception_5b/output (159) inception_5b/output ← inception_5b/1x1 inception_5b/output ← inception_5b/3x3 inception_5b/output ← inception_5b/5x5 inception_5b/output ← inception_5b/pool_proj inception_5b/output → inception_5b/output Setting up inception_5b/output TRAIN Top shape for layer 159 ‘inception_5b/output’ 5 1024 40 40 (8192000) Creating layer ‘pool5/drop_s1’ of type ‘Dropout’ Layer’s types are Ftype:FLOAT Btype:FLOAT Fmath:FLOAT Bmath:FLOAT Created Layer pool5/drop_s1 (160) pool5/drop_s1 ← inception_5b/output pool5/drop_s1 → pool5/drop_s1 Check failed: status == CUDNN_STATUS_SUCCESS (8 vs. 0) CUDNN_STATUS_EXECUTION_FAILED, device 0

I am using a 2070 super

Server: 9dca63a42e15 DIGITS version: 6.1.1 Caffe version: 0.17.0 Caffe flavor: NVIDIA My brain is soup at this point please help me out. caffe_output.log

I have not be able to create one model yet

I am also unable to install the source digits without crashing Ubuntu. Today is May 11 and I started trying to have it work since the 7th please could you help me out. I am really exited about this tool.