Javascript image annotation tool based on image segmentation.

Overview

JS Segment Annotator

Javascript image annotation tool based on image segmentation.

  • Label image regions with mouse.
  • Written in vanilla Javascript, with require.js dependency (packaged).
  • Pure client-side implementation of image segmentation.

A browser must support HTML canvas to use this tool.

There is an online demo.

Importing data

Prepare a JSON file that looks like the following. The required fields are labels and imageURLs. The annotationURLs are for existing data and can be omitted. Place the JSON file inside the data/ directory.

{
  "labels": [
    "background",
    "skin",
    "hair",
    "dress",
    "glasses",
    "jacket",
    "skirt"
  ],
  "imageURLs": [
    "data/images/1.jpg",
    "data/images/2.jpg"
  ],
  "annotationURLs": [
    "data/annotations/1.png",
    "data/annotations/2.png"
  ]
}

Then edit main.js to point to this JSON file. Open a Web browser and visit index.html.

Known issues

Browser incompatibility

A segmentation result can greatly differ due to the difference in Javascript implementation across Web browsers. The difference stems from numerical precision of floating point numbers, and there is no easy way to produce the exact same result across browsers.

Python tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

import numpy as np
from PIL import Image

# Decode
encoded = np.array(Image.open('data/annotations/1.png'))
annotation = np.bitwise_or(np.bitwise_or(
    encoded[:, :, 0].astype(np.uint32),
    encoded[:, :, 1].astype(np.uint32) << 8),
    encoded[:, :, 2].astype(np.uint32) << 16)

print(np.unique(annotation))

# Encode
Image.fromarray(np.stack([
    np.bitwise_and(annotation, 255),
    np.bitwise_and(annotation >> 8, 255),
    np.bitwise_and(annotation >> 16, 255),
    ], axis=2).astype(np.uint8)).save('encoded.png')

JSON

Use JSON module.

import json

with open('data/example.json', 'r') as f:
    dataset = json.load(f)

Using dataURL

Do the following to convert between dataURL and NumPy format.

from PIL import Image
import base64
import io

# Encode
with io.BytesIO() as buffer:
    encoded.save(buffer, format='png')
    data_url = b'data:image/png;base64,' + base64.b64encode(buffer.getvalue())

# Decode
binary = base64.b64decode(data_url.replace(b'data:image/png;base64,', b''))
encoded = Image.open(io.BytesIO(binary))

Matlab tips

Annotation PNG

The annotation PNG file contains label map encoded in RGB value. Do the following to encode an index map.

% Decode

X = imread('data/annotations/0.png');
annotation = X(:, :, 1);
annotation = bitor(annotation, bitshift(X(:, :, 2), 8));
annotation = bitor(annotation, bitshift(X(:, :, 3), 16));

% Encode

X = cat(3, bitand(annotation, 255), ...
           bitand(bitshift(annotation, -8), 255), ...
           bitand(bitshift(annotation, -16)), 255));
imwrite(uint8(X), 'data/annotations/0.png');

JSON

Use the matlab-json package.

Using dataURL

Get the byte encoding tools.

Do the following to convert between dataURL and Matlab format.

% Decode

dataURL = 'data:image/png;base64,...';
png_data = base64decode(strrep(dataURL, 'data:image/png;base64,', ''));
annotation = imdecode(png_data, 'png');

% Encode

png_data = imencode(annotation, 'png');
dataURL = ['data:image/png;base64,', base64encode(png_data)];

Citation

We appreciate if you cite the following article in an academic paper. The tool was originally developed for this work.

@article{tangseng2017looking,
Author        = {Pongsate Tangseng and Zhipeng Wu and Kota Yamaguchi},
Title         = {Looking at Outfit to Parse Clothing},
Eprint        = {1703.01386v1},
ArchivePrefix = {arXiv},
PrimaryClass  = {cs.CV},
Year          = {2017},
Month         = {Mar},
Url           = {http://arxiv.org/abs/1703.01386v1}
}
Comments
  • Tool crashes on loading Pre-Annotated Images

    Tool crashes on loading Pre-Annotated Images

    Hi @kyamagu ,

    Thank you for sharing this project.

    I'm trying to use custom annotation images and I've created the JSON file accordingly. The tool loads up the relevant classes color-coded over the image. But when I try to correct some segmentation the tool stops working and the following error is printed in the console.

    Browser Console Dump localhost-1505715475195.txt

    SampleJSON untitled )

    Sample Browser Screenshot untitled

    opened by anandcu3 15
  • cant compile paperdoll

    cant compile paperdoll

    Hi, sorry to write it here, i was not able to find a repo for paper doll. I can not compile the paper doll. it seems I have some problem with gco, if I comment it out everything else will work, then when I type in

    result = feature_calculator.apply(config, input_sample) I will get Invalid MEX-file '/BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64': /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /BS/wild-search-gaze/work/paperdoll-v1.0/paperdoll-v1.0/lib/pose/+pose/private/shiftdt.mexa64)

    Error in detect_fast>passmsg (line 128) [score0(:,:,k),Ix0(:,:,k),Iy0(:,:,k)] = shiftdt(child.score(:,:,k), child.w(1,k), child.w(2,k), child.w(3,k), child.w(4,k),child.startx(k),child.starty(k),Nx,Ny,child.step);

    Error in detect_fast (line 43) [msg,parts(k).Ix,parts(k).Iy,parts(k).Ik] = passmsg(parts(k),parts(par));

    Error in pose.estimate>process (line 38) box = detect_fast(im, model, detector_threshold);

    Error in pose.estimate (line 21) boxes{i} = process(model, samples{i}, scale, nms_threshold);

    Error in pose_calculator.apply (line 24) samples(i).(config.output) = pose.estimate(config.model, ...

    Error in feature_calculator.apply (line 25) sample = calculators{j}(config{j}, sample, varargin{:}, 'Encode', false);

    result =

     []
    

    can you help me with it please?

    opened by hosnasattar 11
  • Erase functionality

    Erase functionality

    I want to add an Erase button which erases any selection previously done. I used an button which when clicked calls a function very similar to the function responsible for Brush, which is:

    for (var y = -1 * brushSize; y <= brushSize; y++) {
        for (var x = -1 * brushSize; x <= brushSize; x++) {
          if (x * x + y * y > brushLimit) continue;
          var offset =
            4 *
            ((pos[1] + y) * this.layers.visualization.canvas.width + (pos[0] + x));
          offsets.push(offset);
          labels.push(label);
        }
      }
      annotator._updateAnnotation(offsets, labels);
    

    with one difference:

     labels.push(label);
    

    The last line inside the for loop I've hardcoded 0 being pushed into labels.

     labels.push(0);
    

    0 represents the background white color so on clicking the Erase button and selecting any area within a selection, it looks like stuff is being erased since the color of the label gets replaced by the background. But in truth this isn't actually erasing anything, just continues to add label with white background. So when I export it, the erased area expectedly still shows up.

    Any pointers on how to delete previously selected area?

    opened by Abdul-Ghani 10
  • How to code RGB and index information (8b)

    How to code RGB and index information (8b)

    Hi @kyamagu

    thank you for such a nice tool to annotate pictures. I'd like to ask if it is possible to do following:

    I already customized your tool and add 3 classes (Dress, Trousers, Jacket) as can be seen on this picture: 33 On the next picture below is annotated image from Pascal VOC 2012 for segmentation, containing RGB information + index for object and also the same for background. 11

    However, output from JS segmenter looks like this: 22

    If it is possible, could you please tell us on this example how to convert image from 32b to 8b and how to code the demanded information into image as it is in attached annotated VOC Pascal above. E.g. for background --> index 0 and RGB{0 0 0} for jacket --> index 3 and RGB{192 128 0}.

    Thank you in advance, much appreciated!

    opened by marticsvk 3
  • can't export annotation image with more than 2 images

    can't export annotation image with more than 2 images

    when I put more than 2 images in the imageURLs field in the json file, I can only view the first two images when I visit index.html. I can view all the other images by clicking "Next" though.

    The problem is: I can annotate the first two images and export the annotation pictures. But as for the other images, I am able to annotate the image, but not able to export it: There is no response when I click the "export" button.

    How can I solve this issue?

    p.s. The browser I use is Firefox(v42.0 Mac). index.html appeared to be blank with Chrome(v47.0 Mac) and Safari(v9.0).

    Thanks, john

    opened by johnzw 3
  • Some code using js-segment-annotator

    Some code using js-segment-annotator

    I am working on a code that uses this project, thought you might be interested.

    Repo: https://github.com/AKSHAYUBHAT/Vision Demo: http://vision.akshaybhat.com/ 3

    opened by ghost 3
  • Anyway way to debug when finding boundaries goes wrong?

    Anyway way to debug when finding boundaries goes wrong?

    I love your work here and greatly appreciate you sharing it. I'm not at all overly familiar with some of the concepts but wanted to use this in a proof of concept.

    I was using most of the default values and simply porting the code over to not be AMD modules. I have everything working except for the most important part - I've introduced a bug in the segmentation logic. What I end up with is almost perfect squares -

    image

    I've been combing over settings and options and trying to see what is different but nothing stands out. I know this isn't specific to your work on this library, but wanted to see if you had any tips for debugging this part of the code or if you had any tips for what could be going wrong?

    question 
    opened by plwalters 2
  • Annotated image are mostly zero pixel value

    Annotated image are mostly zero pixel value

    @kyamagu i used this tool for annotating my images for caffe-segnet,i first saw that all images are darker(you can hardly notice edges of annotated objects) but i thought on pixel level it might be different,but after checking in matlab,the whole image is almolt zero pixel.i would be happy if you could point out what i am doing wrong.thx

    opened by eliethesaiyan 2
  • Add a polygon-tool.

    Add a polygon-tool.

    • polygon tool
      • left click: span line between points
      • right click: erase drawn lines
    • keyboard shortcuts:
      • ctrl-key: switch between SLIC and polygon-tool
    • new buttons
    opened by NicolaiHarich 2
  • Image export

    Image export

    I cannot export the labeled image. I guess it was the incompatibility of the browser. Could you please suggest the right browser to use? Thanks in advance.

    SL

    question 
    opened by lsymuyu 1
  • Possible to update the annotation image after editing?

    Possible to update the annotation image after editing?

    I wanted to know if it is possible to have a save option which will update the annotation image in the data/annotations/ folder.

    If this is possible, where should I add this functionality?

    wontfix 
    opened by ghost 1
  • Polygon is difficult to close

    Polygon is difficult to close

    I find a bit difficult to close and complete a polygon. I checked the code but I don't understand where the relevant part is. How is the polygon closure checked?

    opened by FSet89 0
  • Brush tool used in the lateral border paints pixels on the other side of the image

    Brush tool used in the lateral border paints pixels on the other side of the image

    When using brush tool in the lateral border of one image, the area of the brush tool that stand out paints the pixels on the oposite border of the image.

    opened by polalbacar 0
  • The annotation masks disappear when clicked on Next button

    The annotation masks disappear when clicked on Next button

    I am able to export the annotated masks but as soon as I click on the "next" so that I can annotate the other images, the masks previously made are gone. We have to annotate the images again.

    It seems as if as soon as we hit "next", the app is removing the overlaid mask that has just been made and I am left with nothing. I have just cloned the library and started working.

    question 
    opened by nikhil-occipitaltech 5
  • Better annotation display

    Better annotation display

    Hi, Thanks for this awesome tool. May I propose an improvement in order to speed up the annotation process. I saw this in other tools and it was really efficient : the possibility to display raw images only in already annoted zones. I put below two examples of vegetation / soil annotation.

    Only vegetation toto

    Only soil: tata

    I checked your code but Im not so familiar with images overlay with javascript and I failed to display annotations with alpha = 1. Could you give me an advice ?

    enhancement question 
    opened by jerem-lab 3
  • The Upload image size is limited

    The Upload image size is limited

    HI, I just meet a problem when trying to upload images which are more than 50kb,these images showing in Firefox all are grey with nothing to see. Maybe the image compression is ok, beside that, any solutions ? Thank you for advance..

    wontfix 
    opened by wzx918 2
Releases(v2.0.1)
Owner
Kota Yamaguchi
Research scientist
Kota Yamaguchi
Wand is a ctypes-based simple ImageMagick binding for Python

Wand Wand is a ctypes-based simple ImageMagick binding for Python, supporting 2.7, 3.3+, and PyPy. All functionalities of MagickWand API are implement

Eric McConville 1.2k Jan 03, 2023
Simple Python / ImageMagick script to package images into WAD3s for use as GoldSrc textures.

WADs Out For [The] Ladies Simple Python / ImageMagick script to package images into WAD3s for use as GoldSrc textures. Development mostly focused on L

5 Apr 09, 2022
starfish is a Python library for processing images of image-based spatial transcriptomics.

starfish: scalable pipelines for image-based transcriptomics starfish is a Python library for processing images of image-based spatial transcriptomics

199 Dec 08, 2022
A functional and efficient python implementation of the 3D version of Maxwell's equations

py-maxwell-fdfd Solving Maxwell's equations via A python implementation of the 3D curl-curl E-field equations. This code contains additional work to e

Nathan Zhao 12 Dec 11, 2022
This tool allows the user to convert a 16 by 16 image into a list with numbers representing an object/character.

Room Formatter This tool allows the user to convert a 16 by 16 image into a list with numbers representing an object/character. There is cur

Thomas Landstra 1 Nov 13, 2021
Hacking github graph with a easy python script

Hacking-Github-Graph Hacking github graph with a easy python script Requirements git latest version installed. A text editor (eg: vs code, sublime tex

SENPAI LEGEND 1 Nov 01, 2021
Convert bitmap images to seeds for Tiny-83 NFT project.

What is this? This tool allows you to convert any 14p high and 22p wide Bitmap (.bmp) to the seed needed for the Tiny-83 NFT project. Project Twitter:

shib_maximalist 1 Oct 31, 2021
Hello, this project is an example of how to generate a QR Code using python 😁

Hello, this project is an example of how to generate a QR Code using python 😁

Davi Antonaji 2 Oct 12, 2021
sK1 2.0 cross-platform vector graphics editor

sK1 2.0 sK1 2.0 is a cross-platform open source vector graphics editor similar to CorelDRAW, Adobe Illustrator, or Freehand. sK1 is oriented for prepr

sK1 Project 238 Dec 04, 2022
Automatic picture transmission(APT) protocol decoder for NOAA weather satellites

Automatic picture transmission(APT) decoder Automatic picture transmission protocol is used by NOAA satellites. They constantly send a frequency modul

Jayachandra Kasarla 25 Aug 05, 2022
The friendly PIL fork (Python Imaging Library)

Pillow Python Imaging Library (Fork) Pillow is the friendly PIL fork by Alex Clark and Contributors. PIL is the Python Imaging Library by Fredrik Lund

Pillow 10.4k Dec 31, 2022
TRREASURE_IMAGE is python lib by which you can hide anything in a .jpg image with Command-Line Interface[cli] feature

TRREASURE_IMAGE TRREASURE_IMAGE is a python third-party library with Command-Line Interface[cli] feature. Table of Contents General Info Python librar

Fatin Shadab 3 Jun 07, 2022
This Github Action automatically creates a GIF from a given web page to display on your project README

This Github Action automatically creates a GIF from a given web page to display on your project README

Pablo Lecolinet 28 Dec 15, 2022
Python Program that lets you write in your handwriting!

Handwriting with Python Python Program that lets you write in your handwriting! Inspired by: thaisribeiro.in How to run? Install Unidecode and Pillow

Amanda Rodrigues Vieira 2 Oct 25, 2021
Group of interfaces interesting for users

Project: Interface to create GIF animation based on Fourier Series.

5 Aug 17, 2021
Nanosensor Image Processor (NanoImgPro), a python-based image analysis tool for dopamine nanosensors

NanoImgPro Nanosensor Image Processor (NanoImgPro), a python-based image analysis tool for dopamine nanosensors NanoImgPro.py contains the main class

1 Mar 02, 2022
将位图转为彩色矢量 svg 图片

一个将位图描摹为彩色矢量 svg 图片的程序,是一个命令行工具,使用 Python 脚本实现,运行环境 Python3.8+。 ✨ 效果 以一个字帖图片为例,这是 png 格式的位图(370KB): 这是颜

Haujet Zhao 104 Dec 30, 2022
Graphical tool to make photo collage posters

PhotoCollage Graphical tool to make photo collage posters PhotoCollage allows you to create photo collage posters. It assembles the input photographs

Adrien Vergé 350 Jan 02, 2023
Generates images of calendar month tables and can paste them onto suitable photos.

📆 calendizer README Generates images of calendar month tables and can paste them onto suitable photos. A quick way to make your own calendar for prin

Sean Ryan 2 Dec 14, 2022
MikuMikuRig是一款集生成控制器,自动导入动画,自动布料为一体的blender插件

Miku_Miku_Rig MikuMikuRig是一款集生成控制器,自动导入动画,自动布料为一体的blender插件。 MikumiKurig is a Blender plugin that can generates rig, automatically imports animations

小威廉伯爵 342 Dec 29, 2022