CLIPImageClassifier wraps clip image model from transformers

Overview

CLIPImageClassifier

CLIPImageClassifier wraps clip image model from transformers.

CLIPImageClassifier is initialized with the argument classes, these are the texts that we want to classify an image to one of them The executor receives Documents with uri attribute. Each Document's uri represent the path to an image. The executor will read the image and classify it to one of the classes.

The result will be saved inside a new tag called class within the original document. The class tag is a dictionary that contains two things:

  • label: the chosen class from classes.
  • score: the confidence score in the chosen class given by the model.

Usage

Use the prebuilt images from Jina Hub in your Python code, add it to your Flow and classify your images according to chosen classes:

from jina import Flow
classes = ['this is a cat','this is a dog','this is a person']
f = Flow().add(
    uses='jinahub+docker://CLIPImageClassifier',
    uses_with={'classes':classes}
    )
docs = DocumentArray()
doc = Document(uri='/your/image/path')
docs.append(doc)

with f:
    f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

Returns

Document with class tag. This class tag which is a dict.It contains label which is an str and a float confidence score for the image.

GPU Usage

This executor also offers a GPU version. To use it, make sure to pass 'device'='cuda', as the initialization parameter, and gpus='all' when adding the containerized Executor to the Flow. See the Executor on GPU section of Jina documentation for more details.

Here's how you would modify the example above to use a GPU:

from jina import Flow

classes = ['this is a cat','this is a dog','this is a person']	
f = Flow().add(
    uses='jinahub+docker://CLIPImageClassifier',
    uses_with={
    'classes':classes,
    'device':'cuda',
    'gpus':'all'
    }
    )
docs = DocumentArray()
doc = Document(uri='/your/image/path')
docs.append(doc)

with f:
    f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

Reference

CLIP Image model

You might also like...
CLIP+FFT text-to-image
CLIP+FFT text-to-image

Aphantasia This is a text-to-image tool, part of the artwork of the same name. Based on CLIP model, with FFT parameterizer from Lucent library as a ge

A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.
A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.

Ryan Murdock has done it again, combining OpenAI's CLIP and the generator from a BigGAN! This repository wraps up his work so it is easily accessible to anyone who owns a GPU.

CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)
CLIP: Connecting Text and Image (Learning Transferable Visual Models From Natural Language Supervision)

CLIP (Contrastive Language–Image Pre-training) Experiments (Evaluation) Model Dataset Acc (%) ViT-B/32 (Paper) CIFAR100 65.1 ViT-B/32 (Our) CIFAR100 6

Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

 Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized
Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized

VQGAN-CLIP-Docker About Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized This is a stripped and minimal dependency repository for running loca

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

CLIP (Contrastive Language–Image Pre-training) trained on Indonesian data

CLIP-Indonesian CLIP (Radford et al., 2021) is a multimodal model that can connect images and text by training a vision encoder and a text encoder joi

Comments
  • CLIPImageClassifier error

    CLIPImageClassifier error

    I tried to run the following flow on "jinahub+sandbox" but I got the following error could you please share your insight with me? I am running the code from my Jupyter notebook.

    import warnings warnings.filterwarnings("ignore", category=DeprecationWarning) from jina import Flow classes = ['this is a cat','this is a dog','this is a person'] doc = Document(uri='image/dog.jpg') docs = DocumentArray() docs.append(doc) f = Flow().add( uses='jinahub://CLIPImageClassifier',name="classifier", uses_with={'classes':classes})

    with f: f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

    -----------------------error------------------ PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release (raised from /home/ubuntu/pyenv/lib/python3.10/site-packages/pkg_resources/init.py:116) PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release (raised from /home/ubuntu/pyenv/lib/python3.10/site-packages/pkg_resources/init.py:116) UserWarning: VersionConflict(torchvision 0.12.0+cpu (/usr/local/lib/python3.10/dist-packages), Requirement.parse('torchvision==0.10.0')) (raised from /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/hubble/helper.py:483) ftfy or spacy is not installed using BERT BasicTokenizer instead of ftfy. ╭────── 🎉 Flow is ready to serve! ──────╮ │ 🔗 Protocol GRPC │ │ 🏠 Local 0.0.0.0:55600 │ │ 🔒 Private 172.31.17.247:55600 │ │ 🌍 Public 34.221.179.218:55600 │ ╰────────────────────────────────────────╯ ERROR classifier/[email protected] AttributeError("'DocumentArrayInMemory' [07/06/22 16:34:35] object has no attribute 'get_attributes'")
    add "--quiet-error" to suppress the exception details
    ╭────────────── Traceback (most recent call last) ───────────────╮
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ru… │
    │ in process_data │
    │ │
    │ 162 │ │ │ │ if self.logger.debug_enabled: │
    │ 163 │ │ │ │ │ self._log_data_request(requests[0]) │
    │ 164 │ │ │ │ │
    │ ❱ 165 │ │ │ │ return await self._data_request_handler. │
    │ 166 │ │ │ except (RuntimeError, Exception) as ex: │
    │ 167 │ │ │ │ self.logger.error( │
    │ 168 │ │ │ │ │ f'{ex!r}' │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ru… │
    │ in handle │
    │ │
    │ 147 │ │ ) │
    │ 148 │ │ │
    │ 149 │ │ # executor logic │
    │ ❱ 150 │ │ return_data = await self._executor.acall( │
    │ 151 │ │ │ req_endpoint=requests[0].header.exec_endpoin │
    │ 152 │ │ │ docs=docs, │
    │ 153 │ │ │ parameters=params, │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ex… │
    │ in acall
    │ │
    │ 271 │ │ if req_endpoint in self.requests: │
    │ 272 │ │ │ return await self.acall_endpoint(req_end │
    │ 273 │ │ elif default_endpoint in self.requests: │
    │ ❱ 274 │ │ │ return await self.acall_endpoint(__defau │
    │ 275 │ │
    │ 276 │ async def acall_endpoint(self, req_endpoint, **k │
    │ 277 │ │ func = self.requests[req_endpoint] │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ex… │
    │ in acall_endpoint
    │ │
    │ 292 │ │ │ if iscoroutinefunction(func): │
    │ 293 │ │ │ │ return await func(self, **kwargs) │
    │ 294 │ │ │ else: │
    │ ❱ 295 │ │ │ │ return func(self, **kwargs) │
    │ 296 │ │
    │ 297 │ @property │
    │ 298 │ def workspace(self) -> Optional[str]: │
    │ │
    │ /home/ubuntu/pyenv/lib/python3.10/site-packages/jina/serve/ex… │
    │ in arg_wrapper │
    │ │
    │ 177 │ │ │ │ def arg_wrapper( │
    │ 178 │ │ │ │ │ executor_instance, *args, **kwargs │
    │ 179 │ │ │ │ ): # we need to get the summary from th │
    │ the self │
    │ ❱ 180 │ │ │ │ │ return fn(executor_instance, *args, │
    │ 181 │ │ │ │ │
    │ 182 │ │ │ │ self.fn = arg_wrapper │
    │ 183 │
    │ │
    │ /home/ubuntu/.jina/hub-package/9k3zudzu/clip_image_classifier… │
    │ in classify │
    │ │
    │ 56 │ │ for docs_batch in docs.traverse_flat( │
    │ 57 │ │ │ parameters.get('traversal_paths', self.traver │
    │ 58 │ │ ).batch(batch_size=parameters.get('batch_size', s │
    │ ❱ 59 │ │ │ image_batch = docs_batch.get_attributes('blob │
    │ 60 │ │ │ with torch.inference_mode(): │
    │ 61 │ │ │ │ input = self._generate_input_features(cla │
    │ 62 │ │ │ │ outputs = self.model(**input) │
    ╰────────────────────────────────────────────────────────────────╯
    AttributeError: 'DocumentArrayInMemory' object has no attribute
    'get_attributes'
    Exception in thread Thread-107: Traceback (most recent call last): File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/base/grpc.py", line 86, in _get_results async for resp in stub.Call( File "/home/ubuntu/pyenv/lib/python3.10/site-packages/grpc/aio/_call.py", line 326, in _fetch_stream_responses await self._raise_for_status() File "/home/ubuntu/pyenv/lib/python3.10/site-packages/grpc/aio/_call.py", line 236, in _raise_for_status raise _create_rpc_error(await self.initial_metadata(), await grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Unexpected <class 'grpc.aio._call.AioRpcError'>: <AioRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'" debug_error_string = "{"created":"@1657125275.618452649","description":"Error received from peer ipv4:0.0.0.0:58903","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'","grpc_status":2}"

    " debug_error_string = "{"created":"@1657125275.619606817","description":"Error received from peer ipv4:0.0.0.0:55600","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'grpc.aio._call.AioRpcError'>: <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = "Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'"\n\tdebug_error_string = "{"created":"@1657125275.618452649","description":"Error received from peer ipv4:0.0.0.0:58903","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'","grpc_status":2}"\n>","grpc_status":2}"

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1009, in _bootstrap_inner self.run() File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/helper.py", line 1292, in run self.result = asyncio.run(func(*args, **kwargs)) File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete return future.result() File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/mixin.py", line 164, in _get_results async for resp in c._get_results(*args, **kwargs): File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/base/grpc.py", line 155, in _get_results raise e File "/home/ubuntu/pyenv/lib/python3.10/site-packages/jina/clients/base/grpc.py", line 135, in _get_results raise BadClient(msg) from err jina.excepts.BadClient: gRPC error: StatusCode.UNKNOWN Unexpected <class 'grpc.aio._call.AioRpcError'>: <AioRpcError of RPC that terminated with: status = StatusCode.UNKNOWN details = "Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'" debug_error_string = "{"created":"@1657125275.618452649","description":"Error received from peer ipv4:0.0.0.0:58903","file":"src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Unexpected <class 'TypeError'>: format_exception() got an unexpected keyword argument 'etype'","grpc_status":2}"


    AttributeError Traceback (most recent call last) File ~/pyenv/lib/python3.10/site-packages/jina/helper.py:1307, in run_async(func, *args, **kwargs) 1306 try: -> 1307 return thread.result 1308 except AttributeError:

    AttributeError: '_RunThread' object has no attribute 'result'

    During handling of the above exception, another exception occurred:

    BadClient Traceback (most recent call last) Input In [15], in <cell line: 12>() 8 f = Flow().add( 9 uses='jinahub://CLIPImageClassifier',name="classifier", 10 uses_with={'classes':classes}) 12 with f: ---> 13 f.post(on='/classify', inputs=docs, on_done=lambda resp: print(resp.docs[0].tags['class']['label']))

    File ~/pyenv/lib/python3.10/site-packages/jina/clients/mixin.py:173, in PostMixin.post(self, on, inputs, on_done, on_error, on_always, parameters, target_executor, request_size, show_progress, continue_on_error, return_responses, **kwargs) 170 if return_results: 171 return result --> 173 return run_async( 174 _get_results, 175 inputs=inputs, 176 on_done=on_done, 177 on_error=on_error, 178 on_always=on_always, 179 exec_endpoint=on, 180 target_executor=target_executor, 181 parameters=parameters, 182 request_size=request_size, 183 **kwargs, 184 )

    File ~/pyenv/lib/python3.10/site-packages/jina/helper.py:1311, in run_async(func, *args, **kwargs) 1308 except AttributeError: 1309 from jina.excepts import BadClient -> 1311 raise BadClient( 1312 'something wrong when running the eventloop, result can not be retrieved' 1313 ) 1314 else: 1316 raise RuntimeError( 1317 'you have an eventloop running but not using Jupyter/ipython, ' 1318 'this may mean you are using Jina with other integration? if so, then you ' 1319 'may want to use Client/Flow(asyncio=True). If not, then ' 1320 'please report this issue here: https://github.com/jina-ai/jina' 1321 )

    BadClient: something wrong when running the eventloop, result can not be retrieved

    opened by sk-haghighi 4
Releases(v0.2)
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
Deep learned, hardware-accelerated 3D object pose estimation

Isaac ROS Pose Estimation Overview This repository provides NVIDIA GPU-accelerated packages for 3D object pose estimation. Using a deep learned pose e

NVIDIA Isaac ROS 41 Dec 18, 2022
Attempt at implementation of a simple GAN using Keras

Simple GAN This is my attempt to make a wrapper class for a GAN in keras which can be used to abstract the whole architecture process. Simple GAN Over

Deven96 7 May 23, 2019
Illuminated3D This project participates in the Nasa Space Apps Challenge 2021.

Illuminated3D This project participates in the Nasa Space Apps Challenge 2021.

Eleftheriadis Emmanouil 1 Oct 09, 2021
This repository contains Prior-RObust Bayesian Optimization (PROBO) as introduced in our paper "Accounting for Gaussian Process Imprecision in Bayesian Optimization"

Prior-RObust Bayesian Optimization (PROBO) Introduction, TOC This repository contains Prior-RObust Bayesian Optimization (PROBO) as introduced in our

Julian Rodemann 2 Mar 19, 2022
Leaderboard and Visualization for RLCard

RLCard Showdown This is the GUI support for the RLCard project and DouZero project. RLCard-Showdown provides evaluation and visualization tools to hel

Data Analytics Lab at Texas A&M University 246 Dec 26, 2022
Simple reference implementation of GraphSAGE.

Reference PyTorch GraphSAGE Implementation Author: William L. Hamilton Basic reference PyTorch implementation of GraphSAGE. This reference implementat

William L Hamilton 861 Jan 06, 2023
MetaTTE: a Meta-Learning Based Travel Time Estimation Model for Multi-city Scenarios

MetaTTE: a Meta-Learning Based Travel Time Estimation Model for Multi-city Scenarios This is the official TensorFlow implementation of MetaTTE in the

morningstarwang 4 Dec 14, 2022
2D&3D human pose estimation

Human Pose Estimation Papers [CVPR 2016] - 201511 [IJCAI 2016] - 201602 Other Action Recognition with Joints-Pooled 3D Deep Convolutional Descriptors

133 Jan 02, 2023
Rainbow: Combining Improvements in Deep Reinforcement Learning

Rainbow Rainbow: Combining Improvements in Deep Reinforcement Learning [1]. Results and pretrained models can be found in the releases. DQN [2] Double

Kai Arulkumaran 1.4k Dec 29, 2022
Code to reproduce the experiments in the paper "Transformer Based Multi-Source Domain Adaptation" (EMNLP 2020)

Transformer Based Multi-Source Domain Adaptation Dustin Wright and Isabelle Augenstein To appear in EMNLP 2020. Read the preprint: https://arxiv.org/a

CopeNLU 36 Dec 05, 2022
An excellent hash algorithm combining classical sponge structure and RNN.

SHA-RNN Recurrent Neural Network with Chaotic System for Hash Functions Anonymous Authors [摘要] 在这次作业中我们提出了一种新的 Hash Function —— SHA-RNN。其以海绵结构为基础,融合了混

Houde Qian 5 May 15, 2022
Implement Decoupled Neural Interfaces using Synthetic Gradients in Pytorch

disclaimer: this code is modified from pytorch-tutorial Image classification with synthetic gradient in Pytorch I implement the Decoupled Neural Inter

Andrew 114 Dec 22, 2022
Adversarial Reweighting for Partial Domain Adaptation

Adversarial Reweighting for Partial Domain Adaptation Code for paper "Xiang Gu, Xi Yu, Yan Yang, Jian Sun, Zongben Xu, Adversarial Reweighting for Par

12 Dec 01, 2022
A way to store images in YAML.

YAMLImg A way to store images in YAML. I made this after seeing Roadcrosser's JSON-G because it was too inspiring to ignore this opportunity. Installa

5 Mar 14, 2022
NeuPy is a Tensorflow based python library for prototyping and building neural networks

NeuPy v0.8.2 NeuPy is a python library for prototyping and building neural networks. NeuPy uses Tensorflow as a computational backend for deep learnin

Yurii Shevchuk 729 Jan 03, 2023
A collection of resources and papers on Diffusion Models, a darkhorse in the field of Generative Models

This repository contains a collection of resources and papers on Diffusion Models and Score-based Models. If there are any missing valuable resources

5.1k Jan 08, 2023
Transformer Huffman coding - Complete Huffman coding through transformer

Transformer_Huffman_coding Complete Huffman coding through transformer 2022/2/19

3 May 19, 2022
Trainable Bilateral Filter Layer (PyTorch)

Trainable Bilateral Filter Layer (PyTorch) This repository contains our GPU-accelerated trainable bilateral filter layer (three spatial and one range

FabianWagner 26 Dec 25, 2022
Measuring if attention is explanation with ROAR

NLP ROAR Interpretability Official code for: Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Toke

Andreas Madsen 19 Nov 13, 2022
Automated Attendance Project Using Face Recognition

dependencies for project: cmake 3.22.1 dlib 19.22.1 face-recognition 1.3.0 openc

Rohail Taha 1 Jan 09, 2022