Unified API to facilitate usage of pre-trained "perceptor" models, a la CLIP

Related tags

Deep LearningMulti-Modal-Comparators
Overview

mmc

installation

git clone https://github.com/dmarx/Multi-Modal-Comparators
cd 'Multi-Modal-Comparators'
pip install poetry
poetry build
pip install dist/mmc*.whl

# optional final step:
#poe napm_installs
python src/mmc/napm_installs/__init__.py

To see which models are immediately available, run:

python -m mmc.loaders

That optional poe napm_installs step

For the most convenient experience, it is recommended that you perform the final poe napm_installs step. Omitting this step will make your one-time setup faster, but will make certain use cases more complex.

If you did not perform the optional poe napm_installs step, you likely received several warnings about models whose loaders could not be registered. These are models whose codebases depend on python code which is not trivially installable. You will still have access to all of the models supported by the library as if you had run the last step, but their loaders will not be queryable from the registry (see below) and will need to be loaded via the appropriate mmc.loader directly, which may be non-trivial to identify without the ability to query it from mmc's registry.

As a concrete example, if the napm step is skipped, the model [cloob - corwsonkb - cloob_laion_400m_vit_b_16_32_epochs] will not appear in the list of registered loaders, but can still be loaded like this:

from mmc.loaders import KatCloobLoader

model = KatCloobLoader(id='cloob_laion_400m_vit_b_16_32_epochs').load()

Invoking the load() method on an unregistered loader will invoke napm to prepare any uninstallable dependencies required to load the model. Next time you run python -m mmc.loaders, the CLOOB loader will show as registered and spinning up the registry will longer emit a warning for that model.

Usage

TLDR

# spin up the registry
from mmc import loaders

from mmc.mock.openai import MockOpenaiClip
from mmc.registry import REGISTRY

cloob_query = {architecture='cloob'}
cloob_loaders = REGISTRY.find(**cloob_query)

# loader repl prints attributes for uniquely querying
print(cloob_loaders)

# loader returns a perceptor whose API is standardized across mmc
cloob_model = cloob_loaders[0].load()

# wrapper classes are provided for mocking popular implementations
# to facilitate drop-in compatibility with existing code
drop_in_replacement__cloob_model = MockOpenaiClip(cloob_model)

Querying the Model Registry

Spin up the model registry by importing the loaders module:

from mmc import loaders

To see which models are available:

from mmc.registry import REGISTRY

for loader in REGISTRY.find():
    print(loader)

You can constrain the result set by querying the registry for specific metadata attributes

# all CLIP models
clip_loaders = REGISTRY.find(architecture='clip')

# CLIP models published by openai
openai_clip_loaders = REGISTRY.find(architecture='clip', publisher='openai')

# All models published by MLFoundations (openCLIP)
mlf_loaders = REGISTRY.find(publisher='mlfoundations)'

# A specific model
rn50_loader = REGISTRY.find(architecture='clip', publisher='openai', id='RN50')
# NB: there may be multiple models matching a particular "id". the 'id' field
# only needs to be unique for a given architecture-publisher pair.

All pretrained checkpoints are uniquely identifiable by a combination of architecture, publisher, and id.

The above queries return lists of loader objects. If model artifacts (checkpoints, config) need to be downloaded, they will only be downloaded after the load() method on the loader is invoked.

loaders = REGISTRY.find(...)
loader = loaders[0] # just picking an arbitrary return value here, remember: loaders is a *list* of loaders
model = loader.load()

The load() method returns an instance of an mmc.MultiModalComparator. The MultiModalComparator class is a modality-agnostic abstraction. I'll get to the ins and outs of that another time.

API Mocking

You want something you can just drop into your code and it'll work. We got you. This library provides wrapper classes to mock the APIs of commonly used CLIP implementations. To wrap a MultiModalComparator so it can be used as a drop-in replacement with code compatible with OpenAI's CLIP:

from mmc.mock.openai import MockOpenaiClip

my_model = my_model_loader.load()
model = MockOpenaiClip(my_model)

MultiMMC: Multi-Perceptor Implementation

The MultiMMC class can be used to run inference against multiple mmc models in parallel. This form of ensemble is sometimes referred to as a "multi-perceptor".

To ensure that all models loaded into the MultiMMC are compatible, the MultiMMC instance is initialized by specifying the modalities it supports. We'll discuss modality objects in a bit.

from mmc.multimmc import MultiMMC
from mmc.modalities import TEXT, IMAGE

perceptor = MultiMMC(TEXT, IMAGE)

To load and use a model:

perceptor.load_model(
    architecture='clip', 
    publisher='openai', 
    id='RN50',
)

score = perceptor.compare(
    image=PIL.Image.open(...), 
    text=text_pos),
)

Additional models can be added to the ensemble via the load_model() method.

The MultiMMC does not support API mocking because of its reliance on the compare method.

Available Pre-trained Models

Some model comparisons here

# [<architecture> - <publisher> - <id>]
[clip - openai - RN50]
[clip - openai - RN101]
[clip - openai - RN50x4]
[clip - openai - RN50x16]
[clip - openai - RN50x64]
[clip - openai - ViT-B/32]
[clip - openai - ViT-B/16]
[clip - openai - ViT-L/14]
[clip - openai - ViT-L/[email protected]]
[clip - mlfoundations - RN50--openai]
[clip - mlfoundations - RN50--yfcc15m]
[clip - mlfoundations - RN50--cc12m]
[clip - mlfoundations - RN50-quickgelu--openai]
[clip - mlfoundations - RN50-quickgelu--yfcc15m]
[clip - mlfoundations - RN50-quickgelu--cc12m]
[clip - mlfoundations - RN101--openai]
[clip - mlfoundations - RN101--yfcc15m]
[clip - mlfoundations - RN101-quickgelu--openai]
[clip - mlfoundations - RN101-quickgelu--yfcc15m]
[clip - mlfoundations - RN50x4--openai]
[clip - mlfoundations - RN50x16--openai]
[clip - mlfoundations - ViT-B-32--openai]
[clip - mlfoundations - ViT-B-32--laion400m_e31]
[clip - mlfoundations - ViT-B-32--laion400m_e32]
[clip - mlfoundations - ViT-B-32--laion400m_avg]
[clip - mlfoundations - ViT-B-32-quickgelu--openai]
[clip - mlfoundations - ViT-B-32-quickgelu--laion400m_e31]
[clip - mlfoundations - ViT-B-32-quickgelu--laion400m_e32]
[clip - mlfoundations - ViT-B-32-quickgelu--laion400m_avg]
[clip - mlfoundations - ViT-B-16--openai]
[clip - mlfoundations - ViT-L-14--openai]
[clip - sbert - ViT-B-32-multilingual-v1]
[clip - sajjjadayobi - clipfa]

# The following models depend on napm for setup
[clip - navervision - kelip_ViT-B/32]
[cloob - crowsonkb - cloob_laion_400m_vit_b_16_16_epochs]
[cloob - crowsonkb - cloob_laion_400m_vit_b_16_32_epochs]
[clip - facebookresearch - clip_small_25ep]
[clip - facebookresearch - clip_base_25ep]
[clip - facebookresearch - clip_large_25ep]
[slip - facebookresearch - slip_small_25ep]
[slip - facebookresearch - slip_small_50ep]
[slip - facebookresearch - slip_small_100ep]
[slip - facebookresearch - slip_base_25ep]
[slip - facebookresearch - slip_base_50ep]
[slip - facebookresearch - slip_base_100ep]
[slip - facebookresearch - slip_large_25ep]
[slip - facebookresearch - slip_large_50ep]
[slip - facebookresearch - slip_large_100ep]
[simclr - facebookresearch - simclr_small_25ep]
[simclr - facebookresearch - simclr_base_25ep]
[simclr - facebookresearch - simclr_large_25ep]
[clip - facebookresearch - clip_base_cc3m_40ep]
[clip - facebookresearch - clip_base_cc12m_35ep]
[slip - facebookresearch - slip_base_cc3m_40ep]
[slip - facebookresearch - slip_base_cc12m_35ep]

VRAM Cost

The following is an estimate of the amount of space the loaded model occupies in memory:

publisher architecture model_name vram_mb
0 openai clip RN50 358
1 openai clip RN101 294
2 openai clip RN50x4 424
3 openai clip RN50x16 660
4 openai clip RN50x64 1350
5 openai clip ViT-B/32 368
6 openai clip ViT-B/16 348
7 openai clip ViT-L/14 908
8 openai clip ViT-L/[email protected] 908
9 mlfoundations clip RN50--openai 402
10 mlfoundations clip RN50--yfcc15m 402
11 mlfoundations clip RN50--cc12m 402
12 mlfoundations clip RN50-quickgelu--openai 402
13 mlfoundations clip RN50-quickgelu--yfcc15m 402
14 mlfoundations clip RN50-quickgelu--cc12m 402
15 mlfoundations clip RN101--openai 476
16 mlfoundations clip RN101--yfcc15m 476
17 mlfoundations clip RN101-quickgelu--openai 476
18 mlfoundations clip RN101-quickgelu--yfcc15m 476
19 mlfoundations clip RN50x4--openai 732
20 mlfoundations clip RN50x16--openai 1200
21 mlfoundations clip ViT-B-32--openai 634
22 mlfoundations clip ViT-B-32--laion400m_e31 634
23 mlfoundations clip ViT-B-32--laion400m_e32 634
24 mlfoundations clip ViT-B-32--laion400m_avg 634
25 mlfoundations clip ViT-B-32-quickgelu--openai 634
26 mlfoundations clip ViT-B-32-quickgelu--laion400m_e31 634
27 mlfoundations clip ViT-B-32-quickgelu--laion400m_e32 634
28 mlfoundations clip ViT-B-32-quickgelu--laion400m_avg 634
29 mlfoundations clip ViT-B-16--openai 634
30 mlfoundations clip ViT-L-14--openai 1688
32 sajjjadayobi clip clipfa 866
33 crowsonkb cloob cloob_laion_400m_vit_b_16_16_epochs 610
34 crowsonkb cloob cloob_laion_400m_vit_b_16_32_epochs 610
36 facebookresearch slip slip_small_25ep 728
37 facebookresearch slip slip_small_50ep 650
38 facebookresearch slip slip_small_100ep 650
39 facebookresearch slip slip_base_25ep 714
40 facebookresearch slip slip_base_50ep 714
41 facebookresearch slip slip_base_100ep 714
42 facebookresearch slip slip_large_25ep 1534
43 facebookresearch slip slip_large_50ep 1522
44 facebookresearch slip slip_large_100ep 1522
45 facebookresearch slip slip_base_cc3m_40ep 714
46 facebookresearch slip slip_base_cc12m_35ep 714

Contributing

Suggest a pre-trained model

If you would like to suggest a pre-trained model for future addition, you can add a comment to this issue

Add a pre-trained model

  1. Create a loader class that encapsulates the logic for importing the model, loading weights, preprocessing inputs, and performing projections.
  2. At the bottom of the file defining the loader class should be a code snippet that adds each respective checkpoint's loader to the registry.
  3. Add an import for the new file to mmc/loaders/__init__.py. The imports in this file are the reason import mmc.loaders "spins up" the registry.
  4. If the codebase on which the model depends can be installed, update pytproject.toml to install it.
  5. Otherwise, add napm preparation at the top of the loaders load method (see cloob or kelip for examples), and also add napm setup to mmc/napm_installs/__init__.py
  6. Add a test case to tests/test_mmc_loaders.py
  7. Add a test script for the loader (see test_mmc_katcloob as an example)
Owner
David Marx
Engineer / Machine Learning Researcher interested in deep learning, probabilistic ML, generative models, multi-modal SSL, visual understanding, geometric
David Marx
GitHub Repository
Author Disambiguation using Knowledge Graph Embeddings with Literals

Author Name Disambiguation with Knowledge Graph Embeddings using Literals This is the repository for the master thesis project on Knowledge Graph Embe

12 Oct 19, 2022
Efficient face emotion recognition in photos and videos

This repository contains code of face emotion recognition that was developed in the RSF (Russian Science Foundation) project no. 20-71-10010 (Efficien

Andrey Savchenko 239 Jan 04, 2023
Spatial Contrastive Learning for Few-Shot Classification (SCL)

This repo contains the official implementation of Spatial Contrastive Learning for Few-Shot Classification (SCL), which presents of a novel contrastive learning method applied to few-shot image class

Yassine 34 Dec 25, 2022
Official Implementation of Few-shot Visual Relationship Co-localization

VRC Official implementation of the Few-shot Visual Relationship Co-localization (ICCV 2021) paper project page | paper Requirements Use python = 3.8.

22 Oct 13, 2022
Algorithm to texture 3D reconstructions from multi-view stereo images

MVS-Texturing Welcome to our project that textures 3D reconstructions from images. This project focuses on 3D reconstructions generated using structur

Nils Moehrle 766 Jan 04, 2023
A paper using optimal transport to solve the graph matching problem.

GOAT A paper using optimal transport to solve the graph matching problem. https://arxiv.org/abs/2111.05366 Repo structure .github: Files specifying ho

neurodata 8 Jan 04, 2023
This is the official pytorch implementation of Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation(TESKD)

Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation (TESKD) By Zheng Li[1,4], Xiang Li[2], Lingfeng Yang[2,4], Jian Yang[2], Zh

Zheng Li 9 Sep 26, 2022
根据midi文件演奏“风物之诗琴”的脚本 "Windsong Lyre" auto play

Genshin-lyre-auto-play 简体中文 | English 简介 根据midi文件演奏“风物之诗琴”的脚本。由Python驱动,在此承诺， ⚠️ 项目内绝不含任何能够引起安全问题的代码。 前排提示：所有键盘在动但是原神没反应的都是因为没有管理员权限，双击run.bat或者以管理员模式

御坂17032号 386 Jan 01, 2023
Lightweight stereo matching network based on MobileNetV1 and MobileNetV2

MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching

Cognitive Systems Research Group 139 Nov 30, 2022
Official implementation of the paper Chunked Autoregressive GAN for Conditional Waveform Synthesis

PyEmits, a python package for easy manipulation in time-series data. Time-series data is very common in real life. Engineering FSI industry (Financial

Descript 150 Dec 06, 2022
A GPT, made only of MLPs, in Jax

MLP GPT - Jax (wip) A GPT, made only of MLPs, in Jax. The specific MLP to be used are gMLPs with the Spatial Gating Units. Working Pytorch implementat

Phil Wang 53 Sep 27, 2022
CausalNLP is a practical toolkit for causal inference with text as treatment, outcome, or "controlled-for" variable.

CausalNLP CausalNLP is a practical toolkit for causal inference with text as treatment, outcome, or "controlled-for" variable. Install pip install -U

Arun S. Maiya 95 Jan 03, 2023
SMCA replication There are no extra compiled components in SMCA DETR and package dependencies are minimal

Usage There are no extra compiled components in SMCA DETR and package dependencies are minimal, so the code is very simple to use. We provide instruct

22 May 06, 2022
A PyTorch implementation of DenseNet.

A PyTorch Implementation of DenseNet This is a PyTorch implementation of the DenseNet-BC architecture as described in the paper Densely Connected Conv

Brandon Amos 771 Dec 15, 2022
An Artificial Intelligence trying to drive a car by itself on a user created map

An Artificial Intelligence trying to drive a car by itself on a user created map

Akhil Sahukaru 17 Jan 13, 2022
The code for paper "Learning Implicit Fields for Generative Shape Modeling".

implicit-decoder The tensorflow code for paper "Learning Implicit Fields for Generative Shape Modeling", Zhiqin Chen, Hao (Richard) Zhang. Project pag

Zhiqin Chen 353 Dec 30, 2022
traiNNer is an open source image and video restoration (super-resolution, denoising, deblurring and others) and image to image translation toolbox based on PyTorch.

traiNNer traiNNer is an open source image and video restoration (super-resolution, denoising, deblurring and others) and image to image translation to

202 Jan 04, 2023
Combining Reinforcement Learning and Constraint Programming for Combinatorial Optimization

Hybrid solving process for combinatorial optimization problems Combinatorial optimization has found applications in numerous fields, from aerospace to

117 Dec 13, 2022
This is a project based on retinaface face detection, including ghostnet and mobilenetv3

English | 简体中文 RetinaFace in PyTorch Chinese detailed blog：https://zhuanlan.zhihu.com/p/379730820 Face recognition with masks is still robust---------

pogg 59 Dec 21, 2022
A Pytorch Implementation of [Source data‐free domain adaptation of object detector through domain

A Pytorch Implementation of Source data‐free domain adaptation of object detector through domain‐specific perturbation Please follow Faster R-CNN and

1 Dec 25, 2021