Mesh Transformer Jax

A haiku library using the new(ly documented) xmap operator in Jax for model parallelism of transformers.

See enwik8_example.py for an example of using this to implement an autoregressive language model.

Benchmarks

On a TPU v3-8 (see tpuv38_example.py):

~2.7B model

Initialized in 121.842s
Total parameters: 2722382080
Compiled in 49.0534s
it: 0, loss: 20.311113357543945
<snip>
it: 90, loss: 3.987450361251831
100 steps in 109.385s
effective flops (not including attn): 2.4466e+14

~4.8B model

Initialized in 101.016s
Total parameters: 4836720896
Compiled in 52.7404s
it: 0, loss: 4.632925987243652
<snip>
it: 40, loss: 3.2406811714172363
50 steps in 102.559s
effective flops (not including attn): 2.31803e+14

10B model

Initialized in 152.762s
Total parameters: 10073579776
Compiled in 92.6539s
it: 0, loss: 5.3125
<snip>
it: 40, loss: 3.65625
50 steps in 100.235s
effective flops (not including attn): 2.46988e+14

Model parallel transformers in Jax and Haiku

Related tags

Overview

Mesh Transformer Jax

Benchmarks

~2.7B model

~4.8B model

10B model

TODO

Owner

Ben Wang

Project code for weakly supervised 3D object detectors using wide-baseline multi-view traffic camera data: WIBAM.

A High-Level Fusion Scheme for Circular Quantities published at the 20th International Conference on Advanced Robotics

Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation.

Deep Learning for Morphological Profiling

A toolkit for controlling Euro Truck Simulator 2 with python to develop self-driving algorithms.

🔥🔥High-Performance Face Recognition Library on PaddlePaddle & PyTorch🔥🔥

Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary Differential Equations

Model-based 3D Hand Reconstruction via Self-Supervised Learning, CVPR2021

COCO Style Dataset Generator GUI

Image De-raining Using a Conditional Generative Adversarial Network

Reverse engineer your pytorch vision models, in style

simple demo codes for Learning to Teach with Dynamic Loss Functions

Implement face detection, and age and gender classification, and emotion classification.

A collection of awesome resources image-to-image translation.

Face2webtoon - Despite its importance, there are few previous works applying I2I translation to webtoon.

A PyTorch-Based Framework for Deep Learning in Computer Vision

code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

A curated list of awesome resources combining Transformers with Neural Architecture Search

Code for the ICCV2021 paper "Personalized Image Semantic Segmentation"

[NeurIPS'21] "AugMax: Adversarial Composition of Random Augmentations for Robust Training" by Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Animashree Anandkumar, and Zhangyang Wang.