Generate images from texts. In Russian

Overview

ruDALL-E

Generate images from texts

Apache license Downloads Coverage Status pipeline pre-commit.ci status

pip install rudalle==1.1.0rc0

🤗 HF Models:

ruDALL-E Malevich (XL)
ruDALL-E Emojich (XL) (readme here)
ruDALL-E Surrealist (XL)

Minimal Example:

Open In Colab Kaggle Hugging Face Spaces

Example usage ruDALL-E Malevich (XL) with 3.5GB vRAM! Open In Colab

Finetuning example Open In Colab

generation by ruDALLE:

import ruclip
from rudalle.pipelines import generate_images, show, super_resolution, cherry_pick_by_ruclip
from rudalle import get_rudalle_model, get_tokenizer, get_vae, get_realesrgan
from rudalle.utils import seed_everything

# prepare models:
device = 'cuda'
dalle = get_rudalle_model('Malevich', pretrained=True, fp16=True, device=device)
tokenizer = get_tokenizer()
vae = get_vae(dwt=True).to(device)

# pipeline utils:
realesrgan = get_realesrgan('x2', device=device)
clip, processor = ruclip.load('ruclip-vit-base-patch32-384', device=device)
clip_predictor = ruclip.Predictor(clip, processor, device, bs=8)
text = 'радуга на фоне ночного города'

seed_everything(42)
pil_images = []
scores = []
for top_k, top_p, images_num in [
    (2048, 0.995, 24),
]:
    _pil_images, _scores = generate_images(text, tokenizer, dalle, vae, top_k=top_k, images_num=images_num, bs=8, top_p=top_p)
    pil_images += _pil_images
    scores += _scores

show(pil_images, 6)

auto cherry-pick by ruCLIP:

top_images, clip_scores = cherry_pick_by_ruclip(pil_images, text, clip_predictor, count=6)
show(top_images, 3)

super resolution:

sr_images = super_resolution(top_images, realesrgan)
show(sr_images, 3)

text, seed = 'красивая тян из аниме', 6955

Image Prompt

see jupyters/ruDALLE-image-prompts-A100.ipynb

text, seed = 'Храм Василия Блаженного', 42
skyes = [red_sky, sunny_sky, cloudy_sky, night_sky]

Aspect ratio images -->NEW<--

🚀 Contributors 🚀

Supported by

Social Media

Comments
  • Smaller / Distilled model?

    Smaller / Distilled model?

    Will there be a smaller or a distilled model release? The problem with inferencing in google colab is the speeds. 4:32 for one image on a P100, and 2 hours+ for 3 images on K80.

    opened by johnpaulbin 10
  • RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

    RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR

    i use default code and get error after generation 100% please help i use windows and conda

    `◼️ Malevich is 1.3 billion params model from the family GPT3-like, that uses Russian language and text+image multi-modality. x4 --> ready tokenizer --> ready Working with z of shape (1, 256, 32, 32) = 262144 dimensions. vae --> ready ruclip --> ready 100%|██████████████████████████████████████████████████████████████████████████████| 1024/1024 [00:46<00:00, 22.14it/s] Traceback (most recent call last): File "gen.py", line 29, in _pil_images, _scores = generate_images(text, tokenizer, dalle, vae, top_k=top_k, images_num=images_num, top_p=top_p) File "C:\Users\1\anaconda3\lib\site-packages\rudalle\pipelines.py", line 60, in generate_images images = vae.decode(codebooks) File "C:\Users\1\anaconda3\lib\site-packages\rudalle\vae\model.py", line 38, in decode img = self.model.decode(z) File "C:\Users\1\anaconda3\lib\site-packages\rudalle\vae\model.py", line 98, in decode quant = self.post_quant_conv(quant) File "C:\Users\1\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "C:\Users\1\anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 399, in forward return self._conv_forward(input, self.weight, self.bias) File "C:\Users\1\anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 395, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

    import torch torch.backends.cuda.matmul.allow_tf32 = True torch.backends.cudnn.benchmark = True torch.backends.cudnn.deterministic = True torch.backends.cudnn.allow_tf32 = True data = torch.randn([3, 256, 32, 32], dtype=torch.float, device='cuda', requires_grad=True).to(memory_format=torch.channels_last) net = torch.nn.Conv2d(256, 256, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1) net = net.cuda().float().to(memory_format=torch.channels_last) out = net(data) out.backward(torch.randn_like(out)) torch.cuda.synchronize()

    ConvolutionParams data_type = CUDNN_DATA_FLOAT padding = [0, 0, 0] stride = [1, 1, 0] dilation = [1, 1, 0] groups = 1 deterministic = true allow_tf32 = true input: TensorDescriptor 0000020481F094B0 type = CUDNN_DATA_FLOAT nbDims = 4 dimA = 3, 256, 32, 32, strideA = 262144, 1, 8192, 256, output: TensorDescriptor 0000020481F09590 type = CUDNN_DATA_FLOAT nbDims = 4 dimA = 3, 256, 32, 32, strideA = 262144, 1, 8192, 256, weight: FilterDescriptor 000001FFD2E76AF0 type = CUDNN_DATA_FLOAT tensor_format = CUDNN_TENSOR_NHWC nbDims = 4 dimA = 256, 256, 1, 1, Pointer addresses: input: 0000001538C7D000 output: 000000153B87D000 weight: 00000014D3BB0000 `

    opened by bitcoin5000 7
  • Auto cut pictures into separated images

    Auto cut pictures into separated images

    Есть ли какие-нибудь параметры, которые автоматически нарежут и сохранят сгенерированные картинки по отдельности?


    Are there any args that will automatically cut and save separated images?

    opened by Sidiusz 4
  • Gradient checkpointing

    Gradient checkpointing

    This patch enables gradient checkpointing for ruDALLE.

    It's possible to use up to 3x higher batch sizes in memory-limited environments during training.

    Setting the gradient_checkpointing during model.forward makes a checkpoint every gradient_checkpointing layers. 6 is a good starting value.

    opened by neverix 3
  • Feature/dwt vae

    Feature/dwt vae

    add support decoding vae with DWT (discrete wavelet transform):

    allow restore 512x512 images

    thanks a lot @bes for issue https://github.com/sberbank-ai/ru-dalle/issues/42 with this idea 👍

    vae = get_vae(dwt=True)
    
    opened by shonenkov 3
  • optimize image prompts

    optimize image prompts

    This enables caching for image prompts. For some reason, the results change slightly. I tried looking for off-by-one bugs in this, but couldn't find one myself.

    opened by neverix 3
  • The error in ruDall-e code that published in Kaggle

    The error in ruDall-e code that published in Kaggle

    Execution of ruDall-e code in the Kaggle notebook (as is published), in GPU session ends with error:

    ModuleNotFoundError                       Traceback (most recent call last)
    /tmp/ipykernel_29/1914141142.py in <module>
    ----> 1 from rudalle.pipelines import generate_images, show, super_resolution, cherry_pick_by_clip
          2 from rudalle import get_rudalle_model, get_tokenizer, get_vae, get_realesrgan, get_ruclip
          3 from rudalle.utils import seed_everything
    
    ModuleNotFoundError: No module named 'rudalle'
    
    

    The error message refers to this code:

    !pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio==0.7.2 -f https://download.pytorch.org/whl/torch_stable.html > /dev/null
    !pip install rudalle==0.0.1rc1 > /dev/null
    
    opened by XieBaoshi 3
  • Constantly having to redownload models

    Constantly having to redownload models

    Hi, I've noticed that running it on a local jupyter instance will always redownload the model again. Is there a way I can avoid this as I don't want to be waiting for it to finish everytime. Thanks/

    opened by JohnnyRacer 2
  • Problem about the PyTorch vision?

    Problem about the PyTorch vision?

    I have look for the issues but I can't find the same problem. So sorry to bother you. GPU: 截屏2021-12-02 下午6 35 14 my python environment: pytorch=1.8.0&torchvision=0.9.0, cudatoolkit=11.3.1&cudnn =8.2.1. I have tried the rudalle=0.3.0 just following the readme.md, or 0.0.1rc5 by the RTX3090.ipynb, but I only got the following error! 截屏2021-12-02 下午6 38 49

    So I wanna know if any problem in my environment? Waiting for your reply!

    opened by Wang-Xiaodong1899 2
  • image_prompts.py – borders crop not working properly

    image_prompts.py – borders crop not working properly

    From an official documentation:

    borders (dict[str] | int): borders that we croped from pil_image example: {'up': 4, 'right': 0, 'left': 0, 'down': 0} (1 int eq 8 pixels)

    Up crop works just fine. But if I will pass as a crop argument something other than "Up" in the result, I will get an AssertionError: telegram-cloud-photo-size-2-5197407051389712641-y

    Thank you for a fantastic algo ✨

    opened by DenisSergeevitch 2
  • Не запускается generate_images

    Не запускается generate_images

    Пытаюсь запустить на device = 'cpu'. Пример из README самый первый

    Падает с таким трейсбеком. Что я делаю не так?

    ◼️ Malevich is 1.3 billion params model from the family GPT3-like, that uses Russian language and text+image multi-modality.
    x4 --> ready
    tokenizer --> ready
    Working with z of shape (1, 256, 32, 32) = 262144 dimensions.
    vae --> ready
    ruclip --> ready
      0%|          | 0/1024 [00:00<?, ?it/s]
    Traceback (most recent call last):
      File "%projectfolder%\test\venv\lib\site-packages\rudalle\pipelines.py", line 46, in generate_images
        logits, has_cache = dalle(out, attention_mask,
      File "%projectfolder%\test\venv\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
      File "%projectfolder%\test\venv\lib\site-packages\rudalle\dalle\fp16.py", line 51, in forward
        return fp16_to_fp32(self.module(*(fp32_to_fp16(inputs)), **kwargs))
      File "%projectfolder%\test\venv\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
      File "%projectfolder%\test\venv\lib\site-packages\rudalle\dalle\model.py", line 150, in forward
        transformer_output, present_has_cache = self.transformer(
      File "%projectfolder%\test\venv\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
      File "%projectfolder%\test\venv\lib\site-packages\rudalle\dalle\transformer.py", line 76, in forward
        hidden_states, present_has_cache = layer(hidden_states, mask, has_cache=has_cache, use_cache=use_cache)
      File "%projectfolder%\test\venv\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
      File "%projectfolder%\test\venv\lib\site-packages\rudalle\dalle\transformer.py", line 146, in forward
        layernorm_output = self.input_layernorm(hidden_states)
      File "%projectfolder%\test\venv\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
        return forward_call(*input, **kwargs)
      File "%projectfolder%\test\venv\lib\site-packages\torch\nn\modules\normalization.py", line 173, in forward
        return F.layer_norm(
      File "%projectfolder%\test\venv\lib\site-packages\torch\nn\functional.py", line 2346, in layer_norm
        return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
    RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'
    
    opened by Xoma163 2
  • Add optional resume_download argument to help download large models

    Add optional resume_download argument to help download large models

    It's kinda pain to download large models with unstable network connection. For instance, i've started seeing this type of error (see screenshot). It breaks download process and you have to start again from zero bytes downloaded.

    However, cached_download(..) function in huggingface_hub has resume_download argument that can be used to restart download without loosing progress. See this line. So i think it would be helpful to add it as optional argument(defaults to False) to the get_rudalle_model(..) so users can turn it on if they have unstable internet.

    opened by Rexhaif 0
  • kandinsky model not available

    kandinsky model not available

    Nice to see the update! There is an auth error with the kandinsky model. Not sure if this is intended as there seem to be some token requirement. Could you clarify?

    opened by xavierleung 0
  • RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.

    RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1.

    What might be causing this ?

    RuntimeError: nvrtc: error: failed to open libnvrtc-builtins.so.11.1. Make sure that libnvrtc-builtins.so.11.1 is installed correctly. nvrtc compilation failed:

    #define NAN __int_as_float(0x7fffffff)
    #define POS_INFINITY __int_as_float(0x7f800000)
    #define NEG_INFINITY __int_as_float(0xff800000)
    
    
    template<typename T>
    __device__ T maximum(T a, T b) {
      return isnan(a) ? a : (a > b ? a : b);
    }
    
    template<typename T>
    __device__ T minimum(T a, T b) {
      return isnan(a) ? a : (a < b ? a : b);
    }
    
    
    #define __HALF_TO_US(var) *(reinterpret_cast<unsigned short *>(&(var)))
    #define __HALF_TO_CUS(var) *(reinterpret_cast<const unsigned short *>(&(var)))
    #if defined(__cplusplus)
      struct __align__(2) __half {
        __host__ __device__ __half() { }
    
      protected:
        unsigned short __x;
      };
    
      /* All intrinsic functions are only available to nvcc compilers */
      #if defined(__CUDACC__)
        /* Definitions of intrinsics */
        __device__ __half __float2half(const float f) {
          __half val;
          asm("{  cvt.rn.f16.f32 %0, %1;}\n" : "=h"(__HALF_TO_US(val)) : "f"(f));
          return val;
        }
    
        __device__ float __half2float(const __half h) {
          float val;
          asm("{  cvt.f32.f16 %0, %1;}\n" : "=f"(val) : "h"(__HALF_TO_CUS(h)));
          return val;
        }
    
      #endif /* defined(__CUDACC__) */
    #endif /* defined(__cplusplus) */
    #undef __HALF_TO_US
    #undef __HALF_TO_CUS
    
    typedef __half half;
    
    extern "C" __global__
    void fused_mul_mul_mul_mu_5065363705190979294(half* t0, half* aten_mul) {
    {
      float t0_1 = __half2float(t0[(8192 * (((512 * blockIdx.x + threadIdx.x) / 8192) % 128) + ((512 * blockIdx.x + threadIdx.x) / 1048576) * 1048576) + (512 * blockIdx.x + threadIdx.x) % 8192]);
      aten_mul[(8192 * (((512 * blockIdx.x + threadIdx.x) / 8192) % 128) + ((512 * blockIdx.x + threadIdx.x) / 1048576) * 1048576) + (512 * blockIdx.x + threadIdx.x) % 8192] = __float2half((t0_1 * 0.5f) * ((tanhf((t0_1 * 0.7978845834732056f) * ((t0_1 * 0.04471499845385551f) * t0_1 + 1.f))) + 1.f));
    }
    }
    
    opened by c0ffymachyne 1
  • Bad syntax in collab

    Bad syntax in collab

    In https://colab.research.google.com/drive/1wGE-046et27oHvNlBNPH07qrEQNE04PQ?usp=sharing#scrollTo=GdOYJvwZSB-D

    it should be a couple of quotes (") in the text parameter:

    text = Что бы ни # @param

    Should be:

    text = "Что бы ни" # @param

    Thanks!

    opened by Jakeukalane 1
Releases(v1.1.0)
Owner
AI Forever
Creating ML for the future. AI projects you already know. We are non-profit organization with members from all over the world.
AI Forever
Machine Learning toolbox for Humans

Reproducible Experiment Platform (REP) REP is ipython-based environment for conducting data-driven research in a consistent and reproducible way. Main

Yandex 662 Nov 20, 2022
Submanifold sparse convolutional networks

Submanifold Sparse Convolutional Networks This is the PyTorch library for training Submanifold Sparse Convolutional Networks. Spatial sparsity This li

Facebook Research 1.8k Jan 06, 2023
Official Pytorch Implementation of: "Semantic Diversity Learning for Zero-Shot Multi-label Classification"(2021) paper

Semantic Diversity Learning for Zero-Shot Multi-label Classification Paper Official PyTorch Implementation Avi Ben-Cohen, Nadav Zamir, Emanuel Ben Bar

28 Aug 29, 2022
This script runs neural style transfer against the provided content image.

Neural Style Transfer Content Style Output Description: This script runs neural style transfer against the provided content image. The content image m

Martynas Subonis 0 Nov 25, 2021
Classifying audio using Wavelet transform and deep learning

Audio Classification using Wavelet Transform and Deep Learning A step-by-step tutorial to classify audio signals using continuous wavelet transform (C

Aditya Dutt 17 Nov 29, 2022
Code for the Paper: Alexandra Lindt and Emiel Hoogeboom.

Discrete Denoising Flows This repository contains the code for the experiments presented in the paper Discrete Denoising Flows [1]. To give a short ov

Alexandra Lindt 3 Oct 09, 2022
[NeurIPS 2021] Well-tuned Simple Nets Excel on Tabular Datasets

[NeurIPS 2021] Well-tuned Simple Nets Excel on Tabular Datasets Introduction This repo contains the source code accompanying the paper: Well-tuned Sim

52 Jan 04, 2023
Multistream CNN for Robust Acoustic Modeling

Multistream Convolutional Neural Network (CNN) A multistream CNN is a novel neural network architecture for robust acoustic modeling in speech recogni

ASAPP Research 37 Sep 21, 2022
A FAIR dataset of TCV experimental results for validating edge/divertor turbulence models.

TCV-X21 validation for divertor turbulence simulations Quick links Intro Welcome to TCV-X21. We're glad you've found us! This repository is designed t

0 Dec 18, 2021
imbalanced-DL: Deep Imbalanced Learning in Python

imbalanced-DL: Deep Imbalanced Learning in Python Overview imbalanced-DL (imported as imbalanceddl) is a Python package designed to make deep imbalanc

NTUCSIE CLLab 19 Dec 28, 2022
a reimplementation of UnFlow in PyTorch that matches the official TensorFlow version

pytorch-unflow This is a personal reimplementation of UnFlow [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 134 Nov 20, 2022
Face recognition project by matching the features extracted using SIFT.

MV_FaceDetectionWithSIFT Face recognition project by matching the features extracted using SIFT. By : Aria Radmehr Professor : Ali Amiri Dependencies

Aria Radmehr 4 May 31, 2022
Bayesian algorithm execution (BAX)

Bayesian Algorithm Execution (BAX) Code for the paper: Bayesian Algorithm Execution: Estimating Computable Properties of Black-box Functions Using Mut

Willie Neiswanger 38 Dec 08, 2022
A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets

HOW TO USE THIS PROJECT A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets Based on DeepLabCut toolbox, we run wit

1 Jan 10, 2022
A scikit-learn-compatible module for estimating prediction intervals.

MAPIE - Model Agnostic Prediction Interval Estimator MAPIE allows you to easily estimate prediction intervals (or prediction sets) using your favourit

588 Jan 04, 2023
Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors

PSML paper: Towards Improving Embedding Based Models of Social Network Alignment via Pseudo Anchors PSML_IONE,PSML_ABNE,PSML_DEEPLINK,PSML_SNNA: numpy

13 Nov 27, 2022
Fast, Attemptable Route Planner for Navigation in Known and Unknown Environments

FAR Planner uses a dynamically updated visibility graph for fast replanning. The planner models the environment with polygons and builds a global visi

Fan Yang 346 Dec 30, 2022
Tensorflow implementation of "BEGAN: Boundary Equilibrium Generative Adversarial Networks"

BEGAN in Tensorflow Tensorflow implementation of BEGAN: Boundary Equilibrium Generative Adversarial Networks. Requirements Python 2.7 or 3.x Pillow tq

Taehoon Kim 922 Dec 21, 2022
We utilize deep reinforcement learning to obtain favorable trajectories for visual-inertial system calibration.

Unified Data Collection for Visual-Inertial Calibration via Deep Reinforcement Learning Update: The lastest code will be updated in this branch. Pleas

ETHZ ASL 27 Dec 29, 2022
Semi-SDP Semi-supervised parser for semantic dependency parsing.

Semi-SDP Semi-supervised parser for semantic dependency parsing. This repo contains the code used for the semi-supervised semantic dependency parser i

12 Sep 17, 2021