Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

Code for "On Memorization in Probabilistic Deep Generative Models"

Codes for the compilation and visualization examples to the HIF vegetation dataset

Topic Modelling for Humans

SigOpt wrappers for scikit-learn methods

The official implementation of the research paper "DAG Amendment for Inverse Control of Parametric Shapes"

4th place solution for the SIGIR 2021 challenge.

This repository provides code for "On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness".

Pytorch Implementation of Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

Code for binary and multiclass model change active learning, with spectral truncation implementation.

Unofficial implement with paper SpeakerGAN: Speaker identification with conditional generative adversarial network

Pytorch Implementation of Residual Vision Transformers(ResViT)

Official source code of paper 'IterMVS: Iterative Probability Estimation for Efficient Multi-View Stereo'

EfficientDet (Scalable and Efficient Object Detection) implementation in Keras and Tensorflow

Target Propagation via Regularized Inversion

Experiments and code to generate the GINC small-scale in-context learning dataset from "An Explanation for In-context Learning as Implicit Bayesian Inference"

KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

Who calls the shots? Rethinking Few-Shot Learning for Audio (WASPAA 2021)

StyleGAN2 Webtoon / Anime Style Toonify

For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training.