Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Last update: Dec 29, 2022

Related tags

Deep Learning wechsel

Overview

WECHSEL

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

arXiv: https://arxiv.org/abs/2112.06598

Models from the paper are available on the HuggingFace Hub:

Installation

We distribute a Python Package via PyPI:

pip install wechsel

Alternatively, clone the repository, install requirements.txt and run the code in wechsel/.

Example usage

Transferring English roberta-base to Swahili:

import torch
from transformers import AutoModel, AutoTokenizer
from datasets import load_dataset
from wechsel import WECHSEL, load_embeddings

source_tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModel.from_pretrained("roberta-base")

target_tokenizer = source_tokenizer.train_new_from_iterator(
    load_dataset("oscar", "unshuffled_deduplicated_sw", split="train")["text"],
    vocab_size=len(source_tokenizer)
)

wechsel = WECHSEL(
    load_embeddings("en"),
    load_embeddings("sw"),
    bilingual_dictionary="swahili"
)

target_embeddings, info = wechsel.apply(
    source_tokenizer,
    target_tokenizer,
    model.get_input_embeddings().weight.detach().numpy(),
)

model.get_input_embeddings().weight.data = torch.from_numpy(target_embeddings)

# use `model` and `target_tokenizer` to continue training in Swahili!

Bilingual dictionaries

We distribute 3276 bilingual dictionaries from English to other languages for use with WECHSEL in dicts/.

Citation

Please cite WECHSEL as

@misc{minixhofer2021wechsel,
      title={WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models}, 
      author={Benjamin Minixhofer and Fabian Paischer and Navid Rekabsaz},
      year={2021},
      eprint={2112.06598},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

Related tags

Overview

WECHSEL

Installation

Example usage

Bilingual dictionaries

Citation

Owner

Institute of Computational Perception

Official implementation for Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder at NeurIPS 2020

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

Videocaptioning.pytorch - A simple implementation of video captioning

A machine learning project which can detect and predict the skin disease through image recognition.

This is a work in progress reimplementation of Instant Neural Graphics Primitives

Source code and data from the RecSys 2020 article "Carousel Personalization in Music Streaming Apps with Contextual Bandits" by W. Bendada, G. Salha and T. Bontempelli

3D Human Pose Machines with Self-supervised Learning

Machine Learning Platform for Kubernetes

code release for USENIX'22 paper `On the Security Risks of AutoML`

Geometric Deep Learning Extension Library for PyTorch

MADT: Offline Pre-trained Multi-Agent Decision Transformer

Official implementation of "Implicit Neural Representations with Periodic Activation Functions"

A set of examples around hub for creating and processing datasets

Tutorial on active learning with the Nvidia Transfer Learning Toolkit (TLT).

Pytorch GUI(demo) for iVOS(interactive VOS) and GIS (Guided iVOS)

Dynamic Capacity Networks using Tensorflow

PyTorch implementation of Neural Combinatorial Optimization with Reinforcement Learning.

Annotate datasets with a semi-trained or fully trained YOLOv5 model

Evaluating different engineering tricks that make RL work

SlideGraph+: Whole Slide Image Level Graphs to Predict HER2 Status in Breast Cancer