Long Expressive Memory (LEM)

Last update: Dec 17, 2022

Overview

Long Expressive Memory for Sequence Modeling

This repository contains the implementation to reproduce the numerical experiments of the paper Long Expressive Memory for Sequence Modeling

Requirements

To setup the environment, install the requirements using python 3.7.10 with

pip install -r requirements.txt

Experiments

This repository contains the codes to reproduce the results of the following experiments for the proposed LEM:

Adding Task
EigenWorms
FitzHugh-Nagumo
Google12 (V2)
Heart rate prediction
noisy CIFAR-10
Permuted Sequential MNIST
PennTree Bank (word- and char-level)
Sequential MNIST

All datasets get either synthetically generated or downloaded automatically, providing a full data pipline for each experiment.
Note that the PTB experiments are based on the open-source tensorflow project LAnguage Modelling Benchmarks (lamb) for tuning and testing Tensorflow language models. In order to reproduce these experiments, please follow the steps explained in the README of the PTB directory.

Results

The results of LEM for each of the experiments are:

Experiment	Result
EigenWorms	92.3% mean test accuracy
FitzHugh-Nagumo	0.002 test L^2 loss
Google12 (V2)	95.7% test accuracy
Heart rate prediction	0.85 test L^2 loss
noisy CIFAR-10	60.5% test accuracy
Permuted Sequential MNIST	96.6% test accuracy
PennTree Bank char-level (single layer)	1.25 test bits-per-character
PennTree Bank word-level (single layer)	72.8 test perplexity
Sequential MNIST	99.5% test accuracy

Long Expressive Memory (LEM)

Related tags

Overview

Long Expressive Memory for Sequence Modeling

Requirements

Experiments

Results

Owner

Konstantin Rusch

Supporting code for short YouTube series Neural Networks Demystified.

Official PyTorch implementation for "Low Precision Decentralized Distributed Training with Heterogenous Data"

Deep learning with dynamic computation graphs in TensorFlow

Auto-Lama combines object detection and image inpainting to automate object removals

For auto aligning, cropping, and scaling HR and LR images for training image based neural networks

Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA results for single-image motion deblurring, image deraining, image denoising (synthetic and real data), and dual-pixel defocus deblurring.

A Nim frontend for pytorch, aiming to be mostly auto-generated and internally using ATen.

Official pytorch implementation of the paper: "SinGAN: Learning a Generative Model from a Single Natural Image"

A very lightweight monitoring system for Raspberry Pi clusters running Kubernetes.

CONditionals for Ordinal Regression and classification in tensorflow

Implementation of "Fast and Flexible Temporal Point Processes with Triangular Maps" (Oral @ NeurIPS 2020)

PyTorch implementation of Algorithm 1 of "On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models"

Self-Supervised CNN-GCN Autoencoder

This repo is developed for Strong Baseline For Vehicle Re-Identification in Track 2 Ai-City-2021 Challenges

CasualHealthcare's Pneumonia detection with Artificial Intelligence (Convolutional Neural Network)

Learned image compression

A more easy-to-use implementation of KPConv based on PyTorch.

Official implementation of "Dynamic Anchor Learning for Arbitrary-Oriented Object Detection" (AAAI2021).

This is a TensorFlow implementation for C2-Rec

Self-Supervised Deep Blind Video Super-Resolution