Implementation of Kronecker Attention in Pytorch

Last update: May 06, 2022

Overview

Kronecker Attention Pytorch

Implementation of Kronecker Attention in Pytorch. Results look less than stellar, but if someone found some context where this architecture works well, please post in the issues and let everyone know.

Install

$ pip install kronecker_attention_pytorch

Usage

import torch
from kronecker_attention_pytorch import KroneckerSelfAttention

attn = KroneckerSelfAttention(
    chan = 32,
    heads = 8,
    dim_heads = 64
)

x = torch.randn(1, 32, 256, 512)
attn(x) # (1, 32, 256, 512)

Citations

@article{Gao_2020,
   title={Kronecker Attention Networks},
   url={http://dx.doi.org/10.1145/3394486.3403065},
   journal={Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining},
   publisher={ACM},
   author={Gao, Hongyang and Wang, Zhengyang and Ji, Shuiwang},
   year={2020},
   month={Aug}
}

You might also like...

Attention-driven Robot Manipulation (ARM) which includes Q-attention

Attention-driven Robotic Manipulation (ARM) This codebase is home to: Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation I

84 Dec 29, 2022

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

LESA Introduction This repository contains the official implementation of Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Cont

20 Dec 31, 2021

Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch

Lie Transformer - Pytorch (wip) Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch. Only the SE3 version will be present in thi

78 Oct 26, 2022

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch.

SE3 Transformer - Pytorch Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. May be needed for replicating Alphafold2 resu

207 Dec 23, 2022

Implementation of TabTransformer, attention network for tabular data, in Pytorch

Tab Transformer Implementation of Tab Transformer, attention network for tabular data, in Pytorch. This simple architecture came within a hair's bread

420 Jan 5, 2023

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

Perceiver - Pytorch Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch Install $ pip install perceiver-pytorch Usage

876 Dec 29, 2022

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

489 Jan 7, 2023

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

TimeSformer This is an official pytorch implementation of Is Space-Time Attention All You Need for Video Understanding?. In this repository, we provid

1k Dec 31, 2022

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

Neural Attention Distillation This is an implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep

84 Jan 4, 2023

Releases(0.0.5)

0.0.5(Sep 12, 2020)

Source code(tar.gz)
Source code(zip)
0.0.4(Aug 29, 2020)

Source code(tar.gz)
Source code(zip)
0.0.3(Aug 27, 2020)

Source code(tar.gz)
Source code(zip)
0.0.2(Aug 27, 2020)

Source code(tar.gz)
Source code(zip)
0.0.1(Aug 27, 2020)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Official implementation of Sparse Transformer-based Action Recognition

STAR Official implementation of S parse T ransformer-based A ction R ecognition Dataset download NTU RGB+D 60 action recognition of 2D/3D skeleton fro

15 Nov 02, 2022

Official PyTorch implementation of the paper Image-Based CLIP-Guided Essence Transfer.

TargetCLIP- official pytorch implementation of the paper Image-Based CLIP-Guided Essence Transfer This repository finds a global direction in StyleGAN

221 Dec 13, 2022

Implementation of the famous Image Manipulation\Forgery Detector "ManTraNet" in Pytorch

Who has never met a forged picture on the web ? No one ! Everyday we are constantly facing fake pictures touched up in Photoshop but it is not always

77 Dec 16, 2022

Educational 2D SLAM implementation based on ICP and Pose Graph

slam-playground Educational 2D SLAM implementation based on ICP and Pose Graph How to use: Use keyboard arrow keys to navigate robot. Press 'r' to vie

19 Dec 17, 2022

Video Contrastive Learning with Global Context

Video Contrastive Learning with Global Context (VCLR) This is the official PyTorch implementation of our VCLR paper. Install dependencies environments

143 Dec 26, 2022

NumPy로 구현한 딥러닝 라이브러리입니다. (자동 미분 지원)

Deep Learning Library only using NumPy 본 레포지토리는 NumPy 만으로 구현한 딥러닝 라이브러리입니다. 자동 미분이 구현되어 있습니다. 자동 미분 자동 미분은 미분을 자동으로 계산해주는 기능입니다. 아래 코드는 자동 미분을 활용해 역전파

17 Aug 16, 2022

StyleMapGAN - Official PyTorch Implementation

StyleMapGAN - Official PyTorch Implementation StyleMapGAN: Exploiting Spatial Dimensions of Latent in GAN for Real-time Image Editing Hyunsu Kim, Yunj

425 Dec 23, 2022

Snapchat-filters-app-opencv-python - Here we used opencv and other inbuilt python modules to create filter application like snapchat

Snapchat like filter App using opencv python Backend : opencv and python Library

2 Jul 19, 2022

This repository contains the code for using the H3DS dataset introduced in H3D-Net: Few-Shot High-Fidelity 3D Head Reconstruction

H3DS Dataset This repository contains the code for using the H3DS dataset introduced in H3D-Net: Few-Shot High-Fidelity 3D Head Reconstruction Access

72 Dec 10, 2022

Implementation of Kronecker Attention in Pytorch

Related tags

Overview

Kronecker Attention Pytorch

Install

Usage

Citations

You might also like...

Attention-driven Robot Manipulation (ARM) which includes Q-attention

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch

Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch.

Implementation of TabTransformer, attention network for tabular data, in Pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

The official pytorch implementation of our paper "Is Space-Time Attention All You Need for Video Understanding?"

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

Releases(0.0.5)

0.0.5(Sep 12, 2020)

0.0.4(Aug 29, 2020)

0.0.3(Aug 27, 2020)

0.0.2(Aug 27, 2020)

0.0.1(Aug 27, 2020)

Owner

Phil Wang

Official implementation of Sparse Transformer-based Action Recognition

Official PyTorch implementation of the paper Image-Based CLIP-Guided Essence Transfer.

Implementation of the famous Image Manipulation\Forgery Detector "ManTraNet" in Pytorch

Educational 2D SLAM implementation based on ICP and Pose Graph

Video Contrastive Learning with Global Context

NumPy로 구현한 딥러닝 라이브러리입니다. (자동 미분 지원)

StyleMapGAN - Official PyTorch Implementation

Snapchat-filters-app-opencv-python - Here we used opencv and other inbuilt python modules to create filter application like snapchat

This repository contains the code for using the H3DS dataset introduced in H3D-Net: Few-Shot High-Fidelity 3D Head Reconstruction

Point Cloud Registration Network

transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

Quantify the difference between two arbitrary curves in space

Unofficial implementation of MLP-Mixer: An all-MLP Architecture for Vision

This repo contains the implementation of YOLOv2 in Keras with Tensorflow backend.

[ICCV2021] 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds

Cockpit is a visual and statistical debugger specifically designed for deep learning.

A TensorFlow implementation of FCN-8s

This repository contains a PyTorch implementation of the paper Learning to Assimilate in Chaotic Dynamical Systems.

LWCC: A LightWeight Crowd Counting library for Python that includes several pretrained state-of-the-art models.

Deep Learning and Reinforcement Learning Library for Scientists and Engineers 🔥