BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information

Python implementation of "Single Image Haze Removal Using Dark Channel Prior"

pq is a jq-like Pickle file viewer

Code release for Local Light Field Fusion at SIGGRAPH 2019

Reinforcement learning algorithms in RLlib

MMdet2-based reposity about lightweight detection model: Nanodet, PicoDet.

Use AI to generate a optimized stock portfolio

Code for the paper "Graph Attention Tracking". (CVPR2021)

[NeurIPS 2021 Spotlight] Code for Learning to Compose Visual Relations

Evolving neural network parameters in JAX.

Package for extracting emotions from social media text. Tailored for financial data.

Code to reproduce the experiments in the paper "Transformer Based Multi-Source Domain Adaptation" (EMNLP 2020)

Official implementation of "Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform", ICCV 2021

Spatially-Adaptive Pixelwise Networks for Fast Image Translation, CVPR 2021

On the Analysis of French Phonetic Idiosyncrasies for Accent Recognition

Fewshot-face-translation-GAN - Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.

Implementation of ICCV19 Paper "Learning Two-View Correspondences and Geometry Using Order-Aware Network"

Keras-tensorflow implementation of Fully Convolutional Networks for Semantic Segmentation（Unfinished）

git《Beta R-CNN: Looking into Pedestrian Detection from Another Perspective》(NeurIPS 2020) GitHub:[fig3]

A tutorial showing how to train, convert, and run TensorFlow Lite object detection models on Android devices, the Raspberry Pi, and more!