BERT-Syntax

Assesing the syntactic abilities of BERT.

What

Evaluate Google's BERT-Base and BERT-Large models on the syntactic agreement datasets from Linzen, Goldberg and Dupoux 2016 and Marvin and Linzen 2018 and Gulordava et al 2018.

This is quite messy, as I hacked it together between things here and there. But I also believe it is accurate. This lists the data files and shows how to run the evaluation. For more details and results, see the arxiv report.

Data Files

Data taken from the github repos of Linzen, Goldberg and Dupoux (LGD), Marvin and Linzen (ML), and Gulordava et al.

File	Description
`marvin_linzen_dataset.tsv`	stimuli from Marvin and Linzen. I dumped it from the pickle files of ML
`wiki.vocab`	from LGD, used for verb inflections (wiki.vocab)
`lgd_dataset.tsv`	processed data from LGD
`generated.tab`	data from Gulordava et al (generated.tab)

lgd_dataset.tsv is created by

wget http://tallinzen.net/media/rnn_agreement/agr_50_mostcommon_10K.tsv.gz
gunzip agr_50_mostcommon_10K.tsv.gz
python make_linzen_goldberg_testset.py > lgd_dataset.tsv

Obtaining the results

pip install pytorch_pretrained_bert

python eval_bert.py > results/lgd_results_large.txt
python eval_bert.py base > results/lgd_results_base.txt
python eval_bert.py marvin > results/marvin_results_large.txt
python eval_bert.py marvin base > results/marvin_results_base.txt
python eval_bert.py gul > results/gulordava_results_large.txt
python eval_bert.py gul base > results/gulordava_results_base.txt

Generating tables (for the PDF)

python gen_marvin_tbl.py 
python gen_lgd_tbl.py
python gen_gul_tbl.py

Assessing syntactic abilities of BERT

Related tags

Overview

BERT-Syntax

What

Data Files

Obtaining the results

Generating tables (for the PDF)

Owner

Yoav Goldberg

Code for ACL'2021 paper WARP 🌀 Word-level Adversarial ReProgramming

Structured Edge Detection Toolbox

Train CPPNs as a Generative Model, using Generative Adversarial Networks and Variational Autoencoder techniques to produce high resolution images.

TYolov5: A Temporal Yolov5 Detector Based on Quasi-Recurrent Neural Networks for Real-Time Handgun Detection in Video

Segmentation and Identification of Vertebrae in CT Scans using CNN, k-means Clustering and k-NN

Python periodic table module

Pytorch implementation for reproducing StackGAN_v2 results in the paper StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

PHOTONAI is a high level python API for designing and optimizing machine learning pipelines.

Project repo for Learning Category-Specific Mesh Reconstruction from Image Collections

基于YoloX目标检测+DeepSort算法实现多目标追踪Baseline

Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Data

Lightweight plotting to the terminal. 4x resolution via Unicode.

Official implementation of the paper DeFlow: Learning Complex Image Degradations from Unpaired Data with Conditional Flows

New approach to benchmark VQA models

TensorFlow-LiveLessons - "Deep Learning with TensorFlow" LiveLessons

Generating Radiology Reports via Memory-driven Transformer

PyTorch implementation of the end-to-end coreference resolution model with different higher-order inference methods.

Autoencoder - Reducing the Dimensionality of Data with Neural Network

MACE is a deep learning inference framework optimized for mobile heterogeneous computing platforms.

Convert onnx models to pytorch.