🤗Datasets is a lightweight library providing two main features:
- one-line dataloaders for many public datasets: one liners to download and pre-process any of the
major public datasets (in 467 languages and dialects!) provided on the HuggingFace Datasets Hub. With a simple command like
squad_dataset = load_datasets("squad"), get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX), - efficient data pre-processing: simple, fast and reproducible data pre-processing for the above public datasets as well as your own local datasets in CSV/JSON/text. With simple commands like
tokenized_dataset = dataset.map(tokenize_exemple), efficiently prepare the dataset for inspection and ML model evaluation and training.
🤗Datasets also provides access to +15 evaluation metrics and is designed to let the community easily add and share new datasets and evaluation metrics.
🤗Datasets has many additional interesting features:
- Thrive on large datasets:
🤗Datasetsnaturally frees the user from RAM memory limitation, all datasets are memory-mapped using an efficient zero-serialization cost backend (Apache Arrow). - Smart caching: never wait for your data to process several times.
- Lightweight and fast with a transparent and pythonic API (multi-processing/caching/memory-mapping).
- Built-in interoperability with NumPy, pandas, PyTorch, Tensorflow 2 and JAX.
🤗Datasets originated from a fork of the awesome TensorFlow Datasets and the HuggingFace team want to deeply thank the TensorFlow Datasets team for building this amazing library. More details on the differences between 🤗Datasets and tfds can be found in the section Main differences between 🤗Datasets and tfds.
Installation
With pip
🤗Datasets can be installed from PyPi and has to be installed in a virtual environment (venv or conda for instance)
pip install datasets
With conda
🤗Datasets can be installed using conda as follows:
conda install -c huggingface -c conda-forge datasets
Follow the installation pages of TensorFlow and PyTorch to see how to install them with conda.
For more details on installation, check the installation page in the documentation: https://huggingface.co/docs/datasets/installation.html
Installation to use with PyTorch/TensorFlow/pandas
If you plan to use 🤗Datasets with PyTorch (1.0+), TensorFlow (2.2+) or pandas, you should also install PyTorch, TensorFlow or pandas.
For more details on using the library with NumPy, pandas, PyTorch or TensorFlow, check the quick tour page in the documentation: https://huggingface.co/docs/datasets/quicktour.html
Usage
🤗Datasets is made to be very simple to use. The main methods are:
datasets.list_datasets()to list the available datasetsdatasets.load_dataset(dataset_name, **kwargs)to instantiate a datasetdatasets.list_metrics()to list the available metricsdatasets.load_metric(metric_name, **kwargs)to instantiate a metric
Here is a quick example:
from datasets import list_datasets, load_dataset, list_metrics, load_metric
# Print all the available datasets
print(list_datasets())
# Load a dataset and print the first example in the training set
squad_dataset = load_dataset('squad')
print(squad_dataset['train'][0])
# List all the available metrics
print(list_metrics())
# Load a metric
squad_metric = load_metric('squad')
# Process the dataset - add a column with the length of the context texts
dataset_with_length = squad_dataset.map(lambda x: {"length": len(x["context"])})
# Process the dataset - tokenize the context texts (using a tokenizer from the 🤗Transformers library)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
tokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)
For more details on using the library, check the quick tour page in the documentation: https://huggingface.co/docs/datasets/quicktour.html and the specific pages on:
- Loading a dataset https://huggingface.co/docs/datasets/loading_datasets.html
- What's in a Dataset: https://huggingface.co/docs/datasets/exploring.html
- Processing data with
🤗Datasets: https://huggingface.co/docs/datasets/processing.html - Writing your own dataset loading script: https://huggingface.co/docs/datasets/add_dataset.html
- etc.
Another introduction to 🤗Datasets is the tutorial on Google Colab here:
Add a new dataset to the Hub
We have a very detailed step-by-step guide to add a new dataset to the
You will find the step-by-step guide here to add a dataset to this repository.
You can also have your own repository for your dataset on the Hub under your or your organization's namespace and share it with the community. More information in the documentation section about dataset sharing.
Main differences between 🤗Datasets and tfds
If you are familiar with the great Tensorflow Datasets, here are the main differences between 🤗Datasets and tfds:
- the scripts in
🤗Datasetsare not provided within the library but are queried, downloaded/cached and dynamically loaded upon request 🤗Datasetsalso provides evaluation metrics in a similar fashion to the datasets, i.e. as dynamically installed scripts with a unified API. This gives access to the pair of a benchmark dataset and a benchmark metric for instance for benchmarks like SQuAD or GLUE.- the backend serialization of
🤗Datasetsis based on Apache Arrow instead of TF Records and leverage python dataclasses for info and features with some diverging features (we mostly don't do encoding and store the raw data as much as possible in the backend serialization cache). - the user-facing dataset object of
🤗Datasetsis not atf.data.Datasetbut a built-in framework-agnostic dataset class with methods inspired by what we like intf.data(like amap()method). It basically wraps a memory-mapped Arrow table cache.
Disclaimers
Similar to TensorFlow Datasets, 🤗Datasets is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use them. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.
If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a GitHub issue. Thanks for your contribution to the ML community!
BibTeX
If you want to cite this framework you can use this:
@article{2020HuggingFace-datasets,
title={Datasets},
author={Thomas Wolf and Quentin Lhoest and Patrick von Platen and Yacine Jernite and Mariama Drame and Julien Plu and Julien Chaumond and Clement Delangue and Clara Ma and Abhishek Thakur and Suraj Patil and Joe Davison and Teven Le Scao and Victor Sanh and Canwen Xu and Nicolas Patry and Angie McMillan-Major and Simon Brandeis and Sylvain Gugger and François Lagunas and Lysandre Debut and Morgan Funtowicz and Anthony Moi and Sasha Rush and Philipp Schmidd and Pierric Cistac and Victor Muštar and Jeff Boudier and Anna Tordjmann},
journal={GitHub. Note: https://github.com/huggingface/datasets},
volume={1},
year={2020}
}
![Whisper Event - RuntimeError: The size of tensor a (504) must match the size of tensor b (448) at non-singleton dimension 1 100% 1000/1000 [2:52:21<00:00, 10.34s/it]](https://avatars.githubusercontent.com/u/12885107?v=4)
14 Nov 15, 2022
45 Nov 29, 2022
652 Jan 06, 2023
22 Dec 27, 2022
3 Nov 04, 2022
3 May 19, 2022
16 Feb 24, 2022
29.8k Jan 02, 2023
8 Apr 13, 2022
18 Dec 20, 2022
2 Nov 05, 2021
46 Dec 05, 2022
1.4k Dec 29, 2022
71 Nov 25, 2022
20 Dec 12, 2022
101 Dec 03, 2022
2 Oct 02, 2022
8 Oct 13, 2022
11 Mar 28, 2022
293 Jan 02, 2023