🦉 Modern high-performance serialization utilities for Python (JSON, MessagePack, Pickle)

Related tags

Data Serializationpythonyamlserializationjsonmsgpackpython-3picklepython-2ujson
Overview

srsly: Modern high-performance serialization utilities for Python

This package bundles some of the best Python serialization libraries into one standalone package, with a high-level API that makes it easy to write code that's correct across platforms and Pythons. This allows us to provide all the serialization utilities we need in a single binary wheel. Currently supports JSON, JSONL, MessagePack, Pickle and YAML.

Azure Pipelines PyPi conda GitHub Python wheels

Motivation

Serialization is hard, especially across Python versions and multiple platforms. After dealing with many subtle bugs over the years (encodings, locales, large files) our libraries like spaCy and Prodigy have steadily grown a number of utility functions to wrap the multiple serialization formats we need to support (especially json, msgpack and pickle). These wrapping functions ended up duplicated across our codebases, so we wanted to put them in one place.

At the same time, we noticed that having a lot of small dependencies was making maintenance harder, and making installation slower. To solve this, we've made srsly standalone, by including the component packages directly within it. This way we can provide all the serialization utilities we need in a single binary wheel.

srsly currently includes forks of the following packages:

Installation

⚠️ Note that v2.x is only compatible with Python 3.6+. For 2.7+ compatibility, use v1.x.

srsly can be installed from pip. Before installing, make sure that your pip, setuptools and wheel are up to date.

pip install -U pip setuptools wheel
pip install srsly

Or from conda via conda-forge:

conda install -c conda-forge srsly

Alternatively, you can also compile the library from source. You'll need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler (XCode command-line tools on macOS / OS X or Visual C++ build tools on Windows), pip, virtualenv and git installed.

pip install -r requirements.txt  # install development dependencies
python setup.py build_ext --inplace  # compile the library

API

JSON

📦 The underlying module is exposed via srsly.ujson. However, we normally interact with it via the utility functions only.

function srsly.json_dumps

Serialize an object to a JSON string. Falls back to json if sort_keys=True is used (until it's fixed in ujson).

data = {"foo": "bar", "baz": 123}
json_string = srsly.json_dumps(data)
Argument Type Description
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 0.
sort_keys bool Sort dictionary keys. Defaults to False.
RETURNS str The serialized string.

function srsly.json_loads

Deserialize unicode or bytes to a Python object.

data = '{"foo": "bar", "baz": 123}'
obj = srsly.json_loads(data)
Argument Type Description
data str / bytes The data to deserialize.
RETURNS - The deserialized Python object.

function srsly.write_json

Create a JSON file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_json("/path/to/file.json", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 2.

function srsly.read_json

Load JSON from a file or standard input.

data = srsly.read_json("/path/to/file.json")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
RETURNS dict / list The loaded JSON content.

function srsly.write_gzip_json

Create a gzipped JSON file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_gzip_json("/path/to/file.json.gz", data)
Argument Type Description
path str / Path The file path.
data - The JSON-serializable data to output.
indent int Number of spaces used to indent JSON. Defaults to 2.

function srsly.read_gzip_json

Load gzipped JSON from a file.

data = srsly.read_gzip_json("/path/to/file.json.gz")
Argument Type Description
path str / Path The file path.
RETURNS dict / list The loaded JSON content.

function srsly.write_jsonl

Create a JSONL file (newline-delimited JSON) and dump contents line by line, or write to standard output.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_jsonl("/path/to/file.jsonl", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
lines iterable The JSON-serializable lines.
append bool Append to an existing file. Will open it in "a" mode and insert a newline before writing lines. Defaults to False.
append_new_line bool Defines whether a new line should first be written when appending to an existing file. Defaults to True.

function srsly.read_jsonl

Read a JSONL file (newline-delimited JSON) or from JSONL data from standard input and yield contents line by line. Blank lines will always be skipped.

data = srsly.read_jsonl("/path/to/file.jsonl")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
skip bool Skip broken lines and don't raise ValueError. Defaults to False.
YIELDS - The loaded JSON contents of each line.

function srsly.is_json_serializable

Check if a Python object is JSON-serializable.

assert srsly.is_json_serializable({"hello": "world"}) is True
assert srsly.is_json_serializable(lambda x: x) is False
Argument Type Description
obj - The object to check.
RETURNS bool Whether the object is JSON-serializable.

msgpack

📦 The underlying module is exposed via srsly.msgpack. However, we normally interact with it via the utility functions only.

function srsly.msgpack_dumps

Serialize an object to a msgpack byte string.

data = {"foo": "bar", "baz": 123}
msg = srsly.msgpack_dumps(data)
Argument Type Description
data - The data to serialize.
RETURNS bytes The serialized bytes.

function srsly.msgpack_loads

Deserialize msgpack bytes to a Python object.

msg = b"\x82\xa3foo\xa3bar\xa3baz{"
data = srsly.msgpack_loads(msg)
Argument Type Description
data bytes The data to deserialize.
use_list bool Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS - The deserialized Python object.

function srsly.write_msgpack

Create a msgpack file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_msgpack("/path/to/file.msg", data)
Argument Type Description
path str / Path The file path.
data - The data to serialize.

function srsly.read_msgpack

Load a msgpack file.

data = srsly.read_msgpack("/path/to/file.msg")
Argument Type Description
path str / Path The file path.
use_list bool Don't use tuples instead of lists. Can make deserialization slower. Defaults to True.
RETURNS - The loaded and deserialized content.

pickle

📦 The underlying module is exposed via srsly.cloudpickle. However, we normally interact with it via the utility functions only.

function srsly.pickle_dumps

Serialize a Python object with pickle.

data = {"foo": "bar", "baz": 123}
pickled_data = srsly.pickle_dumps(data)
Argument Type Description
data - The object to serialize.
protocol int Protocol to use. -1 for highest. Defaults to None.
RETURNS bytes The serialized object.

function srsly.pickle_loads

Deserialize bytes with pickle.

pickled_data = b"\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03foo\x94\x8c\x03bar\x94\x8c\x03baz\x94K{u."
data = srsly.pickle_loads(pickled_data)
Argument Type Description
data bytes The data to deserialize.
RETURNS - The deserialized Python object.

YAML

📦 The underlying module is exposed via srsly.ruamel_yaml. However, we normally interact with it via the utility functions only.

function srsly.yaml_dumps

Serialize an object to a YAML string. See the ruamel.yaml docs for details on the indentation format.

data = {"foo": "bar", "baz": 123}
yaml_string = srsly.yaml_dumps(data)
Argument Type Description
data - The JSON-serializable data to output.
indent_mapping int Mapping indentation. Defaults to 2.
indent_sequence int Sequence indentation. Defaults to 4.
indent_offset int Indentation offset. Defaults to 2.
sort_keys bool Sort dictionary keys. Defaults to False.
RETURNS str The serialized string.

function srsly.yaml_loads

Deserialize unicode or a file object to a Python object.

data = 'foo: bar\nbaz: 123'
obj = srsly.yaml_loads(data)
Argument Type Description
data str / file The data to deserialize.
RETURNS - The deserialized Python object.

function srsly.write_yaml

Create a YAML file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_yaml("/path/to/file.yml", data)
Argument Type Description
path str / Path The file path or "-" to write to stdout.
data - The JSON-serializable data to output.
indent_mapping int Mapping indentation. Defaults to 2.
indent_sequence int Sequence indentation. Defaults to 4.
indent_offset int Indentation offset. Defaults to 2.
sort_keys bool Sort dictionary keys. Defaults to False.

function srsly.read_yaml

Load YAML from a file or standard input.

data = srsly.read_yaml("/path/to/file.yml")
Argument Type Description
path str / Path The file path or "-" to read from stdin.
RETURNS dict / list The loaded YAML content.

function srsly.is_yaml_serializable

Check if a Python object is YAML-serializable.

assert srsly.is_yaml_serializable({"hello": "world"}) is True
assert srsly.is_yaml_serializable(lambda x: x) is False
Argument Type Description
obj - The object to check.
RETURNS bool Whether the object is YAML-serializable.
Comments
  • No module named 'srsly.ujson.ujson'

    No module named 'srsly.ujson.ujson'

    I'm installing srsly on an AWS lambda but I keep getting an error : No module named 'srsly.ujson.ujson'

    It's install with srsly inside its package but can't be access. I tried to change all declaration to a global import ujson which i downloaded without any success ...

    Any ideas ?

    PS : I'm installing a Spacy lambda which require srsly

    opened by antoineGit 8
  • Fix Typing for `JSONOutput` and `JSONOutputBin`

    Fix Typing for `JSONOutput` and `JSONOutputBin`

    Description

    Add Tuple[Any, ...] for JSONOutput and JSONOutputBin. Otherwise type hinting will indicate mismatching types if tuples are an expected type, e.g. for values: Tuple[Any, ...] = srsly.load(value).

    Also fixes CI image for Linux with Python 3.6.

    Related to https://github.com/explosion/srsly/pull/79.

    Types of change

    Bug fix.

    Checklist

    • [x] I confirm that I have the right to submit this contribution under the project's MIT license.
    • [x] I ran the tests, and all new and existing tests passed.
    • [x] My changes don't require a change to the documentation, or if they do, I've added all required information.
    bug 
    opened by rmitsch 5
  • Update UltraJSON

    Update UltraJSON

    opened by jhe921 4
  • Update Ultrajson

    Update Ultrajson

    Hi,

    Ultrajson recently got updated, I'm uncertain how it works with forks and srsly being standalone but is it possible to update the version used in srsly to use this latest version?

    //Tim

    opened by AsymptoticBrain 3
  • How to use `srsly.msgpack_dumps` with my custom class?

    How to use `srsly.msgpack_dumps` with my custom class?

    I want to serialize my custom class with srsly.msgpack_dumps, because it is stored in spacy.Doc. In other words, doc.to_disk fails because my custom class cannotn be serialized with srsly.msgpack_dumps. How to make my custom class to be able to save?

    opened by tamuhey 3
  • Memory leaks in ujson

    Memory leaks in ujson

    We've replaced the usage of json with srsly.ujson a while ago for that free performance boost since we are doing lots of JSON encoding/decoding and we already have it installed as part of spacy, but now we had to move back because of some terrible memory leaks:

    import json
import random
import string
import psutil
from srsly import ujson as json

sample = lambda x: ''.join(
  random.choice(string.ascii_uppercase + string.digits) for _ in range(x))

process = psutil.Process()

for i in range(10):
  data = json.dumps({sample(99): sample(100000) for k in range(50)})
  json.loads(data)
  print(process.memory_info())

    Output with ujson:

    pmem(rss=24203264, vms=4400664576, pfaults=19049, pageins=0)
pmem(rss=29409280, vms=4414173184, pfaults=33898, pageins=0)
pmem(rss=34557952, vms=4419309568, pfaults=47960, pageins=0)
pmem(rss=39714816, vms=4424429568, pfaults=62855, pageins=0)
pmem(rss=44838912, vms=4429549568, pfaults=77571, pageins=0)
pmem(rss=50081792, vms=4434751488, pfaults=92307, pageins=0)
pmem(rss=55312384, vms=4439973888, pfaults=107288, pageins=0)
pmem(rss=60440576, vms=4445093888, pfaults=122711, pageins=0)
pmem(rss=65806336, vms=4451422208, pfaults=137578, pageins=0)
pmem(rss=70934528, vms=4456542208, pfaults=151382, pageins=0)

    Output with stdlib json:

    pmem(rss=17154048, vms=4385366016, pfaults=17692, pageins=0)
pmem(rss=17317888, vms=4403191808, pfaults=32047, pageins=0)
pmem(rss=17317888, vms=4403191808, pfaults=46541, pageins=0)
pmem(rss=17317888, vms=4403191808, pfaults=61035, pageins=0)
pmem(rss=17358848, vms=4403191808, pfaults=75539, pageins=0)
pmem(rss=17383424, vms=4403191808, pfaults=90039, pageins=0)
pmem(rss=17383424, vms=4403191808, pfaults=104533, pageins=0)
pmem(rss=17420288, vms=4403191808, pfaults=119036, pageins=0)
pmem(rss=17420288, vms=4403191808, pfaults=133530, pageins=0)
pmem(rss=17420288, vms=4403191808, pfaults=148024, pageins=0)

    Benchmark ran on python3.7 on macos, but same leak exists on Debian with python27. You can increase the range from 10 and you will eventually run of our memory.

    Days were spent on this issue, because I would never suspect the JSON library to be at fault, but it is. I don't know if it affects Spacy in any way.

    Could be related: https://github.com/esnme/ultrajson/pull/270

    bug 
    opened by sadovnychyi 3
  • Wordwises works, X-Ray doesn't

    Wordwises works, X-Ray doesn't

    Wordwise seems to run fine, X-Ray errors :

    Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 83, in start_work
File "calibre_plugins.worddumb.parse_job", line 37, in do_job
File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\spacy_3.1.1_3.8\spacy\__init__.py", line 11, in <module>
File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\thinc_8.0.8_3.8\thinc\__init__.py", line 5, in <module>
File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\thinc_8.0.8_3.8\thinc\config.py", line 14, in <module>
File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\__init__.py", line 1, in <module>
File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\_json_api.py", line 6, in <module>
File "C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\ujson\__init__.py", line 1, in <module>
ModuleNotFoundError: No module named 'srsly.ujson.ujson'

    pip says srsly is installed correctly, and C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\ujson exists and has ujson.c inside among other files..

    (unrelated? : I had to manually add blis_0.7.4_3.8 to the worddumb-libs on Windows 10 to clear an other error)

    C:\Users\USERNAME\AppData\Roaming\calibre\plugins\worddumb-libs\srsly_2.4.1_3.8\srsly\ujson_init_.py line 1 is :

    from .ujson import decode, encode, dump, dumps, load, loads # noqa: F401

    Called with args: ((1587, 'MOBI', 'BBGL0Z779A', 'book.mobi', <calibre.ebooks.metadata.book.base.Metadata object at 0x082C44C0>, {'spacy': 'en_core_web_', 'wiki': 'en'}), False, True) {'notifications': <queue.Queue object at 0x082C46B8>, 'abort': <threading.Event object at 0x082C45F8>, 'log': <calibre.utils.logging.GUILog object at 0x082C45C8>}

    Windows 10 Python 3.9 Calibre - 5.24

    Installed the plugin today.

    opened by benninkcorien 2
  • Bug: Cannot install if Cython is not already installed

    Bug: Cannot install if Cython is not already installed

    Summary

    The latest version 1 tag (v1.0.5) altered setup.py to directly import from Cython. This means that install fails with No module named 'Cython' unless you have manually installed it.

    Projects that use srsly as a dependency will need to be edited to manually install Cython now, whereas before setuptools handed this install.

    This also effects version 2 and the master branch.

    Edit

    I've looked into this and I think the expectation is that pyproject.toml will handle this cython pre-install. I am using python 3.6.5, I suspect this python version (or its setuptools) is not reading this file.

    Reproduce

    $ docker run -w /home/circleci circleci/python:3.6.5 bash -c "python3 -m venv venv; . venv/bin/activate; pip install srsly==1.0.5" 

    Collecting srsly==1.0.5
  Downloading https://files.pythonhosted.org/packages/c7/08/abe935f33b69a08d365b95e62b47ef48f93a69ab734e623248a8a4079ecb/srsly-1.0.5.tar.gz (86kB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-vjnjnimo/srsly/setup.py", line 11, in <module>
        from Cython.Build import cythonize
    ModuleNotFoundError: No module named 'Cython'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-vjnjnimo/srsly/

    I give a docker route to reproduce to match the environment perfectly.

    Entities

    https://github.com/explosion/srsly/pull/44 https://github.com/explosion/srsly/blob/v1.0.5/setup.py#L11

    opened by DomHudson 2
  • srsly.write_json segfaults trying to write numpy types

    srsly.write_json segfaults trying to write numpy types

    Hi,

    I was using srsly for json serialization and it killed my jupyter kernels without producing any errors. It turns out it's due to ujson segmentation faults trying to serialize types it doesn't understand. It's quite easy to reproduce though:

    import numpy as np
import srsly
f = np.float32()
srsly.write_json("x.json", f)

    Believe this has now been fixed in ujson as part of https://github.com/ultrajson/ultrajson/commit/53f85b1bd6e4f27a3e4cdc605518c48a6c7e7e9e (issue https://github.com/ultrajson/ultrajson/issues/294). Doing an equivalent test with the latest ujson (version 3.0) gives a sensible error.

    Prior to that fix, it looked like some people were abandoning ujson (https://github.com/noirbizarre/flask-restplus/issues/589) and there was a Pandas fork (https://github.com/pandas-dev/pandas/tree/master/pandas/_libs/src/ujson/python) before it started being maintained again.

    Are there any plans to update the version used by srsly?

    opened by jjjamie 2
  • srsly.read_json is broken

    srsly.read_json is broken

    code:

    import srsly

print(srsly.read_json("foo.json"))

    outputs:

    Traceback (most recent call last):
  File "/tmp/foo.py", line 4, in <module>
    print(srsly.read_json("foo.json"))
  File "/Users/yohei_tamura/work/lilly/.venv/lib/python3.8/site-packages/srsly/_json_api.py", line 52, in read_json
    return ujson.load(f)
ValueError: Expected object or value

    srsly version: 2.2.0 python version: 3.8.1

    opened by tamuhey 2
  • Release v1.0.0 was removed from PyPI.org

    Release v1.0.0 was removed from PyPI.org

    The release package for 1.0.0 was removed from PyPI.org. Doing this causes problems for many automated build systems and pipelines. Removing packages from repos is frowned upon. Please replace the package in PyPI. It it is broken in some way, please release a v1.0.1 instead of removing 1.0.0.

    pip3 install srsly==1.0.0
Collecting srsly==1.0.0
  Could not find a version that satisfies the requirement srsly==1.0.0 (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.1.0, 0.2.0, 2.0.0.dev0, 2.0.0.dev1, 2.0.0)
No matching distribution found for srsly==1.0.0
    Releases_·_explosion_srsly srsly_·_PyPI
    opened by robbwagoner 2
Releases(v2.4.5)
Owner
Explosion
A software company specializing in developer tools for Artificial Intelligence and Natural Language Processing
Explosion
GitHub Repository
Python wrapper around rapidjson

python-rapidjson Python wrapper around RapidJSON Authors: Ken Robbins [email prot

469 Jan 04, 2023
Corset is a web-based data selection portal that helps you getting relevant data from massive amounts of parallel data.

Corset is a web-based data selection portal that helps you getting relevant data from massive amounts of parallel data. So, if you don't need the whole corpus, but just a suitable subset (indeed, a c

13 Nov 10, 2022
A lightweight library for converting complex objects to and from simple Python datatypes.

marshmallow: simplified object serialization marshmallow is an ORM/ODM/framework-agnostic library for converting complex datatypes, such as objects, t

marshmallow-code 6.4k Jan 02, 2023
Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy

orjson orjson is a fast, correct JSON library for Python. It benchmarks as the fastest Python library for JSON and is more correct than the standard j

4.1k Dec 30, 2022
Extended pickling support for Python objects

cloudpickle cloudpickle makes it possible to serialize Python constructs not supported by the default pickle module from the Python standard library.

1.3k Jan 05, 2023
Ultra fast JSON decoder and encoder written in C with Python bindings

UltraJSON UltraJSON is an ultra fast JSON encoder and decoder written in pure C with bindings for Python 3.6+. Install with pip: $ python -m pip insta

3.9k Jan 02, 2023
simplejson is a simple, fast, extensible JSON encoder/decoder for Python

simplejson simplejson is a simple, fast, complete, correct and extensible JSON http://json.org encoder and decoder for Python 3.3+ with legacy suppo

1.5k Dec 31, 2022
🦉 Modern high-performance serialization utilities for Python (JSON, MessagePack, Pickle)

srsly: Modern high-performance serialization utilities for Python This package bundles some of the best Python serialization libraries into one standa

Explosion 329 Dec 28, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Google 19.6k Jan 01, 2023
serialize all of python

dill serialize all of python About Dill dill extends python's pickle module for serializing and de-serializing python objects to the majority of the b

The UQ Foundation 1.8k Jan 07, 2023
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Protocol Buffers 57.6k Jan 03, 2023
MessagePack serializer implementation for Python msgpack.org[Python]

MessagePack for Python What's this MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JS

MessagePack 1.7k Dec 29, 2022
Python library for serializing any arbitrary object graph into JSON. It can take almost any Python object and turn the object into JSON. Additionally, it can reconstitute the object back into Python.

jsonpickle jsonpickle is a library for the two-way conversion of complex Python objects and JSON. jsonpickle builds upon the existing JSON encoders, s

1.1k Jan 02, 2023
Generic ASN.1 library for Python

ASN.1 library for Python This is a free and open source implementation of ASN.1 types and codecs as a Python package. It has been first written to sup

Ilya Etingof 223 Dec 11, 2022
Python bindings for the simdjson project.

pysimdjson Python bindings for the simdjson project, a SIMD-accelerated JSON parser. If SIMD instructions are unavailable a fallback parser is used, m

Tyler Kennedy 562 Jan 08, 2023
Crappy tool to convert .scw files to .json and and vice versa.

SCW-JSON-TOOL Crappy tool to convert .scw files to .json and vice versa. How to use Run main.py file with two arguments: python main.py scw2json or j

Fred31 5 May 14, 2021