Feature request, make it possible to specify a fallback function that is called when the encoder encounters an unsupported object. The fallback function can then handle the unsupported object using only msgspec compatible types. When decoding, a corresponding fallback function are then called to handle the decoding of the object.

Motivation

In Dask/Distributed we are discussing the replacement of msgpack. The library should be:

• fast
• secure (no arbitrary code execution when decoding)
• support basic Python types
• not convert list to tuples (can be disabled in msgpack but with a performance penalty)
• not convert numpy scalars to python scalars
• support a fallback mechanism

Currently msgspec supports encoding and decoding only timezone-aware datetime.datetime objects, holding strict conformance to RFC3339. Naive datetime.datetime objects can be encoded using a custom enc_hook, but there's no way to decode a naive datetime.datetime object.

I would like to expand our builtin support for datetime types to include:

• datetime.datetime (both aware and naive)
• datetime.date
• datetime.time (both aware and naive)

Here's the plan I've come up with:

Encoding

To support encoding, we add a support_naive_datetimes keyword argument to msgspec.*.decode and msgspec.*.Decoder to configure the treatment of naive datetimes. This would take one of:

• False: the default. Naive datetime and time objects error on encoding.
• True: allow encoding naive datetime and time objects. These will be encoded as their RFC3339 compatible counterparts, just missing the offset component
• "UTC": naive datetime and time objects will be treated as if they have a UTC timezone.
• a tzinfo object: naive datetime and time objects will be treated as if they have this timezone.

I'm not attached to the keyword name (or boolean options), so if someone can think of a nicer spelling I'd be happy. I think this supports all the common options.

One benefit of supporting these options builtin is that we no longer have the weird behavior of enc_hook only being called for naive datetime.datetime objects. This would admittedly be less weird if Python had different types for aware and naive datetimes.

I could hear an argument that the default should be True (encoding naive datetimes/times by default), but I'm hesitant to make that change. Having an error by default if you're using a naive datetime will force users to think about timezones early on - if they really want a naive datetime they can explicitly opt into it. Supporting naive datetimes/times by default could let programming errors slip by, since most times the user does want an aware datetime rather than a naive datetime.

Decoding

To support decoding, we want to handle the following use cases:

• Only decode RFC3339 compatible datetimes and times (requiring a timezone)
• Only decode naive datetimes and times (require no timezone)
• Decode any datetime or time object (naive or aware)

Since msgspec will only ever decode an object into a datetime if type information is provided, then the natural place to enable this configuration is through our existing type annotations system. The question then is - what does an unannotated datetime.datetime mean?

I want msgspec to make it easy to do the right thing, and (within reason) possible to do the flexible thing. As such, I'd argue that raw datetime.datetime and datetime.time annotations should only decode timezone-aware objects. This means that by default APIs built with msgspec are compatible with json-schema (which lacks a naive datetime/time format), and common web languages like golang (which requires RFC3339 compatible strings in JSON by default).

To support naive-datetime or any-datetime types, we'd add a new config to Meta annotations. Something like:

from msgspec import Struct, Meta
from typing import Annotated
import datetime

class Example(Struct):
    aware_only_datetime: datetime.datetime
    aware_only_time: datetime.time
    date: datetime.date  # date objects have no timezone
    naive_only_datetime: Annotated[datetime.datetime, Meta(timezone=False)]
    naive_only_time: Annotated[datetime.time, Meta(timezone=False)]
    any_datetime: Annotated[datetime.datetime, Meta(timezone=None)]
    any_time: Annotated[datetime.time, Meta(timezone=None)]

Like above, I don't love the timezone=True (aware), timezone=False (naive), timezone=None (aware or naive) syntax, if anyone can think of a better API spelling please let me know.

We could also add type aliases in a new submodule msgspec.types to make this easier to spell (since datetimes are common):

from msgspec import Struct
from msgspec.types import NaiveDatetime

# NaiveDatetime = Annotated[datetime.datetime, Meta(timezone=False)]

class Example(Struct):
    naive_only_datetime: NaiveDatetime

Msgpack Complications

Currently we use msgpack's timestamp extension (https://github.com/msgpack/msgpack/blob/master/spec.md#timestamp-extension-type) when encoding datetimes to msgpack. This extension by design only supports timezone-aware datetimes. msgpack has no standard representation for naive datetimes (or time/date objects in general). To handle this, I plan to encode naive datetimes as strings in the same format as JSON. This is an edge case that I don't expect to affect most users. I think the main benefit of supporting it is parity between types supported by both protocols.

Given the close similarity between msgspec.Struct and dataclasses.dataclass, am curious to what extent dataclasses can be used with msgspec and what this entails.

When encoding a dataclass, instance attributes that aren't dataclass-fields are included in the result:

import dataclasses
import msgspec


@dataclasses.dataclass
class Foo:
    id: int


foo = Foo(id=1)
foo.bar = "test"


print(msgspec.json.encode(foo))
print(msgspec.json.encode(dataclasses.asdict(foo)))

assert msgspec.json.encode(foo) == msgspec.json.encode(dataclasses.asdict(foo))

This is problematic because it differs from dataclasses.asdict, and in turn makes mgspec incompatible with pydantic.dataclasses:

from dataclasses import asdict

import msgspec
from pydantic.dataclasses import dataclass


@dataclass
class Foo:
    id: int


foo = Foo(id=1)


print(msgspec.json.encode(foo))
print(msgspec.json.encode(asdict(foo)))

I have an existing encoding that almost-but-not-quite fits msgspec. The "almost" is that tags are integer rather than string. Would it make sense to accept non-string tags? I think the only non-string case that would be valuable is for integers.

It would be helpful if JSON schema generation was supported

class User(msgspec.Struct):
    """A new type describing a User"""
    name: str
    groups: Set[str] = set()
    email: Optional[str] = None

schema = User.json_schema()

Similar to the functionality seen in https://github.com/s-knibbs/dataclasses-jsonschema

Hello,

I've got a service written in Python that reads data from ElasticSearch frequently 24/7.

Recently I've migrated it from orjson to msgspec.json. Since then, the service runs out of memory pretty quickly.

Following https://docs.python.org/3/library/tracemalloc.html#pretty-top I'm able to capture the top 10 lines that contributes large memory usage, which turns out it's the decode(...) method from msgspec.json.Decoder

Top 10 lines
#1: /app/elasticsearch_repository.py:69: 26821.3 KiB
    self.__decoder.decode(
#2: /app/prepare_data_service.py:280: 3649.2 KiB
    cache_info_rt = decoder.decode(cache_info_data)
#3: /usr/local/lib/python3.10/multiprocessing/reduction.py:51: 1568.1 KiB
    cls(buf, protocol).dump(obj)
#4: /usr/local/lib/python3.10/linecache.py:137: 628.2 KiB
    lines = fp.readlines()
#5: /usr/local/lib/python3.10/json/decoder.py:353: 82.7 KiB
    obj, end = self.scan_once(s, idx)
#6: /usr/local/lib/python3.10/multiprocessing/queues.py:122: 40.2 KiB
    return _ForkingPickler.loads(res)
#7: /usr/local/lib/python3.10/tracemalloc.py:67: 11.9 KiB
    return (self.size, self.count, self.traceback)
#8: /app/elasticsearch_repository.py:68: 3.7 KiB
    return [
#9: /usr/local/lib/python3.10/http/client.py:1293: 3.6 KiB
    self.putrequest(method, url, **skips)
#10: /app/venv/lib/python3.10/site-packages/elasticsearch/client/utils.py:347: 1.8 KiB
    return func(*args, params=params, headers=headers, **kwargs)
193 other: 70.2 KiB
Total allocated size: 32880.9 KiB

Here are the structs for decoding:

'''
structs for msgspec
'''
from typing import Dict, List, Optional

from msgspec import Struct


class Query(Struct):
    """
    Struct for Query
    """
    date: str
    depAirport: str
    arrAirport: str


class Request(Struct):
    """
    Struct for Request
    """
    supplier: str
    tripType: str
    fareClass: str
    adultAmount: int
    childAmount: int
    infantAmount: int
    queries: List[Query]
    timestamp: int


class Segment(Struct):
    """
    Struct for Segment
    """
    fareClass: str
    depDate: str
    depTime: str
    flightNo: str
    carrier: str
    orgAirport: str
    arriveDate: str
    arriveTime: str
    dstAirport: str


class Flight(Struct):
    """
    Struct for Flight
    """
    segments: List[Segment]


class Price(Struct):
    """
    Struct for Price
    """
    price: float
    tax: float
    totalPrice: float
    seatsStatus: Optional[str] = None
    currencyCode: Optional[str] = None


class Trip(Struct):
    """
    Struct for Trip
    """
    flights: List[Flight]
    prices: Dict[str, Price]
    extended = Dict[str, str]


class Result(Struct):
    """
    Struct of Result
    """
    trips: List[Trip]


class CacheInfo(Struct):
    """
    Struct of CacheInfo
    """
    request: Request
    result: Result

I read from https://jcristharif.com/msgspec/structs.html#struct-nogc that

structs referencing only scalar values (ints, strings, bools, …) won’t contribute to GC load, but structs referencing containers (lists, dicts, structs, …) will.

Is this related? What's the recommendation to resolve this issue?

Thanks!

My understanding is that msgspec currently only supports basic type/schema validation, but not more complex validations like a regex pattern or arbitrary user functions, which Pydantic and other validators do support.

I think it would be really interesting if msgspec adopted annotated-types (maybe as an optional dependency), which would not only enable an API for these sorts of constraints but also:

• Improve interoperability with other libraries (Pydantic V2 will support annotated-types, so users could go back and forth or share types between the two libraries)
• Get automatic support for other parts of the ecosystem like Hypothesis

When trying to encode subclasses of supported types, some will result in a TypeError, while others work as expected. It would be great to either support subclasses in general, or at least have an explanation of these limitations in the docs.

For comparison, the below example is supported by json, orjson and ujson.

import msgspec


class CustomList(list):
    pass


class CustomStr(str):
    pass


msgspec.json.encode(CustomList())  # this works
msgspec.json.encode(CustomStr())  # this is a TypeError

This adds two functions:

• msgspec.json.schema for generating a single JSON Schema from a given type.
• msgspec.json.schema_components for generating multiple schemas (and a corresponding components mapping) from multiple types.

~Still needs docs and tests.~

Fixes #125.

This resolves #103 by adding steps for Linux and Mac ARM wheels in the CI build. I also bumped the version of pypa/cibuildwheel to v2.4.0.

Mac support: https://cibuildwheel.readthedocs.io/en/stable/faq/#apple-silicon Linux support: https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation

I don't have a great way of testing this. :smile:

URL querystrings/x-www-form-urlencoded forms are structured but untyped messages. The python standard library has a few tools for encoding/decoding these:

In [2]: urllib.parse.parse_qs("x=1&y=true&z=a&z=b")
Out[2]: {'x': ['1'], 'y': ['true'], 'z': ['a', 'b']}

This is annoying to work with manually because the output is always of type dict[str, list[str]]. This means that:

• The string values have to be manually cast to the expected types
• Fields where you expect a single value have to be validated (or only the last value used)
• Missing required fields and default values have to be manually handled

A library like Pydantic may be used to ease some of the ergonomic issues here, but adds extra overhead.

Since msgspec is already useful for parsing JSON payloads into typed & structured objects, we might support a new querystring encoding/decoding that makes use of msgspec's existing type system to handle the decoding and validation. A lot of the code needed to handle this parsing already exists in msgspec, it's mostly just plumbing needed to hook things together. For performance, I'd expect this to be ~as fast as our existing JSON encoder/decoder.

Proposed interface:

# msgspec/querystring.py

def encode(obj: Any) -> bytes:
    """Encode an object as a querystring.

    This returns `bytes` not `str`, since that's what `msgspec` returns for other encodings.
    """
    ...

def decode(buf: bytes | str, type: Type[T] = dict[str, list[str]]) -> T:
    """Decode a querystring.

    If `type` is passed, a value of that type is returned (or an error is raised).

    If `type` is not passed, a `dict[str, list[str]]` is returned containing all passed query parameters.
    This matches the behavior of `urllib.parse.parse_qs`.
    """
    ...

Proposed encoding/decoding scheme:

• Nested objects are not supported due to querystring restrictions. We don't try to do anything complicated like rails or sinatra do (i.e. no foo[][key]=bar stuff).
• A valid type must be a top-level object-like (struct, dataclass, ...) type, mapping fields to value types

The following value types are supported

• int, float, str, and str-like types (datetimes, ...) map to/from their str representations, quoting as needed
• bool serializes to "true"/"false". When deserializing, "", "1" and "0" are also accepted (are there other common values?)
• None serializes as "". When decoding "null" is also accepted.
• Sequences of the above (e.g. list/tuple/...) map to/from multiple values set for a field. So a field a with value ("x", None, True, 3) would be "a=x&a=&a=true&a=3"
• All builtin constraints are also supported

Questions:

• Do the above encodings make sense?
• Do the restrictions on supported types make sense? In particular, note the no-nested-objects/sequences restriction
• Are there other options we'd want to expose on encode/decode? The stdlib also exposes a few options that I've never needed to change:
  • max_num_fields to limit the number of fields when decoding
  • separator to change the character used for separating fields (defaults to &).
• Is msgspec.querystring the best namespace for this format, or is there a better name we could use?
• Does this seem like something that would be useful to have in msgspec? The intent here is for msgpspec to handle much of the parsing/validation that a typical web server would need to handle in a performant and useful way.

Hi again,

In pydantic I can pass any kwargs to a Field and these will be set as a dictionary under the ModelField.field_info.extra dictionary (if extra is allowed). I can also pass an extra dict directly. This is useful for library authors such as myself because it allows us to pass meta data as part of a field defintion.

For example, we have stuff like this: https://starlite-api.github.io/starlite/usage/3-parameters/3-the-parameter-function/.

At present the closest thing possible with msgspec is to use the Meta object with Annotated, and hijack the extra_json_schema dictionary. This though is not a proper solution - first, because this is not about an extra json schema but rather about passing kwargs, and secondly because this is not necessarily about constraints and the usage of Annotated might be redundant in this case.

Hi again,

So I encountered an issue when trying to integrate msgspec into the internal of our libraries, namely - there is no way for me to easily access the field information on a struct.

The __struct_fields__ attribute only offers a tuple of field names. If I want to access the metadata regarding each field I have to result to some rather inefficent methods such as inspecting the attributes, which is very slow.

I would like to regard a simple and fast way to get the following information about a field: its typing, any default value defined for it, and other meta data available.

from msgspec import Struct
from functools import cached_property

class Foo(Struct):
	bar: tuple[str]

	@cached_property
	def bar_inner(self) -> str:
		return self.bar[0]

will fail because Struct does not implement __dict__. could it be possible to implement cached_property for Struct?

Traceback (most recent call last):
  File "/home/scarf/repo/cata/tileset-tools/learnpillow.py", line 26, in <module>
    print(meta.fallback.ascii)
          ^^^^^^^^^^^^^
  File "/home/scarf/.asdf/installs/python/3.11.0rc2/lib/python3.11/functools.py", line 994, in __get__
    raise TypeError(msg) from None
TypeError: No '__dict__' attribute on 'TileConfig' instance to cache 'fallback' property.

@property decorator does work, but would be nice to have a cached one too (if possible)

Case 1

[
  { "height": 10, "width": 10 },                     // <- type 'A'
  { "ASCIITiles.png": { "//": "indices 0 to 79" } }, // <- type 'B'
  { "fallback.png": { "fallback": true } }
  ...
]

it's guaranteed that the array will be shaped like [A, B, B, ...]. I'm not sure how to make msgspec understand that only first element has different type.

Case 2

[
        { "id": "lighting_hidden" },
        {
          "id": "explosion_weak",
          "multitile": true,
          "additional_tiles": "(...)"
        }
]

the first and second element only differs in multitile and additional_tiles. additional_tiles only exist when multiile exists and is true. I wasn't able to turn both objects into tagged unions. could you help me to to correctly distinguish both objects?

Would be great to have an efficient way to serialize msgspec structs to apache arrow, which would also open it up to using parquet and other tools in the arrow ecosystem like duckdb.