Both pipenv install hyperjson and pip install hyperjson produce the following error:

  Could not find a version that satisfies the requirement hyperjson (from versions: )
No matching distribution found for hyperjson

Tried this on Ubuntu 18.10 with python 3.6.7

This sets up a github action to build/test/bench hyperjson across 3.5, 3.6, 3.7 x macos/ubuntu. This is my first github action, so things might not be as optimal as they could be. doesnt help with #60 atm, but deploying wheels looks pretty straightforward now that I'm comfy with actions.

Here is a list of failing tests and their status:

============================ FAILURES =============================
________________ UltraJSONTests.testEncodeSymbols _________________

self = <test_ujson.UltraJSONTests testMethod=testEncodeSymbols>

    def testEncodeSymbols(self):
        s = '\u273f\u2661\u273f'  # ✿♡✿
        encoded = hyperjson.dumps(s)
        encoded_json = hyperjson.dumps(s)
>       self.assertEqual(len(encoded), len(s) * 6 + 2)  # 6 characters + quotes
E       AssertionError: 5 != 20

hyperjson/tests/test_ujson.py:229: AssertionError
_______________ UltraJSONTests.testEncodeUnicodeBMP _______________

self = <test_ujson.UltraJSONTests testMethod=testEncodeUnicodeBMP>

    def testEncodeUnicodeBMP(self):
        s = '\U0001f42e\U0001f42e\U0001F42D\U0001F42D'  # 🐮🐮🐭🐭
        encoded = hyperjson.dumps(s)
        encoded_json = hyperjson.dumps(s)

        if len(s) == 4:
>           self.assertEqual(len(encoded), len(s) * 12 + 2)
E           AssertionError: 6 != 50

hyperjson/tests/test_ujson.py:204: AssertionError
_____________ UltraJSONTests.test_ReadBadObjectSyntax _____________

self = <test_ujson.UltraJSONTests testMethod=test_ReadBadObjectSyntax>

    def test_ReadBadObjectSyntax(self):
        input = '{"age", 44}'
>       self.assertRaises(ValueError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "{\"age\", 44}", Error: expected `:` at line 1 column 7

hyperjson/tests/test_ujson.py:820: JSONDecodeError
_________ UltraJSONTests.test_WriteArrayOfSymbolsFromList _________

self = <test_ujson.UltraJSONTests testMethod=test_WriteArrayOfSymbolsFromList>

    def test_WriteArrayOfSymbolsFromList(self):
        self.assertEqual("[true, false, null]",
>                        hyperjson.dumps([True, False, None]))
E       AssertionError: '[true, false, null]' != '[true,false,null]'
E       - [true, false, null]
E       ?       -      -
E       + [true,false,null]

hyperjson/tests/test_ujson.py:846: AssertionError
________ UltraJSONTests.test_WriteArrayOfSymbolsFromTuple _________

self = <test_ujson.UltraJSONTests testMethod=test_WriteArrayOfSymbolsFromTuple>

    def test_WriteArrayOfSymbolsFromTuple(self):
        self.assertEqual("[true, false, null]",
>                        hyperjson.dumps((True, False, None)))
E       AssertionError: '[true, false, null]' != '[true,false,null]'
E       - [true, false, null]
E       ?       -      -
E       + [true,false,null]

hyperjson/tests/test_ujson.py:850: AssertionError
___________ UltraJSONTests.test_decodeArrayDepthTooBig ____________

self = <test_ujson.UltraJSONTests testMethod=test_decodeArrayDepthTooBig>

    def test_decodeArrayDepthTooBig(self):
        input = '[' * (1024 * 1024)
>       self.assertRaises(RecursionError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "[{{", Error: key must be a string at line 1 column 2

hyperjson/tests/test_ujson.py:397: JSONDecodeError
_______________________________ UltraJSONTests.test_decodeTrueBroken ________________________________

self = <test_ujson.UltraJSONTests testMethod=test_decodeTrueBroken>

    def test_decodeTrueBroken(self):
        input = "tru"
>       self.assertRaises(ValueError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "tru", Error: expected ident at line 1 column 3

hyperjson/tests/test_ujson.py:413: JSONDecodeError
_______________________ UltraJSONTests.test_decodeWithTrailingNonWhitespaces ________________________

self = <test_ujson.UltraJSONTests testMethod=test_decodeWithTrailingNonWhitespaces>

    def test_decodeWithTrailingNonWhitespaces(self):
        input = "{}\n\t a"
>       self.assertRaises(JSONDecodeError, hyperjson.loads, input)
E       _hyperjson.JSONDecodeError: Value: "{}\n\t a", Error: trailing characters at line 2 column 3

hyperjson/tests/test_ujson.py:790: JSONDecodeError
__________________________________ UltraJSONTests.test_dumpToFile ___________________________________

self = <test_ujson.UltraJSONTests testMethod=test_dumpToFile>

    def test_dumpToFile(self):
        f = six.StringIO()
        hyperjson.dump([1, 2, 3], f)
>       self.assertEqual("[1, 2, 3]", f.getvalue())
E       AssertionError: '[1, 2, 3]' != '[1,2,3]'
E       - [1, 2, 3]
E       ?    -  -
E       + [1,2,3]

hyperjson/tests/test_ujson.py:556: AssertionError
_____________________________ UltraJSONTests.test_dumpToFileLikeObject ______________________________

self = <test_ujson.UltraJSONTests testMethod=test_dumpToFileLikeObject>

    def test_dumpToFileLikeObject(self):
        class filelike:
            def __init__(self):
                self.bytes = ''

            def write(self, bytes):
                self.bytes += bytes

        f = filelike()
        hyperjson.dump([1, 2, 3], f)
>       self.assertEqual("[1, 2, 3]", f.bytes)
E       AssertionError: '[1, 2, 3]' != '[1,2,3]'
E       - [1, 2, 3]
E       ?    -  -
E       + [1,2,3]

hyperjson/tests/test_ujson.py:568: AssertionError
_______________________ UltraJSONTests.test_encodeListLongUnsignedConversion ________________________

self = <test_ujson.UltraJSONTests testMethod=test_encodeListLongUnsignedConversion>

    def test_encodeListLongUnsignedConversion(self):
        input = [18446744073709551615,
                 18446744073709551615, 18446744073709551615]
        output = hyperjson.dumps(input)

>       self.assertEqual(input, hyperjson.loads(output))
E       AssertionError: Lists differ: [18446744073709551615, 18446744073709551615, 18446744073709551615] != [1.8446744073709552e+19, 1.8446744073709552e+19, 1.8446744073709552e+19]
E
E       First differing element 0:
E       18446744073709551615
E       1.8446744073709552e+19
E
E       - [18446744073709551615, 18446744073709551615, 18446744073709551615]
E       ?                   ^ ^^^^                ^ ^^^^                ^^^
E
E       + [1.8446744073709552e+19, 1.8446744073709552e+19, 1.8446744073709552e+19]
E       ?   +               +++ ^^^ ^               +++ ^^^ ^               +++ ^

hyperjson/tests/test_ujson.py:495: AssertionError
_________________________ UltraJSONTests.test_encodeLongUnsignedConversion __________________________

self = <test_ujson.UltraJSONTests testMethod=test_encodeLongUnsignedConversion>

    def test_encodeLongUnsignedConversion(self):
        input = 18446744073709551615
        output = hyperjson.dumps(input)

>       self.assertEqual(input, hyperjson.loads(output))
E       AssertionError: 18446744073709551615 != 1.8446744073709552e+19

hyperjson/tests/test_ujson.py:509: AssertionError
_______________________________ UltraJSONTests.test_encodeOrderedDict _______________________________

self = <test_ujson.UltraJSONTests testMethod=test_encodeOrderedDict>

    @unittest.skipIf(sys.version_info < (2, 7), "No Ordered dict in < 2.7")
    def test_encodeOrderedDict(self):
        from collections import OrderedDict
        input = OrderedDict([(1, 1), (0, 0), (8, 8), (2, 2)])
        self.assertEqual('{"1": 1, "0": 0, "8": 8, "2": 2}',
>                        hyperjson.dumps(input))
E       AssertionError: '{"1": 1, "0": 0, "8": 8, "2": 2}' != '{"0":0,"1":1,"2":2,"8":8}'
E       - {"1": 1, "0": 0, "8": 8, "2": 2}
E       + {"0":0,"1":1,"2":2,"8":8}

hyperjson/tests/test_ujson.py:369: AssertionError
___________________________________ UltraJSONTests.test_sortKeys ____________________________________

self = <test_ujson.UltraJSONTests testMethod=test_sortKeys>

    def test_sortKeys(self):
        data = {"a": 1, "c": 1, "b": 1, "e": 1, "f": 1, "d": 1}
        sortedKeys = hyperjson.dumps(data, sort_keys=True)
        self.assertEqual(
>           sortedKeys, '{"a": 1, "b": 1, "c": 1, "d": 1, "e": 1, "f": 1}')
E       AssertionError: '{"a":1,"b":1,"c":1,"d":1,"e":1,"f":1}' != '{"a": 1, "b": 1, "c": 1, "d": 1, "e": 1, "f": 1}'
E       - {"a":1,"b":1,"c":1,"d":1,"e":1,"f":1}
E       + {"a": 1, "b": 1, "c": 1, "d": 1, "e": 1, "f": 1}
E       ?      +  +    +  +    +  +    +  +    +  +    +

hyperjson/tests/test_ujson.py:865: AssertionError
========================= 30 failed, 102 passed, 28 skipped in 9.44 seconds =========================

This PR tries to reduce the warning-noise when cargo clippy is applied.

The most common warning was about passing arguments by value instead of passing by reference:

warning: this argument is passed by value, but not consumed in the function body
   --> src/lib.rs:228:8
    |
228 |     s: PyObject,
    |        ^^^^^^^^ help: consider taking a reference instead: `&PyObject`
    |
    = help: for further information visit https://rust-lang-nursery.github.io/rust-clippy/v0.0.212/index.html#needless_pass_by_value

Some of the warnings I managed to resolve, but for functions annotated with #[pyfunction] the simple usage of &Option<> introduces an error:

error[E0277]: the trait bound `pyo3::PyObject: pyo3::PyTypeInfo` is not satisfied                                                                                                             
   --> src/lib.rs:153:1                                                                                                                                                                       
    |                                                                                                                                                                                         
153 | #[pyfunction]                                                                                                                                                                           
    | ^^^^^^^^^^^^^ the trait `pyo3::PyTypeInfo` is not implemented for `pyo3::PyObject`                                                                                                      
    |                                                                                                                                                                                         
    = note: required because of the requirements on the impl of `pyo3::PyTryFrom` for `pyo3::PyObject`                                                                                        
    = note: required because of the requirements on the impl of `pyo3::FromPyObject<'_>` for `&pyo3::PyObject`                                                                                
    = note: required by `pyo3::ObjectProtocol::extract`   

The way to resolve the error could be the usage the PyObjectRef and also the usage of #[pyfn()] instead of #[pyfunction] (actually could not find #[pyfunction] in the 0.5.0 documentation - could it be deprecated?), but that would introduce the API breakage, so I left it for now.

The result changes are somewhat chaotic, so feel free to close

Like the title states. 0.6.0 introduced some API incompatibilities that I have fixed to the best of my abilities. I also went ahead and updated every dep involved, and fixed all incompatibilities discovered.

I ran profiles, tests, and played around with it in the python shell, and from what I can tell works well again

should fix #57

make build and by extension cargo build, cargo test and cargo bench fail with the following error:

error: linking with `cc` failed: exit code: 1
..... A lot of output ....
undefined reference to `PyExc_TypeError'
collect2: error: ld returned 1 exit status

Perhaps the build command should be modified to match setuptools-rust one?

E.g.

cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module pyo3/python3 --release -- --crate-type cdylib

We should automate the deployment process of hyperjson. Since I had some great experiences with Github actions, I would prefer to write a completely new CI pipeline with it and remove Travis from the project.

The pipeline should...

• run the tests
• publish the Python package for Python 3.5, 3.6, 3.7, and optionally sdist with the help of maturin.
• (optionally) release to crates.io

If someone wants to tackle this, please go ahead. 😊

We can serialize to bool directly without using extract. This gives us a nice performance boost as can be seen in the new plots. The same trick can probably be applied elsewhere. Good places to look for such improvements are the remaining extract calls and the macro handling for casts.

Yesterday I did some profiling using the setup described here. The resulting callgrind file is attached. This can be opened with qcachegrind on Mac or kcachegrind on Linux.

callgrind.out.35583.zip

If you don't have any of those programs handy, I've added a screenshot for the two main bottlenecks that I can see. I'm not an expert, but it looks like we spend a lot of time allocating, converting, and dropping the BTreeMap, which will be converted to a dictionary and returned to Python in the end.

I guess we could save a lot of time by making this part more efficient. E.g. by copying less and instead working on references. Might be mistaken, though. Help and pull requests are very welcome. 😊

Github deprecated the ::add_path command sometime in October which is causing the CI to fail, adding platform specific steps for modifying the PATH should fix this (at least it did in my poetry/py03 project).

Relevant error:

The `add-path` command is disabled. Please upgrade to using Environment Files or opt into unsecure command execution by setting the `ACTIONS_ALLOW_UNSECURE_COMMANDS` environment variable to `true`. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/ 

Bumps pytest from 5.2.4 to 5.3.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

If all status checks pass Dependabot will automatically merge this pull request.

Bumps simplejson from 3.17.2 to 3.18.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps regex from 1.3.1 to 1.5.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps flake8 from 3.9.1 to 3.9.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps autopep8 from 1.5.6 to 1.5.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hi!

I'm happy to find that hyperjson rejects single surrogates as invalid characters but the exception text looks more low-level than expected. Is this a bug? Can the exception message be made more highlevel like "single surrogates not allowed"?

In [8]: hyperjson.__version__
Out[8]: '0.2.4'

In [9]: hyperjson.loads('"\\ud800"')
[..]
JSONDecodeError: Value: PyObject(0x7f481b3a1530), Error: PyErr { type: Py(0x7f48262225c0, PhantomData) }: line 1 column 1 (char 0)

Thanks and best, Sebastian

From our benchmarks we can see that we are consistently slower than everyone else when serializing/deserializing boolean values. We should fix that.

orjson is using an unsafe block to create a reference to a boolean: https://github.com/ijl/orjson/blob/03d55e99a953ce93cedc05f03e4b63b0bcbbcc7a/src/decode.rs#L81-L96

This avoids additional allocations. For comparison, this is our code at the moment:

https://github.com/mre/hyperjson/blob/ded13b4100638aa32fe19dc477f5cfe3e704893c/src/lib.rs#L475-L480

I wonder if we could achieve comparable performance without using unsafe. @konstin, any idea? Maybe there was a recent development in pyo3 that we could leverage here?