A faster and highly-compatible implementation of the Python programming language. The code here is out of date, please follow our blog

Have you plan to move to an other build system like Cmake or ~~autotools~~? This will be in your roadmap #11 ?

Thanks! And sorry to disturb you but this questions seems important.

I tried to fix issue #171 and I also added a line $(LLVM_CONFIGURE_LINE) --disable-bindings in Makefile because if I don't have -disable-bindings flag, I won't be able to configure LLVM properly, there is an error saying since I have enabled bindings then I must install ctypes on my machine, which I have installed.

Back to bug #171, I tried to add some simple identity testing in __contains__ and __eq__ method. But I don't know for sure this is sufficient. I wonder should what does compareInternal do exactly? Should we put the identity test there?

Also, this is my first pull request, I have signed the agreement. Thanks!

Tried to hijack #441 for this, but couldn't once I deleted the original branch.

This changes the build directories to $(SRC_DIR)/build/Debug $(SRC_DIR)/build/Release $(SRC_DIR)/build/Debug-gcc

I think the usual complaint about polluting the source directory is when the object files are right next to source, but I guess we'll see. It's easy to override BUILD_DIR.

This JIT is tightly coupled to the ASTInterpreter, at every CFGBlock* entry/exit on can switch between interpreting and directly executing the generated code without having to do any translations. Generating the code is pretty fast compared to the LLVM tier but the generated code is not as fast as code generated by the higher LLVM tiers. But because the JITed can use runtime ICs, avoids a lot of interpretation overhead and stores CFGBlock locals symbols inside register/stack slots, it's much faster than the interpreter.

Current perf numbers:

1. run:

              pyston (calibration)                      :    1.0s base1: 1.0 (+1.5%)
              pyston django_template.py                 :    5.2s base1: 6.7 (-23.0%)
              pyston pyxl_bench.py                      :    4.6s base1: 5.0 (-8.0%)
              pyston sqlalchemy_imperative2.py          :    5.7s base1: 6.8 (-14.9%)
              pyston django_migrate.py                  :    1.9s base1: 3.1 (-38.4%)
              pyston virtualenv_bench.py                :    6.2s base1: 8.1 (-22.9%)
              pyston interp2.py                         :    3.7s base1: 4.1 (-9.9%)
              pyston raytrace.py                        :    5.8s base1: 6.0 (-3.5%)
              pyston nbody.py                           :    7.3s base1: 7.0 (+4.1%)
              pyston fannkuch.py                        :    6.3s base1: 6.4 (-1.5%)
              pyston chaos.py                           :   17.5s base1: 18.1 (-3.6%)
              pyston fasta.py                           :    4.5s base1: 4.3 (+5.1%)
              pyston pidigits.py                        :    5.8s base1: 5.3 (+8.7%)
              pyston richards.py                        :    1.9s base1: 1.7 (+10.2%)
              pyston deltablue.py                       :    1.4s base1: 1.8 (-20.4%)
              pyston (geomean-10eb)                     :    4.6s base1: 5.1 (-9.5%)

best of 3 runs:

              pyston (calibration)                      :    1.0s base3: 1.0 (-0.5%)
              pyston django_template.py                 :    4.4s base3: 4.6 (-5.4%)
              pyston pyxl_bench.py                      :    3.9s base3: 3.8 (+2.5%)
              pyston sqlalchemy_imperative2.py          :    4.9s base3: 5.1 (-5.1%)
              pyston django_migrate.py                  :    1.7s base3: 1.7 (+1.7%)
              pyston virtualenv_bench.py                :    5.0s base3: 5.2 (-2.4%)
              pyston interp2.py                         :    3.6s base3: 4.1 (-12.4%)
              pyston raytrace.py                        :    5.5s base3: 5.4 (+1.4%)
              pyston nbody.py                           :    7.0s base3: 7.2 (-2.6%)
              pyston fannkuch.py                        :    6.1s base3: 6.3 (-2.0%)
              pyston chaos.py                           :   16.8s base3: 17.5 (-4.2%)
              pyston fasta.py                           :    4.4s base3: 4.1 (+5.8%)
              pyston pidigits.py                        :    5.7s base3: 5.3 (+8.2%)
              pyston richards.py                        :    1.7s base3: 1.5 (+11.3%)
              pyston deltablue.py                       :    1.2s base3: 1.4 (-12.9%)
              pyston (geomean-10eb)                     :    4.2s base3: 4.3 (-1.4%)

because of the increased malloc traffic this PR should benefit a lot from jemalloc...

@dagar First of all, thanks for getting it working, it's definitely a huge help :)

I wonder if there's something we can do about the testing time though, which I assume comes mostly from having to compile LLVM. I don't think we can easily get rid of our source dependency on LLVM, but we end up building the same stuff every time so maybe there's some way to cache it? We could probably come up with some hash of {llvm_rev, patchset}, and map that to the final build products that we use. A simpler option could be to enable ccache for the Travis build and have Travis cache the .ccache directory.

What do you think? Not hugely important at this stage but could be nice.

Mostly moving simple_destructor functions into tp_dealloc instead of their own field. Replace the field with a boolean to indicate whether the tp_dealloc has the safety properties of a simple_destructor. Assign simple_destructor to more builtin types.

Depends on gcfinalizers2 PR.

This works, and results in a pretty decent speedup for django-template.py, from ~10s to ~8s.

opening a PR to let travis-ci do the slow tests. need to clean up the changes and figure out a better way to deal with the state I'm carrying around during unwinding.

There are still a couple of tests failing (including the one with the recursive GC call that I talked about on Gitter... I'm still not sure if that's partially the fault of this diff or not), but I'll go ahead and put it up here. Also, this ended up being one of those diffs where I fix one thing and that leads to fixing another thing...

Anyway, here is what is happening in this diff:

• In typeNew, I add support for __slots__. This adds a member descriptor for each name in the slots array. Also, if __dict__ is in the slots, then it adds a dict (well, really an attrs, since it's pyston).
• Hopefully this should reduce allocations since most objects with slots won't have to allocate attrs.
• To garbage collect the slots, we need to keep track of the slot locations in the type object. CPython lists PyMemberDefs at the end of a type object. We don't need the full PyMemberDef struct (as far as I know--maybe the c api requires it though), but for now, I just put the slot offsets in a variable array in the type object.
• In the process of writing the above, logic, I stumbled across a tangential problem with the way we add a dict (or in our case, attrs) to a class when subclassing a class that is variable-size. This is tricky, since you have to leave the base layout alone, but that means you don't have a fixed offset to put the dict afterwards. CPython solves this by allowing tp_dictoffset to be negative, so you can offset it from the end of the object. From the [https://docs.python.org/2/c-api/typeobj.html#c.PyTypeObject.tp_dictoffset](python docs):

If the value is less than zero, it specifies the offset from the end of the instance structure. A negative offset is more expensive to use, and should only be used when the instance structure contains a variable-length part. This is used for example to add an instance variable dictionary to subtypes of str or tuple. Note that the tp_basicsize field should account for the dictionary added to the end in that case, even though the dictionary is not included in the basic object layout.

In the process of supporting this, I had to make sure that string and tuple subclassed VarBox and set the ob_size field. It's a bit messy and I think more can be done here to clean things up, but I didn't want to dig too far on this since, again, it's kind of tangential.

• Add __format__ function to long object. Use generic format function(object_format) will have many redundant calls and can not work with d format encode.

Mark as "WIP", because it may contains other minor NumPy fixing soon.

Attempt #2 at a PR implementing PyPy's finalization order algorithm. Depends on PR #662 and #666

As far as I can tell, 1/3 of the slowdown for django_template comes from the extra switch statement added in TraceStack which now has 3 different modes, and the other 2/3 of the slowdown comes from always adding unwind info in doSafePoint. fasta.py is suffering a massive slowdown because of large numbers of PyCapsule objects which don't seem to be safe to turn into simple destructors.

              pyston (calibration)                      :    1.0s baseline: 1.0 (+0.1%)
              pyston django_template.py                 :    4.6s baseline: 4.5 (+2.1%)
              pyston pyxl_bench.py                      :    3.7s baseline: 3.6 (+3.6%)
              pyston sqlalchemy_imperative2.py          :    5.2s baseline: 5.1 (+2.8%)
              pyston django_migrate.py                  :    1.8s baseline: 1.7 (+3.3%)
              pyston virtualenv_bench.py                :    5.3s baseline: 5.2 (+3.4%)
              pyston interp2.py                         :    3.9s baseline: 3.9 (-0.2%)
              pyston raytrace.py                        :    5.7s baseline: 5.3 (+7.4%)
              pyston nbody.py                           :    7.1s baseline: 6.8 (+5.8%)
              pyston fannkuch.py                        :    6.1s baseline: 5.8 (+5.1%)
              pyston chaos.py                           :   18.0s baseline: 16.0 (+12.8%)
              pyston fasta.py                           :    5.0s baseline: 4.1 (+22.2%)
              pyston pidigits.py                        :    4.9s baseline: 4.7 (+4.7%)
              pyston richards.py                        :    1.5s baseline: 1.4 (+1.3%)
              pyston deltablue.py                       :    1.4s baseline: 1.3 (+4.0%)
              pyston (geomean-10eb)                     :    4.3s baseline: 4.1 (+5.5%)

This is a simple cache - we still need to generate the IR but if we can find a cache file created for the exact same IR we will load it and skip instruction selection etc...

Our benchmark test suite does not benefit much from this patch because the JITing time is for most tests too short.

• I see a +2% slowdown on the first run against unmodified pyston (we have to do additional hashing, disk writes, pointer relocating)
• on the second run it turns into a -2% speedup.
• django_setup.py improves from 2.3sec to about 1.6sec.

The cache works be creating symbols for all embedded pointers which change from one pyston start to the next using the same executable (Fixed executable specific addresses will be directly emitted in order to reduce the amount of symbols which will need relocations). The IR for the module in text representation will get crc32 hashed (I'm very open to other ideas :-)) and if we find a file with the same hash in the pyston_object_cache directory we will use it. While this approach is very safe, it fails if the IR output has not the same variable names, line numbers,... That's why I changed the variable name assignment to us a incremental index in the name instead of the pointer value as string. Even after this patch there are still a few instances of nondeterministic IR output (but by far the most cases should get handled), I plan to improve this in a follow up patch. On our devel machines we will generate a huge amount of cache files with this patch because the cache file only works for the exact same executable. I plan to generate a hash of the pyston executable and save the cache file in a directory with the hash name. And remove on startup all directories which do not contain this hash. Better ideas? Another issue is that the IR is now more difficult to read because the patchpoints func destinations will all call to a dummy -1 address but there is a option to disable the cache if one has to debug something.

Hi!

I am the maintainer of python-speed benchmark (https://github.com/vprelovac/python-speed) and I could confirm the increased speed in Pyston. It is about 8% overall (not exactly 20% that is claimed) mostly due to much better stack handling (which shows about 40% speed increase). Overall fantastic job and looking forward to next release!

Python 3 python-speed v1.2 using python v3.8.5 string/mem: 2144.74 ms pi calc/math: 2939.3 ms regex: 2950.72 ms fibonnaci/stack: 1845.43 ms

total: 9880.2 ms (lower is better)

Pyston 3 python-speed v1.2 using python v3.8.2 string/mem: 2152.05 ms pi calc/math: 2971.2 ms regex: 2707.27 ms fibonnaci/stack: 1197.36 ms

total: 9027.89 ms (lower is better)

Is there any 2.1 release distribution of pyston which includes libpyston3.8.a or libpyston3.8.so? I'm needing to build other software which requires the libpython*.a, libpython*.so files for Python bindings.

Hi there and congratulations on the new release. I was wondering if you had any guidelines for setting the JIT_MAX_MEM env variable. Also I was wondering if it takes a full number or if = '100MB' would work etc.

pyenv is the de-facto standard for installing and managing multiple Python versions. Travis-CI uses it to build their supported Python interpreters.

We should ensure that installing pyston with pyenv is possible.