Cython implementation of Toolz: High performance functional utilities

Last update: Jan 02, 2023

Related tags

Overview

CyToolz

Cython implementation of the toolz package, which provides high performance utility functions for iterables, functions, and dictionaries.

toolz is a pure Python package that borrows heavily from contemporary functional languanges. It is designed to interoperate seamlessly with other libraries including itertools, functools, and third party libraries. High performance functional data analysis is possible with builtin types like list and dict, and user-defined data structures; and low memory usage is achieved by using the iterator protocol and returning iterators whenever possible.

cytoolz implements the same API as toolz. The main differences are that cytoolz is faster (typically 2-5x faster with a few spectacular exceptions) and cytoolz offers a C API that is accessible to other projects developed in Cython. Since toolz is able to process very large (potentially infinite) data sets, the performance increase gained by using cytoolz can be significant.

See the PyToolz documentation at https://toolz.readthedocs.io and the full API Documentation for more details.

LICENSE

New BSD. See License File.

Install

cytoolz is on the Python Package Index (PyPI):

pip install cytoolz

Dependencies

cytoolz supports Python 3.5+ with a common codebase. It is developed in Cython, but requires no dependecies other than CPython and a C compiler. Like toolz, it is a light weight dependency.

Contributions Welcome

toolz (and cytoolz) aims to be a repository for utility functions, particularly those that come from the functional programming and list processing traditions. We welcome contributions that fall within this scope and encourage users to scrape their util.py files for functions that are broadly useful.

Please take a look at our issue pages for toolz and cytoolz for contribution ideas.

Community

See our mailing list. We're friendly.

Comments

add join

This is ugly and likely inefficient. Help.

toolz.join has some pretty complex yield stuff going on. The work here is pretty much just writing down state explicitly to handle all of that. It's pretty ugly and I assume that it's not any more efficient than the Pure Python solution. Tips appreciated.

opened by mrocklin 28

Problems installing cytoolz on [email protected]

python version 3.10.0
pip version 21.3.1

Running pip install cytoolz is giving me the following error log:

  Using cached cytoolz-0.11.0.tar.gz (477 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: toolz>=0.8.0 in ./env/lib/python3.10/site-packages (from cytoolz) (0.11.1)
Building wheels for collected packages: cytoolz
  Building wheel for cytoolz (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/hristotodorov/Documents/Vyper/env/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/setup.py'"'"'; __file__='"'"'/private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-wheel-czxvyqb8
       cwd: /private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/
  Complete output (75 lines):
  ALERT: Cython not installed.  Building without Cython.
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.9-universal2-3.10
  creating build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/compatibility.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/utils_test.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/_version.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/__init__.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/_signatures.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  creating build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
  copying cytoolz/curried/operator.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
  copying cytoolz/curried/__init__.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
  copying cytoolz/curried/exceptions.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
  copying cytoolz/itertoolz.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/dicttoolz.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/functoolz.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/recipes.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/utils.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/utils.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/__init__.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/recipes.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/functoolz.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/dicttoolz.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/cpython.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  copying cytoolz/itertoolz.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
  creating build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_none_safe.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_utils.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_curried.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_compatibility.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_embedded_sigs.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_functoolz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_inspect_args.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_doctests.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_tlz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_signatures.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/dev_skip_test.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_recipes.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_docstrings.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_dev_skip_test.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_dicttoolz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_serialization.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_curried_toolzlike.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  copying cytoolz/tests/test_itertoolz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
  running build_ext
  building 'cytoolz.dicttoolz' extension
  creating build/temp.macosx-10.9-universal2-3.10
  creating build/temp.macosx-10.9-universal2-3.10/cytoolz
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Users/hristotodorov/Documents/Vyper/env/include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c cytoolz/dicttoolz.c -o build/temp.macosx-10.9-universal2-3.10/cytoolz/dicttoolz.o
  clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g build/temp.macosx-10.9-universal2-3.10/cytoolz/dicttoolz.o -o build/lib.macosx-10.9-universal2-3.10/cytoolz/dicttoolz.cpython-310-darwin.so
  building 'cytoolz.functoolz' extension
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Users/hristotodorov/Documents/Vyper/env/include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c cytoolz/functoolz.c -o build/temp.macosx-10.9-universal2-3.10/cytoolz/functoolz.o
  cytoolz/functoolz.c:23087:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                    ^
  cytoolz/functoolz.c:23087:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
              ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  cytoolz/functoolz.c:23092:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                    ^
  cytoolz/functoolz.c:23092:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
              ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  cytoolz/functoolz.c:23176:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
              ret = _PyGen_Send((PyGenObject*)yf, NULL);
                    ^
  cytoolz/functoolz.c:23176:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
              ret = _PyGen_Send((PyGenObject*)yf, NULL);
                  ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  3 warnings and 3 errors generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for cytoolz
  Running setup.py clean for cytoolz
Failed to build cytoolz
Installing collected packages: cytoolz
    Running setup.py install for cytoolz ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/hristotodorov/Documents/Vyper/env/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/setup.py'"'"'; __file__='"'"'/private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-record-s_w__7qh/install-record.txt --single-version-externally-managed --compile --install-headers /Users/hristotodorov/Documents/Vyper/env/include/site/python3.10/cytoolz
         cwd: /private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/
    Complete output (77 lines):
    ALERT: Cython not installed.  Building without Cython.
    running install
    /Users/hristotodorov/Documents/Vyper/env/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.9-universal2-3.10
    creating build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/compatibility.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/utils_test.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/_version.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/__init__.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/_signatures.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    creating build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
    copying cytoolz/curried/operator.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
    copying cytoolz/curried/__init__.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
    copying cytoolz/curried/exceptions.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/curried
    copying cytoolz/itertoolz.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/dicttoolz.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/functoolz.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/recipes.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/utils.pyx -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/utils.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/__init__.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/recipes.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/functoolz.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/dicttoolz.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/cpython.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    copying cytoolz/itertoolz.pxd -> build/lib.macosx-10.9-universal2-3.10/cytoolz
    creating build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_none_safe.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_utils.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_curried.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_compatibility.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_embedded_sigs.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_functoolz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_inspect_args.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_doctests.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_tlz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_signatures.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/dev_skip_test.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_recipes.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_docstrings.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_dev_skip_test.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_dicttoolz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_serialization.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_curried_toolzlike.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    copying cytoolz/tests/test_itertoolz.py -> build/lib.macosx-10.9-universal2-3.10/cytoolz/tests
    running build_ext
    building 'cytoolz.dicttoolz' extension
    creating build/temp.macosx-10.9-universal2-3.10
    creating build/temp.macosx-10.9-universal2-3.10/cytoolz
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Users/hristotodorov/Documents/Vyper/env/include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c cytoolz/dicttoolz.c -o build/temp.macosx-10.9-universal2-3.10/cytoolz/dicttoolz.o
    clang -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g build/temp.macosx-10.9-universal2-3.10/cytoolz/dicttoolz.o -o build/lib.macosx-10.9-universal2-3.10/cytoolz/dicttoolz.cpython-310-darwin.so
    building 'cytoolz.functoolz' extension
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/Users/hristotodorov/Documents/Vyper/env/include -I/Library/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c cytoolz/functoolz.c -o build/temp.macosx-10.9-universal2-3.10/cytoolz/functoolz.o
    cytoolz/functoolz.c:23087:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                      ^
    cytoolz/functoolz.c:23087:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
                ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                    ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    cytoolz/functoolz.c:23092:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                      ^
    cytoolz/functoolz.c:23092:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
                ret = _PyGen_Send((PyGenObject*)yf, value == Py_None ? NULL : value);
                    ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    cytoolz/functoolz.c:23176:19: error: implicit declaration of function '_PyGen_Send' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
                ret = _PyGen_Send((PyGenObject*)yf, NULL);
                      ^
    cytoolz/functoolz.c:23176:17: warning: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
                ret = _PyGen_Send((PyGenObject*)yf, NULL);
                    ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    3 warnings and 3 errors generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/hristotodorov/Documents/Vyper/env/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/setup.py'"'"'; __file__='"'"'/private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-install-0ebpvw2k/cytoolz_455f12fc06374667a15ffeeafa952f0f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/5y/q2vqps3s0jq7nc73z2zgmpmr0000gn/T/pip-record-s_w__7qh/install-record.txt --single-version-externally-managed --compile --install-headers /Users/hristotodorov/Documents/Vyper/env/include/site/python3.10/cytoolz Check the logs for full command output.

opened by HristoHT 15

Fails to build on Python 3.8

Using Python 3.8, cytoolz cannot build/install.

It appears to fail while compiling the dicttoolz.c file.

See here for the full log: https://pastebin.com/hrYA77um

If you have docker, you can quickly/easily replicate this with docker run python:3.8-rc python -m pip install cytoolz==0.10.0

Note that this appears to be related (and would be resolved) by fixing #134, as I can confirm that cloning the repository and using the 0.10.0 tag, I can successfully build and install the package with python setup.py build --cython after removing all of the generated .c files on python 3.8.

opened by cheeseandcereal 14
dicttoolz.pyx: faster iteration over dicts (10% speedup for keymap and friends)
This speeds up several operations in dicttoolz.pyx by using PyDict_Next instead of a Cython loop over the iteritems. Using the example predicate for keyfilter, I got the following results. Before:

>>> iseven = lambda x: x % 2 == 0 >>> d = dict(enumerate(range(1, 25000))) >>> %timeit keyfilter(iseven, d) 100 loops, best of 3: 3.19 ms per loop

After:

>>> %timeit keyfilter(iseven, d) 100 loops, best of 3: 2.85 ms per loop

I tried to apply the recipe to c_merge_with but found no difference in performance with large dicts, nor with a combination of large and small dicts, so I concluded that dict iteration isn't the bottleneck there (while the code became a lot more complicated). I did not try the many dicts use case.
opened by larsmans 12
add conda.yaml and .binstar.yml files
This provides hooks into an automated build process on binstar build.

We should be able to set it up so that we can do something like the following

conda install -c USERNAME cytoolz

Where USERNAME is something like mrocklin or eriknw or pytoolz-dev. And it will download and install a binary on either Linux or OS-X, no C compiler required.
opened by mrocklin 11

import failure with cython master

With cython master (and cpython master, but that seems to not matter here)

Process Python finished
Python 3.9.0a5+ (heads/master:71a3522ef8, Mar 26 2020, 23:28:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import cytoolz
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-fe6152314c51> in <module>
----> 1 import cytoolz

~/source/other_source/cytoolz/cytoolz/__init__.py in <module>
     22 from . import curried  # sandbox
     23 
---> 24 functoolz._sigs.update_signature_registry()
     25 
     26 from ._version import __version__, __toolz_version__

AttributeError: partially initialized module 'cytoolz' has no attribute 'update_signature_registry' (most likely due to a circular import)

In [3]: import cytoolz
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-fe6152314c51> in <module>
----> 1 import cytoolz

~/.virtualenvs/bleeding/lib/python3.9/site-packages/cytoolz/__init__.py in <module>
     17 
     18 # Always-curried functions
---> 19 flip = functoolz.flip = curry(functoolz.flip)
     20 memoize = functoolz.memoize = curry(functoolz.memoize)
     21 

NameError: name 'functoolz' is not defined

This works correctly with the latest cython on pypi (0.29.16).

I noticed this a month or two ago but only just now tracked down the source of the problem. I am unfortunately out of my depth to try and debug this.

opened by tacaswell 10

Partition
OK, here is a "getting the feet wet" pull request.

At first I just relied on zip and izip_longest, as we do in toolz. This yielded a 30% speedup.

Then I tried making an iterator class and ran into weird issues. I get a fair number of stackoverflows. In particular I get some strange integer wraparound at 256. Does Cython use one byte integers somehow? Anyway, if you have any thoughts on debugging this I'd be interested.

Fail case on my machine

import cytoolz list(cytoolz.partition(5, range(1000)))

oddly this works

list(cytoolz.partition(5, range(200)))

but this doesn't

timeit list(cytoolz.partition(5, range(200)))
opened by mrocklin 10
possible unitialized values in PyTupleObject

The code in sliding_window in itertoolz.pyx and here too can create a PyTupleObject with uninitialized items. I added some print(n) to the code, and the test_sliding_window code did not assign to self.prev[0]. If my guess is correct, this will cause PyPy to crash. All values must be assigned in creating a PyTupleObject. CPython will not check this, and will crash later if you ever access an uninitialized item, PyPy checks it and crashes earlier.

opened by mattip 9
'cytoolz.functoolz.Compose' object has no attribute '__module__'

When porting to cytoolz from toolz, I ran into the problem where the some previous library code was expecting Compose to have __module__ defined. I'm new to the project, but would expect the goal to be to generally to be able to move to cytoolz from toolz without any issues.

Should this be defined in Compose? If so, I can work to add it.

opened by djrtwo 9
Cython Version
I tried building with Cython and got the exception below. I'm pretty sure this is due to using a very old version of Cython as it seems to be something they fixed ( http://trac.cython.org/ticket/542 ).

NotImplementedError("New relative imports.")

It would be nice to know what version Cython cytoolz aims to be compatible with. Could we please get a note in the Readme? If I'm just missing it, I would appreciate a helpful pointer. Thanks in advance.
opened by jakirkham 8
unified testing via `toolz`

In the interest of development speed, the tests from toolz were basically just copied over to cytoolz. This is not easily sustainable. We would like all toolz-related tests to reside in toolz. Moreover, we would like a way to run the tests for toolz such as toolz.tests.run(), and to provide an optional toolz-like package on which to run the tests such as toolz.tests.run(cytoolz).

This issue does not cover testing CyToolz C API, which I regard as very low priority at the moment.

opened by eriknw 8
Bump pypa/cibuildwheel from 2.11.2 to 2.11.4
Bumps pypa/cibuildwheel from 2.11.2 to 2.11.4.

Release notes

Sourced from pypa/cibuildwheel's releases.

v2.11.4

🐛 Fix a bug that caused missing wheels on Windows when a test was skipped using CIBW_TEST_SKIP (#1377)

🛠 Updates CPython 3.11 to 3.11.1 (#1371)

🛠 Updates PyPy 3.7 to 3.7.10, except on macOS which remains on 7.3.9 due to a bug. (#1371)

📚 Added a reference to abi3audit to the docs (#1347)

v2.11.3

✨ Improves the 'build options' log output that's printed at the start of each run (#1352)

✨ Added a friendly error message to a common misconfiguration of the CIBW_TEST_COMMAND option - not specifying path using the {project} placeholder (#1336)

🛠 The GitHub Action now uses Powershell on Windows to avoid occasional incompabilities with bash (#1346)

Changelog

Sourced from pypa/cibuildwheel's changelog.

v2.11.4

24 Dec 2022

🐛 Fix a bug that caused missing wheels on Windows when a test was skipped using CIBW_TEST_SKIP (#1377)

🛠 Updates CPython 3.11 to 3.11.1 (#1371)

🛠 Updates PyPy to 7.3.10, except on macOS which remains on 7.3.9 due to a bug on that platform. (#1371)

📚 Added a reference to abi3audit to the docs (#1347)

v2.11.3

5 Dec 2022

✨ Improves the 'build options' log output that's printed at the start of each run (#1352)

✨ Added a friendly error message to a common misconfiguration of the CIBW_TEST_COMMAND option - not specifying path using the {project} placeholder (#1336)

🛠 The GitHub Action now uses Powershell on Windows to avoid occasional incompabilities with bash (#1346)

Commits

27fc88e Bump version: v2.11.4

a7e9ece Merge pull request #1371 from pypa/update-dependencies-pr

b9a3ed8 Update cibuildwheel/resources/build-platforms.toml

3dcc2ff fix: not skipping the tests stops the copy (Windows ARM) (#1377)

1c9ec76 Merge pull request #1378 from pypa/henryiii-patch-3

22b433d Merge pull request #1379 from pypa/pre-commit-ci-update-config

98fdf8c [pre-commit.ci] pre-commit autoupdate

cefc5a5 Update dependencies

e53253d ci: move to ubuntu 20

e9ecc65 [pre-commit.ci] pre-commit autoupdate (#1374)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 0

Getting a circular dependency error

Traceback (most recent call last):
  File "main.py", line 11, in <module>
    import config
  File "/Users/satwik/code/hop23/hop23-backend/config.py", line 6, in <module>
    import web3
  File "/Users/satwik/code/hop23/hop23-backend/venv/lib/python3.8/site-packages/web3/__init__.py", line 6, in <module>
    from eth_account import (
  File "/Users/satwik/code/hop23/hop23-backend/venv/lib/python3.8/site-packages/eth_account/__init__.py", line 1, in <module>
    from eth_account.account import (
  File "/Users/satwik/code/hop23/hop23-backend/venv/lib/python3.8/site-packages/eth_account/account.py", line 8, in <module>
    from cytoolz import (
  File "/Users/satwik/code/hop23/hop23-backend/venv/lib/python3.8/site-packages/cytoolz/__init__.py", line 1, in <module>
    from .itertoolz import *
  File "cytoolz/itertoolz.pyx", line 17, in init cytoolz.itertoolz
  File "cytoolz/utils.pyx", line 10, in init cytoolz.utils
  File "/Users/satwik/code/hop23/hop23-backend/venv/lib/python3.8/site-packages/toolz/__init__.py", line 19, in <module>
    from . import curried, sandbox
  File "/Users/satwik/code/hop23/hop23-backend/venv/lib/python3.8/site-packages/toolz/curried/__init__.py", line 81, in <module>
    partial = toolz.curry(toolz.partial)
AttributeError: partially initialized module 'toolz' has no attribute 'partial' (most likely due to a circular import)

opened by satwikkansal 0

error 'No such file or directory: 'build/lib.linux-x86_64-3.9/cytoolz/_version.py'

Hello,

I'm trying to create a simple android app using kivy and buildozer on Google Colab, but when i compile the app i am getting this error 'No such file or directory: 'build/lib.linux-x86_64-3.9/cytoolz/_version.py'. I need cytoolz module because i use web3.py, which need it.

I use the last version of cytoolz (0.12.0) with the last toolz and setuptools modules.

How can i resolve this error?

opened by tonyo008 0
Simplify interleave.
Simplified the implementation of interleave and made it slightly faster.

seqs = [range(1000) for i in range(100)] %timeit list(interleave(seqs))

old: 5.07 ms ± 240 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) new: 4.03 ms ± 45.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
opened by groutr 0
Update property declarations.

Update property declarations from the special, deprecated style to the newer style supported by cython. https://cython.readthedocs.io/en/latest/src/userguide/extension_types.html#properties

opened by groutr 0

Releases(0.12.1)

0.12.1(Dec 23, 2022)
Support Python 3.11 (including wheels)

Drop Python 3.5 support

Source code(tar.gz)
Source code(zip)
0.12.0(Jul 11, 2022)
Update to toolz 0.12.0 (#170)

See toolz release notes: https://github.com/pytoolz/toolz/releases/tag/0.12.0

Build wheels automatically with cibuildwheel! (#167, #168, #169, #170)

Support newer Python versions (#165, don't use inspect.getargspec in tests)

Improve import times

Use versioneer.py to manage versions

__version__ and __toolz_version__ are no longer available from cytoolz._version--use cython.__toolz_version__ instead.

This is the first release with automatic wheel building, versioneer for version management, and auto-upload to PyPI 🤞
Source code(tar.gz)
Source code(zip)
0.11.2(Nov 6, 2021)
Support Python 3.10

Source code(tar.gz)
Source code(zip)

Cython implementation of Toolz: High performance functional utilities

Related tags

Overview

CyToolz

LICENSE

Install

Dependencies

Contributions Welcome

Community

Comments

v2.11.4

v2.11.3

v2.11.4

v2.11.3

Releases(0.12.1)

0.12.1(Dec 23, 2022)

0.12.0(Jul 11, 2022)

0.11.2(Nov 6, 2021)

Owner

Functional programming in Python: implementation of missing features to enjoy FP

More routines for operating on iterables, beyond itertools

A functional standard library for Python.

Make your functions return something meaningful, typed, and safe!

Cython implementation of Toolz: High performance functional utilities

粤语编程语言.The Cantonese programming language.

A fancy and practical functional tools

Simple, elegant, Pythonic functional programming.