Hi yesterday i compile nogil python version and i found an unexpected behavior.

On startup my application prints

gc_get_refs(gc): -1
gc_get_refs(gc): -1
gc_get_refs(gc): -1
gc_get_refs(gc): -1
gc_get_refs(gc): -1
Process started (my app output)
gc_get_refs(gc): -1
gc_get_refs(gc): -1

and then after some minutes my app crashed with error:

Fatal Python error: _Py_queue_object: _Py_queue_object called with unowned object
Python runtime state: initialized

Hi, last days my program crash on high load (when users get active) and i get error:

*** stack smashing detected ***: <unknown> terminated

And i don't know how track this. Also i only get it when my application receive much client updates.

Is there a way i can fix this or do anything?

I would like to test this experimental branch on a regular basis on my Apple M1 dev laptop when working on the Scientific Python / PyData related projects so as to be able to report problems or performance improvements encountered with practical application cases.

However the build setup in this branch is a bit too old and the configure refuses to generate the Makefile for this platform:

./configure --with-openssl=$(brew --prefix openssl)
checking for git... found
checking build system type... Invalid configuration `arm64-apple-darwin20.0.0': machine `arm64-apple' not recognized
configure: error: /bin/sh ./config.sub arm64-apple-darwin20.0.0 failed

Would it be possible to sync with branch with a more recent version of CPython (e.g. 3.10)?

EDIT: the error above was probably observed with a version of clang installed from conda-forge. I get a different error at build time when using the system compiler instead, see the discussion below: https://github.com/colesbury/nogil/issues/36#issuecomment-1046302795

Hello, I am trying to compile nogil on Windows which is endping with LNK2019 and LNK1120 errors. I am using VisualStudio Code 2017.

Do you know what might be a reason?

I get an error when trying to import pydantic using the latest image.

pip.log

[email protected]:~# python3
Python 3.9.0a4+ (default, Oct 17 2021, 19:16:30)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pydantic import BaseModel
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pydantic/__init__.py", line 2, in init pydantic.__init__
  File "pydantic/dataclasses.py", line 7, in init pydantic.dataclasses
    import builtins
  File "pydantic/main.py", line 376, in init pydantic.main
  File "pydantic/main.py", line 369, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/utils.py", line 205, in pydantic.utils.generate_model_signature
  File "/usr/local/lib/python3.9/inspect.py", line 3076, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
  File "/usr/local/lib/python3.9/inspect.py", line 2825, in from_callable
    return _signature_from_callable(obj, sigcls=cls,
  File "/usr/local/lib/python3.9/inspect.py", line 2280, in _signature_from_callable
    return _signature_from_builtin(sigcls, obj,
  File "/usr/local/lib/python3.9/inspect.py", line 2091, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))
ValueError: no signature found for builtin <cyfunction BaseModel.__init__ at 0x58f257d9bd0>

It seems that nogil has compatability with tensorflow package. I have a python script that has tensorflow as its dependency, but when I try to run it with nogil, it reports error. Do you have any ideas about how to fix this compatability. Thanks!

ErrorMsg: python3 -m pip install tensorflow Looking in indexes: https://d1yxz45j0ypngg.cloudfront.net/, https://pypi.org/simple ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none) ERROR: No matching distribution found for tensorflow

The following happens when using the issue56 branch as discussed in #56.

$ viztracer --tracer_entries 10000002 ~/code/sanbox/spline_pipeline_randomized_search.py 
sys.version='3.9.10 (heads/issue56:851b1aac3e, May  2 2022, 11:56:36)\n[nogil, GCC 11.2.0]', joblib_backend_name='threading', n_jobs=8, nogil=True
Fitting 5 folds for each of 3 candidates, totalling 15 fits
Segmentation fault (core dumped)

Here is the source code of spline_pipeline_randomized_search.py:

import sys
from time import perf_counter
import numpy as np
from scipy.sparse import csr_matrix
from sklearn.datasets import fetch_covtype
from sklearn.preprocessing import QuantileTransformer
from sklearn.preprocessing import SplineTransformer
from sklearn.preprocessing import PolynomialFeatures
from sklearn.kernel_approximation import Nystroem
from sklearn.linear_model import SGDClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import RandomizedSearchCV
from sklearn.model_selection import train_test_split
from joblib import parallel_backend
from threadpoolctl import threadpool_limits


nogil = getattr(sys.flags, "nogil", False)
joblib_backend_name = "threading" if nogil else "loky"
n_jobs = 8

print(f"{sys.version=}, {joblib_backend_name=!r}, {n_jobs=}, {nogil=}")

X, y = fetch_covtype(return_X_y=True, as_frame=False)
X = X[:, :5]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=2_000, test_size=10_000, random_state=0
)

pipeline = Pipeline(
    [
        ("preproc", None),
        ("splines", SplineTransformer()),
        # ("poly", PolynomialFeatures(degree=2, interaction_only=True)),
        ("kernel", Nystroem(n_components=300, gamma=1e-3, random_state=0)),
        (
            "classifier",
            SGDClassifier(
                early_stopping=True,
                validation_fraction=0.2,
                n_iter_no_change=5,
                random_state=0,
            ),
        ),
    ]
)


param_grid = {
    "preproc": [None, QuantileTransformer()],
    "splines__degree": [2, 3, 4],
    "splines__n_knots": [3, 5, 10],
    "kernel__gamma": np.logspace(-5, 3, 100),
    "kernel__n_components": [10, 50, 100, 500],
    "classifier__loss": ["hinge", "log_loss"],
    "classifier__alpha": np.logspace(-5, 1, 100),
}

tic = perf_counter()
with parallel_backend(joblib_backend_name, n_jobs=n_jobs):
    with threadpool_limits(limits=1 if nogil else None):
        search = RandomizedSearchCV(
            pipeline,
            param_distributions=param_grid,
            n_iter=3,
            verbose=1,
            random_state=0,
        ).fit(X_train, y_train)
toc = perf_counter()
print(f"search completed in {toc - tic:.1f}s")
print(search.best_params_)
print(search.score(X_test, y_test))

Here is the gdb backtrace:

#0  _Py_IncRefShared (op=<unknown at remote 0x5555558aaec8>) at Objects/object.c:2334
#1  0x00007ffff6d59dd7 in _Py_INCREF (op=<optimized out>) at /home/ogrisel/code/nogil/Include/object.h:517
#2  snaptrace_tracefunc ([email protected]=<viztracer.Tracer at remote 0x293469ca280>, [email protected]=Frame 0x29349870a90, for file /home/ogrisel/code/nogil/Lib/inspect.py, line 2819, in <genexpr> (), 
    [email protected]=3, [email protected]=('knots', <Parameter at remote 0x29349811d20>)) at src/viztracer/modules/snaptrace.c:387
#3  0x00005555556a125f in call_profile ([email protected]=0x555555b40d10, frame=Frame 0x29349870a90, for file /home/ogrisel/code/nogil/Lib/inspect.py, line 2819, in <genexpr> (), [email protected]=3, 
    [email protected]=('knots', <Parameter at remote 0x29349811d20>)) at Python/ceval_meta.c:3511
#4  0x00005555556ab049 in vm_profile ([email protected]=0x555555b40d10, [email protected]=0x293445e6890 "\\\002\002M\001W", acc=..., [email protected]=('knots', <Parameter at remote 0x29349811d20>))
    at Python/ceval_meta.c:3644
#5  0x00005555556ab148 in vm_trace_handler (ts=0x555555b40d10, last_pc=0x293445e6890 "\\\002\002M\001W", acc=('knots', <Parameter at remote 0x29349811d20>)) at Python/ceval_meta.c:3778
#6  0x00005555558103c1 in _PyEval_Fast (ts=0x555555b40d10, initial_acc=Register(as_int64 = 10000003), initial_pc=0x293445e6893 "M\001W") at Python/ceval.c:2784
#7  0x00005555556a97e0 in _PyEval_Eval (pc=<optimized out>, acc=..., tstate=0x555555b40d10) at Python/ceval_meta.c:2831
#8  PyEval2_EvalGen ([email protected]=0x293498f0290, opt_value=None) at Python/ceval_meta.c:2862
#9  0x00005555557c23ac in gen_send_internal (gen=0x293498f0290, opt_value=<optimized out>) at Objects/genobject.c:244
#10 0x00005555557a63b0 in PyIter_Next ([email protected]=<generator at remote 0x293498f0290>) at Objects/abstract.c:2702
#11 0x00005555555fbcc1 in PyDict_MergeFromSeq2 ([email protected]=<OrderedDict at remote 0x29349850c10>, [email protected]=<generator at remote 0x293498f0290>, [email protected]=1) at Objects/dictobject.c:1895
#12 0x00005555555fc178 in dict_update_arg (self=<OrderedDict at remote 0x29349850c10>, arg=<generator at remote 0x293498f0290>) at Objects/dictobject.c:1827
#13 0x00005555555fc4a9 in dict_update_common (methname=0x555555882419 "dict", kwds=0x0, args=<optimized out>, self=<OrderedDict at remote 0x29349850c10>) at Objects/dictobject.c:1841
#14 dict_init (self=<OrderedDict at remote 0x29349850c10>, args=<optimized out>, kwds=0x0) at Objects/dictobject.c:2757
#15 0x00005555556242d7 in type_call (type=<optimized out>, args=(<generator at remote 0x293498f0290>,), kwds=0x0) at Objects/typeobject.c:1044
#16 0x00005555555c5a6d in _PyObject_MakeTpCall (tstate=0x555555b40d10, callable=<type at remote 0x29343e6fc10>, args=<optimized out>, nargs=1, keywords=0x0) at Objects/call.c:191
#17 0x00005555556a4bfb in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffccc25348, callable=<type at remote 0x29343e6fc10>, tstate=0x555555b40d10)
    at ./Include/cpython/abstract.h:116
#18 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffccc25348, callable=<type at remote 0x29343e6fc10>, tstate=0x555555b40d10) at ./Include/cpython/abstract.h:103
#19 vm_call_function (ts=0x555555b40d10, acc=...) at Python/ceval_meta.c:1341
#20 0x000055555581127b in _PyEval_Fast (ts=0x555555b40d10, initial_acc=Register(as_int64 = 10000003), initial_pc=0x293446e9e07 "F\017\001") at Python/ceval.c:982
#21 0x00005555556a8d2f in _PyEval_Eval (pc=<optimized out>, acc=..., tstate=0x555555b40d10) at Python/ceval_meta.c:2831
#22 _PyFunction_Vectorcall (func=<function at remote 0x293447750b0>, stack=0x29349801e48, nargsf=<optimized out>, kwnames=<optimized out>) at Python/ceval_meta.c:3228
#23 0x00005555555c7e68 in _PyObject_FastCallDictTstate (kwargs={'return_annotation': <type at remote 0x293446eec10>, '__validate_parameters__': False}, nargsf=2, args=0x7fffccc254c0, 
    callable=<function at remote 0x293447750b0>, tstate=0x555555b40d10) at Objects/call.c:129
#24 _PyObject_Call_Prepend ([email protected]=0x555555b40d10, [email protected]=<function at remote 0x293447750b0>, [email protected]=<Signature at remote 0x293498209d0>, 
    [email protected]=([<Parameter at remote 0x29349811a50>, <Parameter at remote 0x29349811b40>, <Parameter at remote 0x29349811c30>, <Parameter at remote 0x29349811d20>, <Parameter at remote 0x29349811e10>, <Parameter at remote 0x29349811f00>, <Parameter at remote 0x29349811eb0>],), [email protected]={'return_annotation': <type at remote 0x293446eec10>, '__validate_parameters__': False}) at Objects/call.c:387
#25 0x0000555555631d84 in slot_tp_init (self=<Signature at remote 0x293498209d0>, 
    args=([<Parameter at remote 0x29349811a50>, <Parameter at remote 0x29349811b40>, <Parameter at remote 0x29349811c30>, <Parameter at remote 0x29349811d20>, <Parameter at remote 0x29349811e10>, <Parameter at remote 0x29349811f00>, <Parameter at remote 0x29349811eb0>],), kwds={'return_annotation': <type at remote 0x293446eec10>, '__validate_parameters__': False}) at Objects/typeobject.c:7059
#26 0x00005555556242d7 in type_call (type=<optimized out>, 
    args=([<Parameter at remote 0x29349811a50>, <Parameter at remote 0x29349811b40>, <Parameter at remote 0x29349811c30>, <Parameter at remote 0x29349811d20>, <Parameter at remote 0x29349811e10>, <Parameter at remote 0x29349811f00>, <Parameter at remote 0x29349811eb0>],), kwds={'return_annotation': <type at remote 0x293446eec10>, '__validate_parameters__': False}) at Objects/typeobject.c:1044
#27 0x00005555555c5a6d in _PyObject_MakeTpCall (tstate=0x555555b40d10, callable=<type at remote 0x29344574610>, args=<optimized out>, nargs=1, keywords=('return_annotation', '__validate_parameters__'))
    at Objects/call.c:191
#28 0x00005555556a4981 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=9223372036854775809, args=0x7fffc0000bf8, callable=<type at remote 0x29344574610>, tstate=0x555555b40d10)
    at ./Include/cpython/abstract.h:116
#29 _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=9223372036854775809, args=0x7fffc0000bf8, callable=<type at remote 0x29344574610>, tstate=0x555555b40d10) at ./Include/cpython/abstract.h:103
#30 vm_call_cfunction_slow (ts=0x555555b40d10, acc=...) at Python/ceval_meta.c:1296
#31 0x000055555581127b in _PyEval_Fast (ts=0x555555b40d10, initial_acc=Register(as_int64 = 10000003), initial_pc=0x29343d12c28 "F!\001\002K") at Python/ceval.c:982
#32 0x00005555556a8d2f in _PyEval_Eval (pc=<optimized out>, acc=..., tstate=0x555555b40d10) at Python/ceval_meta.c:2831
#33 _PyFunction_Vectorcall (func=<function at remote 0x29348a982b0>, stack=0x29349900008, nargsf=<optimized out>, kwnames=<optimized out>) at Python/ceval_meta.c:3228
#34 0x00005555555c7e68 in _PyObject_FastCallDictTstate (
    kwargs={'train': <numpy.ndarray at remote 0x29349291ce0>, 'test': <numpy.ndarray at remote 0x29349291c70>, 'parameters': {'splines__n_knots': 10, 'splines__degree': 4, 'preproc': <QuantileTransformer(n_quantiles=1000, output_distribution='uniform', ignore_implicit_zeros=False, subsample=100000, random_state=None, copy=True) at remote 0x29349196550>, 'kernel__n_components': 500, 'kernel__gamma': <numpy.float64 at remote 0x29348d1f9c0>, 'classifier__loss': 'hinge', 'classifier__alpha': <numpy.float64 at remote 0x29348d1faa0>}, 'split_progress': (1, 5), 'candidate_progress': (0, 3), 'scorer': <function at remote 0x29348eca1f0>, 'fit_params': {}, 'return_train_score': False, 'return_n_test_samples': True, 'return_times': True, 'return_parameters': False, 'error_score': <float at remote 0x29345562380>, 'verbose': 1}, nargsf=4, 
    args=0x7fffccc25760, callable=<function at remote 0x29348a982b0>, tstate=0x555555b40d10) at Objects/call.c:129
#35 _PyObject_Call_Prepend ([email protected]=0x555555b40d10, cal[email protected]=<function at remote 0x29348a982b0>, 

The py-bt gdb commands returns:

Traceback (most recent call first):
  File "/home/ogrisel/code/nogil/Lib/inspect.py", line 2819, in <genexpr>
    params = OrderedDict((param.name, param) for param in parameters)
  0x0

Code to reproduce:

pip install numpy scipy cython psutil
git clone https://github.com/scikit-learn/scikit-learn
cd scikit-learn
pip install -e .

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import make_classification
import psutil
import gc


X, y = make_classification(n_samples=int(1e4), random_state=42)
print(f"data size: {X.nbytes / 1e6:.1f} MB")

for i in range(5):
    clf = HistGradientBoostingClassifier(max_iter=100).fit(X, y)
    gc.collect()
    print(f"memory usage: {psutil.Process().memory_info().rss / 1e6:.1f} MB")

Using nogil-Cython build of scikit-learn: CPython

$ OPENMP_NUM_THREADS=1 python ~/code/sanbox/debug_memleak.py 
data size: 1.6 MB
memory usage: 701.8 MB
memory usage: 1290.5 MB
memory usage: 1878.0 MB
memory usage: 2466.0 MB
memory usage: 3053.4 MB

Using scikit-learn installed from conda-forge

$ OPENMP_NUM_THREADS=1 python ~/code/sanbox/debug_memleak.py 
data size: 1.6 MB
memory usage: 124.6 MB
memory usage: 124.6 MB
memory usage: 125.1 MB
memory usage: 125.1 MB
memory usage: 125.1 MB

Note: this code is using OpenMP-based threading but the leak still happens when disabling the OpenMP threading layer by setting OPENMP_NUM_THREADS=1 so this problem is probably not related to OpenMP.

Note sure how to debug this. Maybe I could try to use valgrind.

I just tried to use nogil on my side project, and found out there is an error when installing cffi. Seems like there is a confict between nogil and CPython.

Origin error message as following,

Installing cffi (1.15.0): Failed

  EnvCommandError

  Command ['/workspace/.venv/bin/pip', 'install', '--no-deps', 'file:///root/.cache/pypoetry/artifacts/20/b6/2f/d86d84b144949a7c7435d77e30e4731b66f07f492faac0f75b18bebca7/cffi-1.15.0.tar.gz'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/20/b6/2f/d86d84b144949a7c7435d77e30e4731b66f07f492faac0f75b18bebca7/cffi-1.15.0.tar.gz
  Building wheels for collected packages: cffi
    Building wheel for cffi (setup.py): started
    Building wheel for cffi (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /workspace/.venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ned2bb9p/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ned2bb9p/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-fies3lsc
         cwd: /tmp/pip-req-build-ned2bb9p/
    Complete output (36 lines):
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/cffi
    copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-x86_64-3.9
    creating build/temp.linux-x86_64-3.9/c
    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/workspace/.venv/include -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
    c/_cffi_backend.c: In function ‘get_unique_type’:
    c/_cffi_backend.c:4638:20: error: ‘PyObject’ {aka ‘struct _object’} has no member named ‘ob_refcnt’
     4638 |     ((PyObject *)x)->ob_refcnt--;
          |                    ^~
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
    ERROR: Failed building wheel for cffi
    Running setup.py clean for cffi
  Failed to build cffi
  Installing collected packages: cffi
      Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      ERROR: Command errored out with exit status 1:
       command: /workspace/.venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ned2bb9p/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ned2bb9p/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-e26e6dwz/install-record.txt --single-version-externally-managed --compile --install-headers /workspace/.venv/include/site/python3.9/cffi
           cwd: /tmp/pip-req-build-ned2bb9p/
      Complete output (36 lines):
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.9
      creating build/lib.linux-x86_64-3.9/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-3.9
      creating build/temp.linux-x86_64-3.9/c
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/workspace/.venv/include -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
      c/_cffi_backend.c: In function ‘get_unique_type’:
      c/_cffi_backend.c:4638:20: error: ‘PyObject’ {aka ‘struct _object’} has no member named ‘ob_refcnt’
       4638 |     ((PyObject *)x)->ob_refcnt--;
            |                    ^~
      error: command 'gcc' failed with exit status 1
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /workspace/.venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-ned2bb9p/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-ned2bb9p/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-e26e6dwz/install-record.txt --single-version-externally-managed --compile --install-headers /workspace/.venv/include/site/python3.9/cffi Check the logs for full command output.
  WARNING: You are using pip version 21.2.4; however, version 21.3 is available.
  You should consider upgrading via the '/workspace/.venv/bin/python -m pip install --upgrade pip' command.
  

  at /usr/local/lib/python3.9/site-packages/poetry/utils/env.py:1183 in _run
      1179│                 output = subprocess.check_output(
      1180│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1181│                 )
      1182│         except CalledProcessError as e:
    → 1183│             raise EnvCommandError(e, input=input_)
      1184│ 
      1185│         return decode(output)
      1186│ 
      1187│     def execute(self, bin, *args, **kwargs):

System Details:

OS: Ubuntu 20.04.5 (Linux) CPU: 11th Gen Intel® Core™ i7-11850H @ 2.50GHz × 16

I'd tried to install onnxruntime in nogil-3.9.10

Am I need to build ONNXRuntime from the source?

Minimal reproducer:

$ pip install viztracer numpy
$ python -m viztracer -c "import numpy"
Saving trace data, this could take a whileSegmentation fault (core dumped)

Here is the gdb backtrace:

Saving trace data, this could take a while                                      
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
__strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:77
77	../sysdeps/x86_64/multiarch/strlen-evex.S: No such file or directory.
(gdb) bt
#0  __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:77
#1  0x00007ffff7d23ac9 in __GI__IO_fputs (str=0x0, [email protected]=0x5555559b03c0) at iofputs.c:33
#2  0x00007ffff6ec6191 in fprintfeename ([email protected]=0x5555559b03c0, [email protected]=0x4be89f6bcf0) at src/viztracer/modules/eventnode.c:163
#3  0x00007ffff6ec7b32 in snaptrace_dump (self=0x4be7d589e20, args=<optimized out>) at src/viztracer/modules/snaptrace.c:925
#4  0x00005555557bc6ef in method_vectorcall_VARARGS (func=<method_descriptor at remote 0x4be7d42c190>, args=0x7fffffffd3e8, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/descrobject.c:316
#5  0x000055555580de52 in _PyEval_Fast (ts=0x5555559b2570, initial_acc=Register(as_int64 = 0), initial_pc=0x5555558d7d2f <func_vector_call> "\t") at Python/ceval.c:737
#6  0x00005555556a8d1f in _PyEval_Eval (pc=<optimized out>, acc=..., tstate=0x5555559b2570) at Python/ceval_meta.c:2831
#7  _PyFunction_Vectorcall (func=<function at remote 0x4be7d7e4390>, stack=0x7fffffffd4a0, nargsf=<optimized out>, kwnames=<optimized out>) at Python/ceval_meta.c:3228
#8  0x00005555555c7d78 in _PyObject_FastCallDictTstate (kwargs=0x0, nargsf=1, args=0x7fffffffd4a0, callable=<function at remote 0x4be7d7e4390>, tstate=0x5555559b2570) at Objects/call.c:118
#9  _PyObject_Call_Prepend ([email protected]=0x5555559b2570, [email protected]=<function at remote 0x4be7d7e4390>, 
    [email protected]=<Finalize(_weakref=<weakref at remote 0x4be7bcb7e50>, _callback=<method at remote 0x4be7bcb7db0>, _args=(), _kwargs={}, _key=(-1, 0), _pid=379537) at remote 0x4be7bc778d0>, 
    [email protected]=(), [email protected]=0x0) at Objects/call.c:387
#10 0x0000555555631bec in slot_tp_call (
    self=<Finalize(_weakref=<weakref at remote 0x4be7bcb7e50>, _callback=<method at remote 0x4be7bcb7db0>, _args=(), _kwargs={}, _key=(-1, 0), _pid=379537) at remote 0x4be7bc778d0>, args=(), kwds=0x0)
    at Objects/typeobject.c:6819
#11 0x00005555555c5a6d in _PyObject_MakeTpCall (tstate=0x5555559b2570, 
    callable=<Finalize(_weakref=<weakref at remote 0x4be7bcb7e50>, _callback=<method at remote 0x4be7bcb7db0>, _args=(), _kwargs={}, _key=(-1, 0), _pid=379537) at remote 0x4be7bc778d0>, args=<optimized out>, 
    nargs=0, keywords=0x0) at Objects/call.c:191
#12 0x00005555556a4beb in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffffffd5b8, 
    callable=<Finalize(_weakref=<weakref at remote 0x4be7bcb7e50>, _callback=<method at remote 0x4be7bcb7db0>, _args=(), _kwargs={}, _key=(-1, 0), _pid=379537) at remote 0x4be7bc778d0>, tstate=0x5555559b2570)
    at ./Include/cpython/abstract.h:116
#13 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x7fffffffd5b8, 
    callable=<Finalize(_weakref=<weakref at remote 0x4be7bcb7e50>, _callback=<method at remote 0x4be7bcb7db0>, _args=(), _kwargs={}, _key=(-1, 0), _pid=379537) at remote 0x4be7bc778d0>, tstate=0x5555559b2570)
    at ./Include/cpython/abstract.h:103
#14 vm_call_function (ts=0x5555559b2570, acc=...) at Python/ceval_meta.c:1341
#15 0x00005555558111ab in _PyEval_Fast (ts=0x5555559b2570, initial_acc=Register(as_int64 = 0), initial_pc=0x4be7d7a69a1 "F\f") at Python/ceval.c:982
#16 0x00005555556a8d1f in _PyEval_Eval (pc=<optimized out>, acc=..., tstate=0x5555559b2570) at Python/ceval_meta.c:2831
#17 _PyFunction_Vectorcall (func=<function at remote 0x4be7d7e4f70>, stack=0x5555559542f0 <_Py_EmptyTupleStruct+48>, nargsf=<optimized out>, kwnames=<optimized out>) at Python/ceval_meta.c:3228
#18 0x000055555576a272 in atexit_callfuncs (module=<optimized out>) at ./Modules/atexitmodule.c:93
#19 0x000055555576a46d in atexit_run_exitfuncs (self=<optimized out>, unused=<optimized out>) at ./Modules/atexitmodule.c:192
#20 0x0000555555810c89 in _PyEval_Fast (ts=0x5555559b2570, initial_acc=Register(as_int64 = 0), initial_pc=0x5555558d9c71 <cfunc_header_noargs> "\n") at Python/ceval.c:769
#21 0x00005555556a7a11 in _PyEval_Eval (pc=<optimized out>, acc=..., tstate=0x5555559b2570) at Python/ceval_meta.c:2831
[...]

Not sure if this is a bug in viztracer or in nogil CPython.

Hi, I have stepped into the problem while trying to use multiple pandas.HDFStore() objects in the same interpreter but in separate nogil threads. The use case is pretty simple: each thread operates on its own file. The minimum reproduction example is as follows:

from threading import Thread
from multiprocessing import Process
import pandas as pd
import time
import numpy as np

def f1(n: str, t: float):
    fObj = pd.HDFStore(n)
    fObj.append("a",
                pd.DataFrame({"id" : 1, "time": np.zeros(int(10 * t))}),
                data_columns=["id"])
    time.sleep(t)

    fObj.close()

def main():
    t1 = Thread(target=f1, args=("f1.hdf", 1.0))
    t2 = Thread(target=f1, args=("f2.hdf", 2.0))
    
    # t1 = Process(target=f1, args=("f1.hdf", 1.0))
    # t2 = Process(target=f1, args=("f2.hdf", 2.0))

    t1.start()
    t2.start()

    t1.join()
    t2.join()

if __name__ == "__main__":
    main()

I tested that on Windows and Ubuntu 22.04. On Linux it always segfaults while on Windows the script may fail silently or end up with whole range of different exceptions/outcomes:

childCID = self._g_get_lchild_attr(childname, 'CLASS')
  File "tables\hdf5extension.pyx", line 1072, in tables.hdf5extension.Group._g_get_lchild_attr
tables.exceptions.HDF5ExtError: HDF5 error back trace

  File "..\..\src\H5D.c", line 369, in H5Dopen2
    can't open dataset
  File "..\..\src\H5Dint.c", line 1156, in H5D_open
    not found
  File "..\..\src\H5Dint.c", line 1264, in H5D__open_oid
    can't get layout/pline/efl info
  File "..\..\src\H5Dlayout.c", line 436, in H5D__layout_oh_read
    can't initialize chunk cache
  File "..\..\src\H5Dchunk.c", line 608, in H5D__chunk_init
    can't find object for fapl ID

End of HDF5 error back trace

or

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\tables-3.6.1-py3.9-win-amd64.egg\tables\attributeset.py", line 248, in __init__
    self.__getattr__(attr)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python39\lib\site-packages\tables-3.6.1-py3.9-win-amd64.egg\tables\attributeset.py", line 308, in __getattr__
    value = self._g_getattr(self._v_node, name)
  File "tables\hdf5extension.pyx", line 756, in tables.hdf5extension.AttributeSet._g_getattr
tables.exceptions.HDF5ExtError: Can't get type info on attribute PYTABLES_FORMAT_VERSION in node /.

Or one of the files might end up filled with data or even both of them are empty.

It looks to me like a C API issue of HDF package.

As cross-check I tested the script above in different interpreter & threading/MP configurations:

• (nogil) python3.9 + threading -> fails
• (nogil) python3.9 + MP -> passes
• python3.10 + threading -> passes
• python3.10 + MP -> passes

On Ubuntu I tested this with such a set of packages:

autopep8          2.0.0
bitarray          2.6.0
crccheck          1.3.0
dataclasses-json  0.5.7
marshmallow       3.19.0
marshmallow-enum  1.5.1
mypy-extensions   0.4.3
numexpr           2.8.4
numpy             1.22.3
packaging         21.3
pandas            1.3.3
pip               21.3.1
psutil            5.9.4
pycodestyle       2.10.0
pyparsing         3.0.9
python-dateutil   2.8.2
python-vxi11      0.9
pytz              2022.6
PyVISA            1.12.0
PyYAML            6.0
setuptools        58.1.0
six               1.16.0
tables            3.7.0
tomli             2.0.1
typing_extensions 4.4.0
typing-inspect    0.8.0

While on Windows:

bitarray          2.6.0
cffi              1.14.6
crccheck          1.3.0
Cython            0.29.29
dataclasses-json  0.5.7
Jinja2            3.1.2
MarkupSafe        2.1.1
marshmallow       3.18.0
marshmallow-enum  1.5.1
mock              4.0.3
mypy-extensions   0.4.3
numexpr           2.8.0
numpy             1.19.4
packaging         21.3
pandas            1.3.1+1.gafcbd716a9
pdoc              12.2.0
pip               21.3.1
psutil            5.9.3
py-cpuinfo        9.0.0
pybind11          2.6.2
pycparser         2.21
Pygments          2.13.0
pyparsing         3.0.9
python-dateutil   2.8.2
python-vxi11      0.9
pytz              2022.5
PyVISA            1.12.0
PyYAML            5.3.1
pyzmq             24.0.1
setuptools        58.1.0
six               1.16.0
sniffio           1.3.0
tables            3.6.1 AND ALSO 3.5.2 (3.7.0 -> yet unsolved runtime error on package import)
typing_extensions 4.4.0
typing-inspect    0.8.0
wheel             0.37.1

It would be awesome if you (@colesbury) could look into that.

When I try to use nogil with your fab.py on my mac pro M1，I found nogil could not make use m1 8 cores. And it seems like it dose not support arm chip as good as x86. when I use 1 thread run fab.py with nogil, it use 8 seconds, but I use 1 thread run fab.py with CPython, it just use 1 seconds. I also run this fab.py script on my intel x86 server, its performance is so surprised as your tips.

printf Py_GetPythonHome : /usr/local/nogil-py/

printf Py_GetPythonPath : /usr/local/nogil-py/lib/python39.zip:/usr/local/nogil-py/lib:/usr/local/nogil-py/lib/python3.9/site-packages:/usr/local/nogil-py/lib/python3.9:/usr/local/nogil-py/lib/python3.9/lib-dynload:/usr/local/nogil-py/lib/python39.zip:/usr/local/nogil-py/lib/python3.9:/usr/local/nogil-py/lib/python3.9/lib-dynload

Traceback (most recent call last): File "", line 1, in File "/usr/local/nogil-py/lib/python3.9/site-packages/MNN/init.py", line 4, in from _mnncengine import * ImportError: /usr/local/nogil-py/lib/python3.9/site-packages/_mnncengine.nogil-39b-x86_64-linux-gnu.so: undefined symbol: _Py_TrueStruct