The original motivation for this PR was a performance optimization. See the original PR description below.

After nearly 3 years, work on this PR has led to a few related but distinct behavior changes:

• Already merged PR #3971 — Fixes formatting of messages for originally unnormalized Python errors.
• Already merged PR #3982

This PR:

• Secondary errors during normalization of the Python error are clearly reported (see test_flaky_exception_failure_point_init in test_exceptions.py).
• Secondary errors obtaining the error message are clearly reported (see test_flaky_exception_failure_point_str in test_exceptions.py).
• Calling error_already_set::restore() multiple times is clearly reported (preempts accidentally clearing an error, or accidentally restoring the same error twice; see test_error_already_set_double_restore in test_exceptions.py).
• The error_already_set copy constructor is no longer racy.
• detail::error_string() no longer restores the error message. This fixes (in practice inconsequential) bugs in detail/class.h.

The performance gain for exceptions with long tracebacks can be very substantial. See https://github.com/pybind/pybind11/pull/1895#issuecomment-1127700599 for concrete numbers.

A review of the error_already_set development history is under https://github.com/pybind/pybind11/pull/1895#issuecomment-1134206370.

Original PR description below

Prior to this PR throwing error_already_set was expensive due to the eager construction of the error string. The PR changes error_already_set::what to construct the error string on first call.

Before:

1.7 µs ± 9.03 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

After:

1.14 µs ± 1.93 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

Benchmark:

const py::int_ foo() {
    py::dict d;
    try {
        return d["foo"];
    } catch (const py::error_already_set &) {
        return py::int_(42);
    }    
}

PYBIND11_MODULE(foo, m) {
  m.def("foo", &foo);
}

Note that the running time of the benchmark is dominated by stack unwinding, therefore the speedup is only ~30%.

Suggested changelog entry:

``error_already_set`` is now safer and more performant, especially for exceptions with long tracebacks.

Description

Adds Eigen::Tensor support to eigen.h.

Eigen::Tensor allows you to handle np.arrays with more than two dimensions, which is quite handy in many cases.

I also added tests for all the relevant behavior.

Note that the Eigen::TensorMap implementation could be better if we allow std::optional, but I'm not sure if we are allowed to require that in this codebase.

Closes #1377 Closes #2119

Suggested changelog entry:

pybind11/eigen/tensor.h`` adds converters to and from `Eigen::Tensor` and `Eigen::TensorMap`

Description

This PR adds a PYBIND11_SIMPLE_GIL_MANAGEMENT option (cmake, C++ define). The new option may be useful to try when debugging GIL-related issues, to determine if the more complex default implementation is or is not to blame. See comments here for background.

This PR was triggered by https://github.com/pytorch/pytorch/issues/83101. While we could not reproduce the original problem in a unit test, many tests were added to test_gil_scoped.py trying to do so. That work exposed that test_gil_scoped.py has been finding deadlocks for a long time (years), but those were thus far ignored as "flakes". test_gil_scoped.py was updated to make it crystal clear when a DEADLOCK is detected. It turns out deadlocks are detected at a fairly high rate (ballpark 1 in 1000). This needs to be worked on separately. To not continue to pollute the CI results in the meantime, SKIP_IF_DEADLOCK = True was added in test_gil_scoped.py. The GitHub Actions CI logs can be harvested in the future to determine what platforms exactly are affected, and at what rate. Hopefully the information accumulating over time will lead to a better understanding and fixes, so that SKIP_IF_DEADLOCK can be removed eventually.

This PR also adds ThreadSanitizer compatibility to test_gil_scoped.py (closes #2754).

WARNING: Please be careful to not create ODR violations when using the new option: everything that is linked together with mutual symbol visibility needs to be rebuilt.

Suggested changelog entry:

* A `PYBIND11_SIMPLE_GIL_MANAGEMENT` option was added (cmake, C++ define), along with many additional tests in test_gil_scoped.py. The option may be useful to try when debugging GIL-related issues, to determine if the more complex default implementation is or is not to blame. See #4216 for background. WARNING: Please be careful to not create ODR violations when using the option: everything that is linked together with mutual symbol visibility needs to be rebuilt.

In this PR:

• Support NVidia HPC SDK with PG compilers
• Factor out CMake filter function
• Allow CMake to build even if pytest is missing (still shows warning)

Closes #2214 and closes #2407.

This commit includes a few minor modifications to pybind11 that are needed to get simple hello-world style functions to compile and run on the latest PyPy. Types are now supported as well.

The test suite compiles but crashes when executed.

Adds C++17 support.

• ICC can’t handle complex enable if’s inline, like enable_if_t<all_of<is_positional<Args>...>::value>>, but if it is refactored into a separate constexpr function, it works.
  • But MSVC doesn't like it if the function is in the return type.
• Fold expressions cause an internal compiler error, so we have to avoid turning them on even in C++17 mode
• Plain noexcept can cause weird errors with following syntax, noexcept(true) works fine, workaround in tests

This also fixes the NVIDIA HPC SDK compiler in C++17 mode.

Description

The Intel compiler has trouble with pybind11 when compiling in C++17 mode. The first two examples in the code below result in errors like "no instance of function template [...] matches the argument list", while the third one triggers an internal compiler error.

#include <pybind11/pybind11.h>

void test1() {
  pybind11::print("Hello World");
}

void test2(pybind11::object obj) {
   obj();
}

class C {
};

void test3(pybind11::module_ &m) {
	pybind11::class_<C>(m, "C");
}

Details can be found in #2707. @tobiasleibner implemented the fix for the template mismatch, while the workaround for the internal compiler error is mine. Part of @tobiasleibner's work went into #2573 already, which hasn't been merged yet. Since he currently doesn't have time, he suggested I post a pull request.

Related to #2573 Fixes #2714 Fixes #2707

Suggested changelog entry:

* Support ICC and NVIDIA HPC SDK in C++17 mode

Problem:

1. In a given installation PREFIX, pip install pybind11 or conda install pybind11 will install pybind11 headers into a new pybind11 subdirectory of the python include directory, while cmake + make install results in pybind11 headers being put into PREFIX/include/pybind11.

   For example, with a conda installation, pip will give

   PREFIX/include/pythonX.Y/pybind11 while cmake yields PREFIX/include/pybind11.

2. It would be nice to have a consistent installation scheme between the two methods, plus the benefit of having the cmake project config files being made available, in the context of conda.

Solution:

This changes the destination of the headers of pybind into PYTHON_INCLUDE_DIRS/pybind11.

The generated project config cmake files are installed in the same location as before, but reflect the new location of the headers.

Add testing for Intel icc/icpc via the oneAPI images. Intel oneAPI is in a late beta stage, currently shipping oneAPI beta09 with ICC 20.2.

We are adding // [workaround(intel)] near the workarounds to it will be easy to search them out later, and disable them to see if they have been fixed by the Intel compiler team.

Here are the workarounds here:

• py::args() -> py::args{}, in tests, regression in ICC 20+, workaround in tests
• = default instead of {} doesn’t always work (was a recent modernization, so might have been a problem before too), workaround in tests

Suggested changelog entry:

* Support Intel OneAPI compiler (ICC 20.2) and add to CI.

This is an early snapshot of a potential way to support multiple inheritance in a minimally intrusive way. Standard single inheritance is supported using a fast fallback that matches the current implementation.

Somewhat surprisingly, the change and related simplifications actually reduce the size of generated binaries by 4.4% :).

There are probably a ton of cases that I didn't think of, so this is up for public review now.

CMake current fails to run with Clang on linux:

$ CC=clang CXX=clang++ cmake ..
... (various configuration) ...
-- Configuring done
CMake Error at tools/pybind11Tools.cmake:135 (target_compile_options):
  Error evaluating generator expression:

    $<:-flto>

  Expression did not evaluate to a known generator expression
Call Stack (most recent call first):
  tests/CMakeLists.txt:68 (pybind11_add_module)

This commit fixes the generator expressions when the lto flags are not found (they will be empty strings, but the generator expression needs a 0).

Motivation

Acquiring the pybind11 repo (for large projects where the pybind11_add_module isn't appropriate) is admittedly not hard either as a git submodule or as a CMake ExternalProject plus writing a Findpybind11.cmake file to search out the header location. But for the ExternalProject case, the better CMake route is to provide a pybind11Config.cmake in the installation so that consuming projects can use pre-built (installed) pybind11 without writing their own detection.

PR Changes

• [x] Builds a minimal interface library target (pb11) and exports it so that pybind11Config.cmake and pybind11Targets.cmake are installed along with the headers. Instructions for detection and use given in tools/pybind11Config.cmake.in
• [x] Allows passing -DCMAKE_INSTALL_INCLUDEDIR to specify non-standard install location. Defaults to include. Note that this isn't affecting the pybind11_add_module function since it's not exporting interface include dirs.

Questions

• [x] Used pybind11::pybind11 for imported target name. If you want any other capitalization pattern for the project name, let me know and I'll adjust everything.
• [x] I've hard-coded the version at 1.8, but the write_basic_package_version_file(...) four lines and the ConfigVersion.cmake installation can be commented out, if you don't want to deal with versioning in development mode.
• I'm medium proficient at CMake, so please let me know if you see any holes in the procedure, would like some edits, or want the PR split.

CMake Version

• Checked that GNUInstallDirs and CMakePackageConfigHelpers present at 2.8.12 (https://gitlab.kitware.com/cmake/cmake/blob/v2.8.12/Modules/)
• Checked that the $<INSTALL_INTERFACE generator present at 2.8.12 (https://cmake.org/cmake/help/v2.8.12/cmake.html#section_Generators)
• Couldn't find a 2.8.12 to install, but can confirm the exported target exports at 3.0.1

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

master

Problem description

[[email protected] spack-src]# nox -s tests -- -DCMAKE_BUILD_TYPE=Debug -DDOWNLOAD_EIGEN=OFF
……
____________________________________________________________________________ test_run_in_process_direct[_intentional_deadlock] ____________________________________________________________________________

test_fn = <function _intentional_deadlock at 0xffff79cf59d0>

    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    def test_run_in_process_direct(test_fn):
        """Makes sure there is no GIL deadlock when using processes.

        This test is for completion, but it was never an issue.
        """
>       assert _run_in_process(test_fn) == 0

test_fn    = <function _intentional_deadlock at 0xffff79cf59d0>

../../../../tests/test_gil_scoped.py:244:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../../tests/test_gil_scoped.py:161: in _run_in_process
    process.start()
        args       = ()
        kwargs     = {}
        process    = <Process name='Process-72' parent=3362301 initial daemon>
        t_start    = 1672728059.3542063
        target     = <function _intentional_deadlock at 0xffff79cf59d0>
        test_fn    = <function _intentional_deadlock at 0xffff79cf59d0>
        timeout    = 0.1
/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
        self       = <Process name='Process-72' parent=3362301 initial daemon>
/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
        process_obj = <Process name='Process-72' parent=3362301 initial daemon>
/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/context.py:291: in _Popen
    return Popen(process_obj)
        Popen      = <class 'multiprocessing.popen_forkserver.Popen'>
        process_obj = <Process name='Process-72' parent=3362301 initial daemon>
/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/popen_forkserver.py:35: in __init__
    super().__init__(process_obj)
        __class__  = <class 'multiprocessing.popen_forkserver.Popen'>
        process_obj = <Process name='Process-72' parent=3362301 initial daemon>
        self       = <multiprocessing.popen_forkserver.Popen object at 0xffff79213eb0>
/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
        process_obj = <Process name='Process-72' parent=3362301 initial daemon>
        self       = <multiprocessing.popen_forkserver.Popen object at 0xffff79213eb0>
/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/popen_forkserver.py:51: in _launch
    self.sentinel, w = forkserver.connect_to_new_process(self._fds)
        buf        = <_io.BytesIO object at 0xffffa0695220>
        prep_data  = {'authkey': b'\x84\x94\x02$\xf7\x99\xaf<:tl\xccor%\t\xf1_&\xc5\xf7W\xe0\xab\xcf5\x82C>\xc1\x8a\xfd', 'dir': '/home/sta...r5rgfkal5y2/spack-src/.nox/tests-3-8/tmp/tests', 'init_main_from_name': 'pytest.__main__', 'log_to_stderr': False, ...}
        process_obj = <Process name='Process-72' parent=3362301 initial daemon>
        self       = <multiprocessing.popen_forkserver.Popen object at 0xffff79213eb0>
/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/forkserver.py:84: in connect_to_new_process
    self.ensure_running()
        fds        = []
        self       = <multiprocessing.forkserver.ForkServer object at 0xffff7995fd60>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <multiprocessing.forkserver.ForkServer object at 0xffff7995fd60>

    def ensure_running(self):
        '''Make sure that a fork server is running.

        This can be called from any process.  Note that usually a child
        process will just reuse the forkserver started by its parent, so
        ensure_running() will do nothing.
        '''
        with self._lock:
            resource_tracker.ensure_running()
            if self._forkserver_pid is not None:
                # forkserver was launched before, is it still running?
                pid, status = os.waitpid(self._forkserver_pid, os.WNOHANG)
                if not pid:
                    # still alive
                    return
                # dead, launch it again
                os.close(self._forkserver_alive_fd)
                self._forkserver_address = None
                self._forkserver_alive_fd = None
                self._forkserver_pid = None

            cmd = ('from multiprocessing.forkserver import main; ' +
                   'main(%d, %d, %r, **%r)')

            if self._preload_modules:
                desired_keys = {'main_path', 'sys_path'}
                data = spawn.get_preparation_data('ignore')
                data = {x: y for x, y in data.items() if x in desired_keys}
            else:
                data = {}

            with socket.socket(socket.AF_UNIX) as listener:
                address = connection.arbitrary_address('AF_UNIX')
>               listener.bind(address)
E               OSError: AF_UNIX path too long

address    = '/home/stage/root/spack-stage-py-pybind11-master-buhdtf3eegctgi7bjpxzrr5rgfkal5y2/spack-src/.nox/tests-3-8/tmp/pymp-2y6zv958/listener-xwlet19c'
cmd        = 'from multiprocessing.forkserver import main; main(%d, %d, %r, **%r)'
data       = {'sys_path': ['/home/stage/root/spack-stage-py-pybind11-master-buhdtf3eegctgi7bjpxzrr5rgfkal5y2/spack-src/tests', '/ho.../spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python38.zip', ...]}
desired_keys = {'main_path', 'sys_path'}
listener   = <socket.socket [closed] fd=-1, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
self       = <multiprocessing.forkserver.ForkServer object at 0xffff7995fd60>

/home/spack/opt/spack/linux-centos8-aarch64/gcc-8.5.0/python-3.8.8-m3k3pvne56yo7wyd7a3bwf7mugupszik/lib/python3.8/multiprocessing/forkserver.py:138: OSError
========================================================================================= short test summary info =========================================================================================
SKIPPED [1] ../../../../tests/test_builtin_casters.py:145: no <string_view>
SKIPPED [1] ../../../../tests/test_callbacks.py:203: Current PYBIND11_INTERNALS_VERSION too low
SKIPPED [1] ../../../../tests/test_copy_move.py:77: no <optional>
SKIPPED [1] ../../../../tests/test_sequences_and_iterators.py:13: no <optional>
SKIPPED [1] ../../../../tests/test_stl.py:111: no <optional>
SKIPPED [1] ../../../../tests/test_stl.py:143: no <experimental/optional>
SKIPPED [1] ../../../../tests/test_stl.py:175: no <boost/optional>
SKIPPED [1] ../../../../tests/test_stl.py:234: no <filesystem>
SKIPPED [1] ../../../../tests/test_stl.py:253: no <variant>
SKIPPED [1] ../../../../tests/test_stl.py:270: no std::monostate
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_callback_py_obj] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_callback_std_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_callback_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_callback_pure_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_inner_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_inner_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_inner_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_inner_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_nested_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_nested_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_nested_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_cross_module_gil_nested_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_release_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_nested_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[test_multi_acquire_release_cross_module] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_one_thread[_intentional_deadlock] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_callback_py_obj] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_callback_std_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_callback_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_callback_pure_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_inner_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_inner_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_inner_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_inner_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_nested_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_nested_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_nested_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_cross_module_gil_nested_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_release_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_nested_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[test_multi_acquire_release_cross_module] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_parallel[_intentional_deadlock] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_callback_py_obj] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_callback_std_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_callback_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_callback_pure_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_inner_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_nested_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_nested_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_nested_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_cross_module_gil_nested_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_release_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_nested_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_multi_acquire_release_cross_module] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[_intentional_deadlock] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_callback_py_obj] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_callback_std_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_callback_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_callback_pure_virtual_func] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_inner_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_inner_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_inner_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_inner_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_nested_custom_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_nested_custom_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_nested_pybind11_released] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_cross_module_gil_nested_pybind11_acquired] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_release_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_nested_acquire] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[test_multi_acquire_release_cross_module] - OSError: AF_UNIX path too long
FAILED ../../../../tests/test_gil_scoped.py::test_run_in_process_direct[_intentional_deadlock] - OSError: AF_UNIX path too long
=============================================================================== 72 failed, 711 passed, 10 skipped in 28.18s ===============================================================================
gmake[3]: *** [tests/CMakeFiles/pytest.dir/build.make:74: tests/CMakeFiles/pytest] Error 1
gmake[2]: *** [CMakeFiles/Makefile2:385: tests/CMakeFiles/pytest.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:450: tests/CMakeFiles/check.dir/rule] Error 2
gmake: *** [Makefile:274: check] Error 2
nox > Command cmake --build .nox/tests-3-8/tmp --config=Release --target check failed with exit code 2
nox > Session tests-3.8 failed.
nox > Running session tests-3.9
nox > Missing interpreters will error by default on CI systems.
nox > Session tests-3.9 skipped: Python interpreter 3.9 not found.
nox > Running session tests-3.10
nox > Missing interpreters will error by default on CI systems.
nox > Session tests-3.10 skipped: Python interpreter 3.10 not found.
nox > Running session tests-3.11
nox > Missing interpreters will error by default on CI systems.
nox > Session tests-3.11 skipped: Python interpreter 3.11 not found.
nox > Running session tests-pypy3.7
nox > Missing interpreters will error by default on CI systems.
nox > Session tests-pypy3.7 skipped: Python interpreter pypy3.7 not found.
nox > Running session tests-pypy3.8
nox > Missing interpreters will error by default on CI systems.
nox > Session tests-pypy3.8 skipped: Python interpreter pypy3.8 not found.
nox > Running session tests-pypy3.9
nox > Missing interpreters will error by default on CI systems.
nox > Session tests-pypy3.9 skipped: Python interpreter pypy3.9 not found.
nox > Ran multiple sessions:
nox > * tests-3.8: failed
nox > * tests-3.9: skipped
nox > * tests-3.10: skipped
nox > * tests-3.11: skipped
nox > * tests-pypy3.7: skipped
nox > * tests-pypy3.8: skipped
nox > * tests-pypy3.9: skipped

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

Not a regression

The development team requested (https://github.com/scipy/scipy/issues/17644#issuecomment-1365537213) a reproducer in the CI for gh-4420.

• first commit cuts down the CI to prevent wasting
• second commit formalizes some of my reproducing instructions so you can see the failure with the latest release of pybind11 in CI (if you switch down to the 2.10.1 release there you should see the job pass)
• I realise you might prefer to build master from source to debug rather than me pip installing the latest stable release in the repo proper, but obviously this team should be able to alter this workflow to do that if desired..

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.0 (vcpkg)

Problem description

I'm producing a module that uses the date library On c++ initialization (aka on import my_module) the module (and python) hangs since the library is trying to access some data using curl. Windows is blocked in "waitforsingleobject" and linux in "__GI___select", same behaviour. If I disable the auto-check of tzdata the module works normally.

Anuy idea why python/pybind11 interacts badly with curl on init ?

Reproducible example code

I can submit a minimal example if needed.

Is this a regression? Put the last known working version here if it is.

Not a regression

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.2

Problem description

The docs state that passing arguments as reference and pointer should work, but actually, while pointers works well, references dont. They end up beeing copied, which is a blocker if the type is not copy constructible.

class PyPlugin
{
public:
    virtual void takeThisAndModifyR(StandardItem &item) {};
    virtual void takeThisAndModifyP(StandardItem *item) {};
};
class PyPluginTrampoline : PyPlugin
{
public:
    using PyPlugin::PyPlugin;  // Inherit the constructors
    void takeThisAndModifyR(StandardItem &item) override  { PYBIND11_OVERRIDE(void, PyPlugin, takeThisAndModifyR, item); }
    void takeThisAndModifyP(StandardItem *item) override { PYBIND11_OVERRIDE(void, PyPlugin, takeThisAndModifyP, item); }
};

Then in a pyhton plugin I do:

class Plugin(Plugin):
  …
    def takeThisAndModifyR(self, item):
        item.id = "takeThisAndModifyR";

    def takeThisAndModifyP(self, item):
        item.id = "takeThisAndModifyP";
…

In the app i do

auto item = StandardItem("id");
INFO << item.id();
p->takeThisAndModifyP(&item);
INFO << item.id();
p->takeThisAndModifyR(item);
INFO << item.id();

Finally this prints:

10:47:43 [info:python] id
10:47:43 [info:python] takeThisAndModifyP
10:47:43 [warn:albert] [17ms] Failed loading 'find': return_value_policy = copy, but type albert::StandardItem is non-copyable!

Of course I exposed the item class. Left it out for readability. Obviously pybind does not like references. Is this a bug or a known limitation and I should rather look for a workaround?

Related:

• https://github.com/pybind/pybind11/discussions/3991
• https://stackoverflow.com/questions/72500615/how-to-pass-arguments-by-reference-when-call-python-in-c-with-pybind11
• https://pybind11.readthedocs.io/en/stable/faq.html#limitations-involving-reference-arguments
• https://github.com/pybind/pybind11/issues/1123

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

Not a regression

Required prerequisites

• [X] Make sure you've read the documentation. Your issue may be addressed there.
• [X] Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• [X] Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.2

Problem description

__________________________________________________ test_run_in_process_multiple_threads_sequential[test_multi_acquire_release_cross_module] __________________________________________________

test_fn = <function test_multi_acquire_release_cross_module at 0x8cbf75280>

    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    def test_run_in_process_multiple_threads_sequential(test_fn):
        """Makes sure there is no GIL deadlock when running in a thread multiple times sequentially.
    
        It runs in a separate process to be able to stop and assert if it deadlocks.
        """
>       assert _run_in_process(_run_in_threads, test_fn, num_threads=8, parallel=False) == 0

test_fn    = <function test_multi_acquire_release_cross_module at 0x8cbf75280>

../../pybind11-2.10.2/tests/test_gil_scoped.py:234: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

target = <function _run_in_threads at 0x8cbf754c0>, args = (<function test_multi_acquire_release_cross_module at 0x8cbf75280>,), kwargs = {'num_threads': 8, 'parallel': False}
test_fn = <function test_multi_acquire_release_cross_module at 0x8cbf75280>, timeout = 10, process = <Process name='Process-53' pid=32667 parent=30786 stopped exitcode=-SIGTERM daemon>
t_start = 1671766594.3010159, t_delta = 10.004453420639038, @py_assert1 = None

    def _run_in_process(target, *args, **kwargs):
        if len(args) == 0:
            test_fn = target
        else:
            test_fn = args[0]
        # Do not need to wait much, 10s should be more than enough.
        timeout = 0.1 if test_fn is _intentional_deadlock else 10
        process = multiprocessing.Process(target=target, args=args, kwargs=kwargs)
        process.daemon = True
        try:
            t_start = time.time()
            process.start()
            if timeout >= 100:  # For debugging.
                print(
                    "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs)
                )
                print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True)
            process.join(timeout=timeout)
            if timeout >= 100:
                print("\nprocess.pid JOINED", process.pid, flush=True)
            t_delta = time.time() - t_start
            if process.exitcode == 66 and m.defined_THREAD_SANITIZER:  # Issue #2754
                # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output.
                # Maybe this could work:
                # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e
                pytest.skip(
                    "ThreadSanitizer: starting new threads after multi-threaded fork is not supported."
                )
            elif test_fn is _intentional_deadlock:
                assert process.exitcode is None
                return 0
            elif process.exitcode is None:
                assert t_delta > 0.9 * timeout
                msg = "DEADLOCK, most likely, exactly what this test is meant to detect."
                if env.PYPY and env.WIN:
                    pytest.skip(msg)
>               raise RuntimeError(msg)
E               RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.

args       = (<function test_multi_acquire_release_cross_module at 0x8cbf75280>,)
kwargs     = {'num_threads': 8, 'parallel': False}
msg        = 'DEADLOCK, most likely, exactly what this test is meant to detect.'
process    = <Process name='Process-53' pid=32667 parent=30786 stopped exitcode=-SIGTERM daemon>
t_delta    = 10.004453420639038
t_start    = 1671766594.3010159
target     = <function _run_in_threads at 0x8cbf754c0>
test_fn    = <function test_multi_acquire_release_cross_module at 0x8cbf75280>
timeout    = 10

../../pybind11-2.10.2/tests/test_gil_scoped.py:186: RuntimeError
================================================================================== short test summary info ===================================================================================
SKIPPED [1] ../../pybind11-2.10.2/tests/test_builtin_casters.py:145: no <string_view>
SKIPPED [1] ../../pybind11-2.10.2/tests/test_callbacks.py:203: Current PYBIND11_INTERNALS_VERSION too low
SKIPPED [1] ../../pybind11-2.10.2/tests/test_copy_move.py:77: no <optional>
SKIPPED [1] ../../pybind11-2.10.2/tests/test_pytypes.py:397: Not defined: PYBIND11_HANDLE_REF_DEBUG
SKIPPED [1] ../../pybind11-2.10.2/tests/test_sequences_and_iterators.py:13: no <optional>
SKIPPED [1] ../../pybind11-2.10.2/tests/test_stl.py:111: no <optional>
SKIPPED [1] ../../pybind11-2.10.2/tests/test_stl.py:143: no <experimental/optional>
SKIPPED [1] ../../pybind11-2.10.2/tests/test_stl.py:234: no <filesystem>
SKIPPED [1] ../../pybind11-2.10.2/tests/test_stl.py:270: no std::monostate
FAILED ../../pybind11-2.10.2/tests/test_exceptions.py::test_cross_module_exception_translator - RuntimeError
FAILED ../../pybind11-2.10.2/tests/test_gil_scoped.py::test_run_in_process_multiple_threads_sequential[test_multi_acquire_release_cross_module] - RuntimeError: DEADLOCK, most likely, exactly what this test is meant to detect.
==================================================================== 2 failed, 780 passed, 9 skipped in 151.14s (0:02:31) ====================================================================

Reproducible example code

No response

Is this a regression? Put the last known working version here if it is.

Not a regression