A library which implements low-level functions that relate to packaging and distribution of Python

Describe the bug

After the launcher files were updated in 8d2acb51bae32a0237f2d2419b0636bce807b6f7 (0.3.5), processes spawned from the "w" launchers can't be killed anymore. I am facing this issue when trying to spawn and kill the processes from a NodeJS context. Calling childprocess.kill() via NodeJS's child_process module returns success, but the python process keeps running.

This is a major problem for me, because my NodeJS application (GUI application via NW.js / Electron) is running a python CLI application built with the distlib launcher executables, and it needs to use the "w" launchers, because otherwise a console window pops up, which is bad for the user experience.

0.3.3 is the most recent release which is working fine.

0.3.4 had the immediate 3221225477 / 0xC0000005 exit code error, and I already had to ignore this version when building standalone installers of my python application. Fortunately, pip had reverted back to distlib 0.3.3 until recently, so installing via pip was no problem either, but now it has merged the distlib 0.3.5 release, which is once again bad, and the issue needs to be fixed. Related (old 0.3.4 issue): https://github.com/takluyver/pynsist/issues/243#issuecomment-1192054549

Apparently, these are the recent launcher changes: https://bitbucket.org/vinay.sajip/simple_launcher/commits/

To Reproduce

I've created a reproduction repo (already did this for 0.3.4). Please see the build config and build+run scripts for the 0.3.5 issue: https://github.com/bastimeyer/distlib-bug/tree/53f520efec1a6173b7ca037d26c89a6dec8ca633

The resulting GH action logs: https://github.com/bastimeyer/distlib-bug/actions/runs/2718462897

0.3.5-t64 (success): https://github.com/bastimeyer/distlib-bug/runs/7467911880?check_suite_focus=true#step:7:1 0.3.5-w64 (failure): https://github.com/bastimeyer/distlib-bug/runs/7467912062?check_suite_focus=true#step:7:1

0.3.3-t64 (success): https://github.com/bastimeyer/distlib-bug/runs/7467911132?check_suite_focus=true#step:7:1 0.3.3-w64 (success): https://github.com/bastimeyer/distlib-bug/runs/7467911332?check_suite_focus=true#step:7:1

Environment

• OS, including version: Windows (any)
• Version of this library: 0.3.5

Original report by Jason R. Coombs (Bitbucket: jaraco, GitHub: jaraco).

In this pull request for setuptools, anthrotype is working to update Setuptools to use distlib for its script launchers, unifying yet another piece of the python packaging infrastructure.

However, setuptools' executable launchers support one major feature that distlib does not currently, and that's ARM-compatible executables.

Setuptools builds its launches with this script. Can someone update distlib to include builds of the launcher for ARM as well?

Original report by C Anthony Risinger (Bitbucket: anthonyrisinger, GitHub: anthonyrisinger).

the regex:

(\s*#!("[^"]+"|[\w/\\:.]+))\s+(.*)$

...does not reflect all valid shebangs, eg. an @ in the path leads to failure. also, even though .match(...) is used, the regex should be anchored:

^(\s*#!("[^"]+"|[\w/\\:.]+))\s+(.*)$

...anything save \0 and / may be in the path:

^(\s*#!("[^"]+"|\S+))\s+(.*)$

...there are better ways too (such as setuptools method of open() + readline()) but this makes stuff go again.

Original report by Paul Moore (Bitbucket: pmoore, GitHub: pmoore).

See https://github.com/pypa/pip/issues/5223. This is a priority issue, as it means that the wrapper pip.exe executable generated by pip 10 won't work for a Python installation in a directory containing a space (e.g. C:\Program Files)

To reproduce, create a virtualenv with a space in the name. I'm here using the copy of distlib vendored with pip 10 (which is distlib 0.2.6), as that's where the bug was first reported.

The file "a.py" below simply contains

def f():
    print("Hello from", __file__)

To reproduce the issue:

PS 13:30 {19:29.901} E:\Work\Scratch\a b                                                               
>.\Scripts\python.exe                                                                                  
Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)] on win32              
Type "help", "copyright", "credits" or "license" for more information.                                 
>>> import sys                                                                                         
>>> sys.path.append(r'E:\Work\Projects\pip\src\pip\_vendor')                                           
>>> from distlib import scripts                                                                        
>>> s = scripts.ScriptMaker(r'E:\Work\Scratch\a b\x', r'E:\Work\Scratch\a b', add_launchers=True)      
>>> s.make('a=a:f')                                                                                    
['E:\\Work\\Scratch\\a b\\a.exe', 'E:\\Work\\Scratch\\a b\\a-3.6.exe']                                 

This creates a.exe, which when run, gives an error as follows:

>& 'E:\Work\Scratch\a b\a.exe'
Fatal error in launcher: Unable to create process using '""e:\work\scratch\a b\scripts\python.exe"  "E:\Work\Scratch\a b\a.exe" '

Original report by C Anthony Risinger (Bitbucket: anthonyrisinger, GitHub: anthonyrisinger).

results from JSONLocator appear to be unusable sometime within the last 4-6 hrs:

#!python
>>> from distlib.locators import default_locator
>>> # get from JSONLocator only
>>> s = default_locator.locators[0].locate('setuptools')
>>> s.metadata
<Metadata 2.0 setuptools (0.9.1)>
>>> s.metadata.get_requirements()
[...]
  File "/home/anthony/devel/local/zppy/external-library/distlib/database.py", li
ne 349, in run_requires
    return self._get_requirements('run_requires', 'run_may_require')
  File "/home/anthony/devel/local/zppy/external-library/distlib/database.py", li
ne 345, in _get_requirements
    env=self.context))
TypeError: unhashable type: 'dict'

my app is actually using the DependencyFinder, and i know for a fact that earlier today it was properly returning set(, , [...])

it could be only some packages are messed up... i changed my tool to cache JSONLocator metadata so i can load it later when it's local (eg. as a way to endow DirectoryLocator with proper metadata)... however, ATM, i don't seem to be able to query anything without an exception, even when using the defaults:

#!python
from distlib.locators import default_locator, DependencyFinder
DependencyFinder(default_locator).find('gitdb')

...throws the same unhashable type :(

Original report by James Tocknell (Bitbucket: aragilar, GitHub: aragilar).

Currently the distlib.wheel.Wheel only allows for creating a wheel from files or install files from a wheel, not modifying the wheel in place or creating a new wheel wheel from an old one. Use cases for this would be to update the metadata in the wheel from a legacy version to 2.0, or to add dependencies not contained in setup.py (e.g. add numpy dependency to a scipy wheel, which was what I'm trying to do).

A workaround is to install the wheel into some temporary directory, do the required modifications, and then rebuild, or to modify the zip file directly (what I'm doing currently), but this is more likely to create buggy wheels.

Original report by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft).

Is there any reason not to simplify the generated script? Something like.

# -*- coding: utf-8 -*-
if __name__ == '__main__':
    import sys
    from %(module)s import %(func)s
    sys.exit(%(func)s())

I believe would do the exact same thing as you're doing now, with less code.

Describe the bug On Windows using distlib version 0.3.4 and using a bash terminal, when I run an exe file generated by distlib, sys.stderr is set to None. Running the same exe from cmd or powershell works as expected. Generating the exe with distlib 0.3.3 also works as expected.

This was first reported in IPython https://github.com/ipython/ipython/issues/13509. IPython works fine when running python -m IPython or when running ipython.exe from other shells like cmd or powershell. This was later fixed because pip downgraded distlib here but I thought to report it here for visibility.

I'm including a small example that prints sys.stdout and sys.stderr for comparison.

To Reproduce Steps to reproduce the behavior:

1. Download and unzip this file: bugexe.zip
2. Go to that folder and create a venv: python -m venv .venv
3. Activate the venv: source .venv/Scripts/activate
4. Install distlib 0.3.4 python -m pip install distlib==0.3.4
5. Run python make.py to create an executable called foo.exe
6. From bash run: PYTHONPATH=. ./foo.exe It prints this:

sys.stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
sys.stderr=None

7. For comparison, run from cmd: cmd /V /C "set "PYTHONPATH=." && foo.exe"
8. Or powershell: powershell -Command { $env:PYTHONPATH="."; .\foo.exe }

Expected behavior It should print this:

sys.stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
sys.stderr=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>

Environment

• OS: Windows 10
• Using bash version 4.4.23(2)-release (x86_64-pc-msys) that comes bundled with git for windows
• Version of this library: 0.3.4

Additional information This problem doesn't happen with distlib version 0.3.3

Describe the bug

Since CPython 3.9.11 / pip 22.0.4 / distlib 0.3.3, the pip3 script generated at CPython install time has an incorrect shebang line.

To Reproduce Steps to reproduce the behavior:

In a CPython git clone:

1. mkdir /tmp/destdir
2. ./configure --prefix=/usr/local --with-ensurepip=yes
3. make -j
4. DESTDIR=/tmp/destdir make install
5. head /tmp/destdir/usr/local/bin/pip3

Expected behavior

The shebang I see with CPython 3.9.11 / pip 22.0.4 / distlib 0.3.3 is: #!/usr/local/bin/python. This file does not exist.

The shebang I see with CPython 3.9.8 / pip 21.2.4 / distlib 0.3.2 is: #!/usr/local/bin/python3. This file will exist, but it does not at the time the CPython install runs.

Environment

• CentOS 7

Additional information

This issue is a little hard to track down, as it only reproduces during CPython build, where distlib is embedded inside the file Lib/ensurepip/_bundled/pip-22.0.4-py3-none-any.whl.

I suspect that the regression was introduced in this commit: https://github.com/pypa/distlib/commit/ad0ea221431689cb82c89b4942228fb07a87784d

This adds a line to ScriptMaker._get_shebang() which does the following:

        if not os.path.isfile(executable):
            # for Python builds from source on Windows, no Python executables with
            # a version suffix are created, so we use python.exe
            executable = os.path.join(sysconfig.get_config_var('BINDIR'),
                            'python%s' % (sysconfig.get_config_var('EXE')))

In my scenario, executable = /usr/local/bin/python3. This file does not exist, because we ran DESTDIR=/tmp/destdir make install and so the file has been installed to /tmp/destdir/usr/local/bin/python3. However, this is the path that should be used in the script shebang line, and distlib <= 0.3.2 handled the situation correctly. The change introduced in distlib 0.3.3 incorrectly chooses /usr/local/bin/python instead.

The change is subtle, many OS distributors will create a symlink bin/python3 -> bin/python that hides the issue. In my case we do not have such a symlink, thus we get a pip3 executable that fails to start with an error like:

Failed to execute process '/usr/local/bin/pip3'. Reason:
The file '/usr/local/bin/pip3' specified the interpreter '/usr/local/bin/python', which is not an executable command.

Original report by Anonymous.

https://github.com/pypa/wheel/issues/294

They have changed their directory inclusion behavior. Unclear to me if this is a distlib or a wheel bug, but wanted to get a tracking issue here.

Original report by C Anthony Risinger (Bitbucket: anthonyrisinger, GitHub: anthonyrisinger).

For a long time it seemed like pep426 was the way forward, but recently it appears to have fallen out of style (recent build abstraction threads, etc), and even the original author is no longer interested.

We've been using red-dove via distlib for quite a long time, subconsciously knowing this day would come, but alas, we never moved back to pip vanilla.

Would it be possible to re-enable the sync for a while longer? We are currently migrating back to a straight pip build and adjusting our CI appropriately.

    I'll get to it at some point, but it's not a high priority.

Originally posted by @vsajip in https://github.com/pypa/distlib/issues/159#issuecomment-992982760

Hi everybody!

I noticed that, recently, this project moved to the PyPA GitHub organization, and I like to see that!

However, many of our repositories here have renamed its default branch from master to main. For consistency reasons, can we do the same here?

Original report by 林玮 (Jade Lin) (Bitbucket: 林玮 (Jade Lin), ).

The executable path that ScriptMaker got may be a script. For example, the Python executable installed from nix is a bash script like this.

#! /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env'
export NIX_PYTHONEXECUTABLE='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
export NIX_PYTHONPATH='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/lib/python3.9/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/7pjbbmnrch7frgyp7gz19ay0z1173c7y-python3-3.9.2/bin/python3.9"  "$@"

ScriptMaker only validates it and adapts it while running on the Java platform. Maybe refactor to apply the adaption for general cases.

https://bitbucket.org/pypa/distlib/src/4f54a3117db58f6bb00c69ab9a892170b91acd3b/distlib/scripts.py#lines-102:125