A library and tool for generating .pex (Python EXecutable) files

Last update: Jan 01, 2023

Related tags

Distribution pex

Overview

PEX

https://github.com/pantsbuild/pex/workflows/CI/badge.svg?branch=master

Contents

Overview
Installation
Simple Examples
Integrating pex into your workflow
Documentation
Development
Contributing

Overview

pex is a library for generating .pex (Python EXecutable) files which are executable Python environments in the spirit of virtualenvs. pex is an expansion upon the ideas outlined in PEP 441 and makes the deployment of Python applications as simple as cp. pex files may even include multiple platform-specific Python distributions, meaning that a single pex file can be portable across Linux and OS X.

pex files can be built using the pex tool. Build systems such as Pants, Buck, and {py}gradle also support building .pex files directly.

Still unsure about what pex does or how it works? Watch this quick lightning talk: WTF is PEX?.

pex is licensed under the Apache2 license.

Installation

To install pex, simply

$ pip install pex

You can also build pex in a git clone using tox:

$ tox -e package
$ cp dist/pex ~/bin

This builds a pex binary in dist/pex that can be copied onto your $PATH. The advantage to this approach is that it keeps your Python environment as empty as possible and is more in-line with what pex does philosophically.

Simple Examples

Launch an interpreter with requests, flask and psutil in the environment:

$ pex requests flask 'psutil>2,<3'

Or instead freeze your current virtualenv via requirements.txt and execute it anywhere:

$ pex $(pip freeze) -o my_virtualenv.pex
$ deactivate
$ ./my_virtualenv.pex

Run webserver.py in an environment containing flask as a quick way to experiment:

$ pex flask -- webserver.py

Launch Sphinx in an ephemeral pex environment using the Sphinx entry point sphinx:main:

$ pex sphinx -e sphinx:main -- --help

Build a standalone pex binary into pex.pex using the pex console_scripts entry point:

$ pex pex -c pex -o pex.pex

You can also build pex files that use a specific interpreter type:

$ pex pex -c pex --python=pypy -o pypy-pex.pex

Most pex options compose well with one another, so the above commands can be mixed and matched. For a full list of options, just type pex --help.

Integrating pex into your workflow

If you use tox (and you should!), a simple way to integrate pex into your workflow is to add a packaging test environment to your tox.ini:

[testenv:package]
deps = pex
commands = pex . -o dist/app.pex

Then tox -e package will produce a relocateable copy of your application that you can copy to staging or production environments.

Documentation

More documentation about Pex, building .pex files, and how .pex files work is available at https://pex.readthedocs.io.

Development

Pex uses tox for test and development automation. To run the test suite, just invoke tox:

$ tox

If you don't have tox, you can generate a pex of tox:

$ pex tox -c tox -o ~/bin/tox

Tox provides many useful commands and options, explained at https://tox.readthedocs.io/en/latest/. Below, we provide some of the most commonly used commands used when working on Pex, but the docs are worth acquainting yourself with to better understand how Tox works and how to do more advanced commmands.

To run a specific environment, identify the name of the environment you'd like to invoke by running tox --listenvs-all, then invoke like this:

$ tox -e format-run

To run MyPy:

$ tox -e typecheck

All of our tox test environments allow passthrough arguments, which can be helpful to run specific tests:

$ tox -e py37-integration -- -k test_reproducible_build

To run Pex from source, rather than through what is on your PATH, invoke via Python:

$ python -m pex

Contributing

To contribute, follow these instructions: https://www.pantsbuild.org/docs/contributor-overview

Comments

Add manylinux wheel support and fix a few bugs along the way
Adds support for manylinux1_x86_64 and manylinux1_i686 platforms. I've treated them as platforms, because that's how PEP-513 describes them. This fixes #281.

Fixes a number of bugs related to the handling of wheels that have hyphens in the package names (e.g., cassandra-driver and msgpack-python).

Fixes a couple places where str.replace was being called without storing the result (thereby accomplishing nothing since str is immutable)
opened by dan-blanchard 72
Introduce new `--no-compile` flag to not include .pyc in built Pex due to its non-determinism
Problem

Until Python 3.7 via https://www.python.org/dev/peps/pep-0552/, .pyc files were never reproducible because they included the timestamp. See the discussion at https://github.com/pantsbuild/pex/issues/716#issuecomment-487821354.

We are aiming to ensure reproducible builds a la https://github.com/pantsbuild/pex/issues/716 so that we can safely remote PEXes in CI. Reproducible builds also play an important role in security, per https://reproducible-builds.org.

Solution

Allow users to toggle on and off .pyc files via the new --compile / --no-compile flag.

We currently default to including .pyc, because this has been the precedent for 9 years. In a future release, e.g. 1.70, we will change the default to not include .pyc so that builds are reproducible by default, but people can still opt into including the files if they'd like via the flag.

Alternative flag considered: --reproducible

Originally, we were going to use --reproducible to toggle on multiple behaviors like not including .pyc and using a hardcoded timestamp instead of system time. However, after deciding that in a future release we are going to default to reproducible, it doesn't make sense for people to ever call --not-reproducible, but it does make sense to opt out of specific behaviors like wanting to use --include-pyc. So, we use behavior-specific flags instead of a universal flag.

Alternative solution: ensure our own .pyc timestamp

It may be possible to still include .pyc and have their timestamp be deterministic https://github.com/pantsbuild/pex/pull/718#issuecomment-488202220. This is not pursued for now because it is more complex than necessary, especially because currently we do not ship .pyc files if multiple platforms are used in the PEX. But it could be a followup PR if deemed worth it.

Result

When --no-compile is set, PEXes will no longer include .pyc. This results in all of the reproducible build acceptance tests passing the exploded pex portion of their tests, excluding the bdist_wheel test.

Impact on performance

Not using .pyc has a slight speedup of ~0.2 seconds to the creation of simple PEXes.

time python -m pex -o normal.pex --compile averaged around 0.49 seconds

time python -m pex -o repro.pex --no-compile averaged around 0.31 seconds

Not using .pyc has a slight slowdown of ~0.06 seconds to the startup time of simple PEXes.

time ./normal.pex --version averaged around 0.26 seconds

time ./repro.pex --version averaged around 0.32 seconds

python -m pex -m pydoc -o pydoc.pex likewise ran 0.07 seconds slower when not including .pyc.

Note that the inclusion of .pyc only impacts startup time, not actual runtime performance. See https://stackoverflow.com/questions/2998215/if-python-is-interpreted-what-are-pyc-files for what .pyc files do / why they exist.
opened by Eric-Arellano 27
Improve wheel support in pex.

Wheel handling in pex files is broken. The wheel standard says that wheel files are not designed to be importable archives, but pex treats them as if they are. This causes many standard compliant wheels, including things like tensorflow and opencv, to fail to import in pexes (and thus in pants).

This change modifies wheel handling, so that when a wheel is added to a pex, it's installed in an importable form.

opened by MarkChuCarroll 23
pexrc support

Hi!

Big fan of pex. We are starting to face an issue with our deployments where setting custom PEX_ROOT and PEX_MODULE in some entry scripts. I'm not a huge fan of having to write a wrapper for our pex files and I'd rather drop a .pexrc with my deployment so the pex file itself can see these vars (rather than having to explicitly set them prior to invoking pex).

This PR just adds the ability to drop a ~/.pexrc or .pexrc (relative to the pex itself, in that precedence). Variables() reads this file and updates the environment before looking in os.environ, so anything in .pexrc is overwritten by runtime environment variables.

Let me know what you think, thanks for looking!

opened by lorencarvalho 21

Introduce the spread app layout.

Beyond the top level pex and __main__.py files, the layout is an implementation detail, but the current structure sheds some light on the cache-friendly characteristics:

Given loose sources:

$ cat src/main.py
from pex.version import __version__; print(__version__)

A spread layout PEX looks like:

$ python -mpex pex -Dsrc -emain -opex.spread.venv --spread \
  --seed verbose --venv prepend | jq .
{
  "pex_root": "/home/jsirois/.pex",
  "python": "/usr/bin/python3.9",
  "pex": "/home/jsirois/.pex/venvs/d2f743b9c1ebb156f794419c01b3422653cbdb61/2d1d404c3de23b1810386195be7410700b1feb14/pex"
}
$ tree -a pex.spread.venv
pex.spread.venv
├── .bootstrap
├── .deps
│   └── pex-2.1.46-py2.py3-none-any.whl
├── __main__.py -> src/__main__.py
├── pex -> __main__.py
├── PEX-SPREAD-INFO
└── src
    ├── __main__.py
    ├── main.py
    └── PEX-INFO

And the runtime spreading is suggested by the new PEX-SPREAD-INFO manifest:

{
  "sources": [
    "PEX-INFO",
    "__main__.py",
    "main.py"
  ],
  "spreads": [
    {
      "strip_zip_relpath": false,
      "unpack_relpath": "bootstraps/a54b6ae5e64e5b229388fdffc8adac141f3c416b",
      "zip_relpath": ".bootstrap"
    },
    {
      "strip_zip_relpath": true,
      "unpack_relpath": "installed_wheels/f627f0368a0e29be24aa8cadba74044b9ad990d7/pex-2.1.46-py2.py3-none-any.whl",
      "zip_relpath": ".deps/pex-2.1.46-py2.py3-none-any.whl"
    }
  ]
}

The layout adds new PEX_ROOT caches for the .bootstrap zip and installed wheel chroot zips such that neither bootstraps - which are tied to a version of Pex, nor installed wheel chroot zips, which are constant for a given distribution version, are created more than once.

Closes #1424

opened by jsirois 20

PEX_PATH transitivity

When building a graph of PEXes, each of which contains distributions for a single requirement (*mostly: disregard the pluggy_zipp_importlib-metadata_pytest_attrs cluster), it appears that the embedded pex_path in the PEX-INFO is not consumed. I'm wondering whether:

these failures are expected
providing a flattened transitive PEX_PATH at runtime instead of embedding the direct pex_path deps at build time is likely to be able to allow me to get the behavior I want

As an example: a single requirement PEX containing humbug, with a pex_path containing its requirement PEXes might have PEX-INFO that looks like:

{
  "always_write_cache": false,
  "build_properties": {
    "class": "CPython",
    "pex_version": "2.1.44",
    "platform": "macosx_10_16_x86_64",
    "version": [
      3,
      7,
      7
    ]
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "humbug-0.2.6-py3-none-any.whl": "e772468167e997e051e8c20d71551a11bba19f26"
  },
  "emit_warnings": false,
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "interpreter_constraints": [],
  "pex_hash": "53fc35393a34baa9b5665c00dc19f27a02a3ea07",
  "pex_path": "__reqs/requests.pex:__reqs/types-requests.pex:__reqs/setuptools.pex:__reqs/urllib3.pex:__reqs/certifi.pex:__reqs/idna.pex:__reqs/charset-normalizer.pex:__reqs/pluggy_zipp_importlib-metadata_pytest_attrs.pex:__reqs/typing-extensions.pex:__reqs/packaging.pex:__reqs/iniconfig.pex:__reqs/py.pex:__reqs/toml.pex:__reqs/six.pex:__reqs/pyparsing.pex",
  "requirements": [
    "humbug"
  ],
  "strip_pex_env": true,
  "unzip": false,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "zip_safe": false
}

But consuming it will fail to find the requests distribution embedded in the __reqs/requests.pex PEX:

Below either PexBuilder.set_script or bin.pex.seed_cache:

  ...
  File ".deps/pex-2.1.44-py2.py3-none-any.whl/pex/pex.py", line 118, in resolve
    for dist in env.resolve():
  File ".deps/pex-2.1.44-py2.py3-none-any.whl/pex/environment.py", line 608, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File ".deps/pex-2.1.44-py2.py3-none-any.whl/pex/environment.py", line 695, in resolve_dists
    "{items}".format(pex=self._pex, platform=self._platform, items="\n".join(items))
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /private/var/folders/bg/_r10hqp14kjcpv68yzdk5svc0000gn/T/process-executioneHVQxG/__reqs/humbug.pex.
Needed macosx_10_16_x86_64-cp-37-cp37m compatible dependencies for:
 1: requests
    Required by:
      humbug 0.2.6
    But this pex had no 'requests' distributions.

question answered

opened by stuhood 20

pex is inexplicably slow for a small project

Reproduction

Clone https://github.com/zmanji/git-squash
Checkout 505cf06c598409db9d119a521480e43de3fa8908
Run pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex

With latest pex on macOS.

Expectation

Because of --no-build, --wheel, --no-transitive and a fully resolved requirements.txt from pip-compile building a pex should be very fast. The wheels can be fetched from pypi if they are not cached locally, the package it self a single file, so building the wheel should be very fast and then building the pex should be straight forward

For reference building the wheel for git-squash via python ./setup.py bdist_wheel takes about 0.3s on my computer.

Reality

I see this output

pex . -vv -r requirements.txt --no-build --wheel --no-transitive --inherit-path=false --output-file /Users/zmanji/code/git-squash/dist/git-squash-0.0.1.pex --script git-squash -o dist/out.pex
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Resolving for:
  DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macpex: Hashing pex                                                                                                                                                                                pex: Hashing pex: 67.4ms                                                                                                                            
pex: Isolating pex: 0.1ms
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Building distributions for:
  BuildRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', pex: Building /Users/zmanji/code/git-squash to /Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39                                                                                                                                                                                                                                                                                                                                
pex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Calculating project names for direct requirements:
  LocalProjectRequirement(line=LogicalLine(raw_text='.', processed_text='.', source='<string>', start_line=1, end_line=1), path='/Users/zmanji/code/git-squash', extras=(), marker=None, editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitdb==4.0.7\n', processed_text='gitdb==4.0.7', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=7, end_line=7), requirement=Requirement.parse('gitdb==4.0.7'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitpython==3.1.17\n', processed_text='gitpython==3.1.17', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=9, end_line=9), requirement=Requirement.parse('gitpython==3.1.17'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='smmap==4.0.0\n', processed_text='smmap==4.0.0', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=11, end_line=11), requirement=Rpex: Building pex :: Resolving distributions (['.', 'requirements.txt']) :: Resolving requirements. :: Installing:
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/GitPython-3.1.17-py3-none-any.whl', fingerprint='59d63e79b53503f5403220d3159d8e409c46e107')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/smmap-4.0.0-py2.py3-none-any.whl', fingerprint='5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/gitdb-4.0.7-py3-none-any.whl', fingerprint='9dacbc989a247dabdad014d6397d8482fd55f26c')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-pex: Using cached installation of GitPython-3.1.17-py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/59d63e79b53503f5403220d3159d8e409c46e107/GitPython-3.1.17-py3-none-any.whl                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
pex: Using cached installation of smmap-4.0.0-py2.py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f/smmap-4.0.0-py2.py3-none-any.whl
pex: Using cached installation of gitdb-4.0.7-py3-none-any.whl at /Users/zmanji/.pex/installed_wheels/9dacbc989a247dabdad014d6397d8482fd55f26c/gitdb-4.0.7-py3-none-any.whl
pex: Installing /Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-py3-none-any.whl in /Users/zmanji/.pex/installed_wheels/c355002f497d1f36a0bae44b05e55b4338da8a0f/git_squash-0.0.1-py3-none-any.whl
pex: Set entrypoint to console_script 'git_squash.git_squash:main' in git-squash 0.0.1 (/Users/zmanji/.pex/installed_wheels/c355002f497d1f36a0bae44b05e55b4338da8a0f/git_squash-0.0.1-py3-none-any.whl)
pex: Building pex: 13722.4ms
pex:   Resolving distributions (['.', 'requirements.txt']): 13720.7ms
pex:     Resolving requirements.: 13709.1ms
pex:       Resolving for:
  DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))): 6592.5ms
pex:       Building distributions for:
  BuildRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), source_path='/Users/zmanji/code/git-squash', fingerprint='fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e'): 6475.2ms
pex:       Calculating project names for direct requirements:
  LocalProjectRequirement(line=LogicalLine(raw_text='.', processed_text='.', source='<string>', start_line=1, end_line=1), path='/Users/zmanji/code/git-squash', extras=(), marker=None, editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitdb==4.0.7\n', processed_text='gitdb==4.0.7', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=7, end_line=7), requirement=Requirement.parse('gitdb==4.0.7'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='gitpython==3.1.17\n', processed_text='gitpython==3.1.17', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=9, end_line=9), requirement=Requirement.parse('gitpython==3.1.17'), editable=False)
  PyPIRequirement(line=LogicalLine(raw_text='smmap==4.0.0\n', processed_text='smmap==4.0.0', source='/Users/zmanji/code/git-squash/requirements.txt', start_line=11, end_line=11), requirement=Requirement.parse('smmap==4.0.0'), editable=False): 4.3ms
pex:       Installing:
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/GitPython-3.1.17-py3-none-any.whl', fingerprint='59d63e79b53503f5403220d3159d8e409c46e107')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/smmap-4.0.0-py2.py3-none-any.whl', fingerprint='5eb1d8910b7c93de92bd8db25d0a8293b2d1de6f')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/private/var/folders/1j/9xbcby_57tx1kmn6s8zfsbww0000gn/T/tmpvty3ch2e/Users.zmanji..pyenv.versions.3.9.4.bin.python3.9/gitdb-4.0.7-py3-none-any.whl', fingerprint='9dacbc989a247dabdad014d6397d8482fd55f26c')
  InstallRequest(target=DistributionTarget(interpreter=PythonInterpreter('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', PythonIdentity('/Users/zmanji/.pyenv/versions/3.9.4/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 4)))), wheel_path='/Users/zmanji/.pex/built_wheels/local_projects/git-squash/fa682a73c509acd8bf4e7e52f387f55f3d0b3e6e/cp39-cp39/git_squash-0.0.1-py3-none-any.whl', fingerprint='c355002f497d1f36a0bae44b05e55b4338da8a0f'): 430.6ms
Saving PEX file to dist/out.pex
pex: Zipping PEX file.: 148.4ms
pex . -vv -r requirements.txt --no-build --wheel --no-transitive    --script   9.65s user 4.27s system 96% cpu 14.348 total

Notice:

pex:     Resolving requirements.: 13709.1ms

This is very unexpected and there is no information in the output as to why it takes so long. With all of the caching PEX uses, I would expect this to be very fast.

answered

opened by zmanji 20

Use pip for resolving and building distributions

Now that we can vendor distributions, we might consider vendoring pip to get pip-compatible resolution and wheel building behavior. This would likely kill a fair amount of PEX code and open issues at the expense of a larger PEX tool binary and, without care, a larger PEX runtime. Since pip should be largely (completely?) un-needed at runtime, this might have little downside.
enhancement resolver

opened by jsirois 20
Problem running PEX files with restricted permissions

When running PEX files, PexInfo determines the install_cache path based on the pex_root path, which in turn can be set in the PEX-INFO file - but isn't.

That means it defaults to ~/.pex for writing the install cache - which fails when that path isn't writeable for the user. That can happen - is common practice, even - when securing a system against unwanted file system access. Writing to the home directory, for example, can modify environment variables in unwanted ways.

There's a number of things that could be improved in the approach of defaulting to ~/.pex, but different requirements here would conflict with each other - so I'll leave that discussion to other people. The main point, though, is that it would be very, very useful if one could set the pex_root value that goes into the PEX-INFO file, so that you can effectively choose other paths for the install_cache. If additionally, install_cache could be overridden as well, that'd be excellent.
enhancement

opened by jfinkhaeuser 20
Pex silently builds bad pexfile
This is on OSX against pex 1.4.2 using python 2.7.13. Same command works against pex 1.3.2

Command:

$ pex -r requirements.txt -o env.pex

Contents of requirements.txt:

# === Main Dependencies flask>=0.0 pathlib2 dulwich pandas codenamize pyOpenSSL>=16.2.0 jupyter

When attempting to run env.pex:

$ ./env.pex Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for: ipython appnope

Interestingly, changing the requirements.txt to include appnope and ipython==5.7 explicitly fixes it against pex 1.4.2.

As mentioned, the same works either way against pex 1.3.2.
bug
opened by kmannislands 20

"Resource deadlock avoided" when creating multiple `--venv` PEXs in parallel

Testing out some change in Pants, I create some (>10 < 60) number of processes responsible for creating a PEX venv (--venv) using a PEX lockfile (--lock) specifying some requirement strings which among the processes are likely highly overlapping yet unique subset of my lockfile.

The stacktrace is at the end of this issue. The issue is regularly reproducible in our work repo, but not reproducible in Pants' own repo with the relevant Pants code running (Pants is only relevant because that's how I'm opening the N processes).

Let me know what else you need to help debug and solve.

Stacktrace:

Traceback (most recent call last):                                                                                                                                            File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 504, in execute          
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)                                                                                                     File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 409, in _wrap_coverage   
    return runner(*args)                                                                                                                                                      File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 440, in _wrap_profiling  
    return runner(*args)                                                                                                                                                      File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 560, in _execute         
    return self.execute_entry(self._pex_info.entry_point)                                                                                                                   
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 696, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/unzipped_pexes/15004f7f47d92f2a9c0b754a1c6d2997f332ea44/.bootstrap/pex/pex.py", line 728, in execute_pkg_resources
    return runner()
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 724, in main
    do_main(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 745, in do_main
    pex_builder = build_pex(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/bin/pex.py", line 594, in build_pex
    resolve_from_lock(
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 198, in resolve_from_lock
    pool.map(
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/result.py", line 101, in catch
    return func(*args, **kwargs)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/resolve/lock_resolver.py", line 46, in store_downloadable_artifact
    return self.store(artifact=downloadable_artifact.artifact)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/resolve/lockfile/download_manager.py", line 40, in store
    with atomic_directory(download_dir, exclusive=True) as atomic_dir:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/common.py", line 470, in atomic_directory
    fcntl.lockf(lock_fd, fcntl.LOCK_EX)  # A blocking write lock.
OSError: [Errno 35] Resource deadlock avoided

bug resolver

opened by thejcannon 19

A loose layout, venv-with-symlink PEX creates brittle symlinks

Pex version 2.1.113

I found this through Pants, but ultimately seems like a PEX issue/fix.

$ pex --no-venv-copies --layout=loose --venv prepend awscli --script aws -o  awscli.pex
$ python awscli.pex help
<help output>
$ mv awscli.pex awscli2.pex/
$ python awscli2.pex
Traceback (most recent call last):
  File "/home/josh/.pex/venvs/ea7cb7998ed2db4f784178989a4034013cd10e01/ddab8011daaee380698ac2fb9701af18c90c03f6/bin/aws", line 19, in <module>
    import awscli.clidriver
ModuleNotFoundError: No module named 'awscli

Cannot use pex with statically linked python

due to https://github.com/pypa/pip/issues/6543 it's not possible for pex to resolve dependencies with a statically linked python.

To reproduce in a clean chroot.

$ wget https://github.com/indygreg/python-build-standalone/releases/download/20221220/cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz
$ tar xvzf cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz 
$ wget https://github.com/pantsbuild/pex/releases/download/v2.1.119/pex
$ ./python/bin/python3 ./pex 'fortune' -o fortune.pex
pid 31 -> /root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/bin/python -sE /root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /root/.pex/pip_cache download --dest /root/.pex/downloads/resolver_download.oodws27t/python.bin.python3.10 fortune --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 92, in run
    session = self.get_default_session(options)
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 78, in get_default_session
    self._session = self.enter_context(self._build_session(options))
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 88, in _build_session
    session = PipSession(
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/network/session.py", line 248, in __init__
    self.headers["User-Agent"] = user_agent()
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/network/session.py", line 135, in user_agent
    zip(["lib", "version"], libc_ver()),
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 94, in libc_ver
    glibc_version = glibc_version_string()
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 18, in glibc_version_string
    return glibc_version_string_confstr() or glibc_version_string_ctypes()
  File "/root/.pex/venvs/ffb7a52e5a35c86fd46ad0fdd44b7ac577115b21/58beba50ae31e0c54f07dbf739954fb14f7a9c36/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 52, in glibc_version_string_ctypes
    process_namespace = ctypes.CDLL(None)
  File "/python/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported

Changing the pip version does not work because the vendored pip is used to fetch the newer pip. I see two possible solutions:

pex3 is released and vendors a newer pip.
two pips are vendored much like how two packaging versions are vendored, and when a statically linked python is detected we use the newer pip.

opened by zmanji 2

Release 2.1.120
Backlog:

[ ] Support REPL command history #2019

[ ] Allow system_site_packages=True in Virtualenv.create() #1973

[ ] Cleanup sys.path after pex is imported #1959

[ ] importing a pex leaks bootstrap vendored code #1954

[ ] The --compile option is only useful for Python 2.7 #1939

[ ] Universal lock creation failure for disjoint IC and Requires-Python is correct but the error message from Pip / Pex is confusing. #1927

[ ] The __pex__ import hook does not handle registering top level PEP-420 packages #1910

[ ] Expose setuptools and wheel version flags for use with non-vendored Pips. #1895

[ ] Plumb --pip-version to Platform tag calculation. #1894

[ ] PEX_INHERIT_PATH+default shebangs don't interact well with virtualenvs #1344

[ ] Pex interpreter cache is not upgrade / downgrade safe. #1176

[ ] Provide a better resolve error message for a Platform DistributionTarget when --resolve-local-platforms. #1070

[ ] Remove PEX_PYTHON_PATH from build time interpreter selection. #1075

release
opened by jsirois 1
Specifying `--platform`in `lock create` impedes referring to sdists
Both:

pex3 lock create --platform manylinux_2_31_x86_64-cp-38-cp38 timeout-decorator~=0.4

pex3 lock create --style sources --platform manylinux_2_31_x86_64-cp-38-cp38 timeout-decorator~=0.4

result in an error because timeout-decorator is an sdist-only package.
opened by thejcannon 0
Allow system_site_packages=True in Virtualenv.create()

Currently PEX could create virtualenv to populate bundled packages, and in PEX code it allows to create venv with "--system-site-packages", but there is no way to specify this parameter at runtime.

We want to run with venv and system-site-packages enabled, can we have an extra environment variable to specify this behavior?
enhancement feature request

opened by PhoenixHe-TSP 5

Releases(v2.1.119)

v2.1.119(Dec 15, 2022)
2.1.119

This release brings two new features. The venv pex tool now just warns when using --compile and there is a *.pyc compile error instead of failing to create the venv. Also, a new PEX_DISABLE_VARIABLES env var knob is added to turn off reading all PEX_* env vars from the environment.

Ignore compile error for PEX_TOOLS=1 (#2002)

Add PEX_DISABLE_VARIABLES to lock down a PEX run. (#2014)

Source code(tar.gz)
Source code(zip)
pex(3.88 MB)
v2.1.118(Dec 9, 2022)
2.1.118

This is a very tardy hotfix release for a regression introduced in Pex 2.1.91 by #1785 that replaced sys.argv[0] with its fully resolved path. This prevented introspecting the actual file path used to launch the PEX which broke BusyBox-alike use cases.

There is also a new --non-hermetic-scripts option accepted by the venv tool to allow running console scripts with PYTHONPATH adjustments to the sys.path.

Remove un-needed realpathing of sys.argv[0]. (#2007)

Add --non-hermetic-scripts option to venv tool. (#2010)

Source code(tar.gz)
Source code(zip)
pex(3.88 MB)
v2.1.117(Nov 29, 2022)
2.1.117

This release fixes a bug introduced in Pex 2.1.109 where the released Pex PEX could not be executed by PyPy interpreters. More generally, any PEX created with interpreter constraints that did not specify the Python implementation, e.g.: ==3.8.*, were interpreted as being CPython specific, i.e.: CPython==3.8.*. This is now fixed, but if the intention of a constraint like ==3.8.* was in fact to restrict to CPython only, interpreter constraints need to say so now and use CPython==3.8.* explicitly.

Fix interpreter constraint parsing. (#1998)

Source code(tar.gz)
Source code(zip)
pex(3.88 MB)
v2.1.116(Nov 25, 2022)
2.1.116

This release fixes a bug in --resolve-local-platforms when --complete-platform was used.

Check for --complete-platforms match when --resolve-local-platforms (#1991)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.115(Nov 24, 2022)
2.1.115

This release brings some attention to the pex3 lock export sub-command to make it more useful when inter-operating with pip-tools.

Sort requirements based on normalized project name when exporting (#1992)

Use raw version when exporting (#1990)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.114(Nov 21, 2022)
2.1.114

This release brings two fixes for --venv mode PEXes.

Only insert "" to head of sys.path if a venv PEX runs in interpreter mode (#1984)

Map pex python path interpreter to realpath when creating venv dir hash. (#1972)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.113(Nov 3, 2022)
2.1.113

This is a hotfix release that fixes errors installing wheels when there is high parallelism in execution of Pex processes. These issues were a regression introduced by #1961 included in the 2.1.112 release.

Restore AtomicDirectory non-locked good behavior. (#1974)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.112(Oct 22, 2022)
2.1.112

This release brings support for the latest Pip release and includes some internal changes to help debug intermittent issues some users are seeing that implicate what may be file locking related bugs.

Add support for --pip-version 22.3. (#1953)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.111(Oct 19, 2022)
2.1.111

This release fixes resolving requirements from a lock using arbitrary equality (===).

In addition, you can now "inject" runtime environment variables and arguments into PEX files such that, when run, the PEX runtime ensures those environment variables and command line arguments are passed to the PEXed application. See PEX Recipes for more information.

Fix lock resolution to handle arbitrary equality. (#1951)

Support injecting args and env vars in a PEX. (#1948)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.110(Oct 10, 2022)
2.1.110

This release fixes Pex runtime sys.path scrubbing for cases where Pex is not the main entry point. An important example of this is in Lambdex where the AWS Lambda Python runtime packages (boto3 and botocore) are leaked into the PEX runtime sys.path.

Fix sys.path scrubbing. (#1946)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.109(Oct 6, 2022)
2.1.109

This release brings musllinux wheel support and a fix for a regression introduced in Pex 2.1.105 by #1902 that caused PEX_PATH= (an exported PEX_PATH with an empty string value) to raise an error in almost all use cases.

Vendor latest packaging; support musllinux wheels. (#1937)

Don't treat PEX_PATH= as . like other PATHS. (#1938)

Source code(tar.gz)
Source code(zip)
pex(3.87 MB)
v2.1.108(Oct 4, 2022)
2.1.108

This release fixes a latent PEX boot performance bug triggered by requirements with large extras sets.

Fix slow PEX boot time when there are many extras. (#1929)

Source code(tar.gz)
Source code(zip)
pex(3.67 MB)
v2.1.107(Oct 1, 2022)
2.1.107

This release fixes an issue handling credentials in git+ssh VCS urls when creating locks.

Fix locks for git+ssh with credentials. (#1923)

Source code(tar.gz)
Source code(zip)
pex(3.66 MB)
v2.1.106(Sep 30, 2022)
2.1.106

This release fixes a long standing bug in handling direct reference requirements with a local version component.

Unquote path component of parsed url requirements (#1920)

Source code(tar.gz)
Source code(zip)
pex(3.66 MB)
v2.1.105(Sep 20, 2022)
2.1.105

This is a fix release which addresses issues related to build time work_dir creation, virtualenv, and sh_boot support.

In the unlikely event of a UUID collision in atomic workdir creation, pex could overwrite an existing directory and cause a corrupt state. When building a shell bootable --sh-boot pex the --runtime-pex-root was not always respected based on the condition of the build environment, and the value of the PEX_ROOT.

Fail on atomic_directory work_dir collision. (#1905)

Use raw_pex_root when constructing sh_boot pexes. (#1906)

Docs.

Add support for offline downloads (#1898)

Source code(tar.gz)
Source code(zip)
pex(3.65 MB)
v2.1.104(Aug 28, 2022)
2.1.104

This release brings a long-awaited upgrade of the Pip Pex uses, but behind a --pip-version 22.2.2 flag you must opt in to. Pex will then use that version of Pip if it can (your Pex operations target Python >=3.7) and warn and fall back to the older vendored Pip (20.3.4) if it can't. To turn the need to fallback to older Pip from a warning into a hard error you can also specify --no-allow-pip-version-fallback.

The pex3 lock update command now gains the ability to update just the index and find links repos the lock's artifacts originate from by using a combination of --no-pypi, --index & --find-links along with --pin to ensure the project versions stay pinned as they are in the lockfile and just the repos they are downloaded from is altered. Consult the CLI --help for --fingerprint-mismatch {ignore,warn,error} to gain more control over repo migration behavior.

There are several bug fixes as well dealing with somewhat esoteric corner cases involving changing a PEX --layout from one form to another and building artifacts using certain interpreters on macOS 11.0 (aka: 10.16).

Add support for Pip 22.2.2. (#1893)

Make lock update sensitive to artifacts. (#1887)

Ensure locally built wheel is consumable locally. (#1886)

Ensure --output always overwrites destination. (#1883)

Source code(tar.gz)
Source code(zip)
pex(3.65 MB)
v2.1.103(Aug 5, 2022)
2.1.103

This release fixes things such that pex lockfiles can be created and updated using the Pex PEX when local projects are involved.

Fix pex3 lock ... when run from the Pex PEX. (#1874)

Source code(tar.gz)
Source code(zip)
pex(3.63 MB)
v2.1.102(Jul 27, 2022)
2.1.102

This is a hotfix release that fixes a further corner missed by #1863 in the Pex 2.1.101 release whereby Pex would fail to install platform-specific packages on Red Hat based OSes.

In addition, an old but only newly discovered bug in --inherit-path={prefer,fallback} handling is fixed. Previously only using PEX_INHERIT_PATH={prefer,fallback} at runtime worked properly.

In the process of fixing the old --inherit-path={prefer,fallback} bug, also fix another old bug handling modern virtualenv venvs under Python 2.7 during zipapp execution mode PEX boots.

Fix wheel installs: account for purelib & platlib. (#1867)

Fix --inhert-path handling. (#1871)

Error using pex + virtualenv >=20.0.0 + python 2.7 (#992)

Source code(tar.gz)
Source code(zip)
pex(3.63 MB)
v2.1.101(Jul 25, 2022)
2.1.101

This release fixes a corner-case revealed by python-certifi-win32 1.6.1 that was not previously handled when installing certain distributions.

Make wheel install site-packages detection robust. (#1863)

Source code(tar.gz)
Source code(zip)
pex(3.63 MB)
v2.1.100(Jul 23, 2022)
2.1.100

This release fixes a hole in the lock creation --target-system feature added in #1823 in Pex 2.1.95.

Fix lock creation --target-system handling. (#1858)

Source code(tar.gz)
Source code(zip)
pex(3.63 MB)
v2.1.99(Jul 14, 2022)
2.1.99

This release fixes a concurrency bug in the pex --lock ... artifact downloading.

Fix pex --lock ... concurrent download errors. (#1854)

Source code(tar.gz)
Source code(zip)
pex(3.63 MB)
v2.1.98(Jul 13, 2022)
2.1.98

This releases fixes regressions in foreign --platform handling and artifact downloading introduced by #1787 in Pex 2.1.91.

In addition, PEXes can now be used as sys.path entries. Once on the sys.path, via PYTHONPATH or other means, the code in the PEX can be made importable by first importing __pex__ either as its own stand-alone import statement; e.g.: import __pex__; import psutil or as a prefix of the code to import from the PEX; e.g.: from __pex__ import psutil.

Tags should be patched for --platform. (#1846)

Add support for importing from PEXes. (#1845)

Fix artifact downloads for foreign platforms. #1851

Source code(tar.gz)
Source code(zip)
pex(3.63 MB)
v2.1.97(Jul 12, 2022)
2.1.97

This release patches a hole left by #1828 in the Pex 2.1.95 release whereby, although you could run a PEX under a too-long PEX_ROOT you could not build a PEX under a tool-long PEX_ROOT.

Avoid ENOEXEC for Pex internal --venvs. (#1843)

Source code(tar.gz)
Source code(zip)
pex(3.62 MB)
v2.1.96(Jul 12, 2022)
2.1.96

This is a hotfix release that fixes --venv mode PEX_EXTRA_SYS_PATH propogation introduced in Pex 2.1.95 to only apply to sys.executable and not other Pythons.

Fix --venv PEX PEX_EXTRA_SYS_PATH propagation. (#1837)

Source code(tar.gz)
Source code(zip)
pex(3.62 MB)
v2.1.95(Jul 9, 2022)
2.1.95

This release brings two new pex3 lock features for --style universal locks.

By default, universal locks are created to target all operating systems. This can cause problems when you only target a subset of operating systems and a lock transitive dependency that is conditional on an OS you do not target is not lockable. The new --target-system {linux,mac,windows} option allows you to restrict the set of targeted OSes to work around this sort of issue. Since PEX files currently only support running on Linux and Mac, specifying --target-system linux --target-system mac is a safe way to pre-emptively avoid these sorts of locking issues when creating a universal lock.

Previously you could not specify the --platform\s or --complete-platform\s you would be using later to build PEXes with when creating a universal lock. You now can, and Pex will verify the universal lock can support all the specified platforms.

As is usual there are also several bug fixes including properly propagating PEX_EXTRA_SYS_PATH additions to forked Python processes, fixing pex3 lock export to only attempt to export for the selected target and avoiding too long shebang errors for --venv mode PEXes in a robust way.

Fix PEX_EXTRA_SYS_PATH propagation. (#1832)

Fix pex3 lock export: re-use --lock resolver. (#1831)

Avoid ENOEXEC for --venv shebangs. (#1828)

Check lock can resolve platforms at creation time. (#1824)

Support restricting universal lock target os. (#1823)

Source code(tar.gz)
Source code(zip)
pex(3.62 MB)
v2.1.94(Jul 1, 2022)
2.1.94

This is a hotfix release that fixes a regression introduced in Pex 2.1.93 downloading certain sdists when using pex --lock ....

Fix pex --lock ... handling of sdists. (#1818)

Source code(tar.gz)
Source code(zip)
pex(3.62 MB)
v2.1.93(Jun 21, 2022)
2.1.93

This release brings several new features in addition to bug fixes.

When creating a PEX the entry point can now be any local python script by passing --exe path/to/python-script.

The pex3 lock update command now supports a -dry-dun check mode that exits non-zero to indicate that a lock needs updating and the -p / --project targeted update arguments can now be new projects to attempt to add to the lock.

On the bug fix front, traditional zipapp mode PEX files now properly scrub sys.displayhook and sys.excepthook and their teardown sequence has now been simplified fixing logging to stderr late in teardown.

Finally, pex3 lock create now logs when requirement resolution is taking a long time to provide some sense of progress and suggest generic remedies and pex --lock now properly handles authentication.

Support adding new requirements in a lock update. (#1797)

Add pex3 lock update --dry-run check mode. (#1799)

Universal locks no longer record a platform_tag. (#1800)

Support python script file executable. (#1807)

Fix PEX scrubbing to account for sys.excepthook. (#1810)

Simplify PEX teardown / leave stderr in tact. (#1813)

Surface pip download logging. (#1808)

Use pip download instead or URLFetcher. (#1811)

Source code(tar.gz)
Source code(zip)
pex(3.62 MB)
v2.1.92(May 31, 2022)
2.1.92

This release adds support for locking local projects.

Add support for local project locking. #1792

Source code(tar.gz)
Source code(zip)
pex(3.62 MB)
v2.1.91(May 30, 2022)
2.1.91

This release fixes --sh-boot mode PEXes to have an argv0 and exported PEX environment variable consistent with standard Python boot PEXes; namely the absolute path of the originally invoked PEX.

Fix --sh-boot argv0. (#1785)

Source code(tar.gz)
Source code(zip)
pex(3.59 MB)
v2.1.90(May 25, 2022)
2.1.90

This release fixes Pex handling of sdists to be atomic and also fixes lock files to be emitted ending with a newline. In addition, many typos in Pex documentation were fixed in a contribution by Kian-Meng Ang.

Ensure Pip cache operations are atomic. (#1778)

Ensure that lockfiles end in newlines. (#1774)

Fix typos (#1773)

Source code(tar.gz)
Source code(zip)
pex(3.58 MB)

A library and tool for generating .pex (Python EXecutable) files

Related tags

Overview

PEX

Comments

Problem

Solution

Alternative flag considered: --reproducible

Alternative solution: ensure our own .pyc timestamp

Result

Impact on performance

Releases(v2.1.119)

v2.1.119(Dec 15, 2022)

2.1.119

v2.1.118(Dec 9, 2022)

2.1.118

v2.1.117(Nov 29, 2022)

2.1.117

v2.1.116(Nov 25, 2022)

2.1.116

v2.1.115(Nov 24, 2022)

2.1.115

v2.1.114(Nov 21, 2022)

2.1.114

v2.1.113(Nov 3, 2022)

2.1.113

v2.1.112(Oct 22, 2022)

2.1.112

v2.1.111(Oct 19, 2022)

2.1.111

v2.1.110(Oct 10, 2022)

2.1.110

v2.1.109(Oct 6, 2022)

2.1.109

v2.1.108(Oct 4, 2022)

2.1.108

v2.1.107(Oct 1, 2022)

2.1.107

v2.1.106(Sep 30, 2022)

2.1.106

v2.1.105(Sep 20, 2022)

2.1.105

v2.1.104(Aug 28, 2022)

2.1.104

v2.1.103(Aug 5, 2022)

2.1.103

v2.1.102(Jul 27, 2022)

2.1.102

v2.1.101(Jul 25, 2022)

2.1.101

v2.1.100(Jul 23, 2022)

2.1.100

v2.1.99(Jul 14, 2022)

2.1.99

v2.1.98(Jul 13, 2022)

2.1.98

v2.1.97(Jul 12, 2022)

2.1.97

v2.1.96(Jul 12, 2022)

2.1.96

v2.1.95(Jul 9, 2022)

2.1.95

v2.1.94(Jul 1, 2022)

2.1.94

v2.1.93(Jun 21, 2022)

2.1.93

v2.1.92(May 31, 2022)

2.1.92

v2.1.91(May 30, 2022)

2.1.91

v2.1.90(May 25, 2022)

2.1.90

Owner

Pants Build

Core utilities for Python packages

An example of repository data as bundles

The Application can convert the .py file into exe for faster transformation and can result to build an app in a single click

Python virtualenvs in Debian packages

Subpar is a utility for creating self-contained python executables. It is designed to work well with Bazel.

auto packaging for iOS

Alternative flag considered: `--reproducible`

Alternative solution: ensure our own `.pyc` timestamp