Lets start with very cool project. Been playing around with it and see lots of cases where I needed this in the past.

(venv) api goose$ python setup.py install && python setup.py bdist_xar

(venv) api goose$ ./dist/api.xar
Directories/modules found:  ['ctrl_api', 'setup', 'api', 'compare_api_mappings', 'ctrl_decorators', 'ctrl_logging', 'dev_code', 'endpoints', 'providers', 'settings', 'test_xar', 'tests']
Traceback (most recent call last):
  File "/mnt/xarfuse/uid-501/5348637a/__run_xar_main__.py", line 32, in <module>
    __invoke_main()
  File "/mnt/xarfuse/uid-501/5348637a/__run_xar_main__.py", line 29, in __invoke_main
    runpy._run_module_as_main(module, False)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/xarfuse/uid-501/5348637a/ctrl_api/api.py", line 11, in <module>
    from providers.auth.authenticate import (
ModuleNotFoundError: No module named 'providers'

So what seems to work best is:

try:
    from providers.auth.authenticate import (
        authenticate,
        retrieve_refresh_token
    )
except ImportError:
    from .providers.auth.authenticate import (
        authenticate,
        retrieve_refresh_token
    )

Seems relative imports is the default.

I looked quick and didn't see it mentioned in the docs.

Perhaps mention the relative imports of this is the expected behaviour?

trying out the xar distutils extension, but having trouble getting it running w/in pipenv

basically don't see the bdist_xar extension when running distutils:

pipenv run python setup.py --help-commands
Standard commands:
  build            build everything needed to install
  build_py         "build" pure Python modules (copy to build directory)
  build_ext        build C/C++ extensions (compile/link to build directory)
  build_clib       build C/C++ libraries used by Python extensions
  build_scripts    "build" scripts (copy and fixup #! line)
  clean            clean up temporary files from 'build' command
  install          install everything from build directory
  install_lib      install all Python modules (extensions and pure Python)
  install_headers  install C/C++ header files
  install_scripts  install scripts (Python or otherwise)
  install_data     install data files
  sdist            create a source distribution (tarball, zip file, etc.)
  register         register the distribution with the Python package index
  bdist            create a built (binary) distribution
  bdist_dumb       create a "dumb" built distribution
  bdist_rpm        create an RPM distribution
  bdist_wininst    create an executable installer for MS Windows
  check            perform some checks on the package
  upload           upload binary package to PyPI

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

on mac and have installed xar via brew per the instructions: https://code.fb.com/data-infrastructure/xars-a-more-efficient-open-source-system-for-self-contained-executables/

in the pipenv i install the extensions by cloning xar, cding into the directory and:

pipenv run pip install .

still dont get the extension, but if I import it in my setup.py, it works fine, until the build directory is cleaned:

from xar.commands.bdist_xar import bdist_xar
setup(
...
    cmdclass={'bdist_xar': bdist_xar},
...
)

then:

› per python setup.py bdist_xar
running bdist_xar
running bdist_wheel
running build
running build_py
creating build
creating build/lib
copying collector.py -> build/lib
removing 'build/bdist.macosx-10.12-x86_64/xar' (and everything under it)
error: [Errno 2] No such file or directory: 'build/bdist.macosx-10.12-x86_64/xar'

any direction is much appreciated

D31507009 moved the logging macros to a different header, but this header wasn't added to CMakeLists. This diff fixes the CMake build.

Tested with:

mkdir build; cd build; cmake ..; make

Fixes #49

I have a feeling the open source build has been broken (at least for MacOS) since https://github.com/facebookincubator/xar/commit/a4c59465d7550c0e26041114d213076bc9ad49ac

I was getting a linking error against Logging.h, repro'd with both clang and gcc. Explicitly adding that dependency to the xarlib_srcs set has the binary compiling successfully for me now.

Perhaps there is a better fix for this, but packaging selenium, I get existing file errors unless I patch https://github.com/facebookincubator/xar/blob/master/xar/xar_builder.py#L392 to force=True

e.g.

-        wheel.install(sys_paths, xar_paths, force=False)
+        wheel.install(sys_paths, xar_paths, force=True)

Any suggestions or should I look into passing through a --force flag?

edit: to clarify, packaging an executable with selenium as a dependency.

Hi ,

It's a great one. Thanks for making one. I am seeking one help. I want to make XAR for windows and CentOs also. Could you please help me by letting me know how to do the same.

Thanks & regards, Shashank Singh

Hi all,

I've received the error below when executing a xar executable file:

./hello
/dev/shm/uid-1000/8d11d214-ns-4026531840/hello.py: 3: /dev/shm/uid-1000/8d11d214-ns-4026531840/hello.py: Syntax error: "(" unexpected

Executable python script

cat hellopy/hello.py 
#!/usr/bin/python

for i in range(100):
    print i

Build xar

make_xar --python-interpreter python3 --raw hellopy --raw-executable hello.py --output hello

Any hint is welcome.

OS: Ubuntu 16.04 LTS

When using xar in a docker container, there is an issue with the FUSE driver:

$ ./test.xar
fuse: device not found, try 'modprobe fuse' first FATAL /opt/xar/xar/XarExecFuse.cpp:486: squashfuse_ll failed with exit status 1

[1]    46915 abort      ./test.xar
$ modprobe fuse
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.9.87-linuxkit-aufs/modules.dep.bin'
modprobe: FATAL: Module fuse not found in directory /lib/modules/4.9.87-linuxkit-aufs

AFAIK the container environment is responsible for the latter.

Is there any way to use xar without this kernel module, in order avoid privileged containers or mounting the fuse socket? (Probably really important when using docker containers on windows)

I'd like to use Xar to improve the import-time performance of Zulip, but we don't have a setup.py file with a fixed set of executables. Instead, we have a requirements.txt file for our standard virtualenv, and individual command source it and then use it.

There's only 2-3 entry points that we really care about (e.g. the Django standard manage.py that you mention in your benchmarks is one), so it'd be fine if we need to list the entrypoints, but I can't figure out from your documentation how to create a xar archive for such a command without first creating a setup.py file.

I noticed that this package only officially supports Python 2.6 and 3.6; is there a reason you'd expect it to not work on with 3.5/3.7, or have you just not tested it for those?

Is xar limited to Python or could it be used for other types of payloads as well? Could it be used as more generic image format for something like AppImage?

Hey, thanks for a cool project— I noted that when I install xar in a venv, it doesn't work:

$ path/to/my/xar_env/bin/python setup.py bdist_xar
running bdist_xar
removing 'build/bdist.linux-x86_64/xar' (and everything under it)
error: [Errno 2] No such file or directory: 'build/bdist.linux-x86_64/xar'

I see the same result when I source bin/activate in that virtualenv and also if my project is installed into the virtualenv. It only works as expected if I pip3 install xar system-wide (undesirable, especially for many CI setups). Is there something I'm missing here to make this work? Thanks again.

It is particularly a problem because xar also seems to pick up (and then complain about) system dependency versions even when it has downloaded the correct/newer ones, eg:

Collecting paramiko-expect
  Using cached paramiko_expect-0.2.8-py2.py3-none-any.whl (11 kB)
  Saved ./build/bdist.linux-x86_64/xar/downloads/paramiko_expect-0.2.8-py2.py3-none-any.whl
Collecting paramiko>=1.10.1
  File was already downloaded path/to/my/project/build/bdist.linux-x86_64/xar/downloads/paramiko-2.7.1-py2.py3-none-any.whl
Collecting bcrypt>=3.1.3
  File was already downloaded path/to/my/project/build/bdist.linux-x86_64/xar/downloads/bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl
Collecting cryptography>=2.5
  File was already downloaded path/to/my/project/build/bdist.linux-x86_64/xar/downloads/cryptography-2.9.2-cp35-abi3-manylinux2010_x86_64.whl
Collecting pynacl>=1.0.1
  File was already downloaded path/to/my/project/build/bdist.linux-x86_64/xar/downloads/PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl
Collecting cffi>=1.1
  File was already downloaded path/to/my/project/build/bdist.linux-x86_64/xar/downloads/cffi-1.14.0-cp35-cp35m-manylinux1_x86_64.whl
Collecting six>=1.4.1
  File was already downloaded path/to/my/project/build/bdist.linux-x86_64/xar/downloads/six-1.15.0-py2.py3-none-any.whl
Collecting pycparser
  File was already downloaded path/to/my/project/build/bdist.linux-x86_64/xar/downloads/pycparser-2.20-py2.py3-none-any.whl
Successfully downloaded paramiko-expect paramiko bcrypt cryptography pynacl cffi six pycparser
WARNING: You are using pip version 20.0.2; however, version 20.1.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
removing 'build/bdist.linux-x86_64/xar' (and everything under it)
Traceback (most recent call last):
  File "setup.py", line 25, in <module>
    'tqdm'
  File "/usr/local/lib/python3.5/dist-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.5/dist-packages/xar/commands/bdist_xar.py", line 231, in run
    deps = self._deps(dist)
  File "/usr/local/lib/python3.5/dist-packages/xar/commands/bdist_xar.py", line 193, in _deps
    requires, extras=extras, installer=self.installer
  File "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line 792, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (cryptography 1.2.3 (/usr/lib/python3/dist-packages), Requirement.parse('cryptography>=2.5'), {'paramiko'})

I don't think any of this would be an issue if xar were used from a clean environment.

I'm using Ubuntu; I can reproduce this behaviour on both Ubuntu Xenial (Python 3.5) and Ubuntu Focal (Python 3.8).

I've got a Python CLI tool that I'm trying to distribute internally at my company, and xar is really close to a perfect fit. There's just one niggling issue for me, and that's the way OSXFUSE (for obvious reasons) always mounts the squashfs with an icon on the Mac OS Desktop.

The biggest issue is the name, it ends up being called OSXFUSE Volume 0 (squashfuse_ll) or something similar, which is a little rough. It would be great if we could pass the -o volname= option and either set it to the name of the package, or make it customizable.

Even better would be if there's a way to hide the volume altogether, but that's probably asking a bit much. I haven't been able to figure out if it's actually possible to hide a OSXFUSE volume at all.

I just ran make_xar --python /usr/lib/python3.7/site-packages/IPython --output ipython.xar on Manjaro Linux (Arch derivative) and got this output:

make_xar --python /usr/lib/python3.7/site-packages/
IPython --output ipython.xar                                                                     
2019-11-12 21:01:04,679 INFO Squashing /tmp/tmpgk_12zc1 to /tmp/tmp4pyijqfo
mksquashfs: -force-gid invalid gid or unknown group      
Traceback (most recent call last):                                                               
  File "/home/redacted/.local/bin/make_xar", line 10, in <module>
    sys.exit(main())                                                                             
  File "/home/redacted/.local/lib/python3.7/site-packages/xar/make_xar.py", line 147, in main
    xar.build(opts.output, squashfs_options)                                                     
  File "/home/redacted/.local/lib/python3.7/site-packages/xar/xar_builder.py", line 269, in build
    self._staging, tmp_xar, self._shebang, xar_header, squashfs_options
  File "/home/redacted/.local/lib/python3.7/site-packages/xar/xar_builder.py", line 224, in _build_st
aging_dir                                                                                        
    xar.go()
  File "/home/redacted/.local/lib/python3.7/site-packages/xar/xar_util.py", line 104, in go
    subprocess.check_call(cmd)
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['mksquashfs', '/tmp/tmpgk_12zc1', '/tmp/tmp5mqduxu2', '-
noappend', '-noI', '-noX', '-force-uid', 'nobody', '-force-gid', 'nogroup', '-b', '262144', '-com
p', 'gzip']' returned non-zero exit status 1.

The Ubuntu package contains the libsquashfuse_ll.so dynamic library. It is possible to use it instead of the executable itself for xarexec_fuse. We shouldn't wait for debian to package this binary to move forward.

want to deploy python apps with xar, but need to include python 3.7 binary to support old centos6. used pex to build to that usage. but xar seems dose not include python binary itself. how can I do that?