Module for remote in-memory Python package/module loading through HTTP/S

Overview

httpimport

Python's missing feature!

The feature has been suggested in Python Mailing List

Remote, in-memory Python package/module importing through HTTP/S

PyPI - Python Version PyPI version Build Status Coverage Badge

CPython 2.7 CPython 3.4 CPython 3.7 Pypy 2.7 Pypy 3.6 Jython 2.7.1

A feature that Python2/3 misses and has become popular in other languages is the remote loading of packages/modules.

httpimport lets Python2/3 packages and modules to be imported directly in Python interpreter's process memory, through remote URIs, and more...

Examples

Load a simple package/module through HTTP/S

>>> with httpimport.remote_repo(['package1','package2','package3'], 'http://my-codes.example.com/python_packages'):
... 	import package1
...

Load directly from a GitHub/BitBucket/GitLab repo

  • Load a python file from a github-gist (using this gist):
import httpimport

url = "https://gist.githubusercontent.com/operatorequals/ee5049677e7bbc97af2941d1d3f04ace/raw/e55fa867d3fb350f70b2897bb415f410027dd7e4"
with httpimport.remote_repo(["hello"], url):
    import hello
hello.hello()
>>> with httpimport.github_repo('operatorequals', 'covertutils', branch = 'master'):
...     import covertutils
... # Also works with 'bitbucket_repo' and 'gitlab_repo'

Load a package/module from HTTP/S directory directly to a variable

>>> module_object = httpimport.load('package1', 'http://my-codes.example.com/python_packages')
>>> module_object
<module 'package1' from 'http://my-codes.example.com/python_packages/package1/__init__.py'>

Load a package/module that depends on other packages/modules in different HTTP/S directories

>>> # A depends on B and B depends on C (A, B, C : Python modules/packages in different domains):
>>> # A exists in "repo_a.my-codes.example.com" |
>>> # B exists in "repo_b.my-codes.example.com" | <-- Different domains
>>> # C exists in "repo_c.my-codes.example.com" |
>>> with httpimport.remote_repo(['C'], 'http://repo_c.my-codes.example.com/python_packages'):
...  with httpimport.remote_repo(['B'], 'http://repo_b.my-codes.example.com/python_packages'):
...   with httpimport.remote_repo(['A'], 'http://repo_a.my-codes.example.com/python_packages'):
...   import A
... # Asks for A, Searches for B, Asks for B, Searches for C, Asks for C --> Resolves --> Imports A
>>>

Load Python packages from archives served through HTTP/S

>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.bz2'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.gz'):
>>> # with httpimport.remote_repo(['test_package'], 'http://example.com/packages.tar.xz'): <-- Python3 Only
>>> with httpimport.remote_repo(['test_package'], 'http://example.com/packages.zip'):
... 	import test_package
...
>>>

Serving a package through HTTP/S

$ ls -lR
test_web_directory/:                                                         
total 16                                                                     
drwxrwxr-x. 4 user user 4096 Sep  9 20:54 test_package                       
[...]                  
                                                                             
test_web_directory/test_package:                                             
total 20                                                                     
drwxrwxr-x. 2 user user 4096 Sep  9 20:54 a                                  
drwxrwxr-x. 2 user user 4096 Sep  9 20:54 b                                  
-rw-rw-r--. 1 user user   33 Sep  9 20:54 __init__.py                        
-rw-rw-r--. 1 user user  160 Sep  9 20:54 module1.py                         
-rw-rw-r--. 1 user user  160 Sep  9 20:54 module2.py                         
                                                                             
test_web_directory/test_package/a:                                           
total 4                                                                      
-rw-rw-r--. 1 user user  0 Sep  9 20:54 __init__.py                          
-rw-rw-r--. 1 user user 41 Sep  9 20:54 mod.py                               
                                                                             
test_web_directory/test_package/b:                                           
total 4
-rw-rw-r--. 1 user user  0 Sep  9 20:54 __init__.py
-rw-rw-r--. 1 user user 41 Sep  9 20:54 mod.py

$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Usage

Importing Remotely

add_remote_repo() and remove_remote_repo()

These 2 functions will add and remove to the default sys.meta_path custom HttpImporter objects, given the URL they will look for packages/modules and a list of packages/modules its one can serve.

>>> import test_package### Contexts

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named test_package
>>>
>>> from httpimport import add_remote_repo, remove_remote_repo
>>> # In the given URL the 'test_package/' is available
>>> add_remote_repo(['test_package'], 'http://localhost:8000/') #  
>>> import test_package
>>>
>>> remove_remote_repo('http://localhost:8000/')
>>> import test_package.module1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named module1

The load() function (as of 0.5.10)

The load() function was added to make module loading possible without Namespace pollution. It is used to programmatically load a module in a variable, and call its objects directly from that variable.

>>> import httpimport
>>> pack1 = httpimport.load('test_package','http://localhost:8000/')
>>> pack1
<module 'test_package' from 'http://localhost:8000//test_package/__init__.py'>
>>>

Contexts

The remote_repo() context

Adding and removing remote repos can be a pain, especially if there are packages that are available in more than one repos. So the with keyword does the trick again:

>>> from httpimport import remote_repo
>>>
>>> with remote_repo(['test_package'], 'http://localhost:8000/') :
...     from test_package import module1
...
>>>
>>> from test_package import module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name module2

>>> module1.dummy_str
'Constant Loaded'
>>> module1.dummy_func
<function dummy_func at 0x7f7a8a170410>

The Github Use Case!

The dedicated github_repo() context:
>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>
What about branches?
>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', branch='py3_compatibility' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>
And ad-hoc commits too?

What if you need to stick to a fixed -known to work- commit?

>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', commit='cf3f78c77c437edf2c291bd5b4ed27e0a93e6a77' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>

The newer sibling bitbucket_repo() (as of 0.5.9)

>>> with bitbucket_repo('atlassian', 'python-bitbucket', module='pybitbucket'):
...     import pybitbucket
...
>>>

Another sibling gitlab_repo() (as of 0.5.17)

>>> with gitlab_repo('harinathreddyk', 'python-gitlab', module='gitlab'):
...     from gitlab import const
...
>>>
The domain parameter for gitlab_repo()

You can point to your own installation of GitLab by using the domain parameter:

>>> with gitlab_repo('self', 'myproject', module='test_package', domain='127.0.0.1:8080'):
...     import test_package
...
>>>

This covers the posibility of using httpimport to target local development environments, which is a strong use case for httpimport.

Import remote (encrypted) ZIP files (as of 0.5.18)

After version 0.5.18 the add_remote_repo and the load functions, as well as the remote_repo context got the zip and zip_pwd parameters. By pointing to a HTTP/S URL containing a ZIP file, it is possible to remotely load modules/packages included in it, without downloading the ZIP file to disk!

>>> with httpimport.remote_repo(
...     ['test_package'], base_url='http://localhost:8000/test_package.zip',
...     ):
...    import test_package
...
>>>

Using a ZIP password (zip_pwd parameter)

>>> with httpimport.remote_repo(
...     ['test_package'], base_url='http://localhost:8000/test_package.enc.zip',
...     zip_pwd=b'[email protected]!'
...     ):
...    import test_package
...
>>>

Life suddenly got simpler for Python module testing!!!

Imagine the breeze of testing Pull Requests and packages that you aren't sure they are worth your download.

Recursive Dependencies

If package A requires module B and A exists in http://example.com/a_repo/, while B exists in http://example.com/b_repo/, then A can be imported using the following technique:

>>> from httpimport import remote_repo
>>> with remote_repo(['B'],"http://example.com/b_repo/") :
...     with remote_repo(['A'],"http://example.com/a_repo/") :
...             import A
... 
[!] 'B' not found in HTTP repository. Moving to next Finder.
>>> 
>>> A
<module 'A' from 'http://example.com/a_repo/A/__init__.py'>
>>> B
<module 'B' from 'http://example.com/a_repo/B.py'>
>>> 

Any combination of packages and modules can be imported this way!

The [!] Warning was emitted by the HttpImporter object created for A, as it couldn't locate B, and passed control to the next Finder object, that happened to be the HttpImporter object created for B!

Debugging...

>>> from httpimport import *
>>>
>>> import logging
>>> logging.getLogger('httpimport').setLevel(logging.DEBUG)
>>>
>>> with github_repo('operatorequals','covertutils') :
...     import covertutils
...
FINDER=================
[!] Searching covertutils
[!] Path is None
[@] Checking if connection is HTTPS secure >
[@] Checking if in declared remote module names >
[@] Checking if built-in >
[@] Checking if it is name repetition >
[*]Module/Package 'covertutils' can be loaded!
LOADER=================
[+] Loading covertutils
[+] Trying to import as package from: 'https://raw.githubusercontent.com/operatorequals/covertutils/master//covertutils/__init__.py'
[+] Importing 'covertutils'
[+] Ready to execute 'covertutils' code
[+] 'covertutils' imported succesfully!
>>>

Beware: Huge Security Implications!

Using the httpimport with HTTP URLs is highly discouraged outside the localhost interface!

As HTTP traffic isn't encrypted and/or integrity checked (unlike HTTPS), it is trivial for a remote attacker to intercept the HTTP responses (via an ARP MiTM probably), and add arbitrary Python code to the downloaded packages/modules. This will directly result in Remote Code Execution to your current user's context! In other words, you get totally F*ed...

Preventing the disaster (setting httpimport.INSECURE flag):

>>> import httpimport
>>>
>>> # Importing from plain HTTP ...
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
[-] 'httpimport.INSECURE' is not set! Aborting...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "httpimport.py", line 302, in load
    raise ImportError("Module '%s' cannot be imported from URL: '%s'" % (module_name, url) )
ImportError: Module 'test_module' cannot be imported from URL: 'http://localhost:8000/'
>>> # ... Throws Error!
>>>
>>> # Importing from plain HTTP has to be DELIBERATELY enabled!
>>> httpimport.INSECURE = True
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
<module 'test_module' from 'http://localhost:8000//test_module.py'>
>>> # Succeeded!

You have been warned! Use HTTPS URLs with httpimport!

Minification

This project has started to suggest stager code for HTTP/S RATs made with covertutils. The Documentation for minifying and using httpimport for such purposes can be found here.

Further minification can be achieved by python-minifier, also available in PyPI. So a minified version can be obtained as follows:

pip install python-minifer    # the "pyminify" command
curl https://raw.githubusercontent.com/operatorequals/httpimport/master/httpimport.py | sed 's#log.*#pass#g' | grep -v "import pass" | pyminify - > httpimport_min.py

size reduction:

# Original Size Count
$ curl https://raw.githubusercontent.com/operatorequals/httpimport/0.7.1/httpimport.py |  wc 
[...]
504    1914   18876
# Minified Size Count
$ curl https://raw.githubusercontent.com/operatorequals/httpimport/0.7.1/httpimport.py | sed 's#log.*#pass#g' | grep -v "import pass" | pyminify - | wc 
[...]
177     936   12141

Contributors

  • ldsink - The RELOAD flag and Bug Fixes
  • lavvy - the load() function
  • superloach - Deprecation of imp module in Python3 in favour of importlib
  • yanliakos - Bug Fix
Owner
John Torakis
It is all about what puzzles we prefer to delve into
John Torakis
Pylexa - Artificial Assistant made with Python

Pylexa - Artificial Assistant made with Python Alexa is a famous artificial assistant used massively across the world. It is a substitute of Alexa whi

\_PROTIK_/ 4 Nov 03, 2021
Cute study buddy that helps you study with the Pomodoro technique!

study-buddy Cute study buddy that helps you study with the Pomodoro (or Animedoro) technique! Kirby The Kirby folder has a Kirby, pink-themed Pomodoro

Ethan Emmanuel 1 Jan 19, 2022
An app about keyboards, originating from the design of u/Sonnenschirm

keebapp-backend An app about keyboards, originating from the design of u/Sonnenschirm Setup Firstly, ensure that the environment for python is install

8 Sep 04, 2022
Материалы для курса VK Углубленный Python, весна 2022

VK Углубленный Python, весна 2022 Материалы для курса VK Углубленный Python, весна 2022 Лекции и материалы (слайды, домашки, код с занятий) Введение,

10 Nov 02, 2022
Calculadora-basica - Calculator with basic operators

Calculadora básica Calculadora com operadores básicos; O programa solicitará a d

Vitor Antoni 2 Apr 26, 2022
Structured, dependable legos for Starknet development.

cairomate • Structured, dependable legos for starknet development. Directory Structure contracts ├─ defi │ ├─ ChainlinkPriceOracle — "Simple price or

andreas 127 Nov 23, 2022
Stopmagic gives you the power of creating amazing Stop Motion animations faster and easier than ever before.

Stopmagic gives you the power of creating amazing Stop Motion animations faster and easier than ever before. This project is maintained by Aldrin Mathew.

Aldrin's Art Factory 67 Dec 31, 2022
This is the improvised version of Dobot Magician which can be implemented for Dobot M1

pydobotM1 This is the edited driver for Dobot M1 version of the original pydobot library intended for use with the Dobot Magician. Here's what you nee

Shaik Abdullah 2 Jul 11, 2022
Ssma is a tool that helps you collect your badges in a satr platform

satr-statistics-maker ssma is a tool that helps you collect your badges in a satr platform 🎖️ Requirements python = 3.7 Installation first clone the

TheAwiteb 3 Jan 04, 2022
Fabric mod where anyone can PR anything, concerning or not. I'll merge everything as soon as it works.

Guess What Will Happen In This Fabric mod where anyone can PR anything, concerning or not (Unless it's too concerning). I'll merge everything as soon

anatom 65 Dec 25, 2022
Virtual webcam that takes real webcam footage and replaces the background in order to have Virtual Backgrounds in MS Teams for Linux where the feature is unimplemented.

Background Remover The Need It's been good long while since Microsoft first released a Teams version for Linux and yet, one of Teams' coolest features

Dylan Turner 80 Dec 20, 2022
Arcpy Tool developed for ArcMap 10.x that checks DVOF points against TDS data and creates an output feature class as well as a check database.

DVOF_check_tool Arcpy Tool developed for ArcMap 10.x that checks DVOF points against TDS data and creates an output feature class as well as a check d

3 Apr 18, 2022
We'll be using HTML, CSS and JavaScript for the frontend

We'll be using HTML, CSS and JavaScript for the frontend. Nothing to install in specific. Open your text-editor and start coding a beautiful front-end.

Mugada sai tilak 1 Dec 15, 2021
It is Keqin Wang first project in CMU, trying to use DRL(PPO) to control a 5-dof manipulator to draw line in space.

5dof-robot-writing this project aim to use PPO control a 5 dof manipulator to draw lines in 3d space. Introduction to the files the pybullet environme

Keqin Wang 4 Aug 22, 2022
Coronavirus Tracker API

Coronavirus Tracker API Provides up-to-date data about Coronavirus outbreak. Includes numbers about confirmed cases, deaths and recovered. Support mul

Francisco Laguna 1 Oct 31, 2020
PyToQlik is a library that allows you to integrate Qlik Desktop with Jupyter notebooks

PyToQlik is a library that allows you to integrate Qlik Desktop with Jupyter notebooks. With it you can: Open and edit a Qlik app inside a Ju

BIX Tecnologia 16 Sep 09, 2022
Convert-Decimal-to-Binary-Octal-and-Hexadecimal

Convert-Decimal-to-Binary-Octal-and-Hexadecimal We have a number in a decimal number, and we have to convert it into a binary, octal, and hexadecimal

Maanyu M 2 Oct 08, 2021
Checkers Project Built Using Python

Checkers Project Built Using Python

Meekness Anyaeche 1 Nov 08, 2021
WordlistPasswordGenerator - Shuhfab Basheer

WordlistPasswordGenerator - Shuhfab Basheer Python wordlist generator MAINTAINER

1 Dec 31, 2021
addons to the turtle package that help you drew stuff more quickly

TurtlePlus addons to the turtle package that help you drew stuff more quickly --------------

1 Nov 18, 2021