Application of pip2pi, pypiserver and Apache in PIP local source configuration

pip Introduce ：
pip yes python More popular and powerful package management tools ,pip Is about to replace setuptools(setuptools It can also be seen as easy_install A substitute for , install setuptools after
Will use easy_install Command line to install packages ), however ,pip It's still built on setuptools Based on , install pip Must be installed before setuptools.

pip It's powerful ：
Usage:   
  pip [options]

Commands:
 install                    Install packages.
 uninstall                  Uninstall packages.
 freeze                     Output installed packages in requirements format.
 list                       List installed packages.
 show                       Show information about installed packages.
 search                     Search PyPI for packages.
 wheel                      Build wheels from your requirements.
 help                       Show help for commands.

General Options:
  -h,--help                 Show help.
 --isolated                 Run pip in an isolated mode, ignoring environment variables anduser configuration.
  -v,--verbose              Give more output. Option is additive, and can be used up to 3times.
  -V,--version              Show version and exit.
  -q,--quiet                Give less output.
  --log               Path to a verbose appending log.
  --proxy            Specify a proxy in the form [user:[email protected]]proxy.server:port.
  --retries        Maximum number of retries each connection should attempt (default 5times).
  --timeout            Set the socket timeout (default 15 seconds).
  --exists-action    Defaultaction when a path already exists: (s)witch, (i)gnore, (w)ipe,(b)ackup.
  --trusted-host   Mark this host as trusted,even though it does not have valid or any HTTPS.
  --cert              Path to alternate CA bundle.
  --client-cert       Path to SSL client certificate, a single file containing theprivate key and the certificate in PEM format.
  --cache-dir          Store the cache data in.
 --no-cache-dir             Disable the cache.
  --disable-pip-version-check
                             Don't periodically check PyPI to determine whether a new version ofpip is available for download. Implied with --no-index.

pip Default source (index-pypi=http://pypi.python.org/simple),pip install and pipserach The command defaults to pip.python.org Search and download pypi package
pypi A powerful feature of is ： Write all the dependent packages and their versions of the application into one requirements.txt In the file of , And then through pipinstall -r requirements.txt
According to requirements.txt Installation sequence in pypi package .pip During installation , The default installation files are cached in HOME/.cache/pip Inside the directory , Change the directory and return to production 1,2,3....,a,b,c
The subdirectory of is used to store cache files . therefore , If you want to reinstall again, you don't have to go through the Internet , By the following order ：pip install pypi_nmae--src $HOME/.cache perhaps pip install -r requirements.txt --src$HOME/.cache
This can be done locally cache Directory for installation .

[[email protected] http]# pwd
/root/.cache/pip/http
[[email protected] http]# ll
total 0
drwx------.  9 root root 62 Jul 27 16:22 0
drwx------.  6 root root 38 Jul 27 16:23 1
drwx------. 13 root root 94 Jul 27 16:23 2
drwx------. 11 root root 78 Jul 27 16:22 3
drwx------. 13 root root 94 Jul 27 16:22 4
drwx------. 10 root root 70 Jul 27 16:23 5
drwx------. 12 root root 86 Jul 27 16:22 6
drwx------. 11 root root 78 Jul 27 16:22 7
drwx------.  9 root root 62 Jul 27 16:22 8
drwx------. 13 root root 94 Jul 27 16:23 9
drwx------. 12 root root 86 Jul 27 16:21 a
drwx------. 13 root root 94 Jul 27 16:23 b
drwx------. 11 root root 78 Jul 27 16:23 c
drwx------. 11 root root 78 Jul 27 16:23 d
drwx------.  9 root root 62 Jul 27 16:23 e
drwx------. 11 root root 78 Jul 27 16:17 f

pip Construction of local source ：

pip The idea of building a local source is ： To download pypi package , Then designate pip Local sources
Here are two methods ：
1： Create a new directory to store pypi package , Such as /root/pypi
2. Merge all dependencies into one file ： Such as find . -name requirements.txt -exec cat {} \;>pip_requirement.all
3. Start the download pypi Package to the specified directory ：
#!/bin/bash

PIP_REQUIRE="pip_requirement.all"

while read LINE
do
  if [[ $LINE =~ ^[a-zA-Z] ]]
  then
    echo$LINE
    pip install$LINE -d /root/pypi  # Download only and do not install
  fi
done < $PIP_REQUIRE
This step will put pip_requirement.all Download all dependent packages specified in to /root/pypi Directory . During the download process , Default back to the official website pypi.python.org
To download ,GFW Resulting in slow download , You can use the domestic Douban source to download , Very fast relative , The specific methods ：
edit .pip/pip.conf file , Add the following statement ：
[global]
trusted-host = pypi.douban.com
index-url = http://pypi.douban.com/simple

Or alicloud source ：
[global]
trusted-host=mirrors.aliyun.com
index-url=http://mirrors.aliyun.com/pypi/simple/

sentence trusted-host=xxx Because pip The new version is that http The host is not untrusted , Don't let you connect .

4. Simpler batch pypi The package download method is to install pip2pi Tools ：pip install pip2pi
Download a single package ：pip2tgz /save/to/path pypi_nmae
Bulk download ：pip2tgz /save/to/path -r requirements.txt
In this way, the requirements.txt All inside pypi Download the package to the specified directory ( If the Apache Deploy pip Source , Recommendation award this path is set to Apache The default home directory for ：
/var/www/html/pypi, Such as pip2tgz /var/www/html/pypi -r/root/openstack/keystone/requirements.txt)

5. Generate pypi Of index：dir2pi --normalize-package-names/var/www/html/pypi. The order will be in pypi Generated inside simple file ,simple The file names in are standardized
packeage name ：
[[email protected] pypi]# cd /var/www/html/pypi/simple
[[email protected] simple]# ll
total 8
drwxr-xr-x. 2 root root   62 Jul28 17:53 aioeventlet
drwxr-xr-x. 2 root root   50 Jul28 17:53 alembic
drwxr-xr-x. 2 root root   57 Jul28 17:53 amqp
drwxr-xr-x. 2 root root   50 Jul28 17:53 anyjson
drwxr-xr-x. 2 root root   64 Jul28 17:53 appdirs
drwxr-xr-x. 2 root root   65 Jul28 17:53 argparse
drwxr-xr-x. 2 root root   46 Jul28 17:53 babel
drwxr-xr-x. 2 root root   60 Jul28 17:53 bandit
drwxr-xr-x. 2 root root   60 Jul28 17:53 bashate
drwxr-xr-x. 2 root root   67 Jul28 17:53 beautifulsoup4
drwxr-xr-x. 2 root root   67 Jul28 17:53 cachetools
drwxr-xr-x. 2 root root   47 Jul28 17:53 cffi
drwxr-xr-x. 2 root root   54 Jul28 17:53 contextlib2
simple Each subdirectory in the directory will generate a corresponding index.html file , The contents of the document are as follows ： WebOb-1.4.1.tar.gz


6. To configure Apache The server .
Apache The server is mainly configured /etc/httpd/conf/httpd.conf file , As far as this article is concerned , Do not change the default DocumentRoot"/var/www/html"
directory Field add Options Indexes MultiViews FollowSymlinks
Indexes: Indicate that there is no index.html It is allowed to list files and directories ;
MultiViews： Allow multiple views to list according to the content
FollowSymlinks： Allow symbolic connections

hold deny The place is changed to ：Require all granted

restart Apache：systemctl restart httpd.service
see Apache Monitoring situation ：netstat -ntl |grep 80
Browser input ：http://host_ip/pypi/simple , The page will display all in the form of hyperlinks pypi package ：





Click anywhere to download ！

7. Set up pip Local sources .
You can install packages in the following form ：pip install --index-url http://host_ip/pypi/simple -rrequirements.txt
But it's more troublesome , You can modify ~/.pip/pip.conf To specify the default local pip Source

[global]
trusted-host = host_ip
index-url = http://host_ip/pypi/simple

such , That is, local pip Source installation ：pip install -r requirements.txt
You will find that it is very fast , Faster than watercress ！ Besides , If you need to download in bulk pypi package , as follows ：pip uninstall -rrequirements.txt -y
-y The parameter indicates that you can uninstall without confirmation .

Another kind of construction pip The way of local source is to use pypiserver Package to build ：pip install pypiserver
pypi-server -h
pypi-server [OPTIONS] [PACKAGES_DIRECTORY...]
  start PyPI compatible package server servingpackages from
  PACKAGES_DIRECTORY. If PACKAGES_DIRECTORY is notgiven on the
  command line, it uses the default~/packages.  pypiserver scans this
  directory recursively for packages. It skipspackages and
  directories starting with a dot. Multiplepackage directories can be
  specified.

pypi-server understands the following options:

  -p, --port PORT
    listen onport PORT (default: 8080)

  -i, --interface INTERFACE
    listen oninterface INTERFACE (default: 0.0.0.0, any interface)

  -a, --authenticate (UPDATE|download|list),...
   comma-separated list of (case-insensitive) actions toauthenticate
    (requiresgiving also the -P option). For example to password-protect
    packageuploads & downloads while leaving listings public, give:
     -a update,download.
    By default,only 'update' is password-protected.

  -P, --passwords PASSWORD_FILE
    use apachehtpasswd file PASSWORD_FILE to set usernames & passwords
    used forauthentication of certain actions (see -a option).

  --disable-fallback
    disableredirect to real PyPI index for packages not found in the
    localindex

  --fallback-url FALLBACK_URL
    for packagesnot found in the local index, this URL will be used to
    redirect to(default: http://pypi.python.org/simple)

  --server METHOD
    use METHODto run the server. Valid values include paste,
    cherrypy,twisted, gunicorn, gevent, wsgiref, auto. The
    default isto use "auto" which chooses one of paste, cherrypy,
    twisted orwsgiref.

  -r, --root PACKAGES_DIRECTORY
    [deprecated]serve packages from PACKAGES_DIRECTORY

  -o, --overwrite
    allowoverwriting existing package files

  --welcome HTML_FILE
    uses theASCII contents of HTML_FILE as welcome message response.

  -v
    enableverbose logging;  repeate for moreverbosity.

  --log-file
    writelogging info into this FILE.

  --log-frmt
    the loggingformat-string.  (see `logging.LogRecord` classfrom standard python library)
    [Default:%(asctime)s|%(levelname)s|%(thread)d|%(message)s]

  --log-req-frmt FORMAT
    aformat-string selecting Http-Request properties to log; setto  '%s' to see them all.
    [Default:%(bottle.request)s]
    
  --log-res-frmt FORMAT
    aformat-string selecting Http-Response properties to log; setto  '%s' to see them all.
    [Default:%(status)s]

  --log-err-frmt FORMAT
    aformat-string selecting Http-Error properties to log; setto  '%s' to see them all.
    [Default:%(body)s: %(exception)s
%(traceback)s]

  --cache-control AGE
    Add"Cache-Control: max-age=AGE, public" header to packagedownloads.
    Pip 6+ needsthis for caching.


pypi-server -h
pypi-server --help
  show this help message

pypi-server --version
  show pypi-server's version

pypi-server -U [OPTIONS] [PACKAGES_DIRECTORY...]
  update packages in PACKAGES_DIRECTORY. Thiscommand searches
  pypi.python.org for updates and shows a pipcommand line which
  updates the package.

The following additional options can be specified with -U:

  -x
    execute thepip commands instead of only showing them

  -d DOWNLOAD_DIRECTORY
    downloadpackage updates to this directory. The default is to use
    thedirectory which contains the latest version of the package to
    beupdated.

  -u
    allowupdating to unstable version (alpha, beta, rc, dev versions)

Visit https://pypi.python.org/pypi/pypiserver for moreinformation.

pypi-server The configuration of is relatively simple , No more configuration is required apache service .
Sum up ,pip The local source can be configured in two ways ：
1.pip2pi + apache
2.pip2pi + pypiserver

The first way is to use pip2pi Of pip2tgz -r requirements.txt Command batch download python package , And then use it dir2pipackges_dir Generate index, Then configure
Apache The server , adopt http Visit local repos To build a local source ;

The second chapter is about downloading python The package part also uses pip2pi Command to download , Then store python The package directory is used as pypiserver The bag warehouse , By starting pypiserver Service
Listening port ：pypi-server -i host_ip -p port packges_dir1 packges_dir12 ...packges_dir1n &
adopt netstat -ntl You can view the port listening status , adopt lsof -i:port You can see the application name and process listening to the port ID

pypi-server No need to use dir2pi Generate index, Just take what you need python Download the package locally , Then start pypiserver Just listen ！

pypi-server have access to --fall-back Parameter specifies when the required package cannot be found ,pip Where to search for packages , Such as ：
pypi-server -i 192.168.142.10 -p 8080 --fallback-urlhttp://192.168.142.10/pypi/simple/ pypi &

pypiserver monitor 192.168.142.10：8080 port , Use at the same time url=http://192.168.142.10/pypi/simple/ As a candidate index-url（ This column is the previously configured
Apacheindex）, If in ~/pypi The required... Was not found in the directory python package , Then to http://192.168.142.10/pypi/simple Middle search .

start-up pypiserver After monitoring , For ease of installation , Yes ~.pip/pip.conf To configure ：
[global]

##############douban mirrors###############
#trusted-host=pypi.douban.com
#index-url = http://pypi.douban.com/simple

##############pip2pi&apache local repos##############
#trusted-host=192.168.142.10
#index-url = http://192.168.142.10/pypi/simple

################pypiserver local repos################
trusted-host=192.168.142.10
index-url = http://192.168.142.10:8080/simple/


When it's done , that will do pip install -r requirements.txt Installation ！

notes ： If you want to stop pypiserver, need lsof -i:8080 See the process ID , then kill PID！