I've finally managed to refactor the proxy support changes from requests lib into urllib3.

Test results:

 [email protected] [~/development/urllib3] > python test/test_proxy.py
TEST Proxy: http://192.168.56.1:8888
HTTPS OK
HTTP OK

---
TEST Proxy: socks4://192.168.56.1:8889
HTTPS OK
HTTP OK

---
TEST Proxy: socks5://192.168.56.1:8889
HTTPS OK
HTTP OK

---
TEST: No proxy via PoolManager
HTTPS OK
HTTP OK

---
TEST: No proxy via connection_from_url
HTTPS OK
HTTP OK

Could one of the core devs please review this patch and make sure you are happy with it. Once signed off, I will go ahead and do a documentation update as well (want to make sure everyone's happy first).

I've used as many of the suggestions from other tickets as possible, but please let me know if I have missed anything or if it needs to be changed.

My sincere apologies it has taken so long to get this done, spare time is like gold dust :(

This references the following discussions:

https://github.com/shazow/urllib3/pull/56 https://github.com/kennethreitz/requests/pull/478 https://github.com/kennethreitz/requests/issues/362

Others involved: @senko @wolever @kennethreitz @shazow @Anorov

We should follow the lead of Chromium. We should deprecate support for use of common names when a certificate doesn't provide a subjectAltName.

This will bring us into compliance with the >10 year old RFC 2818

This PR adds all the select methods to a new util module called wait. Uses the same ordering used by selectors.py in the standard library to determine which selector should be used. Also factored in a way to allow easy patching and testing. Similar to #998 but uses a different way to break up the different methods to wait for I/O events. Also adds tests to exercise the selectors which can probably be extended.

This Pull Request resolves #50 adding HTTPS (CONNECT) proxy support to ProxyManager with minimal API and code changes.

A short summary of the changes:

• ProxyManager is a subclass of PoolManager now and its logic has been changed completely, though its constructor still accepts HTTPConnectionPool as the main argument so it's backwards compatible
• Tiny change to VerifiedHTTPSConnection.connect() that introduces CONNECT tunneling support (copied from HTTPConnection.connect() code)
• HTTPSConnectionPool._new_conn() has been changed, but its logic for non-proxied connections stays the same
• HTTPSConnectionPool._new_conn() actually establishes the connection for proxied requests by calling HTTPSConnection.connect(), otherwise Host: header is set improperly due to httplib logic. As stated above, the behavior for non-proxied requests hasn't changed
• tornado-proxy has been added for tests
• TornadoServerThread has been changed a bit - now it's possible to run multiple server threads in a single test case
• Added proxy tests
• Docs updated

All tests run fine with Python 2.6, 2.7 and 3.2 on Mac OS 10.7.5, except for sporadic timeouts and SSL errors that occur against the unpatched library as well (and should be fixed separately). The code doesn't introduce any new regressions.

Possible TODOs and improvements:

• Proxy authentication (though that could be another libraries' responsibility, like requests)
• When accepting proxy_pool as the first argument for ProxyManager, copy all pool properties and use them for other pool instances generation or even use it as the pool for HTTP proxy requests
• Usage of HTTPConnectionPool and, especially, HTTPSConnectionPool against proxy isn't really clear and requires some external logic - much like ProxyManager.connection_from_host()

We (mostly @pquentin and I) have been working on a proof of concept for adding pluggable async support to urllib3, with the hope of eventually getting this into the upstream urllib3. It's reached the point where there's still lots of missing bits, but there's an end-to-end demo working and we don't see any major obstacles to getting everything else working, so I wanted to start getting feedback from the urllib3 maintainers about whether this looks like something you'd be interested in eventually merging, and what it would take to get there.

Demo

So: hi! Check it out – a single py2.py3 wheel that keeps the classic synchronous API, and on python 3.6+ it can also run in async mode on both Trio and Twisted:

# Demo from commit 1ca67ee53e18f823d0cb in the python-trio/urllib3 bleach-spike branch
$ curl -O https://vorpus.org/~njs/tmp/async-urllib3-demo.zip
$ unzip async-urllib3-demo.zip
$ cd async-urllib3-demo
$ ls
sync-demo.py
async-demo.py
urllib3-2.0.dev0+bleach.spike.proof.of.concept.dont.use-py2.py3-none-any.whl

$ virtualenv -p python3.6 py36-venv
$ py36-venv/bin/pip install trio twisted[tls] urllib3-2.0.dev0+bleach.spike.proof.of.concept.dont.use-py2.py3-none-any.whl

$ py36-venv/bin/python sync-demo.py
--- urllib3 using synchronous sockets ---
URL: http://httpbin.org/uuid
Status: 200
Data: b'{\n  "uuid": "a2c28245-47b8-4a50-b64c-da09d27bf626"\n}\n'

$ py36-venv/bin/python async-demo.py
--- urllib3 using Trio ---
URL: http://httpbin.org/uuid
Status: 200
Data: b'{\n  "uuid": "dab50c1a-1b20-483f-903e-fe74494629f2"\n}\n'

--- urllib3 using Twisted ---
URL: http://httpbin.org/uuid
Status: 200
Data: b'{\n  "uuid": "72196b66-7caa-40dd-9c7c-af65bb4f7fb6"\n}\n'

$ virtualenv -p python2 py2-venv
$ py2-venv/bin/pip install urllib3-2.0.dev0+bleach.spike.proof.of.concept.dont.use-py2.py3-none-any.whl
$ py2-venv/bin/python sync-demo.py
--- urllib3 using synchronous sockets ---
URL: http://httpbin.org/uuid
Status: 200
Data: '{\n  "uuid": "2a2fce90-d853-4940-b111-0969f92b7678"\n}\n'

Things that (probably) don't work yet include: HTTPS, timeouts, proper connection reuse, using python 2 to run setup.py bdist_wheel, running the test suite. OTOH some non-trivial things do work, like chunked transfer encoding, and there's at least code in there for proxy support and handling early responses from the server.

what sorcery is this

This is based on @Lukasa's "v2" branch, so it's using h11. (Goodbye httplib :wave:.) The v2 branch is currently stalled in a pretty half-finished state (e.g. the I/O layer is a bunch of raw select loops open-coded everywhere that IIRC don't work; I'm sure @Lukasa would have cleaned it up if he had more time); we cleaned all that up and made it work. The major new code is here: https://github.com/python-trio/urllib3/blob/bleach-spike/urllib3/_async/connection.py

And then we made the low-level I/O pluggable – there's a small-ish ad hoc API providing the set of I/O operations that urllib3 actually needs, and several implementations using different backends. Note that the I/O backend interface is internal, so we can adjust it as needed; there's no attempt to provide a generic abstract I/O interface that would work for anyone else. The code for the backends is here: https://github.com/python-trio/urllib3/tree/bleach-spike/urllib3/_backends

Then we started adding async/await annotations to the rest of urllib3's code. That gave us a version that could work on Trio and Twisted. But wait, we also want to support python 2! And everyone using the existing synchronous API on python 3 too, for that matter. But we don't want to maintain two copies of the code. Unfortunately, Python absolutely insists that async APIs and synchronous APIs be loaded from different copies of the code; there's no way around this. (Mayyybe if we dropped python 2 support and were willing to maintain a giant shim layer then it would be possible, but I don't think either of those things is true.)

Solution: we maintain one copy of the code – the version with async/await annotations – and then a little script maintains the synchronous copy by automatically stripping them out again. It's not beautiful, but as far as I can tell all the alternatives are worse.

Currently the async version (source of truth) lives in urllib3/_async/, and then setup.py has the code to automatically generate urllib3/_sync/... at build time. (This script should probably get factored out into a separate project.) The script is not at all clever; you can see the exact set of transformations in the file, but basically it just tokenizes the source, deletes async and await tokens, and renames a few other tokens like __aenter__ → __enter__. There's no complicated AST manipulation and comments are preserved. Then urllib3/__init__.py imports things from urllib3._sync and (if it's running on a new enough Python) urllib3._async.

The resulting dev experience is sort of half-way between that of a pure-python project and one with a C extension: instead of an edit/run cycle, you need to do an edit/compile/run cycle. But you still end up with a nice py2.py3-none-any wheel at the end, so you don't need to stress out about providing binary builds for different platforms, and the builds are extremely fast because it's just some shallow text manipulation, not invoking a C compiler.

Oh, and for backcompat we also added some shim files like urllib3/connectionpool.py, that just re-export stuff from the corresponding files in urllib3/_sync/..., since these submodules are documented as part of the API. This also has the benefit of making it easier to avoid accidentally exporting things in the future; urllib3's public API is perhaps larger than it should be.

Importantly, this basic strategy would also work for libraries that use urllib3, so it provides a path forward for higher-level projects like requests, botocore, etc. to provide dual sync/async APIs while supporting python 2 + multiple async backends.

Backwards compatibility

So far this is looking surprisingly good. Switching to h11 means losing most of urllib3.connection, since that's directly exposing httplib APIs. In async mode we can't support lazily loading response.data, but that's fine, and we can still support it in sync mode if we want. Right now the branch is keeping the "close" operations synchronous, but we might want to switch to making them async, because it might make things easier for HTTP/2 later. If we do switch to async close, then that will force some broader changes – in particular RecentlyUsedContainer assumes that it can close things from __setitem__ and __delitem__, which can't be made async. It wouldn't be hard to rewrite RecentlyUsedContainer, but right now technically it's a public API.

I'm not sure about some of the more obscure stuff like urllib3.contrib. In particular urllib3.contrib.pyopenssl, urllib3.contrib.socks, and urllib3.contrib.securetransport seem difficult to handle in a I/O-backend-agnostic way – maybe we could hack them to keep working on the sync backend only? And I don't know what to think about the appengine support. Maybe it's fine because it doesn't actually use urllib3 internals at all?

But overall, my impression is that we could keep quite a high degree of backwards compatibility if we want. Perhaps we'd want to break things for other reasons if we're doing a v2, but that's a separate discussion.

Questions

Basically at this point it seems clear that this approach can work. And what's left is just straightforward work. So:

• Does this seem like the direction we want urllib3 to go?

• If so, then how would you like that process to go?

I know @Lukasa is wary of the code generation approach, but the alternatives seem to be (a) maintaining a ton of redundant code, (b) not supporting async in urllib3, in which case the community will... end up maintaining a ton of redundant code as each I/O library separately implements their own fake copy of requests. And I don't believe that we have enough HTTP experts in the community to do that well; I think there are huge benefits to concentrating all our maintainers on a single library if we can.

CC: @kennethreitz @markrwilliams

@sethmlarson had a cute idea: ur//ib3, kinda inspired by Mozilla's moz://a.

Would be fun to make stickers for future PyCons and whatnot.

Maybe someone is inspired to take a shot at this?

Or maybe we can convince a designer of one of the corporate sponsors to take a stab at it? :D

HPKP is cool, and I think we should be able to do it.

This issue tracks us (me?) spiking out how best to do it. I believe we can do it, and continue to be Python HTTP security wizards.

People who may want to watch this issue: @sigmavirus24 @reaperhulk @dstufft @tiran.

This branch contains the most substantial chunk of work I have ever done on urllib3: the beginnings of an effort to remove our dependency on httplib.

Removing our dependency on httplib has been a long-held desire of the urllib3 project. In fact, it's so desired that #776 (our wishlist of features for v2.0) includes in a section entitled "unlikely longshots" the phrase "Get rid of httplib dependence".

This branch does exactly that, by bringing the http.client implementation from Python 3.5 that we were already used to interacting with into our codebase and then ripping its guts out and replacing them with @njsmith's h11 library.

As you can see from the commit log, my work on this began back in October, but I've sat on it for the past two months in order to get it to a place where I had something worth showing. Now I finally do. On my machine, a run of tox -e py27 passes all tests, albeit with incomplete test coverage:

py27 runtests: commands[1] | nosetests
SSS....................S....................................................................................................................................S.....................................................S........................................................................................................................................................................................................................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS...............................S..S..............................S................................................................................
Name                      Stmts   Miss  Cover   Missing
-------------------------------------------------------
urllib3                      20      0   100%   
urllib3._collections        148      0   100%   
urllib3.connection          515    103    80%   80-82, 158, 162, 175, 229, 261, 285, 295-300, 306-338, 341-364, 368-372, 375, 385-392, 396-399, 408, 449, 453, 471, 567, 571-572, 602-611, 615, 651-652, 657, 729-732, 736-740, 755, 781, 830, 855-856, 868-869, 905-906, 972-973, 985, 1005
urllib3.connectionpool      240      1    99%   256
urllib3.contrib               0      0   100%   
urllib3.contrib.socks        35      0   100%   
urllib3.exceptions           73      2    97%   216, 219
urllib3.fields               62      0   100%   
urllib3.filepost             32      0   100%   
urllib3.poolmanager         124      0   100%   
urllib3.request              32      0   100%   
urllib3.response            232      1    99%   453
urllib3.util                  1      0   100%   
urllib3.util.connection      47      0   100%   
urllib3.util.request         37      0   100%   
urllib3.util.response        19      5    74%   54-65
urllib3.util.retry          121      0   100%   
urllib3.util.selectors      295     45    85%   101, 360-423
urllib3.util.ssl_            64      0   100%   
urllib3.util.timeout         48      0   100%   
urllib3.util.url             97      0   100%   
urllib3.util.wait            14      1    93%   21
-------------------------------------------------------
TOTAL                      2256    158    93%   
----------------------------------------------------------------------
Ran 604 tests in 13.646s

OK (SKIP=42)
___________________________________ summary ____________________________________
  py27: commands succeeded
  congratulations :)

Initially, my goal was to do this in the least-disruptive way possible: that's why I began by trying to replicate the logic of httplib inside urllib3 but on top of h11. I would have succeeded in that goal if I'd been able to avoid rewriting any tests in any substantive way.

Unfortunately, as you can see from this change, I did not succeed at that. In particular, it was simply not possible to continue with some behaviours that urllib3 had previously promised, such as maintaining header case. I also had to bend over backwards to preserve some other behaviours urllib3 users have expected, such as reporting chunk boundaries, which is probably not something we should ever have promised to do.

From my perspective, then, I think that if we want to continue this work it needs to be considered a breaking change for urllib3. Any change this substantial simply cannot be done in a non-breaking way. A huge number of things will change: our tolerance of errors (we're much stricter now because h11 is much more insistent about being spec-compliant), our wire format (h11 does some case normalization on header field names), and maybe even some of our output formats.

This is also still very much a work in progress. There are TODO statements everywhere. The code is poorly factored: it's a wacky in-between state between trying to maintain the httplib abstraction and a total rejection of it in favour of something much clearer. A whole lot of our code that worked around httplib flaws is no longer necessary and can be removed, but hasn't yet been. A better underlying implementation that doesn't perform socket writes would also be possible, as would some refactoring of code I wrote to pull out common features. And the Python 3 tests don't pass, mostly because of the fact that right now this new branch emits headers as bytestrings rather than unicode strings.

However, at this point it's no longer feasible for me to work on this branch alone. Further progress requires actual governance decisions, and I am not qualified to make those on my own, nor should I. urllib3 has a great collection of core maintainers and regular contributors whose opinions and work I value, and would like to solicit to help me. As I see it, we need to answer the following questions:

1. Is this worth doing? Do we want to be rid of httplib so badly that we will perform a massive codebase rewrite to get rid of it?
2. If so, are we happy with using h11?
   1. If we are, do we depend on it using pip, or vendor it? How does our decision affect Requests?
3. Should this be the beginning of a v2.0 branch that will contain other bits of work from #776?
   1. If it should, how many breakages do we want to roll into one change? Do we want to go all in and leave a massive codebase change?
4. What pieces of functionality do we want to retain from the old codebase? In particular, do we want:
   1. To still have stream() expose chunk boundaries?
   2. To return str header field names and values on Python 3?
   3. To return different types for header field names and values on Python 3?
   4. To implement buffered I/O in the low-level primitives?
   5. To maintain the weird two-level Response structure where we have a urllib3 Response object that contains a httplib Response object within it?
5. How would we want to manage a rollout of this functionality?
6. How do we want to review it?
7. What kind of timeline do we want for releasing it?
8. If we decide to go ahead with a v2.0, should we freeze the current master branch for everything except bug/security fixes until we complete v2.0?

When @shazow originally made me interim lead maintainer, he told me that I had complete authority because he could always undo any change I made that he didn't like. I think this particular change is an exception: I don't think I have any degree of "lead maintainer" authority on this. We're going to need @shazow to express some opinions too.

While I'm here, I'd also like to tag a few community members that are particularly likely to be interested in this proposal:

/cc @njsmith @durin42 @dstufft

(Re: issue #236)

First go at implementing this. Not 100% sure of the scope so there might be some features missing/extraneous items, would like to start discussion regardless.

Is it normal that the update to version 1.25 makes kodi start up much slower? With version 1.22 it takes about 20 seconds to start up my rpi with kodi and all the add-ons. After the latest update of a couple of says ago to 1.25.1 and also with 1.25.2 which I installed from zip it takes about 45 seconds to start up with all the add-ons. Is this to be expected and if so will the slow down only in the startup or can it also be in the use of kodi and addons?

Thank you

This is currently very rough and early. I want to make sure I'm on the right track before continuing.

Outstanding questions:

1. What functionality should be covered by the connection pool? (retries, etc.)
2. Should I just inherit from TestConnectionPool and null out the tests that aren't applicable to GAE instead of duplicating code?
3. How should this connection class be wired in? On GAE Sandbox environments w/o sockets, I assume this should be default.
4. What exact edge cases should this class be aware; e.g, where does URLFetch behavior differ from what's expected of httplib/sockets? Do the tests already cover this?

[_shazow edit: Fixes #664]_

Bumps ossf/scorecard-action from 2.1.1 to 2.1.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps actions/setup-python from 4.3.1 to 4.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Subject

When the Retry-After header contains a float, for example "0.503", then parse_retry_after raises an invalid header error.

Environment

OS Windows-10-10.0.19045-SP0 Python 3.10.3 urllib3 1.26.13

Steps to Reproduce

I cant really give an easy way to reproduce... the tedious way: generate python code with openapi-generator from this api https://github.com/SpaceTradersAPI/api-docs/tree/2.0.0-rc.2 hit any endpoint often enough to trigger the ratelimit which should be 2 requests/sec

EDIT: pull my repo change the main() in api.py a bit to send enough requests and see the ratelimit apply https://github.com/feba66/spacetraders-sdk

Expected Behavior

parse the float correctly and wait the correct time before retrying

Actual Behavior

the regex match returns false because a dot is present and then tries to parse the date from "0.503" which fails

possible fix

change the regex to allow for a float and change the conversion from int(retry_after) to float(retry_after)

If a chunked response is truncated exactly before sending a new chunk (that is, before the first byte of the chunk size), urllib3 throws InvalidChunkLength(got length b'\\n', 0 bytes read). This is confusing as no chunk length was sent (a valid chunk length line would also be \r\n-terminated and we didn't see those either).

If the chunk length is the empty string, send a different error message.