As far as I know currently it's not possible to specify the password for the client side certificate you're using for authentication. This is a bit of a problem because you typically always want to password protect your .pem file which contains the private key. openssl won't even let you create one without a password.

I'm having a somewhat odd issue. A get() request seems to work fine with requests-2.5.1, but after upgrading to requests 2.5.2, the same URL leads to CERTIFICATE_VERIFY_FAILED. cc @mpharrigan

$ pip install requests==2.5.1
[ ... snip, installs just fine ...]

$ python -c 'import requests; print(requests.get("http://conda.binstar.org/omnia/linux-64/fftw3f-3.3.3-1.tar.bz2"))'
<Response [200]>

$ pip install requests==2.5.2
[ ... snip, installs just fine ...]

$ python -c 'import requests; print(requests.get("http://conda.binstar.org/omnia/linux-64/fftw3f-3.3.3-1.tar.bz2"))'
Traceback (most recent call last):
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 341, in _make_request
    self._validate_conn(conn)
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 762, in _validate_conn
    conn.connect()
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/site-packages/requests/packages/urllib3/connection.py", line 238, in connect
    ssl_version=resolved_ssl_version)
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/site-packages/requests/packages/urllib3/util/ssl_.py", line 256, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/ssl.py", line 364, in wrap_socket
    _context=self)
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/ssl.py", line 578, in __init__
    self.do_handshake()
  File "/Users/rmcgibbo/miniconda/envs/3.4.2/lib/python3.4/ssl.py", line 805, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:
[... snip ...]

At least for Python 3 -- charset_normalizer doesn't support Python2, so for that chardet is still used -- this means the "have chardet" path is also still tested.

Although using the (non-vendored) chardet library is fine for requests itself, but using a LGPL dependency the story is a lot less clear for downstream projects, particularly ones that might like to bundle requests (and thus chardet) in to a single binary -- think something similar to what docker-compose is doing. By including an LGPL'd module it is no longer clear if the resulting artefact must also be LGPL'd.

By changing out this dependency for one under MIT we remove all license ambiguity.

As an "escape hatch" I have made the code so that it will use chardet first if it is installed, but we no longer depend upon it directly, although there is a new extra added, requests[lgpl]. This should minimize the impact to users, and give them an escape hatch if charset_normalizer turns out to be not as good. (In my non-exhaustive tests it detects the same encoding as chartdet in every case I threw at it)

I've read https://github.com/psf/requests/pull/4115, https://github.com/psf/requests/issues/3389, and https://github.com/chardet/chardet/issues/36#issuecomment-768281452 so I'm aware of the history, but I hope that the approach in this PR will allow this to be merged, as right now, the Apache Software Foundation doesn't allow projects to depend upon LGPL'd code (this is something I'm trying to get changed, but it is a very slow process)

Example with pip 6.1.1 (but same with pip 8.1.1):

> pip install requests
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting requests
  Using cached requests-2.14.0-py2.py3-none-any.whl
  Exception:
  Traceback (most recent call last):
    File "D:\VEnvs\testpip\lib\site-packages\pip\basecommand.py", line 246, in main
      status = self.run(options, args)
    File "D:\VEnvs\testpip\lib\site-packages\pip\commands\install.py", line 342, in run
      requirement_set.prepare_files(finder)
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 345, in prepare_files
      functools.partial(self._prepare_file, finder))
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 290, in _walk_req_to_install
      more_reqs = handler(req_to_install)
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 557, in _prepare_file
      set(req_to_install.extras) - set(dist.extras)
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2758, in extras
      return [dep for dep in self._dep_map if dep]
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2781, in _dep_map
      self.__dep_map = self._compute_dependencies()
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2814, in _compute_dependencies
      common = frozenset(reqs_for_extra(None))
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2811, in reqs_for_extra
      if req.marker_fn(override={'extra':extra}):
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\_markerlib\markers.py", line 113, in marker_fn
      return eval(compiled_marker, environment)
    File "<environment marker>", line 1, in <module>
  NameError: name 'platform_system' is not defined

So, installing requests 2.14.0 implicitly requires pip >= 9.x. Should be at least in the release note as a disclaimer, or be fixed if it's not on purpose.

I am building a basic load generator and started running into file descriptor limits, I havent seen any documentation pertaining to how to release resources, so either I am doing it wrong and the docs need updated, or requests is leaking file descriptors somewhere (without support for keepalive I am slightly confused at why any files would be left open at all)

Here is the first issue. https://github.com/kennethreitz/requests/issues/2906

Python 3.5.1 (https://www.python.org/downloads/) Virtualenv 14.0.5 Mac OS X 10.11.3

First, I created a virtualenv and pip install requests[security]

Then I got

>>> from cryptography.hazmat.backends.openssl.backend import backend
>>> print(backend.openssl_version_text())
OpenSSL 1.0.2f  28 Jan 2016

which was what I expected.

Everything worked great for about an hour.

Then, some of my own scripts crashed which was normal. After that, when I try to run my script again, I got following exceptions

ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:645)

requests.packages.urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:645)

requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:645)

So I opened another Python console and

>>> requests.get("https://www.google.com")
<Response [200]>
>>> requests.get("https://www.telegram.org")
Traceback (most recent call last):
  File "VirtualenvPath/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "VirtualenvPath/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "VirtualenvPath/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 784, in _validate_conn
    conn.connect()
  File "VirtualenvPath/lib/python3.5/site-packages/requests/packages/urllib3/connection.py", line 252, in connect
    ssl_version=resolved_ssl_version)
  File "VirtualenvPath/lib/python3.5/site-packages/requests/packages/urllib3/util/ssl_.py", line 305, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 376, in wrap_socket
    _context=self)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 747, in __init__
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 983, in do_handshake
    self._sslobj.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/ssl.py", line 628, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "VirtualenvPath/lib/python3.5/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "VirtualenvPath/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "VirtualenvPath/lib/python3.5/site-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "VirtualenvPath/lib/python3.5/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "VirtualenvPath/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "VirtualenvPath/lib/python3.5/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "VirtualenvPath/lib/python3.5/site-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:645)
>>> 

So I rebooted, uninstalled all these libs and pip install them again. Everything worked again.

But after 1 hour or so, same exception again.

Related to #1083, perhaps. Standard requests.get() for this particular site/page https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html results in:

>>> import requests
>>> requests.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Using request-toolbelt's SSLAdapter to try various ssl versions, they all fail, it would seem... see following tracebacks.

TLSv1:

>>> adapter = SSLAdapter('TLSv1')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

SSLv3:

>>> adapter = SSLAdapter('SSLv3')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure

SSLv2:

>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)

Note the last one gives a Connection reset by peer error, which differs from the others, but I'm pretty sure SSLv2 isn't supported by the server anyhow.

For fun, I tried to pass through some more appropriate headers through on the last request as well:

>>> headers = {
...     'Accept': u"text/html,application/xhtml+xml,application/xml",
...     'User-Agent': u"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36",
...     'Accept-Encoding': u"gzip,deflate",
...     'Accept-Language': u"en-US,en;q=0.8"
... }
>>> adapter = SSLAdapter('SSLv2')
>>> s = requests.Session()
>>> s.mount('https://', adapter)
>>> s.get('https://docs.apitools.com/2014/04/24/a-small-router-for-openresty.html', headers=headers)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jaddison/.virtualenvs/techtown/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.apitools.com', port=443): Max retries exceeded with url: /2014/04/24/a-small-router-for-openresty.html (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)

No dice there either. Here's what the HTTPS connection info in Chrome on Mac looks like:

I'm not positive, but some googling indicates it's likely a cipher list issue, which is more urllib3, I think?

I tried to modify DEFAULT_CIPHER_LIST in pyopenssl, but started running into import errors. At this point it seemed like things were just broken, and there wasn't really a proper way to approach fixing this yet.

Version information: OSX Mavericks Python 2.7.5 OpenSSL 0.9.8y 5 Feb 2013 - (from python -c "import ssl; print ssl.OPENSSL_VERSION") requests 2.2.1 requests-toolbelt 0.2.0 urllib3 1.8

I'm getting a strange error when using Requests in Python 3.3 (other flavors of Python 3 do not get this error):

Traceback (most recent call last):
  File "C:\Python33\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 421, in urlopen
    body=body, headers=headers)
  File "C:\Python33\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 273, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Python33\lib\http\client.py", line 1049, in request
    self._send_request(method, url, body, headers)
  File "C:\Python33\lib\http\client.py", line 1087, in _send_request
    self.endheaders(body)
  File "C:\Python33\lib\http\client.py", line 1045, in endheaders
    self._send_output(message_body)
  File "C:\Python33\lib\http\client.py", line 890, in _send_output
    self.send(msg)
  File "C:\Python33\lib\http\client.py", line 828, in send
    self.connect()
  File "C:\Python33\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 104, in connect
    ssl_version=resolved_ssl_version)
  File "C:\Python33\lib\site-packages\requests\packages\urllib3\util.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Python33\lib\ssl.py", line 210, in wrap_socket
    _context=self)
  File "C:\Python33\lib\ssl.py", line 310, in __init__
    raise x
  File "C:\Python33\lib\ssl.py", line 306, in __init__
    self.do_handshake()
  File "C:\Python33\lib\ssl.py", line 513, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python33\lib\site-packages\requests\adapters.py", line 211, in send
    timeout=timeout
  File "C:\Python33\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 465, in urlopen
    raise MaxRetryError(self, url, e)
requests.packages.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='analysiscenter.veracode.com', port=443): Max retries exceeded with url: /api/2.0/getappbuilds.do (Caused by <class 'ConnectionResetError'>: [WinError 10054] An existing connection was forcibly closed by the remote host)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\tfs\Vertafore_TFSDev\CQ\python\veracode\pythontestscriptonation.py", line 33, in <module>
    print(v.get_app_builds())
  File "C:\tfs\Vertafore_TFSDev\CQ\python\veracode\apiwrapper.py", line 184, in get_app_builds
    {}
  File "C:\tfs\Vertafore_TFSDev\CQ\python\veracode\apiwrapper.py", line 57, in request
    r = requests.get(URL, params=data, auth=username_password)
  File "C:\Python33\lib\site-packages\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Python33\lib\site-packages\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python33\lib\site-packages\requests\sessions.py", line 354, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python33\lib\site-packages\requests\sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python33\lib\site-packages\requests\adapters.py", line 246, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='analysiscenter.veracode.com', port=443): Max retries exceeded with url: /api/2.0/getappbuilds.do (Caused by <class 'ConnectionResetError'>: [WinError 10054] An existing connection was forcibly closed by the remote host)
[Finished in 61.0s with exit code 1]

This automation has run for months on Python 3.2 and these errors have never occurred. I don't really know enough about requests to investigate this issue, but I'd be happy to help recreate the issue or debug if someone else can. Perhaps there's a bug in how Requests is handling HTTPS requests with Python 3.3? (Did 3.3 change how urllib works?...I don't know offhand...)

Again, I'm not getting any of these issues in Python 3.2 or Python 3.1. Please help! :)

We really need to automate this and have an official workflow and policy for security releases.

Fixes #1655

TODO:

• [x] Automate CA Bundle generation
• [ ] Establish documented security policy

See Also:

• PEP 453 Security Considerations
• kennethreitz/cert-playground

>>> r = requests.get('http://httpbin.org/ip')

>>> print r.request.dump()
REQUESTS/2.9.1 GET http://httpbin.org/ip
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.9.1

Please see the following example

_data = """callCount=1 
page=/internetsales/iSnapVehicle.xhtml?_gk=_c083894BB-4392-5290-8580-DE159D03B2B0_k9C5EF4F0-0269-3CC9-0994-633224433265

httpSessionId=
scriptSessionId=DD51AC690A8BC3BEDB1AEDF1B2A7A9DD575
c0-scriptName=PresentationRulesFacade
c0-methodName=execute
c0-id=0
c0-param0=string:VehicleForm c0-param1=string:VehicleForm%3Amake
c0-param2=string:%7B%22VehicleForm%3AdaysDriven%22%20%3A%20%221%22%2C%20%22VehicleForm%3AmilesDriven%22%20%3A%20%220%22%2C%20%22VehicleForm%3Ayear%22%20%3A%20%221998%22%2C%20%22VehicleForm%3AestimatedMileage%22%20%3A%20%22%22%2C%20%22VehicleForm%3AodometerReading%22%20%3A%20%22%22%2C%20%22VehicleForm%3Amake%22%20%3A%20%22AUDI%22%2C%20%22VehicleForm%3Ayear-txt%22%20%3A%20%22%22%2C%20%22VehicleForm%3Amake-txt%22%20%3A%20%22%22%2C%20%22VehicleForm%3Amodel-txt%22%20%3A%20%22%22%2C%20%22VehicleForm%3AmodelDesc%22%20%3A%20%22Other%22%2C%20%22VehicleForm%3Amodel%22%20%3A%20%22Other%22%2C%20%22VehicleForm%3Abodystyle%22%20%3A%20%22VAN%22%7D
c0-param3=boolean:false
batchId=327
"""

print len(_data)
_headers = {
    'referer' : 'https://sales2.geico.com/internetsales/iSnapVehicle.xhtml?_gk=_c083894BB-4392-5290-8580-DE159D03B2B0_k9C5EF4F0-0269-3CC9-0994-633224433265'
}
r = requests.post(
    url = 'https://sales2.geico.com/internetsales/dwr/call/plaincall/PresentationRulesFacade.execute.dwr',
    data = _data,
    headers = _headers
)

print r.request.headers
{'referer': 'https://sales2.geico.com/internetsales/iSnapVehicle.xhtml?_gk=_c083894BB-4392-5290-8580-DE159D03B2B0_k9C5EF4F0-0269-3CC9-0994-633224433265', 'Accept-Encoding': 'identity, deflate, compress, gzip', 'User-Agent': 'python-requests/0.7.3'}

In this example, the 'Content-Length' is missing.

Haven't got any spare time to try and patch the bug - as this was only a quick test to explore new libs.

Cal

[revised post 3 - multiple edits due to mistake on bug report]

Sometimes proper reason is missing in the Error thrown by the raise_for_status method.

Exception has occurred: HTTPError 405 Client Error: for url: https://bla.com

This is becasue the reason is an empty string on #Line1010

But it would be helpful to maybe try and deduct a reason, if not provided by the server (i.e. just a status code is provided)

My very ugly implementation of this would look like (just before Line1010)

from http import HTTPStatus    # somewhere on top 
.... 

if not reason: 
    for msg, code in HTTPStatus.__dict__.items(): 
        if code == self.status_code: 
            reason = msg 
            break

I am sure you guys will find a one liner.

I would expect a more clear error message as follows

Exception has occurred: HTTPError 405 Client Error: METHOD_NOT_ALLOWED for url: https://bla.com

Reproduction Steps

Simply try sending a POST message to a server that does not accept POST. The server should return a proper status_code back but the reason should/could be empty. This might not be an edge case and can be see in some popular servers (eg. I saw this in Jira REST API).

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null      
  },
  "charset_normalizer": {
    "version": "2.1.1"   
  },
  "cryptography": {      
    "version": ""        
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.9"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.28.1"
  },
  "system_ssl": {
    "version": "1010111f"
  },
  "urllib3": {
    "version": "1.26.13"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

python requests supports a no_proxy environment variable with a IPv4 CIDR e.g.

e.g

+ export http_proxy no_proxy
+ http_proxy=http://localhost
+ python -c 'import requests.utils ; print(requests.utils.should_bypass_proxies("http://1.2.3.4", None))'
False
+ no_proxy=1.2.3.0/24
+ python -c 'import requests.utils ; print(requests.utils.should_bypass_proxies("http://1.2.3.4", None))'
True

but doesn't support a IPv6 CIDR e.g.

+ http_proxy=http://localhost
+ python -c 'import requests.utils ; print(requests.utils.should_bypass_proxies("http://[1:2:3::4]", None))'
False
+ no_proxy=1:2:3::/64
+ python -c 'import requests.utils ; print(requests.utils.should_bypass_proxies("http://[1:2:3::4]", None))'
False

Expected Result

For proxy to be bypassed if a the url matches the IPv6 CIDR in no_proxy

Actual Result

IPv6 CIDR ignored

Reproduction Steps

as above

System Information

[[email protected] ~]$ python -m requests.help /usr/lib/python3.11/site-packages/requests/help.py:24: DeprecationWarning: 'urllib3.contrib.pyopenssl' module is deprecated and will be removed in a future release of urllib3 2.x. Read more in this issue: https://github.com/urllib3/urllib3/issues/2680 from urllib3.contrib import pyopenssl { "chardet": { "version": "5.1.0" }, "charset_normalizer": { "version": "2.1.0" }, "cryptography": { "version": "37.0.2" }, "idna": { "version": "3.3" }, "implementation": { "name": "CPython", "version": "3.11.1" }, "platform": { "release": "6.0.13-300.fc37.x86_64", "system": "Linux" }, "pyOpenSSL": { "openssl_version": "30000050", "version": "21.0.0" }, "requests": { "version": "2.28.1" }, "system_ssl": { "version": "30000050" }, "urllib3": { "version": "1.26.12" }, "using_charset_normalizer": false, "using_pyopenssl": true }

       except MaxRetryError as e:
            if isinstance(e.reason, ConnectTimeoutError):
                # TODO: Remove this in 3.0.0: see #2811
                if not isinstance(e.reason, NewConnectionError):
                    raise ConnectTimeout(e, request=request)

            if isinstance(e.reason, ResponseError):
                raise RetryError(e, request=request)

            if isinstance(e.reason, _ProxyError):
                raise ProxyError(e, request=request)

            if isinstance(e.reason, _SSLError):
                # This branch is for urllib3 v1.22 and later.
                raise SSLError(e, request=request)

            raise ConnectionError(e, request=request)

When using adapter and urllib3....Retry I don't catch the documented exception "MaxRetryError"

Errors will be wrapped in MaxRetryError unless retries are disabled, in which case the causing exception will be raised.

This seems to have been addressed https://github.com/psf/requests/issues/4389, https://github.com/psf/requests/issues/5639 and https://github.com/psf/requests/issues/1198, although not as straightforward.

The question is why is Requests rethrowing the enclosed Exception, making it impossible to use urllib3 logic of the Retry class, where errors can be retried until a maximum number and then the MaxRetryError exception is raised.

This is additionally confusing because it's not documented.

The history of a requests contains a self-reference to the history owner. The history will be endless.

Expected Result

If I traverse recursive the complete history of a requests, this will be a finally graph. The history of a request is a tree without cycles.

R1 (history: 2)
     R2 (no history)
     R3 (no history)

Actual Result

If I traverse recursive the complete history of a requests, the program breaks with recursive error. RecursionError: maximum recursion depth exceeded while calling a Python object The history contains a self-reference to the history owner.

The history of a request is a graph with a cycle.

R1 (history: 2)
     R2 (no history)
          R3 (history: 1)
               R3 (history: 1)
                    R3 (history: 1)
                         R3 (history: 1)
                              R3 (history: 1)
                                   ....

id=140537834271072 history=2
	index=0 id=140537834079136
	index=1 id=140537834080960
id=140537834079136 history=0
id=140537834080960 history=1
	index=0 id=140537834080960
id=140537834080960 history=1
	index=0 id=140537834080960
id=140537834080960 history=1
	index=0 id=140537834080960
id=140537834080960 history=1
	index=0 id=140537834080960

....


id=140537834080960 history=1
	index=0 id=140537834080960
id=140537834080960 history=1
	index=0 id=140537834080960
Traceback (most recent call last):
  File "/Users/andreas/PycharmProjects/cce/main.py", line 12, in <module>
    history(requests.get('https://coord.info/GC8T8E8'))
  File "/Users/andreas/PycharmProjects/cce/main.py", line 9, in history
    history(item)
  File "/Users/andreas/PycharmProjects/cce/main.py", line 9, in history
    history(item)
  File "/Users/andreas/PycharmProjects/cce/main.py", line 9, in history
    history(item)
  [Previous line repeated 993 more times]
  File "/Users/andreas/PycharmProjects/cce/main.py", line 5, in history
    print(f"id={id(r)} history={len(r.history)}")
RecursionError: maximum recursion depth exceeded while calling a Python object

Process finished with exit code 1

Reproduction Steps

import requests


def history(r):
    print(f"id={id(r)} history={len(r.history)}")
    for index, item in enumerate(r.history):
        print(f"\tindex={index} id={id(item)}")
    for index, item in enumerate(r.history):
        history(item)


history(requests.get('https://coord.info/GC8T8E8'))

System Information

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.1.1"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.2"
  },
  "platform": {
    "release": "21.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.28.1"
  },
  "system_ssl": {
    "version": "1010109f"
  },
  "urllib3": {
    "version": "1.26.13"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Context: uploading files to AWS S3 via requests. Files may be empty. When attempting to upload an empty file, a response of 501: Not implemented is received. This seems to be because, when content length is 0, a Transfer-Encoding: chunked header is automatically added. S3 does not support the encoding.

Expected Result

When an empty file PUT is attempted via requests.put, a Content-Length: 0 header should be set on the request. A Transfer-Encoding: chunked header should not be set automatically.

Actual Result

When an empty file PUT is attempted via requests.put, a Transfer-Encoding: chunked header is set. No Content-Length header is set. If a Content-Length: 0 header is set explicitly, the Transfer-Encoding: chunked header is still set.

Reproduction Steps

If AWS account, bucket, credentials for principal with PutObject permissions aren't available:

import requests

file = open(<path_to_empty_file>, "rb")
session = requests.session()

link = "http://httpbin.org/put"

# using requests.put yields the same results. Using session.put here to examine request
# with S3 instead of httpbin.org, the call below results in a 501. Note the request headers
res_no_CL = session.put(link, data=file, headers={"Content-Type": "text/plain"})
print(res_no_CL.request.headers)
# with S3 instead of httpbin.org, the call below results in a 501. With httpbin.org it returns
# ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
# so can't examine headers, but I assume behaviour is consistent
res_explicit_CL = session.put(link, data=file, headers={"Content-Type": "text/plain", "Content-Length": "0"})
print(res_explicit_CL.request.headers)

If AWS prerequisites above are available:

import requests
import boto3

s3 = boto3.client('s3')
file = open(<path_to_empty_file>, "rb")
session = requests.session()

link = s3.generate_presigned_url(ClientMethod="put_object", Params={"Bucket": "<your_bucket>", "Key": "<path_in_bucket>", "ContentType": "text/plain"})

# using requests.put yields the same results. Using session.put here to examine request
# the call below results in a 501. Note the request headers
res_no_CL = session.put(link, data=file, headers={"Content-Type": "text/plain"})
print(res_no_CL.request.headers)
# the call below results in a 501. Note the request headers
res_explicit_CL = session.put(link, data=file, headers={"Content-Type": "text/plain", "Content-Length": "0"})
print(res_explicit_CL.request.headers)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.1.1"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.4"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.5"
  },
  "platform": {
    "release": "5.10.102.1-microsoft-standard-WSL2",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.28.1"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.26.12"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}