Implementation of the idea discussed in #308. The tests don't pass yet and I should also implement additional tests for this, but hopefully this helps with the discussion.

So just discovered funny case, i have my pylons app served with proxy prefix middleware. i set in my ini file

[filter:proxy-prefix]
use = egg:PasteDeploy#prefix
prefix = /pref
...
[app:main]
filter-with = proxy-prefix

now entering an url server/pref/ works as expected but server/pref// causes waitress to trigger a 301 redirect to /pref/// that triggers another redirect and we end up in a loop.

good old Paste#http server works as expected in such case.

If we are in a trusted proxy situation whereby waitress is running behind nginx/haproxy/any other reverse proxy (whose IP matches what is configured, or if the socket we receive the connection on is a unix socket) then we want to pull a variety of information from the X-Forwarded-* headers or Forwarded which has superseded it.

The algorithm will pull X-Forwarded-* headers, unless there is a Forwarded header present and trusted_proxy_headers is set to just forwarded, in which case it takes priority.

By default we will use the proxy Forwarded header to set the following:

• wsgi.url_scheme (Proto)
• SERVER_NAME (Using Host param)
• SERVER_PORT (Using Proto param to set a default, overriden by Host param)
• HTTP_HOST (Using Host param, adding the SERVER_PORT as necessary)
• REMOTE_ADDR (Using For param, first For is used as it is client IP address)
• REMOTE_PORT (Using For param, as above, if a port is available)

This off course also means that Waitress now supports https://tools.ietf.org/html/rfc7239 's Forwarded header! Yay for being standards compliant.

Waitress also now knows how to clean up the headers and thus the WSGI environment, so that untrusted headers are no longer passed through to downstream WSGI applications using clear_untrusted_proxy_headers.

Closes https://github.com/Pylons/waitress/issues/207

My organization runs a Pyramid web application that we deploy to AWS Elastic Container Service (Fargate). Waitress serves the Pyramid application with nginx as a reverse proxy on a Debian buster image. We found upon taking waitress 2.1.0 we saw an immediate performance degradation. Specifically for response sizes above ~50kb (compressed by nginx), we see a ~5-10x slowdown on responses, resulting in numerous 504s for our various APIs. Upon rolling back to version 2.0.0, performance is restored. This is on Python 3.7.12.

I use locust to run performance tests on the web application, which allows me to quickly isolate and identify the root cause of such dramatic performance changes. All system level dependencies are identical (those installed by apt), as are other Python libraries.

Pre-update (waitress 2.0.0):

Response time percentiles (approximated in ms)
 Type     Name            50%    66%    75%    80%    90%    95%    98%    99%   99.9%  99.99%   100%
--------|------------|---------|------|------|------|------|------|------|------|------|------|------|
 All     Aggregated       750    980   1200   1400   1800   2300   4000   4600   6500   6500   6500 

Post update (waitress 2.1.0):

Response time percentiles (approximated in ms)
 Type     Name             50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100%
-------|------------|---------|------|------|------|------|------|------|------|------|------|------|
 All     Aggregated      780    1500   2600   5300  21000  57000  60000  61000  61000  61000  61000

This test was run on the same environment - only difference is the waitress version. Most routes that return small responses are not affected, but as soon as response size gets beyond a certain threshold the performance falls off a cliff.

Any thoughts why this might be? Something obvious I am missing? It is entirely possible there is a mistake or misconfiguration on my end, but such a slowdown is very suspicious and I believe isolated to this version.

Dockerfile

nginx.conf

As discussed in issue #204, the pull request adds the new parameter sockets to serve and create_server which can be used to create waitress instances using already existing and bound sockets, e.g. for socket activation with systemd or launchd.

The changes are tested with several additional test cases.

In addition to the code changes, it adds a new page to the documentation and a new entry to describe the new parameter sockets in the page for arguments. Adding the entry in arguments is done under the assumption that the next version will be 1.1.1 to supply the version number for versionadded.

The changes only add the parameter, they do not handle init system specific tasks like creating sockets from file descriptors because we want to avoid dependencies to init systems.

Closes #204

This change is 

I just might be wrong because if this indeed a race condition it should be breaking more things. Anyway, I got this exception (line numbers might be wrong due to debug statements):

Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/.../lib/python3.8/site-packages/aqt/mediasrv.py", line 89, in run
    self.server.run()
  File "/.../lib/python3.8/site-packages/waitress/server.py", line 323, in run
    self.asyncore.loop(
  File "/.../lib/python3.8/site-packages/waitress/wasyncore.py", line 285, in loop
    poll_fun(timeout, map)
  File "/.../lib/python3.8/site-packages/waitress/wasyncore.py", line 211, in poll
    r, w, e = select.select(r, w, e, timeout)
OSError: [Errno 9] Bad file descriptor

This error was extremely rare but since I was getting it while running tests I could just run a lot of them until one failed, which I did, and I think the problem is a follows.

1. First, thread Thread-1 that the app I'm testing is launching, one that runs waitress server, assembles the descriptor lists for select: https://github.com/Pylons/waitress/blob/603d2c12ce09fcd1be0b7a5734b4fca339820286/src/waitress/wasyncore.py#L154-L166

2. Then, thread waitress-0 deletes one of the channels, in my case it was <waitress.channel.HTTPChannel 127.0.0.1:54044 at 0x7f10ec052400>, and immediately closes the socket: https://github.com/Pylons/waitress/blob/603d2c12ce09fcd1be0b7a5734b4fca339820286/src/waitress/wasyncore.py#L460-L470

   Stack of waitiress-0 at the moment:

     File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap
       self._bootstrap_inner()
     File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
       self.run()
     File "/usr/lib/python3.8/threading.py", line 870, in run
       self._target(*self._args, **self._kwargs)
     File "/.../lib/python3.8/site-packages/waitress/task.py", line 84, in handler_thread
       task.service()
     File "/.../lib/python3.8/site-packages/waitress/channel.py", line 426, in service
       task.service()
     File "/.../lib/python3.8/site-packages/waitress/task.py", line 168, in service
       self.execute()
     File "/.../lib/python3.8/site-packages/waitress/task.py", line 451, in execute
       self.channel.write_soon(app_iter)
     File "/.../lib/python3.8/site-packages/waitress/channel.py", line 377, in write_soon
       (flushed, exception) = self._flush_exception(self._flush_some)
     File "/.../lib/python3.8/site-packages/waitress/channel.py", line 132, in _flush_exception
       return (flush(), False)
     File "/.../lib/python3.8/site-packages/waitress/channel.py", line 270, in _flush_some
       num_sent = self.send(chunk)
     File "/.../lib/python3.8/site-packages/waitress/wasyncore.py", line 479, in send
       self.handle_close()
     File "/.../lib/python3.8/site-packages/waitress/channel.py", line 317, in handle_close
       wasyncore.dispatcher.close(self)
     File "/.../lib/python3.8/site-packages/waitress/wasyncore.py", line 519, in close
       traceback.print_stack()
   

3. Then, thread Thread-1 is trying to see if the file descriptor of the socked closed above is writable, which leads to the the exception above: https://github.com/Pylons/waitress/blob/603d2c12ce09fcd1be0b7a5734b4fca339820286/src/waitress/wasyncore.py#L171-L177

Python 3.8.10, waitress 2.1.1, Ubuntu 20.04 LTS focal @ WSL2

hi, if waitress provide a way to specify public and private keys for ssl it would be cool using ssl_context, or ssl_context='adhoc' which werkzeug provide by generating a certificate.

Here is a suggestion to use the X_FORWARDED_PROTO on cloud based reversed proxies with no fixed ip.

I don't see any personal objection on allowing the wildcard if it is documented well enough for this particular case.

Cheers

We use waitress in devpi like this:

  serve(app, host=host, port=port, threads=50)

and on Windows got #138 with waitress 1.0. To work around that I tried to change to the following:

  serve(app, listen="%s:%s" % (host, port), threads=50)

But then I got ValueError: Invalid host/port specified.

This only happens on Windows, Linux and OS X is fine. For now we pinned to waitress<1.

With devpi-server I can now confirm a memory leak somehow related to waitress. I observed it on Debian and FreeBSD, but not on macOS. The last test I did was replacing waitress with cheroot and the leak is completely gone with that. I'm currently working on creating a minimal example for reproduction, but wanted to give a heads up in case someone else noticed something and started investigating. It is certainly possible it is related in how we use waitress in devpi-server, but I don't think we do anything special. I don't want to switch to cheroot, because it lacks some features and seems to be less performant.

This is something of a selfish pull request.

At my work, we often use waitress by itself without the likes of Paste acting as the runner as we generally delegate serving static assets and the like to the reverse proxy server sitting in front of the application. For applications like that, we use a simple command line runner specific to waitress, effectively a thin wrapper around waitress.serve. I wrote it as it simplified our deployment story quite a bit.

This pull request adapts that code to integrate with waitress properly. Just so as you know, this code and the original runner were written on my own time, not the company's time, so there are no copyright complications involved in merging it into waitress if you think it's worthwhile.

The test_gardenpath test fails on OpenIndiana because the string returned from strerror() is implementation specific.

Simple test case in C:

#include <stdio.h>
#include <string.h>
#include <errno.h>

int main(void)
{
        printf("%s\n", strerror(EPERM));
        return 0;
}

On linux it prints: Operation not permitted, but on OpenIndiana (and thus on all illumos based distros, and likely Solaris too) it prints: Not owner.

Here is the test failure:

________________________ Test__strerror.test_gardenpath ________________________

self = <tests.test_wasyncore.Test__strerror testMethod=test_gardenpath>

    def test_gardenpath(self):
>       self.assertEqual(self._callFUT(1), "Operation not permitted")

tests/test_wasyncore.py:1199:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.9/vendor-packages/teamcity/diff_tools.py:33: in _patched_equals
    old(self, first, second, msg)
E   AssertionError: 'Not owner' != 'Operation not permitted'
E   - Not owner
E   + Operation not permitted

According to setup.cfg the pytest-cover package is required for testing. This is unfortunate since pytest-cover was merged back to pytest-cov and is no longer maintained. Please switch waitress to use pytest-cov instead of pytest-cover.

Hi Folks, We have a BZ to get the fix done in 2.1.1 https://github.com/Pylons/waitress/commit/9e0b8c801e4d505c2ffc91b891af4ba48af715e0 to the old versions available in EPEL

This backport was done for RHEL7 via CVE-2022-24761.

I created this FORK with an initial backport to v1.4.4. I wonder if we can create a new tag as v1.4.5 including this fix, + some help to validate it

One particular user keeps submitting a request to my server that causes it to crash with an uncaptured python exception. Below is the request and corresponding error:

[2022-Jul-03 01:05] XXX.XX.XX.XXX  http /? 405 METHOD NOT ALLOWED
[2022-Jul-03 01:06] XXX.XX.XX.XXX GET http /nice ports,/Trinity.txt.bak? 404 NOT FOUND
[2022-Jul-03 01:06] XXX.XX.XX.XXX  http /? 405 METHOD NOT ALLOWED
ERROR:waitress:uncaptured python exception, closing channel <waitress.server.TcpWSGIServer listening XXX.XX.XXX.XX at XXXXXXXX> (<class 'OSError'>:[Errno 22] Invalid argument [/XXXX/XXXX/XXXX/waitress/wasyncore.py|read|104] /XXXX/XXXX/XXXX/waitress/wasyncore.py|handle_read_event|492] /XXXX/XXXX/XXXX/waitress/server.py|handle_accept|316] [/XXXX/XXXX/XXXX/waitress/server.py|set_socket_options|372])
WARNING:waitress:unhandled close event

My server is a Flask app running on a Mac OS with waitress 2.1.2. I glanced into the server.py code and it looks like an exception handler is supposed to catch this situation (at least for Linux), but it doesn't seem to be working (or more likely, I don't fully understand how it should work). The other odd thing is that the same request from the same user does not always trigger the error: sometimes the server continues even after receiving the 3 identical requests that precede the error message. I haven't yet figured out how to trigger the error myself, but it looks like the method is just an empty string (note the extra space after the IP address).

Thanks in advance!

this will help users to hide the server details from waitress generated error response with the help of ident value. Some organization don't want to expose there server details to user, since it will help the attackers perform attacks based on the known vulnerability of the server. So instead of showing generated by waitress, we can show a generic text.

See discussions: astropy/astropy#11091 pypa/pip#7413

[[email protected] waitress-2.1.2]$ grep -r pkg_resources
docs/conf.py:import pkg_resources
docs/conf.py:version = pkg_resources.get_distribution("waitress").version