gunicorn 'Green Unicorn' is a WSGI HTTP Server for UNIX, fast clients and sleepy applications.

Last update: Jan 01, 2023

Overview

Gunicorn

Gunicorn 'Green Unicorn' is a Python WSGI HTTP Server for UNIX. It's a pre-fork worker model ported from Ruby's Unicorn project. The Gunicorn server is broadly compatible with various web frameworks, simply implemented, light on server resource usage, and fairly speedy.

Feel free to join us in #gunicorn on Freenode.

Documentation

The documentation is hosted at https://docs.gunicorn.org.

Installation

Gunicorn requires Python 3.x >= 3.5.

Install from PyPI:

$ pip install gunicorn

Usage

Basic usage:

$ gunicorn [OPTIONS] APP_MODULE

Where APP_MODULE is of the pattern $(MODULE_NAME):$(VARIABLE_NAME). The module name can be a full dotted path. The variable name refers to a WSGI callable that should be found in the specified module.

Example with test app:

$ cd examples
$ gunicorn --workers=2 test:app

Contributing

See our complete contributor's guide for more details.

License

Gunicorn is released under the MIT License. See the LICENSE file for more details.

Comments

Sockets

Service is running on a kubernetes pod, and out of nowhere and without any specific causes, it happens off and on:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
    req = six.next(parser)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
    self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
    super(Request, self).__init__(cfg, unreader)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
    unused = self.parse(self.unreader)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
    self.headers = self.parse_headers(data[:idx])
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
    remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected
[2018-11-04 17:57:55 +0330] [31] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle
    req = six.next(parser)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/parser.py", line 41, in __next__
    self.mesg = self.mesg_class(self.cfg, self.unreader, self.req_count)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 181, in __init__
    super(Request, self).__init__(cfg, unreader)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 54, in __init__
    unused = self.parse(self.unreader)
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 230, in parse
    self.headers = self.parse_headers(data[:idx])
  File "/usr/local/lib/python3.6/site-packages/gunicorn/http/message.py", line 74, in parse_headers
    remote_addr = self.unreader.sock.getpeername()
OSError: [Errno 107] Transport endpoint is not connected

Investigation help wanted

opened by andjedani 87

CRITICAL WORKER TIMEOUT when running Flask app

It seems there have been already several reports related to [CRITICAL] WORKER TIMEOUT error but it just keeps popping up. Here is my issue.

I'm running this Flask hello world application.

from flask import Flask
application = Flask(__name__)

@application.route('/')
def hello_world():
    return 'Hello, World!'

The gunicorn command is this one:

gunicorn -b 0.0.0.0:5000 --log-level=debug hello

And this is the console output:

[2018-06-05 14:56:21 +0200] [11229] [INFO] Starting gunicorn 19.8.1
[2018-06-05 14:56:21 +0200] [11229] [DEBUG] Arbiter booted
[2018-06-05 14:56:21 +0200] [11229] [INFO] Listening at: http://0.0.0.0:5000 (11229)
[2018-06-05 14:56:21 +0200] [11229] [INFO] Using worker: sync
[2018-06-05 14:56:21 +0200] [11232] [INFO] Booting worker with pid: 11232
[2018-06-05 14:56:21 +0200] [11229] [DEBUG] 1 workers
[2018-06-05 14:56:32 +0200] [11232] [DEBUG] GET /
[2018-06-05 14:56:57 +0200] [11232] [DEBUG] Closing connection. 
[2018-06-05 14:57:16 +0200] [11232] [DEBUG] GET /
[2018-06-05 14:57:47 +0200] [11229] [CRITICAL] WORKER TIMEOUT (pid:11232)
[2018-06-05 14:57:47 +0200] [11232] [INFO] Worker exiting (pid: 11232)
[2018-06-05 14:57:47 +0200] [11324] [INFO] Booting worker with pid: 11324

Can you please clearly explain why do I get the error and if it's expected in this example? How do I fix it or if it's an expected behavior why critical error then?

Investigation unconfirmed

opened by bigunyak 83

Gunicorn sync workers time out on docker + AWS

Using gunicorn 19.4.5 inside a docker image, which is running on Ubuntu 14.04 LTS on AWS, I see constant worker timeouts when running with >1 sync worker and the default 30s timeout period. This may be related to #588 or #942, although this is a significantly newer version of gunicorn so I couldn't tell for sure.

Circumstantial evidence over several runs seems to suggest that one of the workers is not affected, and it's only the N-1 remaining workers that keep failing and restarting. However, I was only able to get a few data points that suggest this, so don't take it as a strong signal.

Here are things I checked:

Do the issues happen outside of docker, or when using the same docker image, but not on AWS? No, I was only able to replicate the issue with docker on AWS.
Does the application take >30s to initialize? No, the application finishes initializing in under 1 second.
Does the application receive a request that causes it to hang? The issue popped up without any requests being made to the application.
Does the application have broken initialization code that confuses gunicorn? The issue popped up when running gunicorn in the same configuration (with >1 sync worker), but with a different application. This new application runs without problems on gunicorn when using gevent workers.
Does docker's filesystem fail to update the ctime? See the below log -- the ctime seems to usually advance fine. Docker's filesystem doesn't allow stat -L on deleted files, but preventing gunicorn from unlinking the file also results in stat -L showing that ctime is being updated as usual.

I added some code to workertmp.py to log the read ctime on each check, and here is one such log (interleaved between all workers):

[2016-01-29 00:21:38 +0000] [3238] [INFO] Starting gunicorn 19.4.0
[2016-01-29 00:21:38 +0000] [3238] [INFO] Listening at: http://0.0.0.0:5000 (3238)
[2016-01-29 00:21:38 +0000] [3238] [INFO] Using worker: sync
[2016-01-29 00:21:38 +0000] [3243] [INFO] Booting worker with pid: 3243
[2016-01-29 00:21:38 +0000] [3244] [INFO] Booting worker with pid: 3244
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026899.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026899.03
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026900.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026899.03
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026900.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026899.03
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026900.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026899.03
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026900.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026899.03
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026904.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026899.03
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026904.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026905.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026904.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026905.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026904.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026905.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026904.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026905.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026904.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026909.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026910.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026909.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026910.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026909.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026910.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026909.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026910.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026909.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026914.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026933.73
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026934.74
FOOBAR: modify /tmp/wgun[2016-01-29 00:22:25 +0000] [3238] [CRITICAL] WORKER TIMEOUT (pid:3243)
[2016-01-29 00:22:25 +0000] [3243] [INFO] Worker exiting (pid: 3243)
[2016-01-29 00:22:25 +0000] [3330] [INFO] Booting worker with pid: 3330
icorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunic54026934.74
FOOBAR: modify /tmp/wgun[2016-01-29 00:22:25 +0000] [3238] [CRITICAL] WORKER TIMEOUT (pid:3243)
[2016-01-29 00:22:25 +0000] [3243] [INFO] Worker exiting (pid: 3243)
[2016-01-29 00:22:25 +0000] [3330] [INFO] Booting worker with pid: 3330
icorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time:fy] [3238] [CRITICAL] WORKER TIMEOUT (pid:3243)
[2016-01-29 00p/2:25 +0000] [3243] [INFO] Worker exiting (pid: 3243)
[2016-0ic29 00:22:25 +0000] [3330] [INFO] Booting worker with pid: 33myicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorntiyjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt45ime: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026945.82
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026946.01
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026946.01
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026946.01
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8corn-myjLwI time:fy] [3238] [CRITICAL] WORKER TIMEOUT (pid:32BA)
[2016-01-29 00p/2:25 +0000] [3243] [INFO] Worker exiting (pdify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
FOOBAR: modify /tmp/wgunicorn-Mw64T1 timeodify /tmp/wgunicorn-myjLwI time: 1454026964.74
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026965.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026965.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026965.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026965.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026969.74
FOO[2016-01-29 00:22:59 +0000] [3238] [CRITICAL] WORKER TIMEOUT (pid:3330)
[2016-01-29 00:22:59 +0000] [3330] [INFO] Worker exiting (pid: 3330)
icorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026935.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026939.74
FOOBAR: modify /tmp/wgunicorn-LwI time: 1454026965.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 tI time: 1454026940.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026940.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026940.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026940.0
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026944.74
FOOBAR: modify /tmp/wgunicorn-RhAFmt time: 1454026915.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.0
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026945.8
[2016-01-29 00:22:59 +0000] [3396] [INFO] Booting worker with pid: 3396
BAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjL26949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjL26949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjL26949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjL26949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjL26949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
FOOBAR: modify /tmp/wgunicorn-myjL26949.74
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
FOOBAR: modify /tmp/w79.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026979.97
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
80.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454029.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454029.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454029.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026979.95
FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
 WORKER TIMEOUT (pid:3396)
 [2016-01-29 00:23:31 +0000] [3396] [INFO] Worker exiting (pid: 3396)
 BAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026970.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026974.74
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw64T1 time: 1454026949.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454026975.0
 FOOBAR: modify /tmp/wgunicorn-Mw6nicorn-k3uZLy time: 1454026980.16
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027005.0
 FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027010.0
 FOOBAR: modify /tmp/wgunicorn-k3uZLy time: 1454026980.16
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027010.0
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027011.08
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027011.28
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027011.28
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027011.28
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmprn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-Znicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027011.06
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027028.98
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027030.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027030.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wguicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027030.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027030.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wguicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027030.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027030.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wguicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027030.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn--ZmVaVS time: 1454027014.74
 FOOBAR: modify /tmp/wgunicorn-myjLwI time: 1454027035.0
 FOOBAR: modify /tmp/wgunicorn-ZmVaVS time: 1454027014.74
 FOOBAR: modify /t[2016-01-29 00:24:05 +0000] [3238] [CRITICAL] WORKER TIMEOUT (pid:3453)
 [2016-01-29 00:24:05 +0000] [3453] [INFO] Worker exiting (pid: 3453)

Feature/IPC Platform/Docker - Bugs -

opened by obi1kenobi 78

"Booting worker" is looping infinitely despite no exit signals

I'm trying to get gunicorn set up on Docker. It works well locally, and the production image is exactly the same as the local image, but I'm getting this strange behaviour on the production Docker engine:

ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [INFO] Starting gunicorn 19.7.1
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [DEBUG] Arbiter booted
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [INFO] Using worker: sync
ml-server_1     | [2017-12-11 13:18:50 +0000] [8] [INFO] Booting worker with pid: 8
ml-server_1     | [2017-12-11 13:18:50 +0000] [1] [DEBUG] 1 workers
ml-server_1     | Using TensorFlow backend.
ml-server_1     | [2017-12-11 13:18:54 +0000] [11] [INFO] Booting worker with pid: 11
ml-server_1     | Using TensorFlow backend.
ml-server_1     | [2017-12-11 13:18:58 +0000] [14] [INFO] Booting worker with pid: 14
ml-server_1     | Using TensorFlow backend.
ml-server_1     | [2017-12-11 13:19:02 +0000] [17] [INFO] Booting worker with pid: 17
ml-server_1     | Using TensorFlow backend.

It looks like gunicorn is booting workers every 4-5 seconds, despite no apparent error messages or exit signals. This behaviour continues indefinitely until terminated.

Is it possible that a worker can exit without logging anything to stderr/stdout, or for the arbiter to spawn workers infinitely?

Since they are the same docker image, they are running exactly the same code, on exactly the same architecture, so I'm really confused what this could be (bug?). Any help greatly appreciated!

Improvement help wanted

opened by bensalilijames 66

detecting scheme=='https' is broken if bound to unix domain socket

commit b07532be752668be5eb5dbd0a8303abf5c219c99 "Forbid contradictory secure scheme headers" broke "request.scheme" and "wsgi.url_scheme" when gunicorn is bound to unix domain socket. When gunicorn is bound to tcp socket everything works fine.

When used with nginx in front of gunicorn 19.8, nginx is passing HTTP_X_FORWARDED_PROTO == https, but in gunicorn/http/message.py in line remote_addr = self.unreader.sock.getpeername() in case of unix domain socket remote_addr is empty string and then secure_scheme_headers is empty dict and scheme ends up being "http" although the web server is https-only and passed all the headers.

I'm using gevent worker with latest everything on python 3.6 but I don't think it matters.

opened by zt-initech 57
Gunicorn freeze

Gunicorn 19.1.1 Django 1.7.1 nginx 1.0.10

After upgrading to the latest gunicorn I am having weird problem (gateway timeout 504) with backend running Django + PostgreSQL. It hangs after few days. I see gunicorn (master + workers) processes running fine, but no traffic is being served.

I am using damontools package for running gunicorn with command:

exec setuidgid $USER gunicorn -b ${TCP} -w 3 --timeout=60 --max-requests=1000 site.wsgi 2>&1

NO entries in daemontools/ django logs. Restarting gunicorn (svc -d service && svc -u service) helps.

Has similar problems some time ago, but upgrading gunicorn resolved the problem. After few months of stable serving, I upgraded gunicorn and the problem appeared again.
Feedback Requested - Bugs -

opened by rtpm 56

gunicorn 19.x causing problems with django-1.7.1 and gevent

Hello,

I'm seeing this error with gunicorn 19.x (tested with 19.0 and 19.1.1) using the gevent worker. I'm using django-1.7.1.

this is the exception:

Traceback:
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  87.                 response = middleware_method(request)
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/contrib/auth/middleware.py" in process_request
  34.         if user and hasattr(user, 'get_session_auth_hash'):
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/utils/functional.py" in inner
  224.             self._setup()
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/utils/functional.py" in _setup
  357.         self._wrapped = self._setupfunc()
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/contrib/auth/middleware.py" in <lambda>
  23.         request.user = SimpleLazyObject(lambda: get_user(request))
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/contrib/auth/middleware.py" in get_user
  11.         request._cached_user = auth.get_user(request)
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/contrib/auth/__init__.py" in get_user
  151.         user_id = request.session[SESSION_KEY]
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py" in __getitem__
  49.         return self._session[key]
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/contrib/sessions/backends/base.py" in _get_session
  175.                 self._session_cache = self.load()
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/contrib/sessions/backends/db.py" in load
  21.                 expire_date__gt=timezone.now()
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/models/manager.py" in manager_method
  92.                 return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py" in get
  351.         num = len(clone)
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py" in __len__
  122.         self._fetch_all()
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py" in _fetch_all
  966.             self._result_cache = list(self.iterator())
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/models/query.py" in iterator
  265.         for row in compiler.results_iter():
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in results_iter
  700.         for rows in self.execute_sql(MULTI):
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
  784.         cursor = self.connection.cursor()
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/backends/__init__.py" in cursor
  163.         self.validate_thread_sharing()
File "/home/sieve/sitroom.sieve.com.br/shared/env/local/lib/python2.7/site-packages/django/db/backends/__init__.py" in validate_thread_sharing
  515.                 % (self.alias, self._thread_ident, thread.get_ident()))

Exception Type: DatabaseError at /
Exception Value: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140049505195808 and this is thread id 61130224.

Here is my gunicorn.conf.py:

##
# Gunicorn config at 
# Managed by Chef - Local Changes will be Nuked from Orbit (just to be sure)
##

# What ports/sockets to listen on, and what options for them.
bind = "127.0.0.1:9300"

# The maximum number of pending connections
backlog = 2048

# What the timeout for killing busy workers is, in seconds
timeout = 300

# How long to wait for requests on a Keep-Alive connection, in seconds
keepalive = 2

# The maxium number of requests a worker will process before restarting
max_requests = 1024

# Whether the app should be pre-loaded
preload_app = True

# How many worker processes
workers = 16

# Type of worker to use
worker_class = "gevent"

# The granularity of error log outputs.
loglevel = "error"

The error occurs every time, just accessing any URL of the django app (that depends on any model, of course) triggers this error. Any ideas? Since changing to gunicorn-18.0 solves the problem, I'm assuming that gunicorn-19.x is doing something different.

If you need any additional information, let me know and I will post it here.

For now I'm using 18.0.

Thanks a lot.

bug :( Feature/Worker Feature/Core Thirdparty/Gevent Deploy Investigation - Bugs -

opened by daltonmatos 53

gunicorn 21
meta issue to track changes needed to release the version 21. After the talk I gave at the FOSDEM I thought it would be interesting to track the different tasks we need for the new version. Feedback and help is more than welcome.

tasks

[ ] release imsg

[ ] add regression test to the supervision

[ ] replace the temporary file notification system by imsg

[ ] add/increase tests for the logging (if needed)

[ ] handle logging via a separate process coupled with imsg

[ ] add new Proc class

[ ] add HTTP 2 support

[x] remove all python2 compatibility bits (PR: #1764)

Project release
opened by benoitc 49
v20 issue: Failed to find application object 'create_app()' in 'app'
I had neglected to pin my version of gunicorn, and the run command started breaking this morning when I re-deployed my app and it automatically upgraded to 20.0.

Downgrading my version of gunicorn back to 19.9 fixed the issue.

This is the command I'm using to run my app:

gunicorn 'app:create_app()' --workers 4 --threads 4 --bind 0.0.0.0:$PORT

The error is:

Failed to find application object 'create_app()' in 'app'
bug :( Feedback Requested Feature/App Investigation
opened by tjwaterman99 47
eventlet worker: ALREADY_HANDLED -> WSGI_LOCAL

Eventlet 0.30.3+ removed wsgi.ALREADY_HANDLED breaking public API in just a patch version increase. Sorry.

Issue with ALREADY_HANDLED: https://github.com/eventlet/eventlet/issues/543 Solution with WSGI_LOCAL: https://github.com/eventlet/eventlet/pull/544

It's recommended to use eventlet>=0.31.0 if one uses websockets, because older versions are vulnerable to DoS attack. https://github.com/eventlet/eventlet/security/advisories/GHSA-9p9m-jm8w-94p2

CI failed in pylint checks on lines I didn't touch.

opened by temoto 46
gaiohttpy.py invalid syntax warning (19.0.0)
Thanks for the new release!

Just pip installed gunicorn 19.0.0, it installs all right, but there's the following warning during installation (Python 2.7.6):

File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/gaiohttp.py", line 67 yield from self.wsgi.close() ^ SyntaxError: invalid syntax

Thanks,
opened by hyao 46
Question regarding the expected behavior of graceful shutdown
Hi

I'm working on critical service running on gunicorn+flask with the following configurations Python 3.9.0 latest gunicorn latest flask worker :gthread threads : 1 workers : 4-20 depends on deployed server

lately I wanted to add graceful shutdown to our application, so I used on_exit hook to register the service in discovery service as 'DOWN'

All works fine but the listening port is closed before on_exit invoke so when I send SIGTERM what actually happens is :

1.listening port closed (service still registers as up in discovery service) 2.Arbiter close listening socket 3.waiting graceful time out 4.kill workers 5.on_exit runs

So what happens is I still get requests while 1-4 happening which can be lost because only in 5 I register my service as 'DOWN'.

As far as I know when parent process close same socket (fd) as child have, the socket should close only if reference count == 0 and I wonder why the reference count is 1 on the listening socket before closing the socket ? I know it supposed to be 1(master) + num of workers but in gunicorn case it's always 1.

So is it intentional ? I couldn't figure out from sock.py and I think parent process should only be responsible for his own socket.

below the code in the arbiter

def halt(self, reason=None, exit_status=0): """ halt arbiter """ self.stop() self.log.info("Shutting down: %s", self.master_name) if reason is not None: self.log.info("Reason: %s", reason) if self.pidfile is not None: self.pidfile.unlink() self.cfg.on_exit(self) sys.exit(exit_status) def stop(self, graceful=True): """\ Stop workers :attr graceful: boolean, If True (the default) workers will be killed gracefully (ie. trying to wait for the current connection) """ unlink = ( self.reexec_pid == self.master_pid == 0 and not self.systemd and not self.cfg.reuse_port ) sock.close_sockets(self.LISTENERS, unlink) self.LISTENERS = [] sig = signal.SIGTERM if not graceful: sig = signal.SIGQUIT limit = time.time() + self.cfg.graceful_timeout # instruct the workers to exit self.kill_workers(sig) # wait until the graceful timeout while self.WORKERS and time.time() < limit: time.sleep(0.1) self.kill_workers(signal.SIGKILL)

Thanks
opened by eranseer 0

Trouble with gunicorn BaseApplication and threading.Thread

Hey, I have some threading/multiprocessing issues if I run gunicorn from a python file using BaseApplication. I extended your example application to somehow reproduce the issue.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
# An example of a standalone application using the internal API of Gunicorn.
#
#   $ python standalone_app.py
#
# This file is part of gunicorn released under the MIT license.
# See the NOTICE for more information.

import multiprocessing

import gunicorn.app.base

from queue import Queue
import threading
import atexit

q = Queue()
workers = []


@atexit.register
def stop_worker():
    for worker in workers:
        print("Trying to stop a worker")
        q.put_nowait(False)
        print(f"queue size: {q.qsize()}")
    for worker in workers:
        worker.join()
    print("All workers are finished")


class Worker(threading.Thread):

    def __init__(self, i, q):
        super().__init__()
        self.i = i
        self.q = q

    def run(self):
        while True:
            print(f"Worker {self.i} started")
            job = self.q.get()
            if not job:
                print(f"Worker {i} is done")
                return
            print(f"Got job {job}")


for i in range(3):
    w = Worker(i, q)
    w.start()
    workers.append(w)


def number_of_workers():
    #return (multiprocessing.cpu_count() * 2) + 1
    return 1


def handler_app(environ, start_response):
    response_body = b'Works fine'
    status = '200 OK'

    response_headers = [
        ('Content-Type', 'text/plain'),
    ]

    start_response(status, response_headers)

    return [response_body]


class StandaloneApplication(gunicorn.app.base.BaseApplication):

    def __init__(self, app, options=None):
        self.options = options or {}
        self.application = app
        super().__init__()

    def load_config(self):
        config = {key: value for key, value in self.options.items()
                  if key in self.cfg.settings and value is not None}
        for key, value in config.items():
            self.cfg.set(key.lower(), value)

    def load(self):
        return self.application


if __name__ == '__main__':
    options = {
        'bind': '%s:%s' % ('127.0.0.1', '8080'),
        'workers': number_of_workers(),
    }
    StandaloneApplication(handler_app, options).run()

(venv) [email protected]:gunicorn python examples/standalone_app.py
Worker 0 started
Worker 1 started
Worker 2 started
[2022-12-28 23:20:24 +0100] [304121] [INFO] Starting gunicorn 20.1.0
[2022-12-28 23:20:24 +0100] [304121] [INFO] Listening at: http://127.0.0.1:8080 (304121)
[2022-12-28 23:20:24 +0100] [304121] [INFO] Using worker: sync
[2022-12-28 23:20:24 +0100] [304125] [INFO] Booting worker with pid: 304125
^C[2022-12-28 23:20:25 +0100] [304121] [INFO] Handling signal: int
[2022-12-28 23:20:25 +0100] [304125] [INFO] Worker exiting (pid: 304125)
Trying to stop a worker
queue size: 1
Trying to stop a worker
queue size: 2
Trying to stop a worker
queue size: 3
All workers are finished
[2022-12-28 23:20:25 +0100] [304121] [INFO] Shutting down: Master
^C^C^C^C^C

What I expected to see: the application terminates clean. But it hangs. In the "real" application, I have a similar setup: a flask backend puts something in the queue (queue.size() increases) but the workers doing the queue.get() never get something back. Also, another strange behaviour I don't understand: if I increase the number of workers, stop_worker will be run multiple times (All workers are finished is printed multiple times). I think I need more knowledge of the inner workings of gunicorn to solve this. Would be nice if you can help me.

UPDATE: to be clearer: the put into the queue works, but not the get. A "Worker x is done" was not printed.

Thank you!

opened by kmille 0

Python 3.11.0 ModuleNotFoundError: No module named 'psycopg2._psycopg' 🐛

Bug Report

At what date and time did you most recently experience the problem? 2022-12-22T15:52:45.994388013Z
Where did you experience the problem? E.g. Azure Web Apps, Azure Functions, Azure Container Registry, or offline use. Azure Web Apps
If your repo is publicly available please share its URL: Private 🚫
What happened? ModuleNotFoundError: No module named 'psycopg2._psycopg'
What did you expect or want to happen? Successful loading of the Web App
How can we reproduce it? 🤷‍♂️
Do you have log output? Please include between the backticks:

   _____
  /  _  \ __________ _________   ____
 /  /_\  \\___   /  |  \_  __ \_/ __ \
/    |    \/    /|  |  /|  | \/\  ___/
\____|__  /_____ \____/ |__|    \___  >
        \/      \/                  \/
A P P   S E R V I C E   O N   L I N U X

Documentation: http://aka.ms/webapp-linux
Python 3.11.0
Note: Any data outside '/home' is not persisted
Starting OpenBSD Secure Shell server: sshd.
App Command Line not configured, will attempt auto-detect
Starting periodic command scheduler: cron.
Launching oryx with: create-script -appPath /home/site/wwwroot -output /opt/startup/startup.sh -virtualEnvName antenv -defaultApp /opt/defaultsite
Found build manifest file at '/home/site/wwwroot/oryx-manifest.toml'. Deserializing it...
Build Operation ID: |yXcV3nEsHeE=.fbb67f3_
Oryx Version: 0.2.20221031.3, Commit: 2b19efca9729673fc259e6a817be3cc0bb73b9d5, ReleaseTagName: 20221031.3
Output is compressed. Extracting it...
Extracting '/home/site/wwwroot/output.tar.gz' to directory '/tmp/8dae428d1852d04'...
App path is set to '/tmp/8dae428d1852d04'
Detected an app based on Django
Generating `gunicorn` command for 'core.wsgi'
Writing output script to '/opt/startup/startup.sh'
Using packages from virtual environment antenv located at /tmp/8dae428d1852d04/antenv.
Updated PYTHONPATH to '/opt/startup/app_logs:/tmp/8dae428d1852d04/antenv/lib/python3.11/site-packages'
[80] [INFO] Starting gunicorn 20.1.0
[80] [INFO] Listening at: http://0.0.0.0:8000 (80)
[80] [INFO] Using worker: sync
[83] [INFO] Booting worker with pid: 83
[80] [CRITICAL] WORKER TIMEOUT (pid:83)
[80] [WARNING] Worker with pid 83 was terminated due to signal 9
[95] [INFO] Booting worker with pid: 95
[95] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.9/dist-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.9/dist-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.9/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.9/dist-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.9/dist-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.9/dist-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/tmp/8dae428d1852d04/core/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
  File "/tmp/8dae428d1852d04/antenv/lib/python3.11/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/tmp/8dae428d1852d04/antenv/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/tmp/8dae428d1852d04/antenv/lib/python3.11/site-packages/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "/tmp/8dae428d1852d04/antenv/lib/python3.11/site-packages/django/apps/config.py", line 123, in create
    mod = import_module(mod_path)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/tmp/8dae428d1852d04/antenv/lib/python3.11/site-packages/django/contrib/postgres/apps.py", line 1, in <module>
    from psycopg2.extras import DateRange, DateTimeRange, DateTimeTZRange, NumericRange
  File "/tmp/8dae428d1852d04/antenv/lib/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ModuleNotFoundError: No module named 'psycopg2._psycopg'
[95] [INFO] Worker exiting (pid: 95)
[80] [INFO] Shutting down: Master
[80] [INFO] Reason: Worker failed to boot.

Please complete the following information

OS: Oryx 0.2.20220825.1, gunicorn 20.1.0
Psycopg version: 2.9.5 psycopg2_binary-2.9.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Python version: 3.11.0
PostgreSQL version: 13.8
pip version: 22.3

Re-add and improve the warning for workers terminated due to a signal
Re-add the warning for workers terminated due to a signal

The warnings are printed later in the main loop, not directly in the signal handler. This should fix the RuntimeError from #2564. See https://stackoverflow.com/q/45680378.

The order of events in the log might be a bit different from before.

Fix false positive "terminated due to signal 15"

During a reload (i.e. SIGHUP sent to master), gunicorn used to send multiple SIGTERMs to each worker. Every time the master got a SIGCHLD from one worker exiting, it sent a SIGTERM to all old workers who are still alive.

If you send a worker another SIGTERM when it's just about to shut down and the Python interpreter is already deinitialized, the signal won't be caught. The process exit status will be WIFSIGNALED SIGTERM.

Explanation:

This PR limits it to sending max one SIGTERM per worker in manage_workers(). I believe this is safe because handle_exit() in workers/base.py just sets self.alive = False anyway. So sending multiple SIGTERMs is never helpful to get a worker "unstuck". If a worker doesn't react to the first SIGTERM because it's in some kind of infinite loop, murder_workers() will notice it eventually, and send it a SIGABRT and a SIGKILL.

The SIGTERMs sent by handle_winch() and stop() are not changed in this PR.

I get the impression Gunicorn maintenance is mostly dead, so I don't have high expectations of getting any reply, but I'll hold out hope that somebody will review this PR one day. :)

Testing

Compare what happens on these three commits. Use any WSGI app.

$ git checkout 20.1.0 # you will see warnings about signal 15 and probably some log errors $ git checkout 01cf8421511dd16405a09564931e4508168a8909 # you will see warnings about signal 15 $ git checkout 3fe47e6a3f1d3a2b4011020b284970efeffc1312 # you will see only INFO $ python3 -m gunicorn -w 10 testapp $ kill -HUP $THE_MAIN_PID

I got the warnings and errors on the first try, but if you can't reproduce the problem, increasing the number of workers might help.
opened by TomiBelan 0
Fix access logging of interrupted and late-throwing requests
This PR fixes some cases where Gunicorn did not write a request to the access log:

If the client closes the connection during the request

If the WSGI app raises an exception after the initial HTTP response code was sent

If the worker times out and is killed by the master (at least for sync workers)

I get the impression Gunicorn maintenance is mostly dead, so I don't have high expectations of getting any reply, but I'll hold out hope that somebody will review this PR one day. :)

How it works

tfw the explanation is much longer than the patch.........

Look at handle() and handle_request() in sync.py and gthread.py, and handle_error() in base.py.

To see why this code change is correct, let's analyze these cases:

Normal success

The same things happen as before, in a slightly different order.

An error is raised before handle_request() is called

No change in this PR.

Some types of errors are logged when handle() calls handle_error().

Some types of errors aren't logged (e.g. connection closed while parsing the request URL).

An error is raised before handle_request() calls resp, environ = wsgi.create(...)

No change in this PR. Some types of errors are logged, some are not.

We can't call log.access() in handle_request anyway, because it needs a valid resp object.

An error is raised before response headers were sent (if resp and not resp.headers_sent)

OSError?

handle_request() reraises it.

NEW: handle_request() writes to the access log in the finally block.

handle() catches the OSError and closes the connection.

(Gunicorn assumes that OSError always means it's pointless to try writing a response. I'm not so sure, but this PR doesn't change the assumption.)

Exception?

handle_request() reraises it.

handle_request() doesn't write it to the access log. Same as before.

handle() catches it and calls handle_error().

handle_error() writes a e.g. HTTP 500 response to the client and writes e.g. code 500 to the access log. Same as before.

BaseException?

(E.g.: worker times out -> master sends SIGABRT -> handle_abort calls sys.exit(1) -> SystemExit is thrown on main thread -> the call stack unwinds and runs finally blocks)

NEW: handle_request() writes to the access log in the finally block.

handle_request() and handle() don't catch it.

An error is raised after response headers were sent (if resp and resp.headers_sent)

OSError? Same as above.

Exception?

handle_request() logs the exception to the error log and raises StopIteration.

NEW: handle_request() writes to the access log in the finally block.

handle() catches the StopIteration and closes the connection.

BaseException? Same as above.

This was tricky to get right. One big obstacle was that log.access() can be called in two places -- handle_request and handle_error. It's important not to log a request twice, hence handle_request() should call log.access() only if it knows that handle() won't call handle_error() for this error type.

There are some edge cases where I gave up (e.g. SSLError, or StopIteration thrown by the WSGI app itself).

It might be better to rewrite it all, but in this PR I optimized for a reasonably small patch.

Note that in cases 4.i and 4.iii, if there isn't any start_response call, the access log line will contain the string None instead of a HTTP status number. This could be easily changed if you don't like it.

Testing

You can use this script to manually test the behavior before vs after the PR.

$ cat testapp.py import errno import time def application(environ, start_response): path = environ['PATH_INFO'] if environ['PATH_INFO'] == '/a': raise ValueError("Hello A") if environ['PATH_INFO'] == '/b': raise OSError("Hello B") if environ['PATH_INFO'] == '/c': raise OSError(errno.EPIPE, "Hello C") if environ['PATH_INFO'] == '/d': time.sleep(100) write = start_response('200 OK', {}) write(b'Hi\n') if environ['PATH_INFO'] == '/e': raise ValueError("Hello E") if environ['PATH_INFO'] == '/f': time.sleep(100) return [b'Env %s\n' % repr(environ).encode('utf-8')] $ python3 -m gunicorn --access-logfile access.log testapp $ curl -v http://localhost:8000/ $ curl -v http://localhost:8000/a $ curl -v http://localhost:8000/b $ curl -v http://localhost:8000/c $ curl -v http://localhost:8000/d # interrupt the request with Ctrl+C $ curl -v http://localhost:8000/d # wait 30 seconds for worker timeout $ curl -v http://localhost:8000/e $ curl -v http://localhost:8000/f # interrupt the request with Ctrl+C $ curl -v http://localhost:8000/f # wait 30 seconds for worker timeout
opened by TomiBelan 0

Gunicorn threads are not being killed, ends up throwing "Can't start new thread" error.

     --access-logfile - \
      --workers 3 \
      --bind unix:/run/staging.sock:7000 \
      --threads 5 \
      --keep-alive 5 \
      --max-requests 5 \
      --max-requests-jitter 4 \
      --graceful-timeout 15 \
      medai_django_boilerplate.wsgi:application
      
      This is my service file.
      I want the threads to be killed after finishing a request on its own.

opened by OceanRahan 0

Releases(20.1.0)

20.1.0(Apr 28, 2021)
document WEB_CONCURRENCY is set by, at least, Heroku

capture peername from accept: Avoid calls to getpeername by capturing the peer name returned by accept

log a warning when a worker was terminated due to a signal

fix tornado usage with latest versions of Django

add support for python -m gunicorn

fix systemd socket activation example

allows to set wsgi application in configg file using wsgi_app

document --timeout = 0

always close a connection when the number of requests exceeds the max requests

Disable keepalive during graceful shutdown

kill tasks in the gthread workers during upgrade

fix latency in gevent worker when accepting new requests

fix file watcher: handle errors when new worker reboot and ensure the list of files is kept

document the default name and path of the configuration file

document how variable impact configuration

document the $PORT environment variable

added milliseconds option to request_time in access_log

added PIP requirements to be used for example

remove version from the Server header

fix sendfile: use socket.sendfile instead of os.sendfile

reloader: use absolute path to prevent empty to prevent0 InotifyError when a file is added to the working directory

Add --print-config option to print the resolved settings at startup.

remove the --log-dict-config CLI flag because it never had a working format (the logconfig_dict setting in configuration files continues to work)

** Breaking changes **

minimum version is Python 3.5

remove version from the Server header

** Documentation **

** Others **

miscellaneous changes in the code base to be a better citizen with Python 3

remove dead code

fix documentation generation

Source code(tar.gz)
Source code(zip)
20.0.4(Nov 26, 2019)
fix binding a socket using the file descriptor

remove support for the bdist_rpm build

Source code(tar.gz)
Source code(zip)
20.0.3(Nov 25, 2019)
fixed load of a config file without a Python extension

fixed socketfromfd.fromfd when defaults are not set

note: we now warn when we load a config file without Python Extension

Source code(tar.gz)
Source code(zip)
20.0.2(Nov 23, 2019)
20.0.2

fix changelog

20.0.1

fixed the way the config module is loaded. __file__ is now available

fixed wsgi.input_terminated. It is always true.

use the highest protocol version of openssl by default

only support Python >= 3.5

added __repr__ method to Config instance

fixed support of AIX platform and musl libc in socketfromfd.fromfd function

fixed support of applications loaded from a factory function

fixed chunked encoding support to prevent any request smuggling <https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn>_

Capture os.sendfile before patching in gevent and eventlet workers. fix RecursionError.

removed locking in reloader when adding new files

load the WSGI application before the loader to pick up all files

note this release add official support for applications loaded from a factory function as documented in Flask and other places.

Source code(tar.gz)
Source code(zip)
20.0.0(Nov 23, 2019)
Fixed fdopen RuntimeWarning in Python 3.8

Added check and exception for str type on value in Response process_headers method.

Ensure WSGI header value is string before conducting regex search on it.

Added pypy3 to list of tested environments

Grouped StopIteration and KeyboardInterrupt exceptions with same body together in Arbiter.run()

Added setproctitle module to extras_require in setup.py

Avoid unnecessary chown of temporary files

Logging: Handle auth type case insensitively

Removed util.import_module

Removed fallback for types.SimpleNamespace in tests utils

Use SourceFileLoader instead instead of execfile_

Use importlib instead of __import__ and eval`

Fixed eventlet patching

Added optional datadog <https://www.datadoghq.com>_ tags for statsd metrics

Header values now are encoded using latin-1, not ascii.

Rewritten parse_address util added test

Removed redundant super() arguments

Simplify futures import in gthread module

Fixed worker_connections` setting to also affects the Gthread worker type

Fixed setting max_requests

Bump minimum Eventlet and Gevent versions to 0.24 and 1.4

Use Python default SSL cipher list by default

handle wsgi.input_terminated extension

Simplify Paste Deployment documentation

Fix root logging: root and logger are same level.

Fixed typo in ssl_version documentation

Documented systemd deployement unit examples

Added systemd sd_notify support

Fixed typo in gthread.py

Added tornado <https://www.tornadoweb.org/>_ 5 and 6 support

Declare our setuptools dependency

Added support to --bind to open file descriptors

Document how to serve WSGI app modules from Gunicorn

Provide guidance on X-Forwarded-For access log in documentation

Add support for named constants in the --ssl-version flag

Clarify log format usage of header & environment in documentation

Fixed systemd documentation to properly setup gunicorn unix socket

Prevent removal unix socket for reuse_port

Fix ResourceWarning when reading a Python config module

Remove unnecessary call to dict keys method

Support str and bytes for UNIX socket addresses

fixed InotifyReloadeder: handle module.__file__ is None

/dev/shm as a convenient alternative to making your own tmpfs mount in fchmod FAQ

fix examples to work on python3

Fix typo in --max-requests documentation

Clear tornado ioloop before os.fork

Miscellaneous fixes and improvement for linting using Pylint

Source code(tar.gz)
Source code(zip)
19.9.0(Jul 3, 2018)
fix: address a regression that prevented syslog support from working (#1668, #1773)

fix: correctly set REMOTE_ADDR on versions of Python 3 affected by Python Issue 30205 <https://bugs.python.org/issue30205>_ (#1755, #1796)

fix: show zero response length correctly in access log (#1787)

fix: prevent raising :exc:AttributeError when --reload is not passed in case of a :exc:SyntaxError raised from the WSGI application. (#1805, #1806)

❗️The internal module gunicorn.workers.async was renamed to gunicorn.workers.base_async since async is now a reserved word in Python 3.7. (#1527)

Source code(tar.gz)
Source code(zip)
19.8.1(Apr 30, 2018)
fix: secure scheme headers when bound to a unix socket (#1766, #1767)

Source code(tar.gz)
Source code(zip)
19.8.0(Apr 28, 2018)
Eventlet 0.21.0 support (#1584)

Tornado 5 support (#1728, #1752)

support watching additional files with --reload-extra-file (#1527)

support configuring logging with a dictionary with --logging-config-dict (#1087, #1110, #1602)

add support for the --config flag in the GUNICORN_CMD_ARGS environment variable (#1576, #1581)

disable SO_REUSEPORT by default and add the --reuse-port setting (#1553, #1603, #1669)

fix: installing inotify on MacOS no longer breaks the reloader (#1540, #1541)

fix: do not throw TypeError when SO_REUSEPORT is not available (#1501, #1491)

fix: properly decode HTTP paths containing certain non-ASCII characters (#1577, #1578)

fix: remove whitespace when logging header values under gevent (#1607)

fix: close unlinked temporary files (#1327, #1428)

fix: parse --umask=0 correctly (#1622, #1632)

fix: allow loading applications using relative file paths (#1349, #1481)

fix: force blocking mode on the gevent sockets (#880, #1616)

fix: preserve leading / in request path (#1512, #1511)

fix: forbid contradictory secure scheme headers

fix: handle malformed basic authentication headers in access log (#1683, #1684)

fix: defer handling of USR1 signal to a new greenlet under gevent (#1645, #1651)

fix: the threaded worker would sometimes close the wrong keep-alive connection under Python 2 (#1698, #1699)

fix: re-open log files on USR1 signal using handler._open to support subclasses of FileHandler (#1739, #1742)

deprecation: the gaiohttp worker is deprecated, see the worker-class documentation for more information (#1338, #1418, #1569)

Source code(tar.gz)
Source code(zip)
19.7.1(Mar 21, 2017)
== 19.7.1 ==

fix: continue if SO_REUSEPORT seems to be available but fails (#1480)

fix: support non-decimal values for the umask command line option (#1325)

Source code(tar.gz)
Source code(zip)
19.7.0(Mar 4, 2017)
== 19.7.0 ==

The previously deprecated gunicorn_django command has been removed. Use the command-line interface instead.

The previously deprecated django_settings setting has been removed. Use the raw-env setting instead.

The default value of ssl-version has been changed from ssl.PROTOCOL_TLSv1 to ssl.PROTOCOL_SSLv23.

fix: initialize the group access list when initgroups is set (#1297)

add environment variables to gunicorn access log format (#1291)

add --paste-global-conf option (#1304)

fix: print access logs to STDOUT (#1184)

remove upper limit on max header size config (#1313)

fix: print original exception on AppImportError (#1334)

use SO_REUSEPORT if available (#1344)

fix leak of duplicate file descriptor for bound sockets.

add --reload-engine option, support inotify and other backends (#1368, #1459)

fix: reject request with invalid HTTP versions

add child_exit callback (#1394)

add support for eventlets _AlreadyHandled object (#1406)

format boot tracebacks properly with reloader (#1408)

refactor socket activation and fd inheritance for better support of SystemD (#1310)

fix: o fds are given by default in gunicorn (#1423)

add ability to pass settings to GUNICORN_CMD_ARGS environnement variable which helps in container world (#1385)

fix: catch access denied to pid file (#1091)

many additions and improvements to the documentation

Breaking Change

Python 2.6.0 is the last supported version

Source code(tar.gz)
Source code(zip)
19.6.0(May 23, 2016)
== 19.6.0 ==

Core

improvement of the binary upgrade behaviour using USR2: remove file lockin (#1270)

add the --capture-output setting to capture stdout/stderr tot the log file (#1271)

Allow disabling sendfile() via the SENDFILE` environment variable (#1252)

fix reload under pycharm (#1129)

Workers

fix: make sure to remove the signal from the worker pipe (#1269)

fix: gthread worker, handle removed socket in the select loop (#1258)

Source code(tar.gz)
Source code(zip)
19.5.0(May 11, 2016)
== 19.5.0 ==

=== Core ===

fix: Ensure response to HEAD request won't have message body

fix: lock domain socket and remove on last arbiter exit (#1220)

improvement: use EnvironmentError instead of socket.error (#939)

add: new $FORWARDDED_ALLOW_IPS environment variable (#1205)

fix: infinite recursion when destroying sockets (#1219)

fix: close sockets on shutdown (#922)

fix: clean up sys.exc_info calls to drop circular refs (#1228)

fix: do post_worker_init after load_wsgi (#1248)

=== Workers ===

fix access logging in gaiohttp worker (#1193)

eventlet: handle QUIT in a new coroutine (#1217)

gevent: remove obsolete exception clauses in run (#1218)

tornado: fix extra "Server" response header (#1246)

fix: unblock the wait loop under python 3.5 in sync worker (#1256)

=== Logging ===

fix: log message for listener reloading (#1181)

Let logging module handle traceback printing (#1201)

improvement: Allow configuring logger_class with statsd_host (#1188)

fix: traceback formatting (#1235)

fix: print error logs on stderr and access logs on stdout (#1184)

=== Documentation ===

Simplify installation instructions in gunicorn.org (#1072)

Fix URL and default worker type in example_config (#1209)

update django doc url to 1.8 lts (#1213)

fix: miscellaneous wording corrections (#1216)

Add PSF License Agreement of selectors.py to NOTICE (:issue: #1226)

document LOGGING overriding (#1051)

put a note that error logs are only errors from Gunicorn (#1124)

add a note about the requirements of the threads workers under python 2.x (#1200)

add access_log_format to config example (#1251)

=== Tests ===

Use more pytest.raises() in test_http.py

Source code(tar.gz)
Source code(zip)
19.4.5(Jan 5, 2016)
== 19.4.5 ==

fix: NameError fileno in gunicorn.http.wsgi (#1178)

Source code(tar.gz)
Source code(zip)
19.4.4(Jan 4, 2016)
== 19.4.4 ==

fix: check if a fileobject can be used with sendfile(2) (#1174)

doc: be more descriptive in errorlog option (#1173)

Source code(tar.gz)
Source code(zip)
19.4.3(Dec 29, 2015)
== 19.4.3 ==

fix: don't check if a file is writable using os.stat with SELINUX (#1171)

Source code(tar.gz)
Source code(zip)
19.4.2(Dec 29, 2015)
== 19.4.2 ==

=== Core ===

improvement: handle HaltServer in manage_workers (#1095)

fix: Do not rely on sendfile sending requested count (#1155)

fix: claridy --no-sendfile default (#1156)

fix: LoggingCatch sendfile failure from no file descriptor (#1160)

=== Logging ===

fix: Always send access log to syslog if syslog is on

fix: check auth before trying to own a file (#1157)

=== Documentation ===

fix: Fix Slowloris broken link. (#1142)

Tweak markup in faq.rst

=== Testing ===

fix: gaiohttp test (#1164)

Source code(tar.gz)
Source code(zip)
19.4.1(Nov 25, 2015)
== 19.4.1 ==

fix tornado worker (#1154)

Source code(tar.gz)
Source code(zip)
19.4(Nov 25, 2015)
Gunicorn 19.4 has been released. This is a new version with some new features and fixes.

Changes

Core

fix: make sure that a user is able to access to the logs after dropping a privilege (#1116)

improvement: inherit the Exception class where it needs to be (#997)

fix: make sure headers are always encodedas latin1 RFC 2616 (#1102)

improvement: reduce arbiter noise (#1078)

fix: don't close the unix socket when the worker exit (#1088)

improvement: Make last logged worker count an explicit instance var (#1078)

improvement: prefix config file with its type (#836)

improvement: pidfile handing (#1042)

fix: catch OSError as well as ValueError on race condition (#1052)

improve support of ipv6 by backporting urlparse.urlsplit from Python 2.7 to Python 2.6.

fix: raise InvalidRequestLine when the line contains maliscious data (#1023)

fix: fix argument to disable sendfile

fix: add gthread to the list of supported workers (#1011)

improvement: retry socket binding up to five times upon EADDRNOTAVAIL (#1004)

breaking change: only honor headers that can be encoded in ascii to comply to the RFC 7230 (See #1151).

Logging

add new parameters to access log (#1132)

fix: make sure that files handles are correctly reopenebd on HUP (#627)

include request URL in error message (#1071)

get username in access logs (#1069)

fix statsd logging support on Python 3 (#1010)

Testing

use last version of mock.

many fixes in Travis CI support

miscellaneous improvements in tests

Thread worker

fix: Fix self.nr usage in ThreadedWorker so that auto restart works as expected (#1031)

Gevent worker

fix quit signal handling (#1128)

add support for Python 3 (#1066)

fix: make graceful shutdown thread-safe (#1032)

Tornado worker

fix ssl options (#1146, #1135)

don't check timeout when stopping gracefully (#1106)

AIOHttp worker

add SSL support (#1105)

Documentation

fix link to proc name setting (#1144)

fix worker class documentation (#1141, #1104)

clarify graceful timeout documentation (#1137)

don't duplicate NGINX config files examples (#1050, #1048)

add web.py framework example (#1117)

update Debian/Ubuntu installations instructions (#1112)

clarify pythonpath setting description (#1080)

tweak some example for python3

clarify sendfile documentation

miscellaneous typos in source code comments (thanks!)

clarify why REMOTE_ADD may not be the user's IP address (#1037)

Misc

fix: reloader should survive SyntaxError (#994)

fix: expose the reloader class to the worker.

Full changelog is available here: https://github.com/benoitc/gunicorn/compare/19.3...19.4

More

You can find the documentation here: http://docs.gunicorn.org/en/19.4/

Latest version is also available on Pypi: https://pypi.python.org/pypi/gunicorn/19.4.0
Source code(tar.gz)
Source code(zip)
19.3(Mar 6, 2015)
Gunicorn 19.3 has been released. This is a new version with some new features and fixes.

Changes:

Core

fix:#978 make sure a listener is inheritable

add check_config class method to workers

fix select timeout in sync worker with multiple connections (#983)

allows workers to access to the reloader. close #984

raise TypeError instead of AssertionError

Logging

make Logger.loglevel a class attribute

Documentation

fix fix syntax errors in examples/gunicorn_rc (#988)

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/19.2.1...19.3

More

You can find the documentation here:

http://docs.gunicorn.org/en/19.3/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/19.3.0
Source code(tar.gz)
Source code(zip)
19.2.1(Feb 4, 2015)
Gunicorn 19.2.1 is a patch release with fixes.

Changes:

Logging

expose loglevel in the Logger class

AsyncIO worker (gaiohttp)

fix #977 fix initial crash

Documentation

document security mailing-list in the contributing page.

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/19.2...19.2.1

More

You can find the documentation here:

http://docs.gunicorn.org/en/19.2.1/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/19.2.1
Source code(tar.gz)
Source code(zip)
19.2(Jan 30, 2015)
Gunicorn 19.2 is a major release with many improvements and fixes.

Changes:

Core

optimize the sync workers when listening on a single interface

add --sendfile settings to enable/disable sendfile. fix #856 .

add the selectors module to the code base. #886

fix #862 add --max-requests-jitter setting to set the maximum jitter to add to the max-requests setting.

fix #899 propagate proxy_protocol_info to keep-alive requests

fix #863 worker timeout: dynamic timeout has been removed, fix a race condition error

fix: Avoid world writable file

fix #917: the deprecated --debug option has been removed.

Logging

fix #941 set logconfig default to paster more trivially

add statsd-prefix config setting: set the prefix to use when emitting statsd metrics

fix #832 log to console by default

fix #845 set the gunicorn loggers from the paste config

Thread Worker

fix #908 make sure the worker can continue to accept requests

Eventlet Worker

fix #867 , Fix eventlet shutdown to actively shut down the workers.

Documentation

Many improvements and fixes have been done, see the detailled changelog for more informations.

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/19.1.1...19.2

More

You can find the documentation here:

http://docs.gunicorn.org/en/19.2/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/19.2.0
Source code(tar.gz)
Source code(zip)
19.1.1(Aug 16, 2014)
Gunicorn 19.1.1 is a minor release with some fixes.

Changes:

Core

fix #835: display correct pid of already running instance

fix : fix PyTest class in setup.py.

Logging

fix #838 : statsd logger, send statsd timing metrics in milliseconds

fix #839 : statsd logger, allows for empty log message while pushing metrics and restore worker number in DEBUG logs

fix #850: add timezone info to logging

fix #853: Respect logger_class setting unless statsd is on

AioHttp worker

fix #830 make sure gaiohttp worker is shipped with gunicorn.

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/19.1...19.1.1

More

You can find the documentation here:

http://docs.gunicorn.org/en/19.1.1/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/19.1.1
Source code(tar.gz)
Source code(zip)
19.1(Jul 26, 2014)
Gunicorn 19.1 is a major release with new features and fixes.

Changes:

Core

fix #785: handle binary type address given to a client socket address

fix graceful shutdown. make sure QUIT and TERMS signals are switched everywhere.

support loading config from module (#799)

fix check for file-like objects (#805)

fix #815 args validation in WSGIApplication.init

fix #787 check if we load a pyc file or not.

Tornado worker

fix #771: support tornado 4.0

fix #783: x_headers error. The x-forwarded-headers option has been removed c4873681299212d6082cd9902740eef18c2f14f1. The discussion is available on #633.

AioHttp worker

fix: fetch all body in input. fix #803

fix: don't install the worker if python < 3.3

fix #822: Support UNIX sockets in gaiohttp worker

Async worker

fix #790 StopIteration shouldn't be catched at this level.

Logging

add statsd logging handler fix #748

Paster

fix #809 Set global logging configuration from a Paste config.

Extra

fix RuntimeError in gunicorn.reloader (#807)

Documentation

update faq: put a note on how `watch logs in the console since many people asked for it.

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/19.0...19.1

More

You can find the documentation here:

http://docs.gunicorn.org/en/19.1/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/19.1.0
Source code(tar.gz)
Source code(zip)
19.0(Jun 12, 2014)
Gunicorn 19.0 is a major release with new features and fixes. This version improve a lot the usage of Gunicorn with python 3 by adding two new workers to it: gthread a fully threaded async worker using futures and gaiohttp a worker using asyncio.

Breaking Changes

Switch QUIT and TERM signals

With this change, when gunicorn receives a QUIT all the workers are killed immediately and exit and TERM is used for the graceful shutdown.

Note: the old behaviour was based on the NGINX but the new one is more correct according the following doc:

https://www.gnu.org/software/libc/manual/html_node/Termination-Signals.html

also it is complying with the way the signals are sent by heroku:

https://devcenter.heroku.com/articles/python-faq#what-constraints-exist-when-developing-applications-on-heroku

Deprecations

run_gunicorn, gunicorn_django and gunicorn_paster are now completely deprecated and will be removed in the next release. Use the gunicorn command instead.

Changes:

core

add aiohttp worker named gaiohttp using asyncio. Full async worker on python 3.

fix HTTP-violating excess whitespace in write_error output

fix: try to log what happened in the worker after a timeout, add a worker_abort hook on SIGABRT signal.

fix: save listener socket name in workers so we can handle buffered keep-alive requests after the listener has closed.

add on_exit hook called just before exiting gunicorn.

add support for python 3.4

fix: do not swallow unexpected errors when reaping

fix: remove incompatible SSL option with python 2.6

add new async gthread worker and --threads options allows to set multiple threads to listen on connection

deprecate gunicorn_django and gunicorn_paster

switch QUIT and TERM signal

reap workers in SIGCHLD handler

add universal wheel support

use email.utils.formatdate in gunicorn.util.http_date

deprecate the --debug option

fix: log exceptions that occur after response start …

allows loading of applications from .pyc files (#693)

fix: issue #691, raw_env config file parsing

use a dynamic timeout to wait for the optimal time. (Reduce power usage)

fix python3 support when notifying the arbiter

add: honor $WEB_CONCURRENCY environment variable. Useful for heroku setups.

add: include tz offset in access log

add: include access logs in the syslog handler.

add --reload option for code reloading

add the capability to load gunicorn.base.Application without the loading of the arguments of the command line. It allows you to embed gunicorn in your own application.

improve: set wsgi.multithread to True for async workers

fix logging: make sure to redirect wsgi.errors when needed

add: syslog logging can now be done to a unix socket

fix logging: don't try to redirect stdout/stderr to the logfile.

fix logging: don't propagate log

improve logging: file option can be overriden by the gunicorn options --error-logfile and --access-logfile if they are given.

fix: dont' override SERVER_* by the Host header

fix: handle_error

add more option to configure SSL

fix: sendfile with SSL

add: worker_int callback (to react on SIGTERM)

fix: don't depend on entry point for internal classes, now absolute modules path can be given.

fix: Error messages are now encoded in latin1

fix: request line length check

improvement: proxy_allow_ips: Allow proxy protocol if "*" specified

fix: run worker's setup method before setting num_workers

fix: FileWrapper inherit from object now

fix: Error messages are now encoded in latin1

fix: don't spam the console on SIGWINCH.

fix: logging -don't stringify T and D logging atoms (#621)

add support for the latest django version

deprecate run_gunicorn django option

fix: sys imported twice

gevent worker

fix: make sure to stop all listeners

fix: monkey patching is now done in the worker

fix: "global name 'hub' is not defined"

fix: reinit hub on old versions of gevent

support gevent 1.0

fix: add subprocess in monket patching

fix: add support for multiple listener

eventlet worker

fix: merge duplicate EventletWorker.init_process method (fixes #657)

fix: missing errno import for eventlet sendfile patch

fix: add support for multiple listener

tornado worker

add gracefull stop support

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/18.0...19.0

More

You can find the documentation here:

http://docs.gunicorn.org/en/19.0/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/19.0.0
Source code(tar.gz)
Source code(zip)
18.0(Aug 27, 2013)
Gunicorn 18.0 is a major release with new features and fixes. With this release the commands gunicorn_django and gunicorn_paster are now deprecated. They will be removed in the next major release R19. You can now easily launch your django or paster applications by only using the gunicorn command line. See the documentation and the following changes for more more information.

Changes

new: add -e/--env command line argument to pass an environment variables to gunicorn

new: add --chdir command line argument to specified directory before apps loading.

new: add wsgi.file_wrapper support in async workers

new: add --paste command line argument to set the paster config file

deprecated: the command gunicorn_django is now deprecated. You should now run your application with the WSGI interface installed with your project (see https://docs.djangoproject.com/en/1.4/howto/deployment/wsgi/gunicorn/) for more infos.

deprecated: the command gunicorn_paste is deprecated. You now should use the new --paste argument to set the configuration file of your paster application.

fix: Removes unmatched leading quote from the beginning of the default access log format string

fix: null timeout

fix: gevent worker

fix: don't reload the paster app when using pserve

fix: after closing for error do not keep alive the connection

fix: responses 1xx, 204 and 304 should not force the connection to be closed

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/17.5...18.0

More

You can find the documentation here:

http://docs.gunicorn.org/en/18.0/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/18.0
Source code(tar.gz)
Source code(zip)
17.5(Jul 3, 2013)
Gunicorn 17.5 is a service release with mostly a number of small corrections and user contributions. But there are some new functions worth mentioning as well:

new: add signals documentation

new: add post_worker_init hook for workers

new: try to use gunicorn.conf.py in current folder as the default config file.

fix graceful timeout with the Eventlet worker

fix: don't raise an error when closing the socket if already closed

fix: fix --settings parameter for django application and try to find the django settings when using the gunicorn command.

fix: give the initial global_conf to paster application

fix: fix 'Expect: 100-continue' support on Python 3

Full changelog is available here:

https://github.com/benoitc/gunicorn/compare/0.17.4...17.5

You can find the documentation here:

http://docs.gunicorn.org/en/17.5/

Latest version is also available on Pypi:

https://pypi.python.org/pypi/gunicorn/17.5

Notes:

New versioning:

With this release, the versioning of Gunicorn is changing. Gunicorn is stable since a long time and there is no point to release a "1.0" now. It should have been done since a long time. 0.17 really meant it was the 17th stable version. From the beginning we have only 2 kind of releases:

major release: releases with major changes or huge features added

services releases: fixes and minor features added

So from now we will apply the following versioning <major>.<service>. For example 17.5 is a service release.
Source code(tar.gz)
Source code(zip)
gunicorn_17.5_all.deb(140.68 KB)

gunicorn 'Green Unicorn' is a WSGI HTTP Server for UNIX, fast clients and sleepy applications.

Related tags

Overview

Gunicorn

Documentation

Installation

Usage

Contributing

License

Comments

tasks

Bug Report

Please complete the following information

See also

Testing

How it works

Testing

Releases(20.1.0)

20.1.0(Apr 28, 2021)

20.0.4(Nov 26, 2019)

20.0.3(Nov 25, 2019)

20.0.2(Nov 23, 2019)

20.0.2

20.0.1

20.0.0(Nov 23, 2019)

19.9.0(Jul 3, 2018)

19.8.1(Apr 30, 2018)

19.8.0(Apr 28, 2018)

19.7.1(Mar 21, 2017)

== 19.7.1 ==

19.7.0(Mar 4, 2017)

== 19.7.0 ==

Breaking Change

19.6.0(May 23, 2016)

== 19.6.0 ==

Core

Workers

19.5.0(May 11, 2016)

19.4.5(Jan 5, 2016)

19.4.4(Jan 4, 2016)

19.4.3(Dec 29, 2015)

19.4.2(Dec 29, 2015)

19.4.1(Nov 25, 2015)

19.4(Nov 25, 2015)

Changes

Core

Logging

Testing

Thread worker

Gevent worker

Tornado worker

AIOHttp worker

Documentation

Misc

More

19.3(Mar 6, 2015)

Changes:

Core

Logging

Documentation

More

19.2.1(Feb 4, 2015)

Changes:

Logging

AsyncIO worker (gaiohttp)

Documentation

More

19.2(Jan 30, 2015)

Changes:

Core

Logging

Thread Worker

Eventlet Worker

Documentation

More

19.1.1(Aug 16, 2014)

Changes:

Core

Logging

AioHttp worker