web.py is a web framework for python that is as simple as it is powerful.

https://pypi.org/project/web.py#history The last production release to PyPI was made in February 2018 and it does not fully support Python 3. Given that the current code in this repo does support Python 3 and given that Python 2 end of life is in 110 days, it seems like a good time for us to make a new production release to PyPI so that projects that depend on web.py can use a production release.

1. Are there any objections to making such a release?
2. Are there any open issues or PRs that are showstoppers to making such a release?
3. Do we have a documented process for how to make a release to PyPI?
4. Do we have the access rights (logins, passwords, certs, etc.) to make are release to PyPI?

setup.py: Add python_requires='>=3.5'

This prevents installation on legacy Python and other versions < Python 3.5 as discussed in https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires

Tasks for making a new release.

• [x] Reread Packaging Python Projects to make sure you miss no steps...
• [x] Bump the version in web/__init__.py
• [x] Update ChangeLog.txt and mark the release date in ChangeLog.txt
• [x] Check setup.py and update if necessary
• [x] Check README.md and update if necessary
• [x] Check MANIFEST.in and update if necessary #628
• [x] Tag the version and mark it as a release in GitHub
• [x] python3 -m pip install --upgrade pip setuptools wheel # Important for PyPI page rendering
• [x] Run python3 setup.py sdist under webpy source code directory to generate package file used to publish to PyPI
• [x] Run tar tf dist/web.py-<version>.tar.gz to make sure all required files are included.
• [x] Publish package on PyPI with command python3 setup.py sdist upload
• [x] Update the documentation on the website (see webpy.github.com repo) webpy/webpy.github.com#137
• [ ] Announce it on the mailing list
• [x] Update the API docs if required (not required)

I am trying to run this old code and there is an error coming from the depths of 0.51 web.py

✗ python index.py                
Traceback (most recent call last):
  File "index.py", line 687, in <module>
    app.run()  # standalone run
  File "/virtualenvs/OpenStreetMap.by-MOXOekcf/lib/python2.7/site-packages/web/application.py", line 375, in run
    return wsgi.runwsgi(self.wsgifunc(*middleware))
  File "index.py", line 685, in <lambda>
    web.wsgi.runwsgi = lambda func, addr=None: web.wsgi.runfcgi(func, addr)
  File "/virtualenvs/OpenStreetMap.by-MOXOekcf/lib/python2.7/site-packages/web/wsgi.py", line 17, in runfcgi
    import flup.server.fcgi as flups
ImportError: No module named flup.server.fcgi

Fixes: #604 This needs careful review because it might have put enough imports in place to pass the tests but other apps might be looking for other imports.

web.py release: 0.40 Python release: 3.7.3 (Debian buster)

While porting a project from Python 2 to Python 3 (thanks for the 0.40 release!), I immediately encountered a bug in web.input(). Apparently, when a POST is sent by requests (as opposed to a web browser), web.py emits an error:

TypeError: write() argument must be str, not bytes

If debug is enabled, the default, then a stream of errors occur with the a similar error. See this issue for more information https://github.com/webpy/webpy/issues/448.

Sure enough, setting web.config.debug = False will prevent the stream of errors, yet the originating error still exists. (Also, I want error handling, I'll address that later).

The source of the error is in cgi.py, as it tries and copy the BytesIO object web.py created into a TemporaryFile that was created without wb+.

Here is the cgi.py write error: https://github.com/python/cpython/blob/ef4ec6ed12d6c6200a85068f60483723298b6ff4/Lib/cgi.py#L704

Here is the web.py BytesIO creation: https://github.com/webpy/webpy/blob/fa27cc510e39af9c32e015c8daf51023a6e80007/web/webapi.py#L443-L444

I am able fix this by monkey patching cgi.py, as the source of the error is make_file: https://github.com/python/cpython/blob/ef4ec6ed12d6c6200a85068f60483723298b6ff4/Lib/cgi.py#L814-L842

Here is a test case:

import cgi
import web
import json
import tempfile

urls = (
    '/', 'index'
)

class index:
    def POST(self):
        inputs = web.input()
        data = json.loads(web.data())
        print(inputs, data)
        return "Hello, world!"

def make_file(self):
    return tempfile.TemporaryFile("wb+")

# Comment out this line to reproduce the error                                                                                                                               
cgi.FieldStorage.make_file = make_file

if __name__ == "__main__":
    # Comment this line to see the stream of errors. Note, make_file
    # also needs to be the default, not patched, version.
    web.config.debug = False
    app = web.application(urls, globals())
    app.run()

You may test this with httpie:

$ http -v POST localhost:8080 a=b

One last comment, when monkey patching cgi.py, the debugging error page works again.

Is there anything else I can submit to help solve this?

With web.py current master, using len on db results is not working.

$ git log -n 1 --oneline
e90573e (HEAD -> master, origin/master, origin/HEAD, anandology/master) Reformat with Black :(

$ python
Python 3.7.4 (v3.7.4:e09359112e, Jul  8 2019, 14:54:52)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import web

>>>
>>> db = web.database(dbn="postgres", db="webpy")
>>>
>>> len(db.query("SELECT 1"))
0.0 (1): SELECT 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'IterBetter' has no len()

There is already a test for this as part of the test suite, but not sure how Travis is letting it go.

Fixes #644

Works with:

• MySQL column type TEXT and VARBINARY
• PostgreSQL column type TEXT and bytea

py3 code i used for testing:

import web

web.config.debug = False

urls = (
    "/count", "count",
)

app = web.application(urls, locals())

conn = web.database(
    dbn='mysql',    # Use `postgres` for PostgreSQL
    host='localhost',
    port=3306,    # Use `5432` for PostgreSQL
    db='<db-name>',
    user='<db-user>',
    pw='<db-password>',
)

session = web.session.Session(
    app=app,
    store=web.session.DBStore(conn, "sessions"),
    initializer={"count": 0},
)

class count:
    def GET(self):
        session.count += 1
        return str(session.count)

if __name__ == "__main__":
    app.run()

This installs v0.38 from Pypi:

$ pip install web.py

However, the latest version on Pypi is 0.40.dev0.

I don't think it is a good idea to release a .devX version to Pypi. One reason is that apparently, Pip ignores it.

Note: If you want people to test a particular branch, they can install it (even using Pip) directly from the github repo, so there is no need to release it to Pypi. For example, to install the current state of the master branch:

$ pip install git+https://https://github.com/webpy/[email protected]

Under heavy load, we sometimes see the following error (see issue #214) Exception in thread Thread-328: File "/pkg/lib/python2.7/site-packages/web/utils.py", line 500, in call update(block=True) File "/pkg/lib/python2.7/site-packages/web/utils.py", line 497, in update self.running[key].release() error: release unlocked lock

This can happen when there is a thread-switch between the test and creation of self.running[key]. In this patch we acquire a lock first to make the test+create atomic.

If the value attribute of a dropdown or radio button element is set to an integer and validation of the form containing that element fails the element's state is lost in the redisplayed form.

If the following form was submitted with the textbox left blank and the first radio button selected the form displayed after validation would show an empty textbox and no radio button selected. simple_form = form.Form( form.Textbox('test', form.notnull, ), form.Radio('example', [1,2,3,4], ) )

I think this happens because the form post is u/str and thus the value checks fail. Maybe this could be fixed by converting the values to strings for the test of equality in the render functions for these elements.

243c243
<             if self.value == value: select_p = ' selected="selected"'

---
>             if self.value == str(value): select_p = ' selected="selected"'
266c266
<             if self.value == arg:

---
>             if self.value == str(arg):

Table name exists injection

When calling db.multiple_insert to insert data, the internally implemented SQL statement uses the form of splicing as follows

    def multiple_insert(self, tablename, values, seqname=None, _test=False):
        """
        Inserts multiple rows into `tablename`. The `values` must be a list of
        dictionaries, one for each row to be inserted, each with the same set
        of keys. Returns the list of ids of the inserted rows.
        Set `seqname` to the ID if it's not the default, or to `False`
        if there isn't one.

            >>> db = DB(None, {})
            >>> db.supports_multiple_insert = True
            >>> values = [{"name": "foo", "email": "[email protected]"}, {"name": "bar", "email": "[email protected]"}]
            >>> db.multiple_insert('person', values=values, _test=True)
            <sql: "INSERT INTO person (email, name) VALUES ('[email protected]', 'foo'), ('[email protected]', 'bar')">
        """
        if not values:
            return []

        if not self.supports_multiple_insert:
            out = [
                self.insert(tablename, seqname=seqname, _test=_test, **v)
                for v in values
            ]
            if seqname is False:
                return None
            else:
                return out

        keys = values[0].keys()
        # @@ make sure all keys are valid

        for v in values:
            if v.keys() != keys:
                raise ValueError("Not all rows have the same keys")

        # enforce query order for the above doctest compatibility with Py3
        keys = sorted(keys)

        sql_query = SQLQuery(
            "INSERT INTO %s (%s) VALUES " % (tablename, ", ".join(keys))
        )

        for i, row in enumerate(values):
            if i != 0:
                sql_query.append(", ")
            SQLQuery.join(
                [SQLParam(row[k]) for k in keys],
                sep=", ",
                target=sql_query,
                prefix="(",
                suffix=")",
            )

When the values parameter is passed by the front-end combination, the current end passes in a json value like {"title":"admin","content":"admin"} to the text parameterdb.multiple_insert("entries" , [text]). The form of the insert statement is "INSERT INTO %s (%s) VALUES"% (tablename, ", ".join(keys)), which causes injection on the table name. There are special symbol filtering and string conversion on the parameters, but the table name is not processed

Input parameters:

{"title": "222222","content)%20VALUES%20('admin'%20or%20updatexml(1,concat(0x7e,(version())),0)%20or%20'')%3b#(": "qweqwe"}

sql_query：

<class 'tuple'>: (<sql: "INSERT INTO entries (content) VALUES ('admin' or updatexml(1,concat(0x7e,(version())),0) or '');#(, title) VALUES ('qweqwe', '222222')">, <sql: 'SELECT last_insert_id();'>)

Finally, it will report an error and execute the sql statement:

pymysql.err.InternalError: (1105, "XPATH syntax error: '~5.7.26'")

So, is it possible not to use splicing or to filter the table name as well

import json
urls = (
    '/api', 'api'
)

app = web.application(urls, globals())

class api:
      def POST(self):
           web.header('Content-Type', 'application/json')
           web.header('Access-Control-Allow-Origin', 'http://localhost:80')
           data = {}
           data["foo"] ="Hello"
           return json.dumps(data)

if __name__ == "__main__":
    app.run()

I am trying to use the default 8080 port of this http://localhost:8080/api from Apache http://localhost page in port 80. However, in the response web.py is not sending 'Access-Control-Allow-Origin' to the browser and I get

Access to fetch at 'http://localhost:8080/api' from origin 'http://localhost' has been blocked by CORS policy:
Response to preflight request doesn't pass access control check: 
No 'Access-Control-Allow-Origin' header is present on the requested resource.

We have an app where we have been setting global flags in the URL pattern with a prefix like (?i) or (?s). It has worked well until recently with Python 3.11 where the re module throws an error that the global flags must be set at the beginning of the pattern. I tracked this down to the fact that webpy is blindly modifying the pattern with anchors like ^ pattern \Z and the re module no longer tolerates the anchor (or any other pattern text) coming before the flags.

This can be fixed in application.py by replacing this call:

result = utils.re_complile(r'%s\z' % (pat,)).match(value)

with the equivalent:

result = utils.re_compile(pat).fullmatch(value)

This allows legal patterns with globals to be passed successfully, without adding any complexity to actually parse/split the pattern and add anchor after the globals.

I also see two other calls with the re_subm utility function. These look like:

what, results = utils.re_subm(r"^%s\Z'" % (pat,), what, value)
...
what, results = utils.re_subm("^" + pat + "$", what, value)

Since there isn't a variant of sub comparable to fullmatch to require a full match without adding anchors to the pattern, you'd have to add something uglier if you want to allow flags here. For example, you can then introduce anchors that won't break the pattern:

def re_set_anchors(pat, start='^', end='$'):
    """Return pattern modified with specified start and end anchors.

    :param pat: the original pattern to be augmented
    :param start: the desired starting anchor or None
    :param end: the desired ending anchor or None
    
    Returns the augmented pattern while preserving any prefixed global
    flags. Also detects and strips existing anchors.
    """
    start = start if start else ''
    end = end if end else ''
    metapat = r'^((\(\?[aiLmsux]+\))*)\^?(.*?)(\$|\\Z)?$'
    return re_compile(metapat).sub(r'\1' + start + r'\3' + end, pat)    
...
what, results = utils.re_subm(re_set_anchors(pat), what, value)

Etags/If-None-Match can be optionally prefixed by W/ to denote a "weak validator" according to MDN. This flag denotes that it is approximately identical, but not byte-identical. MDN suggests W/ shouldn't match for byte ranges. Presumably it should match for anything else though?

Update http.modified to handle W/.

MDN states:

If-None-Match Succeeds if the ETag of the distant resource is different to each listed in this header. By default, unless the etag is prefixed with 'W/', it performs a strong validation.

So if the requester etag is weak, then if the page etag can be weak or strong. If it's strong, then the page's etag must also be strong. I believe that's the correct interpretation.

The cookie for storing the session id in the web browser is set using self._setcookie in line 155 of sessions.py.

The same method is also called to delete the cookie, so it has a parameter called expires which defaults to an empty string. All the other options are taken from self._config instead inside the method, but not this one.

This means the timeout parameter is ignored, since setting web.config.session_parameters['timeout'] to any value does virtually nothing. If the user closes the web browser, then open it again, the cookie with the session id won't be there anymore so the session can't be recovered.

Changing the line 155 fixes the issue for me and sessions are preserved even if I close and reopen the browser, which I think should be the intended behaviour.

From this:

self._setcookie(self.session_id)

To this:

self._setcookie(self.session_id, expires=self._config.timeout)

Changing the self._setcookie definition just below should cause the same effect, since this method is only called twice, but I didn't test it.

From this:

def _setcookie(self, session_id, expires="", **kw):

To this:

def _setcookie(self, session_id, expires=self._config.timeout, **kw):

The Google App Engine code in web/application.py is heavily focused on Python 2 which we no longer support. Should we clean up that code to be compatible with Python 3 or should we drop the custom code for GHE? I do not use GHE.