This question was previously asked about docopt, but now docopt-ng seems to have ended up in the same situation - unmerged/unanswered PRs and issues. It would be really great to have a 'live' docopt for Python.

This is my proposal to fix #33. (With the assumption that compatibility with docopt is a goal of docopt-ng.)

This PR adds two new functions, parse_docstring_sections() and parse_options(); and uses them to parse docstrings accepted by docopt 0.6.2, while retaining docopt-ng's improvements to supported syntax.

Currently, docopt-ng parses option-defaults using a strategy that was in docopt's master branch, but considered unstable by the author, and was not released in docopt. It looks for option descriptions in an "options:" section, which is ended on the first blank line. This has the side-effect that options defined in a man-page style — with blank lines in-between — are not found. Neither are options outside an options: section (docopt allows options to follow the usage with no section heading).

parse_docstring_sections() is used to separate the usage section from the rest of the docstring. The text before the usage is ignored. The usage body (without its header) is parsed for the argument pattern and the usage header with its body is used to print the usage summary help. The text following the usage is parsed for options descriptions, using parse_options(), which supports option the description syntax of both docopt and the current docopt-ng.

Note that docopt 0.6.2 recognises option descriptions in the text prior to the usage section, but this change does not, as it seems like an unintended side-effect of the previous parser's implementation, and seems unlikely to be used in practice.

The testcases have two cases added for docopt 0.6.2 compatibility.

The first commit ("Fix test for missing arg before --") could be merged separately, but my final commit's tests depends on it.

I'm going to have a crack at making the error messages a little more user-friendly, to resolve #5. Especially the Warning: found unmatched (duplicate?) arguments [Argument(None, '.\\10. PowerShell.ps1')] that occurs in response to an end user passing incorrect or too many arguments when running a program using docopt-ng. I need that to be resolved before I can use docopt-ng in place of docopt really.

Before making changes there, I'd like to ensure the current messages are tested, so I can be sure I don't change messages unexpectedly.

This PR mostly adds exception message assertions to all the places where tests trigger an exception or SystemExit with usage.

The first 3 commits are unrelated to these messages assertions — small improvements to the dev experience using this repo. They could go into a separate PR if you prefer, but I've already opened a few, so I didn't want to swamp with too many!

This issue tracks the implementation of the Jazzband guidelines for the project docopt-ng

It was initiated by @itdaniher who was automatically assigned in addition to the Jazzband roadies.

See the TODO list below for the generally required tasks, but feel free to update it in case the project requires it.

Feel free to ping a Jazzband roadie if you have any question.

TODOs

• [x] Fix all links in the docs (and README file etc) from old to new repo
• [x] Add the Jazzband badge to the README file
• [x] Add the Jazzband contributing guideline to the CONTRIBUTING.md or CONTRIBUTING.rst file
• [x] Port Travis config to GitHub Actions following previous examples (see linked GitHub project board)
• [x] Port test coverage testing to Codecov if needed (Codecov)
• [x] Add jazzband account to PyPI project as maintainer role (e.g. URL: https://pypi.org/manage/project/docopt-ng/collaboration/)
• N/A Add jazzband-bot as maintainer to the Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/docopt-ng/users/)
• N/A Add incoming GitHub webhook integration to Read the Docs project (e.g. URL: https://readthedocs.org/dashboard/docopt-ng/integrations/)
• [x] Fix project URL in GitHub project description
• [x] Review project if other services are used and port them to Jazzband
• [x] Decide who is project lead for the project (if at all)
• [x] Set up CI for Jazzband project releases if needed and open ticket if yes

Project details

My project https://github.com/staticjinja/staticjinja uses mypy, but it can't actually use the type hints defined here because we aren't following https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages

PR to follow

@NickCrews would you be OK with auto sorting/formatting imports (probably with isort)?

When I was rebasing the error message PR just now, and a few other branches I've got, probably 3/4 of my merge conflicts were just imports. I'm thinking it would help to use isort with the single line option, so that instead of:

from typing import Any, Callable, Tuple, Type, Union, cast

we get:

from typing import Any
from typing import Callable
from typing import Tuple
from typing import Type
from typing import Union
from typing import cast

Which doesn't look as nice, but means import changes shouldn't cause merge conflicts.

I've got an old CLI program which uses OG docopt. I'm giving it a bit of minor TLC to refresh the tooling, and I tried switching to docopt-ng, but -ng fails to parse my usage string for some reason:

[email protected] /w/rnginline ((bef5200c…)) [127]> poetry run ipython
Python 3.10.5 (main, Jun  6 2022, 12:05:50) [GCC 9.5.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import docopt

In [2]: docopt.__version__
Out[2]: '0.8.1'

In [3]: from rnginline import cmdline

In [4]: docopt.docopt(cmdline.__doc__, argv=['rnginline', '--no-libxml2-compat', '/some/file'])
An exception has occurred, use %tb to see the full traceback.

DocoptExit: Warning: found unmatched (duplicate?) arguments [Option(None, '--no-libxml2-compat', 0, True)]
usage: rnginline [options] <rng-src> [<rng-output>]
       rnginline [options] --stdin [<rng-output>]

/home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3406: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

In [5]:                                                                                                                                                                                                            
Do you really want to exit ([y]/n)? y
[email protected] /w/rnginline ((bef5200c…))> poetry run pip uninstall docopt-ng
Found existing installation: docopt-ng 0.8.1
Uninstalling docopt-ng-0.8.1:
  Would remove:
    /home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/docopt/*
    /home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/docopt_ng-0.8.1.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled docopt-ng-0.8.1
[email protected] /w/rnginline ((bef5200c…))> poetry run pip install docopt
Collecting docopt
  Using cached docopt-0.6.2-py2.py3-none-any.whl
Installing collected packages: docopt
Successfully installed docopt-0.6.2

[email protected] /w/rnginline ((bef5200c…))> poetry run ipython
Python 3.10.5 (main, Jun  6 2022, 12:05:50) [GCC 9.5.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import docopt

In [2]: docopt.__version__
Out[2]: '0.6.2'

In [3]: from rnginline import cmdline

In [4]: docopt.docopt(cmdline.__doc__, argv=['rnginline', '--no-libxml2-compat', '/some/file'])
Out[4]: 
{'-': None,
 '--base-uri': None,
 '--default-base-uri': None,
 '--help': False,
 '--no-libxml2-compat': True,
 '--stdin': False,
 '--traceback': False,
 '--version': False,
 '<rng-output>': '/some/file',
 '<rng-src>': 'rnginline'}

This is the usage string: https://github.com/h4l/rnginline/blob/b1d1c8cda2a17d46627309950f2442021749c07e/rnginline/cmdline.py#L14

I really appreciate your efforts in keeping docopt going, It's a great library.

the idea of forking docopt because it's unmaintained it's cool but the idea of docopt is to be available in tons of languages, while docopt-ng only forked the python implementation. Do you plan to fork all others https://github.com/docopt?

Currently only DocoptTestException is reported, no info on the error is shown if any other error is raised during a testcases.docopt test. This change delegates to the default pytest.Item error reporting implementation for non-docopt errors.

I'd like to be able to support arguments that look like this:

-a --argument-name This is the argument to test

This PR enables dot access as before but converts the name to arguments.argument_name.

This PR aims to improve test reliability of test test_docopt_ng_more_magic_global_arguments_and_dot_access by cleaning state pollution of global variable arguments by assigning with None.

The test can fail in this way by running pip3 install pytest-repeat; python3 -m pytest --count=2 tests/test_docopt_ng.py::test_docopt_ng_more_magic_global_arguments_and_dot_access:

        global arguments
        docopt.docopt(doc, "-v file.py", more_magic=True)
>       assert arguments == {"-v": True, "-q": False, "-r": False, "--help": False, "FILE": "file.py", "INPUT": None, "OUTPUT": None}
E       AssertionError: assert {'--help': Fa...OUTPUT': None} == {'--help': Fa...v': True, ...}
E         Omitting 6 identical items, use -vv to show
E         Left contains 1 more item:
E         {'<FILE>': None}
E         Right contains 1 more item:
E         {'FILE': 'file.py'}

It may be better to clean state pollutions so that some other tests won't fail in the future due to the shared state pollution.

I have started using docopt-dispatch and I find it very intriguing. It makes docopt CLI code super-concise and readable.

Unfortunately, that package both has docopt as a hard dependency and is unmaintained for 7+ years.

If we integrated the dispatching project in docopt-ng some CLI code could look like this, preserving the compatibility with the original docopt spec and staying aligned with the syntax of the original dispatch project:

"""Run something in development or production mode.

Usage: run.py --development <host> <port>
       run.py --production <host> <port>
       run.py remote add <item>
       run.py remote delete <item>
"""
from docopt import dispatch


@dispatch.on('--development')
def development(host, port, **kwargs):
    print('in *development* mode')


@dispatch.on('--production')
def development(host, port, **kwargs):
    print('in *production* mode')


if __name__ == 'main':
    dispatch(__doc__)

Would you accept a PR that adds the dispatching feature to docopt-ng?

I just began using docopt-ng. I was creating my interface from scratch (I currently use a functional but spaghetti-like script), and my docstring read something like:

"""Performous macOS Bundler

Usage:
    macos_bundler.py
    macos_bundler.py [options]
    macos_bundler.py (-h | --help)

Options:
...
"""

But I kept getting the following error and couldn't figure out why.

    raise tokens.error("unmatched '%s'" % token)
docopt.DocoptLanguageError: unmatched '('

I double checked brackets and there were definitely no unmatched ones. I then double-checked the documentation to make sure I hadn't made a mistake with the syntax, and I couldn't find one.

I had read sometimes docopt (I think the original one?) had issues with pipes, so even though that exact syntax appeared in the documentation, I tried to remove the macos_bundler.py (-h | --help) line and, the error went away; but I couldn't use either -h nor --help, because it kept telling me those optons needed an argument.

Upon closer inspection, I noticed I had missed a space between -h --help and the option description.

So, it seems missing writing something such as

"""Command
Usage:
    cli_tool.py (-s | --long)

Options:
    -s --long Some description for this option.
"""

Will produce such an error.

Of course, there is an error there, and one the documentation specifically warns against. However, in this particular scenario, the error message was very obscure.

If you install pipenv install pipreqs pip-upgrader docopt-ng Which sometimes happens, you can't put everything into system or pipx.

Then apps use the wrong docopt.

$ python
>>> import docopt
>>> docopt.__file__
'C:\\Users\\matth\\.virtualenvs\\demo_doc-j-5bRj6B\\lib\\site-packages\\docopt.py'
>>> exit()

$ pip install docopt-ng
Requirement already satisfied: docopt-ng in c:\users\matth\.virtualenvs\demo_doc-j-5brj6b\lib\site-packages (0.7.2)

In C# there was a way to handle this and ensure that two similarly named dlls weren't confused with each other, I don't know what the python way is to handle this scenario.

Description

Using docopt for building utilities with relatively wide range of options is pretty limited because of a huge performance penalty. Namely, a combinatorial explosion may happen in the transform function: the pattern expansion (like ((-a | -b) (-c | -d)) => (-a -c | -a -d | -b -c | -b -d)) has unacceptable computational complexity.

A good example would be the GNU ls utility. See the sample below.

To Reproduce

The script below takes almost 3 seconds to run which is terribly slow for just to parse CLI arguments.

"""
ls with a subset of GNU options (that's not even all of them!)

Usage:
    ls [-a|-A] [--hide=PATTERN] [-I=PATTERN] [-dLR]
       [--color] [-h|--si] [--indicator-style=WORD|-p|-F|--file-type]
       [--format=WORD|-x|-m|-x|-l|-1|-C|-g|-n|-o] [-Giks]
       [--sort=WORD|-f|-U|-S|-t|-v|-X] [--group-directories-first] [-r]
       [--time=WORD|-u|-c] [--time-style=TIME_STYLE]
       [FILES ...]
    ls --help
    ls --version

Arguments:
    FILES
        list of files
"""
from docopt import docopt

args = docopt()

Not seeing any references to an actual docopt formal grammar yet, I am writing a PEG grammar for use in Python with Tatsu (https://tatsu.readthedocs.io). I am hopeful that it will be usable with other PEG parser generators but am not well enough versed in the area to know by how much they differ in syntax.

I would like to work with others interested in developing a formal grammar for docopt. I'm not fixed on Tatsu or even PEG though I suspect PEG grammars will provide more accurate results for the corner cases.

I appended a sketch of the steps I believe necessary to develop the grammar. I've written a grammar for the usage examples section. Now I need to take a step back and write the frame work to snag each major section. Divide and conquer seemed like the best approach. In a day or so, I could post the grammar and test cases to github.

So, any information in this area or would anyone like to collaborate ?

Philip

Unix Utility Usage Grammar

Or, a formal grammar for the docopy language.

First identify which elements must be parsed and which (if any) may be discarded (or perhaps remain free form text).

Characterize the delineations of each section sufficiently to write regular expression matchers for their start and end, or span.

Write a grammar to parse to just the sections out as blocks of free form text. This forms the framework within which the section parsers will operate.

For each section independently write a grammar to parse the it.

One by one, incorporate the section parsers into the framework.