Use Sphinx’s own intersphinx files and APIs to get all symbol names. That’s both more robust and gives us a better introspection into the actual types of symbols.

Todos:

• [x] front page as reported in https://github.com/hynek/doc2dash/issues/27#issuecomment-50266807
• [x] bring coverage back to 100%

Should fix #27 and #25.

Partial fix for #57

• allow INV_TO_TYPE mapping to be overridden via convert_type method
• allow ParserEntry creation to be overridden with create_entry method

Hi, thanks for this tool. I have installed sagemath through nix and it includes documentation using sphinx. I am trying to build a docset for it using

doc2dash -n Sage -f -d ~/.local/share/dashdocs/ -i ~/Downloads/sage_logo_new_hc-nq8.png /nix/store/150yjwckyscc2kisgx93fs40cwk5wwdp-sagedoc-9.6/share/doc/sage/html/en/reference

However this fails with

PermissionError: [Errno 13] Permission denied: '.buildinfo'

The .buildinfo files seem to be part of many sub-directories of the documentation directory and the look like this,

# Sphinx build info version 1
# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
config: f164681b73cecad2fb4d00961dc5623f
tags: 645f666f9bcd5a90fca523b33c5a78b7

I assume these are for nix (I don't know much, sagemath is the only thing I use nix for). Is it possible to ignore these files somehow? Or for doc2dash to continue despite such errors and generate docset based on files it can successfully consume?

Gevent's document is generated by Sphinx, but I ran: doc2dash -n 'Py2_gevent' -I ./gevent/doc/_build/html/contents.html ./gevent/doc/_build/html/

and got the message: "./gevent/doc/_build/html/" does not contain a known documentation format.

doc2dash isn't adding the table of contents metadata with docs that are generated using the latest pydoctor. Happens with the Twisted docs. No errors are shown.

Hi, I'm trying to use doc2dash to generate scikit-learn doc for dash https://github.com/Kapeli/Dash-User-Contributions/pull/3805 https://github.com/Kapeli/Dash-User-Contributions/pull/3812. However, the ci complains about File not found:

Scikit/scikit-learn.tgz: FAIL ❌
File not found at "genindex.html" for entry with row id "3058" of "scikit-learn.docset"
File not found at "py-modindex.html" for entry with row id "3222" of "scikit-learn.docset"
File not found at "py-modindex.html" for entry with row id "3327" of "scikit-learn.docset"
[NOTICE] Some pages loaded online resources, which might mean the docset doesn't work offline.

I am asking for advice on how to solve this problem, here are the commands I tried:

# download scikit-learn docs directly
wget https://scikit-learn.org/stable/_downloads/scikit-learn-docs.zip
unzip -d scikit-learn-docs scikit-learn-docs.zip

# convert to dash
doc2dash --index-page documentation.html --enable-js -n scikit-learn scikit-learn-docs
tar --exclude='.DS_Store' -czvf scikit-learn.tgz scikit-learn.docset

I was trying to use doc2dash with the documentation of quimb but the screen show the following error.

Converting intersphinx docs from "html" to "./html.docset".
Parsing documentation...
Added 2,494 index entries.
Adding table of contents meta data...  [##################################################-------------------------------------------]   53%  00:00:18
Traceback (most recent call last):
  File "/Users/mofeing/Develop/quimb/.venv/bin/doc2dash", line 8, in <module>
    sys.exit(main())
  File "/Users/mofeing/Develop/quimb/.venv/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mofeing/Develop/quimb/.venv/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/mofeing/Develop/quimb/.venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mofeing/Develop/quimb/.venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/mofeing/Develop/quimb/.venv/lib/python3.8/site-packages/doc2dash/__main__.py", line 227, in main
    toc.close()
  File "/Users/mofeing/Develop/quimb/.venv/lib/python3.8/site-packages/doc2dash/parsers/utils.py", line 132, in patch_anchors
    patch_files(pbar)
  File "/Users/mofeing/Develop/quimb/.venv/lib/python3.8/site-packages/doc2dash/parsers/utils.py", line 112, in patch_files
    with codecs.open(full_path, mode="r", encoding="utf-8") as fp:
  File "/Users/mofeing/.pyenv/versions/3.8.2/lib/python3.8/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: './html.docset/Contents/Resources/Documents/calculating%20quantities.html'

Problem was that the repo generates some HTML files with spaces in their names and other files referencing to them encodes their names in URL form. Adding a URL decoding step solves the problem and does not interfere with other filenames.

Some path parse from objects.inv like the one in alembic's would be 'index.html#document-api/ddl#alembic.ddl.mysql.MySQLChangeColumn'. As you see, there is more than one '#' in this path, and it cannot be recognised correctly. That may be the inv file's error. This commit replace the path with head and tail of the original one. It can fix this issue.

Hello,

i would like to create a docset for guzzle. But when running the doc2dash command:

doc2dash _build/html -n guzzle-docs -d /Users/USERNAME/Git/guzzle/docs

i get an error message.

Converting intersphinx docs from "html" to "/Users/USERNAME/Git/guzzle/docs/guzzle-docs.docset".
Parsing documentation...
Added 22 index entries.
Adding table of contents meta data...  [#################################################################################################################################################################------------------------------------------------------------------------------------------------------------]   60%
Traceback (most recent call last):
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/bin/doc2dash", line 8, in <module>
    sys.exit(main())
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/lib/python3.9/site-packages/doc2dash/__main__.py", line 227, in main
    toc.close()
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/lib/python3.9/site-packages/doc2dash/parsers/utils.py", line 134, in patch_anchors
    patch_files(pbar)
  File "/Users/USERNAME/.virtualenvs/doc2dashtest/lib/python3.9/site-packages/doc2dash/parsers/utils.py", line 114, in patch_files
    with codecs.open(full_path, mode="r", encoding="utf-8") as fp:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/USERNAME/Git/guzzle/docs/guzzle-docs.docset/Contents/Resources/Documents/py-modindex.html'

any ideas?

doc2dash version: 2.4.1

Hi,

I'm trying to generate a Dash docset for mypy, unfortunatly it doesn't work and I don't understand what I'm doing wrong.

I cloned the mypy repo I installed doc2dash with pip I go do mypy/docs I type :

>doc2dash -n mypy .

End result is :

"/Users/XXXXX/dev/python/mypy/docs" does not contain a known documentation format.

I tried with enabling the -v option, with no more success.

Any help is welcomed 😃

Doc2Dash seems very cool but unfortunately, I was not able to import BeautifulSoup documentation :S

Steps to reproduce:

wget http://www.crummy.com/software/BeautifulSoup/bs4/download/4.3/beautifulsoup4-4.3.2.tar.gz
tar xzvf beautifulsoup4-4.3.2.tar.gz
cd beautifulsoup4-4.3.2/doc
make html
doc2dash -A build/html -n BeautifulSoup

In Dash, I see "Empty docset". When I open beautifulsoup4-4.3.2/doc/build/html/index.html, the documentation is good. Have I done something wrong ?

Thanks !

When I build the statsmodels docs for Dash, I usually get these types of errors, the 'Can't find anchor ... in ...'. Below are some such examples. Even so, the docs are created, however what I've noticed is that some index entries are repeated, such as the entries for sections (see image below).

Is this a statsmodels issue, or can it be confronted in doc2dash ?

Can't find anchor 'statsmodels.genmod.families.links.CLogLog' (EntryType.CLASS) in 'generated/statsmodels.genmod.families.links.CLogLog.html'.

Can't find anchor 'statsmodels.genmod.families.links.Log' (EntryType.SECTION) in 'generated/statsmodels.genmod.families.links.Log.html'.

Can't find anchor 'statsmodels.genmod.families.links.CLogLog.deriv' (EntryType.METHOD) in 'generated/statsmodels.genmod.families.links.CLogLog.deriv.html'.

Here is the current OS where you run pyoxidizer:

https://github.com/hynek/doc2dash/blob/46bba9b667da609bb38981901dde1e68574d925f/.github/workflows/pyoxidizer.yml#L20

On GitHub Actions, ubuntu-latest is currently being updated to point to ubuntu-22.04, which uses glibc 2.35. Since your binary is not compiled statically, this means for the next doc2dash release you won't support Ubuntu 20.04 and other systems with an older glibc.

And the current release does not support Ubuntu 18.04, as shown by pyoxidizer analyze:

$ pyoxidizer analyze doc2dash

...

glibc
-----

Minimum Version: 2.29
Minimum Distro Versions:
  Debian 11
  Fedora 30
  OpenSUSE 15.3
  RHEL 9
  Ubuntu 19.04

See more details in the documentation: https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_distributing_linux.html#managing-binary-portability-on-linux

This might not be a problem, but I thought you would like to know.

It has become common for python docs to reference jupyter notebooks with external links to http://nbviewer.ipython.org./. (e.g. http://nbviewer.jupyter.org/github/cvxgrp/cvx_short_course/blob/master/applications/portfolio_optimization.ipynb) I have found that these notebooks are often the most useful portion of the docs, however they are not automatically captured by doc2dash. I propose a feature that automatically downloads these notebooks.