How to properly write cross-references to external documentation with intersphinx?

Question:

I’m trying to add cross-references to external API into my documentation but I’m facing three different behaviors.

I am using sphinx(1.3.1) with Python(2.7.3) and my intersphinx mapping is configured as:

{
'python': ('https://docs.python.org/2.7', None),
'numpy': ('http://docs.scipy.org/doc/numpy/', None),
'cv2' : ('http://docs.opencv.org/2.4/', None),
'h5py' : ('http://docs.h5py.org/en/latest/', None)
}

I have no trouble writing a cross-reference to numpy API with :class:`numpy.ndarray` or :func:`numpy.array` which gives me, as expected, something like numpy.ndarray.

However, with h5py, the only way I can have a link generated is if I omit the module name. For example, :class:`Group` (or :class:`h5py:Group`) gives me Group but :class:`h5py.Group` fails to generate a link.

Finally, I cannot find a way to write a working cross-reference to OpenCV API, none of these seems to work:

:func:`cv2.convertScaleAbs`
:func:`cv2:cv2.convertScaleAbs`
:func:`cv2:convertScaleAbs`
:func:`convertScaleAbs`

How to properly write cross-references to external API, or configure intersphinx, to have a generated link as in the numpy case?

Asked By: Gall

||

Answers:

I gave another try on trying to understand the content of an objects.inv file and hopefully this time I inspected numpy and h5py instead of only OpenCV’s one.

How to read an intersphinx inventory file

Despite the fact that I couldn’t find anything useful about reading the content of an object.inv file, it is actually very simple with the intersphinx module.

from sphinx.ext import intersphinx
import warnings


def fetch_inventory(uri):
    """Read a Sphinx inventory file into a dictionary."""
    class MockConfig(object):
        intersphinx_timeout = None  # type: int
        tls_verify = False

    class MockApp(object):
        srcdir = ''
        config = MockConfig()

        def warn(self, msg):
            warnings.warn(msg)

    return intersphinx.fetch_inventory(MockApp(), '', uri)


uri = 'http://docs.python.org/2.7/objects.inv'

# Read inventory into a dictionary
inv = fetch_inventory(uri)
# Or just print it
intersphinx.debug(['', uri])

File structure (numpy)

After inspecting numpy’s one, you can see that keys are domains:

[u'np-c:function',
 u'std:label',
 u'c:member',
 u'np:classmethod',
 u'np:data',
 u'py:class',
 u'np-c:member',
 u'c:var',
 u'np:class',
 u'np:function',
 u'py:module',
 u'np-c:macro',
 u'np:exception',
 u'py:method',
 u'np:method',
 u'np-c:var',
 u'py:exception',
 u'np:staticmethod',
 u'py:staticmethod',
 u'c:type',
 u'np-c:type',
 u'c:macro',
 u'c:function',
 u'np:module',
 u'py:data',
 u'np:attribute',
 u'std:term',
 u'py:function',
 u'py:classmethod',
 u'py:attribute']

You can see how you can write your cross-reference when you look at the content of a specific domain. For example, py:class:

{u'numpy.DataSource': (u'NumPy',
  u'1.9',
  u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.DataSource.html#numpy.DataSource',
  u'-'),
 u'numpy.MachAr': (u'NumPy',
  u'1.9',
  u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.MachAr.html#numpy.MachAr',
  u'-'),
 u'numpy.broadcast': (u'NumPy',
  u'1.9',
  u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html#numpy.broadcast',
  u'-'),
  ...}

So here, :class:`numpy.DataSource` will work as expected.

h5py

In the case of h5py, the domains are:

[u'py:attribute', u'std:label', u'py:method', u'py:function', u'py:class']

and if you look at the py:class domain:

{u'AttributeManager': (u'h5py',
  u'2.5',
  u'http://docs.h5py.org/en/latest/high/attr.html#AttributeManager',
  u'-'),
 u'Dataset': (u'h5py',
  u'2.5',
  u'http://docs.h5py.org/en/latest/high/dataset.html#Dataset',
  u'-'),
 u'ExternalLink': (u'h5py',
  u'2.5',
  u'http://docs.h5py.org/en/latest/high/group.html#ExternalLink',
  u'-'),
 ...}

That’s why I couldn’t make it work as numpy references. So a good way to format them would be :class:`h5py:Dataset`.

OpenCV

OpenCV’s inventory object seems malformed. Where I would expect to find domains there is actually 902 function signatures:

[u':',
 u'AdjusterAdapter::create(const',
 u'AdjusterAdapter::good()',
 u'AdjusterAdapter::tooFew(int',
 u'AdjusterAdapter::tooMany(int',
 u'Algorithm::create(const',
 u'Algorithm::getList(vector<string>&',
 u'Algorithm::name()',
 u'Algorithm::read(const',
 u'Algorithm::set(const'
 ...]

and if we take the first one’s value:

{u'Ptr<AdjusterAdapter>': (u'OpenCV',
  u'2.4',
  u'http://docs.opencv.org/2.4/detectorType)',
  u'ocv:function 1 modules/features2d/doc/common_interfaces_of_feature_detectors.html#$ -')}

I’m pretty sure it is then impossible to write OpenCV cross-references with this file…

Conclusion

I thought intersphinx generated the objects.inv based on the content of the documentation project in an standard way, which seems not to be the case.
As a result, it seems that the proper way to write cross-references is API dependent and one should inspect a specific inventory object to actually see what’s available.

Answered By: Gall

In addition to the detailed answer from @gall, I’ve discovered that intersphinx can also be run as a module:

python -m sphinx.ext.intersphinx 'http://python-eve.org/objects.inv'

This outputs nicely formatted info. For reference: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/intersphinx.py#L390

Answered By: lingfish

The accepted answer no longer works in the new version (1.5.x) …

import requests
import posixpath
from sphinx.ext.intersphinx import read_inventory

uri = 'http://docs.python.org/2.7/'

r = requests.get(uri + 'objects.inv', stream=True)
r.raise_for_status()

inv = read_inventory(r.raw, uri, posixpath.join)
Answered By: Eric

How to use OpenCV 2.4 (cv2) intersphinx

Inspired by @Gall’s answer, I wanted to compare the contents of the OpenCV & numpy inventory files. I couldn’t get sphinx.ext.intersphinx.fetch_inventory to work from ipython, but the following does work:

curl http://docs.opencv.org/2.4/objects.inv | tail -n +5 | zlib-flate -uncompress > cv2.inv
curl https://docs.scipy.org/doc/numpy/objects.inv | tail -n +5 | zlib-flate -uncompress > numpy.inv

numpy.inv has lines like this:

numpy.ndarray py:class 1 reference/generated/numpy.ndarray.html#$ -

whereas cv2.inv has lines like this:

cv2.imread ocv:pyfunction 1 modules/highgui/doc/reading_and_writing_images_and_video.html#$ -

So presumably you’d link to the OpenCV docs with :ocv:pyfunction:`cv2.imread` instead of :py:function:`cv2.imread`. Sphinx doesn’t like it though:

WARNING: Unknown interpreted text role “ocv:pyfunction”.

A bit of Googling revealed that the OpenCV project has its own “ocv” sphinx domain: https://github.com/opencv/opencv/blob/2.4/doc/ocv.py — presumably because they need to document C, C++ and Python APIs all at the same time.

To use it, save ocv.py next to your Sphinx conf.py, and modify your conf.py:

sys.path.insert(0, os.path.abspath('.'))
import ocv
extensions = [
    'ocv',
]
intersphinx_mapping = {
    'cv2': ('http://docs.opencv.org/2.4/', None),
}

In your rst files you need to say :ocv:pyfunc:`cv2.imread` (not :ocv:pyfunction:).

Sphinx prints some warnings (unparseable C++ definition: u'cv2.imread') but the generated html documentation actually looks ok with a link to http://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#cv2.imread. You can edit ocv.py and remove the line that prints that warning.

An additional way to inspect the objects.inv file is with the sphobjinv module.

You can search local or even remote inventory files (with fuzzy matching). For instance with scipy:

$ sphobjinv suggest -t 90 -u https://docs.scipy.org/doc/scipy/reference/objects.inv "signal.convolve2d"

Remote inventory found.

:py:function:`scipy.signal.convolve2d`
:std:doc:`generated/scipy.signal.convolve2d`

Note that you may need to use :py:func: and not :py:function: (I’d be happy to know why).

Answered By: Francis Colas

Stubborn fool that I am, I used 2to3 and the Sphinx deprecated APIs chart to revive @david-röthlisberger’s ocv.py-based answer so it’ll work with Sphinx 2.3 on Python 3.5.

The fixed-up version is here:

https://gist.github.com/ssokolow/a230b27b7ea4a31f7fb40621e6461f9a

…and the quick version of what I did was:

  1. Run 2to3 -w ocv.py && rm ocv.py.bak
  2. Cycle back and forth between running Sphinx and renaming functions to their replacements in the chart. I believe these were the only changes I had to make on this step:
    1. Directive now has to be imported from docutils.parsers.rst
    2. Replace calls to l_(...) with calls to _(...) and remove the l_ import.
  3. Replace calls to env.warn with calls to log.warn where log = sphinx.util.logging.getLogger(__name__).

Then, you just pair it with this intersphinx definition and you get something still new enough to be relevant for most use cases:

'cv2': ('https://docs.opencv.org/3.0-last-rst/', None)
Answered By: ssokolow

For convenience, I made a small extension for aliasing intersphinx cross references. This is useful as sometimes the object inventory gets confused when an object from a submodule is imported from a package’s __init__.py.

See also https://github.com/sphinx-doc/sphinx/issues/5603

###
# Workaround of
# Intersphinx references to objects imported at package level can"t be mapped.
#
# See https://github.com/sphinx-doc/sphinx/issues/5603

intersphinx_aliases = {
    ("py:class", "click.core.Group"):
        ("py:class", "click.Group"),
    ("py:class", "click.core.Command"):
        ("py:class", "click.Command"),
}


def add_intersphinx_aliases_to_inv(app):
    from sphinx.ext.intersphinx import InventoryAdapter
    inventories = InventoryAdapter(app.builder.env)

    for alias, target in app.config.intersphinx_aliases.items():
        alias_domain, alias_name = alias
        target_domain, target_name = target
        try:
            found = inventories.main_inventory[target_domain][target_name]
            try:
                inventories.main_inventory[alias_domain][alias_name] = found
            except KeyError:
                print("could not add to inv")
                continue
        except KeyError:
            print("missed :(")
            continue


def setup(app):
    app.add_config_value("intersphinx_aliases", {}, "env")
    app.connect("builder-inited", add_intersphinx_aliases_to_inv)

To use this, I paste the above code in my conf.py and add aliases to the intersphinx_aliases dictionary.

Answered By: abstrus