Does pip provide a TOML parser for python 3.9?

Question

I want to parse TOML files in python 3.9, and I am wondering if I can do so without installing another package.

Since pip knows how to work with pyproject.toml files, and I already have pip installed, does pip provide a parser that I can import and use in my own code?

Asked By: Arthur Hebert-Ryan

||

Source

Answer 1

For 3.9, pip vendors tomli:

from pip._vendor import tomli

For consenting adults, importing the vendored module should work as usual. However, this is an implementation detail of pip, and it could change without a deprecation period at any moment in a future release of pip. Therefore, for anything apart from a quick hack, it would be safer to install tomli (or some other TOML parser) into the site instead.

Answered By: wim

Answer 2

So, it seems that PIP vendors the toml library tomli. Just after searching for "pyproject.toml" in the pip github repo, I found the following function in the source code::

def load_pyproject_toml(
    use_pep517: Optional[bool], pyproject_toml: str, setup_py: str, req_name: str
) -> Optional[BuildSystemDetails]:
    """Load the pyproject.toml file.
    Parameters:
        use_pep517 - Has the user requested PEP 517 processing? None
                     means the user hasn't explicitly specified.
        pyproject_toml - Location of the project's pyproject.toml file
        setup_py - Location of the project's setup.py file
        req_name - The name of the requirement we're processing (for
                   error reporting)
    Returns:
        None if we should use the legacy code path, otherwise a tuple
        (
            requirements from pyproject.toml,
            name of PEP 517 backend,
            requirements we should check are installed after setting
                up the build environment
            directory paths to import the backend from (backend-path),
                relative to the project root.
        )
    """
    has_pyproject = os.path.isfile(pyproject_toml)
    has_setup = os.path.isfile(setup_py)

    if not has_pyproject and not has_setup:
        raise InstallationError(
            f"{req_name} does not appear to be a Python project: "
            f"neither 'setup.py' nor 'pyproject.toml' found."
        )

    if has_pyproject:
        with open(pyproject_toml, encoding="utf-8") as f:
            pp_toml = tomli.loads(f.read())
    ...

The tomli import at the top:

from pip._vendor import tomli

Checking in the REPL

>>> from pip._vendor import tomli
>>> tomli.loads('[foo]nbar = "baz"')
{'foo': {'bar': 'baz'}}

Of course, this is not part of the public API, and you should really not rely on this.

Answered By: juanpa.arrivillaga

Answer 3

So the workaround I started with was to try for tomllib and, failing that, import the vendored version as mentioned by wim:

try: import tomllib
except ModuleNotFoundError: import pip._vendor.tomli as tomllib

Since the APIs are the same (or close enough for what I needed), this worked well with no further code changes. I also confirmed that a comment above mentioning that it was pip._vendor.toml in Python 3.8 was incorrect; it’s tomli, as it is in all the others.

However, this works with the "modern" version of pip, which is only usable on 3.7 and above. Python 3.6 and below have their own separate versions of getpip.py, and 3.6 does not seem to have a vendored TOML library in it.

Since in my particular case I was just using this to do a tiny bit of parsing on pyproject.toml files before installing those packages and their dependencies, and I am always using a virtual environment for this anyway, I decided the best option was to keep things simple: directly install the tomli package from PyPI and use only it, saving myself the hassle of multiple different checks.

But if you’re supporting only Python 3.7 and up, the above might be one of the easiest solutions.

Answered By: cjs

Does pip provide a TOML parser for python 3.9?

Question:

Answers: