Xpath like query for nested python dictionaries

More work would have to be put into how the XPath-like selector would work.
'/' is a valid dictionary key, so how would

foo={'/':{'/':'eggs'},'//':'ham'}

be handled?

foo.select("///")

would be ambiguous.

Is there any reason for you to the query it the way like the XPath pattern? As the commenter to your question suggested, it just a dictionary, so you can access the elements in a nest manner. Also, considering that data is in the form of JSON, you can use simplejson module to load it and access the elements too.

There is this project JSONPATH, which is trying to help people do opposite of what you intend to do (given an XPATH, how to make it easily accessible via python objects), which seems more useful.

Another alternative (besides that suggested by jellybean) is this:

def querydict(d, q):
  keys = q.split('/')
  nd = d
  for k in keys:
    if k == '':
      continue
    if k in nd:
      nd = nd[k]
    else:
      return None
  return nd

foo = {
  'spam':'eggs',
  'morefoo': {
               'bar':'soap',
               'morebar': {'bacon' : 'foobar'}
              }
   }
print querydict(foo, "/morefoo/morebar")

There is an easier way to do this now.

http://github.com/akesterson/dpath-python

$ easy_install dpath
>>> dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar")

… done. Or if you don’t like getting your results back in a view (merged dictionary that retains the paths), yield them instead:

$ easy_install dpath
>>> for (path, value) in dpath.util.search(YOUR_DICTIONARY, "morefoo/morebar", yielded=True)

… and done. ‘value’ will hold {‘bacon’: ‘foobar’} in that case.

There is the newer jsonpath-rw library supporting a JSONPATH syntax but for python dictionaries and arrays, as you wished.

So your 1st example becomes:

from jsonpath_rw import parse

print( parse('$.morefoo.morebar').find(foo) )

And the 2nd:

print( parse("$.morefoo[0].morebar.bacon").find(foo) )

PS: An alternative simpler library also supporting dictionaries is python-json-pointer with a more XPath-like syntax.

One of the best libraries I’ve been able to identify, which, in addition, is very actively developed, is an extracted project from boto: JMESPath. It has a very powerful syntax of doing things that would normally take pages of code to express.

Here are some examples:

search('foo | bar', {"foo": {"bar": "baz"}}) -> "baz"
search('foo[*].bar | [0]', {
    "foo": [{"bar": ["first1", "second1"]},
            {"bar": ["first2", "second2"]}]}) -> ["first1", "second1"]
search('foo | [0]', {"foo": [0, 1, 2]}) -> [0]

If terseness is your fancy:

def xpath(root, path, sch='/'):
    return reduce(lambda acc, nxt: acc[nxt],
                  [int(x) if x.isdigit() else x for x in path.split(sch)],
                  root)

Of course, if you only have dicts, then it’s simpler:

def xpath(root, path, sch='/'):
    return reduce(lambda acc, nxt: acc[nxt],
                  path.split(sch),
                  root)

Good luck finding any errors in your path spec tho 😉

dict > jmespath

You can use JMESPath which is a query language for JSON, and which has a python implementation.

import jmespath # pip install jmespath

data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}

jmespath.search('root.section.item2', data)
Out[42]: 'value2'

The jmespath query syntax and live examples: http://jmespath.org/tutorial.html

dict > xml > xpath

Another option would be converting your dictionaries to XML using something like dicttoxml and then use regular XPath expressions e.g. via lxml or whatever other library you prefer.

from dicttoxml import dicttoxml  # pip install dicttoxml
from lxml import etree  # pip install lxml

data = {'root': {'section': {'item1': 'value1', 'item2': 'value2'}}}
xml_data = dicttoxml(data, attr_type=False)
Out[43]: b'<?xml version="1.0" encoding="UTF-8" ?><root><root><section><item1>value1</item1><item2>value2</item2></section></root></root>'

tree = etree.fromstring(xml_data)
tree.xpath('//item2/text()')
Out[44]: ['value2']

Json Pointer

Yet another option is Json Pointer which is an IETF spec that has a python implementation:

• https://github.com/stefankoegl/python-json-pointer

From the jsonpointer-python tutorial:

from jsonpointer import resolve_pointer

obj = {"foo": {"anArray": [ {"prop": 44}], "another prop": {"baz": "A string" }}}

resolve_pointer(obj, '') == obj
# True

resolve_pointer(obj, '/foo/another%20prop/baz') == obj['foo']['another prop']['baz']
# True

>>> resolve_pointer(obj, '/foo/anArray/0') == obj['foo']['anArray'][0]
# True

def Dict(var, *arg, **kwarg):
  """ Return the value of an (imbricated) dictionnary, if all fields exist else return "" unless "default=new_value" specified as end argument
      Avoid TypeError: argument of type 'NoneType' is not iterable
      Ex: Dict(variable_dict, 'field1', 'field2', default = 0)
  """
  for key in arg:
    if isinstance(var, dict) and key and key in var:  var = var[key]
    else:  return kwarg['default'] if kwarg and 'default' in kwarg else ""   # Allow Dict(var, tvdbid).isdigit() for example
  return kwarg['default'] if var in (None, '', 'N/A', 'null') and kwarg and 'default' in kwarg else "" if var in (None, '', 'N/A', 'null') else var

foo = {
  'spam':'eggs',
  'morefoo': {
               'bar':'soap',
               'morebar': {'bacon' : 'foobar'}
              }
   }
print Dict(foo, 'morefoo', 'morebar')
print Dict(foo, 'morefoo', 'morebar', default=None)

Have a SaveDict(value, var, *arg) function that can even append to lists in dict…

I reference form this link..

Following code is for json xpath base parse implemented in python :

import json
import xmltodict

# Parse the json string
class jsonprase(object):
    def __init__(self, json_value):
        try:
            self.json_value = json.loads(json_value)
        except Exception :
            raise ValueError('must be a json str value')


    def find_json_node_by_xpath(self, xpath):
        elem = self.json_value
        nodes = xpath.strip("/").split("/")
        for x in range(len(nodes)):
            try:
                elem = elem.get(nodes[x])
            except AttributeError:
                elem = [y.get(nodes[x]) for y in elem]
        return elem

    def datalength(self, xpath="/"):
        return len(self.find_json_node_by_xpath(xpath))

    @property
    def json_to_xml(self):
        try:
            root = {"root": self.json_value}
            xml = xmltodict.unparse(root, pretty=True)
        except ArithmeticError :
            pyapilog().error(e)
        return xml

Test Json :

{
    "responseHeader": {
        "zkConnected": true,
        "status": 0,
        "QTime": 2675,
        "params": {
            "q": "TxnInitTime:[2021-11-01T00:00:00Z TO 2021-11-30T23:59:59Z] AND Status:6",
            "stats": "on",
            "stats.facet": "CountryCode",
            "rows": "0",
            "wt": "json",
            "stats.field": "ItemPrice"
        }
    },
    "response": {
        "numFound": 15162439,
        "start": 0,
        "maxScore": 1.8660598,
        "docs": []
    }
}

Test Code to read the values from above input json.

numFound = jsonprase(ABOVE_INPUT_JSON).find_json_node_by_xpath('/response/numFound')
print(numFound)