Converting identifier naming between camelCase and underscores during JSON serialization/deserialization

Question:

I am working on a python/django application that serves as a web API server to its frontend counterpart. The data exchange between the server and the client is in JSON format with the use of XMLHttpRequest (Javascript). For those that are familiar with both Python and Javascript, you know that they have different identifier naming convention when it comes to variables/methods/attributes; Python uses names_with_underscores while Javascript prefers camelCaseNames. I would like to keep both conventions in their respective worlds and perform conversion on identifiers when data exchange happens.

I have decided to have the conversion performed on the server (Python). In my own opinion, the most logical place for this two-way conversion to happen is during JSON serialization/deserialization. How should I go about implementing this approach? Examples are highly appreciated.

Note that I am on Python 2.7.

Asked By: tamakisquare

||
Source

Answers:

One way to do it using regular expressions,

import re

camel_pat = re.compile(r'([A-Z])')
under_pat = re.compile(r'_([a-z])')

def camel_to_underscore(name):
    return camel_pat.sub(lambda x: '_' + x.group(1).lower(), name)

def underscore_to_camel(name):
    return under_pat.sub(lambda x: x.group(1).upper(), name)

And,

>>> camel_to_underscore('camelCaseNames')
'camel_case_names'
>>> underscore_to_camel('names_with_underscores')
'namesWithUnderscores'

Note: You have to use a function (lambda expression here) for accomplishing the case change but that seems pretty straightforward.

EDIT:

If you truly wanted to intercept and adjust json objects between Python and Javascript you would have to rewrite functionality of the json module. But I think that is much more trouble than it’s worth. Instead something like this would be equivalent and not be too much of a hit performance-wise.

To convert each key in a dict representing your json object, you can do something like this,

def convert_json(d, convert):
    new_d = {}
    for k, v in d.iteritems():
        new_d[convert(k)] = convert_json(v,convert) if isinstance(v,dict) else v
    return new_d

You only need to provide which function to apply,

>>> json_obj = {'nomNom': {'fooNom': 2, 'camelFoo': 3}, 'camelCase': {'caseFoo': 4, 'barBar': {'fooFoo': 44}}}
>>> convert_json(json_obj, camel_to_underscore)
{'nom_nom': {'foo_nom': 2, 'camel_foo': 3}, 'camel_case': {'case_foo': 4, 'bar_bar': {'foo_foo': 44}}}

You can wrap all of this logic in new load and dump functions,

import json

def convert_load(*args, **kwargs):
    json_obj = json.load(*args, **kwargs)
    return convert_json(json_obj, camel_to_underscore)

def convert_dump(*args, **kwargs):
    args = (convert_json(args[0], underscore_to_camel),) + args[1:]
    json.dump(*args, **kwargs)

And use then just as you would json.load and json.dump.

Answered By: Jared

Jared’s answer does not take into account the possibility of arrays with objects in a json object structure.

The solution requires three functions to recursively handle the arrays.

For converting from CamelCase to underscores_with_spaces:

def convert(s):
    a = re.compile('((?<=[a-z0-9])[A-Z]|(?!^)[A-Z](?=[a-z]))')
    return a.sub(r'_1', s).lower()

For the json object

def convertJSON(j):
    out = {}
    for k in j:
        newK = convert(k)
        if isinstance(j[k],dict):
            out[newK] = convertJSON(j[k])
        elif isinstance(j[k],list):
            out[newK] = convertArray(j[k])
        else:
            out[newK] = j[k]
    return out

For arrays within the json object:

def convertArray(a):
    newArr = []
    for i in a:
        if isinstance(i,list):
            newArr.append(convertArray(i))
        elif isinstance(i, dict):
            newArr.append(convertJSON(i))
        else:
            newArr.append(i)
    return newArr

Usage:

convertJSON({
    "someObject": [
        {
            "anotherObject": "CamelCaseValue"
        },
        {
            "anotherObject": "AnotherCamelCaseValue"
        }
    ]
})

Yields:

{
    'some_object': [
        {
            'another_object': 'CamelCaseValue'
        },
        {
            'another_object': 'AnotherCamelCaseValue'
        }
    ]
}
Answered By: Evan Siroky

I have improved on Evan Siroky’s answer.

import re


class convert:
    def __init__(self):
        self.js_to_py_re = re.compile('((?<=[a-z0-9])[A-Z]|(?!^)[A-Z](?=[a-z]))')
        self.py_to_js_re = re.compile(r'_([a-z])')

    def convert_js_to_py(self, s):
        return self.js_to_py_re.sub(r'_1', s).lower()

    def convert_py_to_js(self, s):
        return self.py_to_js_re.sub(lambda x: x.group(1).upper(), s)

    def js_to_py_JSON(self, j):
        out = {}
        for k in j:
            newK = self.convert_js_to_py(k)
            if isinstance(j[k], dict):
                out[newK] = self.js_to_py_JSON(j[k])
            elif isinstance(j[k], list):
                out[newK] = self.js_to_py_array(j[k])
            else:
                out[newK] = j[k]
        return out

    def js_to_py_array(self, a):
        newArr = []
        for i in a:
            if isinstance(i, list):
                newArr.append(self.js_to_py_array(i))
            elif isinstance(i, dict):
                newArr.append(self.js_to_py_JSON(i))
            else:
                newArr.append(i)
        return newArr

    def py_to_js_JSON(self, j):
        out = {}
        for k in j:
            newK = self.convert_py_to_js(k)
            if isinstance(j[k], dict):
                out[newK] = self.py_to_js_JSON(j[k])
            elif isinstance(j[k], list):
                out[newK] = self.py_to_js_array(j[k])
            else:
                out[newK] = j[k]
        return out

    def py_to_js_array(self, a):
        newArr = []
        for i in a:
            if isinstance(i, list):
                newArr.append(self.py_to_js_array(i))
            elif isinstance(i, dict):
                newArr.append(self.py_to_js_JSON(i))
            else:
                newArr.append(i)
        return newArr


if __name__ == '__main__':
    py_to_js = {
        'some_object': [
            {
                'another_object': 'CamelCaseValue'
            },
            {
                'another_object': 'AnotherCamelCaseValue'
            }
        ]
    }
    js_to_py = {
        "someObject": [
            {
                "anotherObject": "CamelCaseValue"
            },
            {
                "anotherObject": "AnotherCamelCaseValue"
            }
        ]
    }
    print convert().py_to_js_JSON(py_to_js)
    print convert().js_to_py_JSON(js_to_py)

The above yields:

{'someObject': [{'anotherObject': 'CamelCaseValue'}, {'anotherObject': 'AnotherCamelCaseValue'}]}
{'some_object': [{'another_object': 'CamelCaseValue'}, {'another_object': 'AnotherCamelCaseValue'}]}
Answered By: viv

I just found this answer after doing this myself for a project with TornadoWeb. So I rewrote it to use recursion, it’s python 3.7, but could be easily adapted to python 2.7 by just changing the items to iteritems

def camel(snake_str):
    first, *others = snake_str.split('_')
    return ''.join([first.lower(), *map(str.title, others)])

def camelize_dict(snake_dict):
    new_dict = {}
    for key, value in snake_dict.items():
        new_key = camel(key)
        if isinstance(value, list):
            new_dict[new_key] = list(map(camelize_dict, value))
        elif isinstance(value, dict):
            new_dict[new_key] = camelize_dict(value)
        else:
            new_dict[new_key] = value
    return new_dict

just import camelize_dict(dictionary)

you can also camelize a string using a lambda:

camel = lambda key: ''.join([key.split('_')[0].lower(), *map(str.title, key.split('_')[1:])])
Answered By: Asitaka

For future googlers, the humps package can do this for you.

import humps
humps.decamelize({'outerKey': {'innerKey': 'value'}})
# {'outer_key': {'inner_key': 'value'}}
Answered By: Fush

I’ve got something even better!

mapper.py

import re


class Mapper:

    def __init__(self):
        pass

    @staticmethod
    def camelcase_to_underscore(camel_case):
        if isinstance(camel_case, dict) or isinstance(camel_case, list):
            return Mapper.dict_camelcase_to_underscore(camel_case)
        else:
            return re.sub('(((?<=[a-z])[A-Z])|([A-Z](?![A-Z]|$)))', '_\1', camel_case).lower().strip('_')


    @staticmethod
    def underscore_to_camelcase(underscore):
        if isinstance(underscore, dict) or isinstance(underscore, list):
            return Mapper.dict_underscore_to_camelcase(underscore)
        else:
            return Mapper.string_underscore_to_camelcase(underscore)

    @staticmethod
    def string_underscore_to_camelcase(underscore):
        if '_' in underscore:
            return re.sub(r'(?!^)_([a-zA-Z])', lambda m: m.group(1).upper(), underscore)
        else:
            return underscore

    @staticmethod
    def underscore_to_titlecase(underscore):
        if isinstance(underscore, dict) or isinstance(underscore, list):
            return Mapper.dict_underscore_to_titlecase(underscore)
        else:
            title_name = underscore.replace('_', ' ').title().replace(' ', '')
            return title_name

    @staticmethod
    def titlecase_to_camelcase(titlecase):
        if isinstance(titlecase, dict) or isinstance(titlecase, list):
            return Mapper.dict_titlecase_to_camelcase(titlecase)
        else:
            if titlecase.isupper():
                return titlecase.lower()
            else:
                val = titlecase[0].lower() + titlecase[1:]
                reg = re.compile('^[A-Z]+')
                front = reg.findall(titlecase)
                if len(front) > 0:
                    if front[0].isupper() and len(front[0]) > 1:
                        s1 = front[0][:-1].lower()
                        val = s1 + titlecase[len(s1):]
                if val[-2:] == "ID":
                    val = val[:-2] + "Id"
                elif val[-3:] == "IDs":
                    val = val[:-3] + "Ids"

                return val

    @staticmethod
    def dict_camelcase_to_underscore(obj):
        if isinstance(obj, dict):
            new_dict = {}
            for key, value in obj.items():
                underscore = Mapper.camelcase_to_underscore(key)
                if isinstance(value, dict) or isinstance(value, list):
                    value = Mapper.camelcase_to_underscore(value)
                new_dict[underscore] = value
            return new_dict
        elif isinstance(obj, list):
            new_list = []
            for o in obj:
                new_item = {}
                if isinstance(o, list):
                    new_item = Mapper.camelcase_to_underscore(o)
                elif isinstance(o, dict):
                    for key, value in o.items():
                        underscore = Mapper.camelcase_to_underscore(key)
                        if isinstance(value, dict) or isinstance(value, list):
                            value = Mapper.camelcase_to_underscore(value)
                        new_item[underscore] = value
                else:
                    new_item = o
                new_list.append(new_item)
            return new_list

    @staticmethod
    def dict_underscore_to_camelcase(obj):
        if isinstance(obj, dict):
            return {
                Mapper.string_underscore_to_camelcase(key) : Mapper.dict_underscore_to_camelcase(value)
                for key, value in obj.items()
            }

        if isinstance(obj, list):
            return [Mapper.dict_underscore_to_camelcase(x) for x in obj] 

        return obj

    @staticmethod
    def dict_underscore_to_titlecase(obj):
        if isinstance(obj, dict):
            new_dict = {}
            for key, value in obj.items():
                titlecase = Mapper.underscore_to_titlecase(key)
                if isinstance(value, dict) or isinstance(value, list):
                    value = Mapper.underscore_to_titlecase(value)
                new_dict[titlecase] = value
            return new_dict
        elif isinstance(obj, list):
            new_list = []
            for o in obj:
                new_dict = {}
                for key, value in o.items():
                    titlecase = Mapper.underscore_to_titlecase(key)
                    if isinstance(value, dict) or isinstance(value, list):
                        value = Mapper.underscore_to_titlecase(value)
                    new_dict[titlecase] = value
                new_list.append(new_dict)
            return new_list

    @staticmethod
    def dict_titlecase_to_camelcase(obj):
        if isinstance(obj, dict):
            new_dict = {}
            for key, value in obj.items():
                camelcase = Mapper.titlecase_to_camelcase(key)
                if isinstance(value, dict) or isinstance(value, list):
                    value = Mapper.titlecase_to_camelcase(value)
                new_dict[camelcase] = value
            return new_dict
        elif isinstance(obj, list):
            new_list = []
            for o in obj:
                new_dict = {}
                if isinstance(o, dict):
                    for key, value in o.items():
                        camelcase = Mapper.titlecase_to_camelcase(key)
                        if isinstance(value, dict) or isinstance(value, list):
                            value = Mapper.titlecase_to_camelcase(value)
                        new_dict[camelcase] = value
                    new_list.append(new_dict)
                else:
                    new_list.append(o)
        return new_list

and you’ve gotta have tests of course!

test_mapper.py

import random
import unittest
import uuid
from unittest.mock import MagicMock

from rest_framework_simplify.mapper import Mapper


class MapperTests(unittest.TestCase):

    def test_camelcase_to_underscore_not_capitalized(self):
        camel_case = 'camelCase'
        underscore = 'camel_case'
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_camelcase_to_underscore_capitalized(self):
        camel_case = 'CamelCase'
        underscore = 'camel_case'
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_camelcase_to_underscore_array_of_numbers(self):
        camel_case = {'camelCase': [1, 10]}
        underscore = {'camel_case': [1, 10]}
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_camelcase_to_underscore_array_of_strings(self):
        camel_case = {'camelCase': ['camelCase']}
        underscore = {'camel_case': ['camelCase']}
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_camelcase_to_underscore_array_of_bools(self):
        camel_case = {'camelCase': [True, False]}
        underscore = {'camel_case': [True, False]}
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_camelcase_to_underscore_empty_array(self):
        camel_case = {'camelCase': []}
        underscore = {'camel_case': []}
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_camelcase_to_underscore_array_of_objects(self):
        camel_case = {'camelCase': [{'camelCase': 1}]}
        underscore = {'camel_case': [{'camel_case': 1}]}
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_camelcase_to_underscore_array_of_mixed_types(self):
        int_type_value = random.randint(1, 10)
        str_type_value = str(uuid.uuid4())[:4]
        bool_type_value = False
        obj_type_value = {'camelCase': 1}
        ary_type_value = [int_type_value, obj_type_value]
        underscore_mock = MagicMock(obj_type_value={'camel_case': 1}, ary_type_value=[int_type_value, {'camel_case': 1}])
        camel_case = {'camelCase': [int_type_value, str_type_value, obj_type_value, ary_type_value, bool_type_value]}
        underscore = {'camel_case': [int_type_value, str_type_value, underscore_mock.obj_type_value, underscore_mock.ary_type_value, bool_type_value]}
        val = Mapper.camelcase_to_underscore(camel_case)
        self.assertEqual(val, underscore)

    def test_underscore_to_camelcase_array_of_mixed_types(self):
        int_type_value = random.randint(1, 10)
        str_type_value = str(uuid.uuid4())[:4]
        bool_type_value = False
        obj_type_value = {'camel_case': 1}
        ary_type_value = [int_type_value, obj_type_value]
        camel_case_mock = MagicMock(obj_type_value={'camelCase': 1}, ary_type_value=[int_type_value, {'camelCase': 1}])
        underscore = {'camel_case': [int_type_value, str_type_value, obj_type_value, ary_type_value, bool_type_value]}
        camel_case = {'camelCase': [int_type_value, str_type_value, camel_case_mock.obj_type_value, camel_case_mock.ary_type_value, bool_type_value]}
        val = Mapper.underscore_to_camelcase(underscore)
        self.assertEqual(val, camel_case)

    def test_underscore_to_camelcase(self):
        underscore = 'camel_case'
        camel_case = 'camelCase'
        val = Mapper.underscore_to_camelcase(underscore)
        self.assertEqual(val, camel_case)

    # I know this is horrible, but we have api's relying on this bug and we cannot fix it safely
    def test_underscore_to_backwards_compatible(self):
        underscore = 'address_line_1'
        camel_case = 'addressLine_1'
        val = Mapper.underscore_to_camelcase(underscore)
        self.assertEqual(val, camel_case)

    def test_underscore_to_camelcase_embedded(self):
        underscore = [{'camel_case': [{'more_camel_case': 5}]}]
        camel_case = [{'camelCase': [{'moreCamelCase': 5}]}]
        val = Mapper.underscore_to_camelcase(underscore)
        self.assertEqual(val, camel_case)

    def test_title_case_full_upper(self):
        upper = 'SSN'
        lower = 'ssn'
        val = Mapper.titlecase_to_camelcase(upper)
        self.assertEqual(val, lower)

    def test_title_case_mixed_bag(self):
        title = 'PMSystemID'
        camel = 'pmSystemId'
        val = Mapper.titlecase_to_camelcase(title)
        self.assertEqual(val, camel)

    def test_underscore_t0_titlecase(self):
        underscore = 'sum_charges'
        title = 'SumCharges'
        val = Mapper.underscore_to_titlecase(underscore)
        self.assertEqual(val, title)

Of course, you could always just pip install rest_framework_simplify and use it for yourself!

https://github.com/Skylude/django-rest-framework-simplify

Answered By: Chase Gibbons

small improvement from the top answer.

import re

camel_pat = re.compile(r'([A-Z])')
under_pat = re.compile(r'_([a-z])')

def camel_to_underscore(name):
    return camel_pat.sub(lambda x: '_' + x.group(1).lower(), name)

def underscore_to_camel(name):
    return under_pat.sub(lambda x: x.group(1).upper(), name)

def convert_json(d, convert):
    if isinstance(d, list):
        return [convert_json(item, convert) for item in d]
    new_d = {}
    for k, v in d.items():
        new_d[convert(k)] = convert_json(v,convert) if isinstance(v,dict) else v
    return new_d
Answered By: Egbert Ke

Here’s my combination of Evan’s and Jared’s answers above, which, in addition to lists, dicts and primitive types (strings, integers and booleans), also supports nested complex objects (data classes, for example):


import json
import re

camel_pat = re.compile(r'([A-Z])')
under_pat = re.compile(r'_([a-z])')


def camel_to_underscore(name):
    return camel_pat.sub(lambda x: '_' + x.group(1).lower(), name)


def underscore_to_camel(name):
    return under_pat.sub(lambda x: x.group(1).upper(), name)


def convert_array(a, convert):
    new_arr = []
    for i in a:
        if isinstance(i, list):
            new_arr.append(convert_array(i, convert))
        elif isinstance(i, dict):
            new_arr.append(convert_json(i, convert))
        else:
            new_arr.append(i)
    return new_arr


def convert_json(j, convert):
    out = {}
    for k in j:
        new_k = convert(k)
        if isinstance(j[k], dict):
            out[new_k] = convert_json(j[k], convert)
        elif isinstance(j[k], list):
            out[new_k] = convert_array(j[k], convert)
        elif isinstance(j[k], str) or isinstance(j[k], int):
            out[new_k] = j[k]
        else:
            out[new_k] = convert_json(vars(j[k]), convert)
    return out


def convert_load(*args, **kwargs):
    json_obj = json.loads(*args, **kwargs)
    return convert_json(json_obj, camel_to_underscore)


def convert_dump(*args, **kwargs):
    args = (convert_json(args[0], underscore_to_camel),) + args[1:]

    return json.dumps(*args, **kwargs)
Answered By: schiavuzzi
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.