How to get the correct info while extracting some particular key value from nested JSON

Question

I want to extract the task name and config corresponding to each task into new variable.

The code that I have shared is not giving me the desired output. Although it is extracting some info but it is not able to extract all the required details.

Here is the json:

old = {
        "tasks": [
            {
                "task_group_id": "Task_group_1",
                "branch": [
                    {
                        "task_id": "Task_Name_1",
                        "code_file_path": "tasks/base_creation/final_base_logic.hql",
                        "language": "hive",
                        "config": {
                            "k1": "v1",
                            "Q1":"W1"
                        },
                        "sequence": 1,
                        "condition": "in_start_date in range [2021-10-01 , 2023-11-04]"
                    }
                ],
                "default": {
                    "task_id": "Task_group_1_default",
                    "code_file_path": "tasks/base_creation/default_base_logic.hql",
                    "language": "hive",
                    "config": {}
                }
            },
            {
                "task_group_id": "Task_group_2",
                "branch": [
                    {
                        "task_id": "Task_Name_2",
                        "code_file_path": "tasks/variables_creation/final_cas_logic.py",
                        "language": "pyspark",
                        "config": {
                            "k2": "v2"
                        },
                        "sequence": 1,
                        "condition": "in_start_date in range [2022-02-01 , 2023-11-04]"
                    },
                    {
                        "task_id": "Task_Name_3",
                        "code_file_path": "tasks/variables_creation/final_sor_logic.py",
                        "language": "pyspark",
                        "config": {
                            "k3": "v3"
                        },
                        "sequence": 2,
                        "condition": "in_start_date in range [2021-10-01 , 2022-01-31]"
                    }
                ],
                "default": {
                    "task_id": "Task_group_2_default",
                    "code_file_path": "tasks/variables_creation/default_variables_logic.py",
                    "language": "pyspark",
                    "config": {}
                }
            }
        ],
        "dependencies": " ['task_group_id_01_Name >> task_group_id_02_Name']"
    }

Here is my code for extracting the info:

o_mod = []
for grp in range(len(old['tasks'])):
    for task_id in range(len(old['tasks'][grp]['branch'])):
        o_mod.append({})
        o_mod[grp]['task_id'] = old['tasks'][grp]['branch'][task_id]['task_id']
        o_mod[grp]['config'] = old['tasks'][grp]['branch'][task_id]['config']
            
print(o_mod)

Here is the output which is wrong:

[{'task_id': 'Task_Name_1', 'config': {'k1': 'v1', 'Q1': 'W1'}},
 {'task_id': 'Task_Name_3', 'config': {'k3': 'v3'}},
 {}]

I want output to look like this (Correct output):

[{'task_id': 'Task_Name_1', 'config': {'k1': 'v1', 'Q1': 'W1'}},
 {'task_id': 'Task_Name_2', 'config': {'k2': 'v2'}},
 {'task_id': 'Task_Name_3', 'config': {'k3': 'v3'}}}]

Asked By: Professor

||

Source

Answer 1

You could use a nested list comprehension over tasks and branch:

o_mod = [ { 'task_id' : b['task_id'], 'config' : b['config'] } for t in old['tasks'] for b in t['branch'] ]

Output:

[
    {
        "task_id": "Task_Name_1",
        "config": {
            "k1": "v1",
            "Q1": "W1"
        }
    },
    {
        "task_id": "Task_Name_2",
        "config": {
            "k2": "v2"
        }
    },
    {
        "task_id": "Task_Name_3",
        "config": {
            "k3": "v3"
        }
    }
]

Answered By: Nick

Answer 2

a multiline alternative if that’s something you fancy.

new_list = []
task_list = old["tasks"]
for task in task_list:
    branch_list = task["branch"]
    for branch in branch_list:
        new_list.append({"task_id":branch["task_id"],"config":branch["config"]})



print(new_list)

Output:

[{‘task_id’: ‘Task_Name_1’, ‘config’: {‘k1’: ‘v1’, ‘Q1’: ‘W1’}}, {‘task_id’: ‘Task_Name_2’,
‘config’: {‘k2’: ‘v2’}}, {‘task_id’: ‘Task_Name_3’, ‘config’: {‘k3’: ‘v3’}}]

Answered By: Zenith_1024

How to get the correct info while extracting some particular key value from nested JSON

Question:

Answers: