parse nested XML and extract attributes + tag text both

Question

My XML looks like this:

<?xml version="1.0" encoding="UTF-8" ?>
<main_heading timestamp="20220113">
<details>
    <offer id="11" new_id="12">
        <level>1&amp;1</level>
        <typ>Green</typ>
        <name>Alpha</name>
        <visits>
            <name>DONT INCLUDE</name>
        </visits>
    </offer>
    <offer id="12" new_id="31">
        <level>1&amp;1</level>
        <typ>Yellow</typ>
        <name>Beta</name>
        <visits>
            <name>DONT INCLUDE</name>
        </visits>
    </offer>
</details>
</main_heading>

I want to parse certain fields into a dataframe.

Expected Output

timestamp   id     new_id   level      name
20220113    11     12       1&amp;1    Alpha
20220113    12     31       1&amp;1    Beta

where NAME nested within the "visits" tag is not included. I just want to consider the outer "name" tag.

timestamp = soup.find('main_heading').get('timestamp')
df[timestamp'] = timestamp

this solves one part

The rest I can do like this:

typ = []
for i in (soup.find_all('typ')):
    typ.append(i.text)

but i don’t want to create several for loops for every new field

Asked By: x89

||

Source

Answer 1

Iterate over the offers and select its previous main_heading:

for e in soup.select('offer'):
    data.append({
        'timestamp': e.find_previous('main_heading').get('timestamp'),
        'id':e.get('id'),
        'id_old':e.get('old_id'),
        'level':e.level.text,
        'typ':e.typ.text,
        'name':e.select_one('name').text
    })

Or in alternative to exclude only some elements and be more generic:

for e in soup.select('offer'):
    
    d = {
        'timestamp': e.find_previous('main_heading').get('timestamp'),
        'id':e.get('id'),
        'id_old':e.get('old_id'),
    }

    d.update({c.name:c.text for c in e.children if c.name is not None and 'visits' not in c.name})

    data.append(d)

Example

from bs4 import BeautifulSoup
import pandas as pd

xml = '''<?xml version="1.0" encoding="UTF-8" ?>
<main_heading timestamp="20220113">
<details>
    <offer id="11" new_id="12">
        <level>1&amp;1</level>
        <typ>Green</typ>
        <name>Alpha</name>
        <visits>
            <name>DONT INCLUDE</name>
        </visits>
    </offer>
    <offer id="12" new_id="31">
        <level>1&amp;1</level>
        <typ>Yellow</typ>
        <name>Beta</name>
        <visits>
            <name>DONT INCLUDE</name>
        </visits>
    </offer>
</details>
</main_heading>
'''
soup = BeautifulSoup(xml,'xml')

data = []

for e in soup.select('offer'):
    data.append({
        'timestamp': e.find_previous('main_heading').get('timestamp'),
        'id':e.get('id'),
        'id_old':e.get('old_id'),
        'level':e.level.text,
        'typ':e.typ.text,
        'name':e.select_one('name').text
    })

pd.DataFrame(data)

Output

	timestamp	id	id_old	level	typ	name
0	20220113	11		1&1	Green	Alpha
1	20220113	12		1&1	Yellow	Beta

Answered By: HedgeHog

parse nested XML and extract attributes + tag text both

Question:

Answers:

Example

Output