How can I scrape the "Time" and other data in the advanced details section using Beautiful Soup

Question

Here is the url that I want to scrape the data ‘https://www.blockchain.com/explorer/transactions/btc/43eebdc59c6c5ce948ccd9cf514b6c2ece9f1289f136c2e3d9d69dcd29304142’

I tried to scrape using Beautiful Soup because it doesn’t have to open the browser like the Selenium does. So I tried to extract the data from the outer section

('section',{'class':'sc-f9148dd7-2 irWxzm'})

the irWxzm section
and then tried to find a little bit deeper to the targeted div tag but I don’t understand why after I extracted the data from ‘section’,{‘class’:’sc-f9148dd7-2 irWxzm’}, it seemed that the data has stopped from the advanced detailsthe data from irWxzm section and I can’t dive deeper to the desired div tag in the first picture.

Here my code that I wrote

import requests
import bs4
from bs4 import BeautifulSoup
url = 'https://www.blockchain.com/explorer/transactions/btc/43eebdc59c6c5ce948ccd9cf514b6c2ece9f1289f136c2e3d9d69dcd29304142' 
res = requests.get(url)
format = '%Y-%m-%d %H:%M'
soup = BeautifulSoup(res.content, 'html.parser')

soup2 = soup.find('section',{'class':'sc-f9148dd7-2 irWxzm'})

print(soup2)

I tried a lot but it can’t find any tag under ‘class’:’sc-f9148dd7-2 irWxzm’ except class="sc-c907597a-0 MqlNG and class="sc-c907597a-3 ctQMfW" according to second picture.

Could you help me find the way to get the data in the advanced details section please.
desired data

Thank you so very much in advance.

Asked By: user20433475

||

Source

Answer 1

The page loads the data from external URL via JavaScript, to load the data you can use next example:

from datetime import datetime

import requests

api_url = "https://www.blockchain.com/explorer/api/transaction?asset=btc&id=43eebdc59c6c5ce948ccd9cf514b6c2ece9f1289f136c2e3d9d69dcd29304142"

data = requests.get(api_url).json()
print(data)

Prints:

{
    "ticker": "btc",
    "transaction": {
        "txid": "43eebdc59c6c5ce948ccd9cf514b6c2ece9f1289f136c2e3d9d69dcd29304142",
        "size": 381,
        "version": 1,
        "locktime": 0,
        "fee": 170170,
        "inputs": [
            {
                "coinbase": False,
                "txid": "667f2825db6b03e349b5e4be7b4c4c5be266c242a6aaa0218480572ffc5a7b37",
                "output": 0,
                "sigscript": "47304402204ba063dca925f759777ed8818027c421cb4052ecf2e3b980c814bc528c73638e02206a3d58ec92d0be9915c14d6c4cef40a01d301286c90d82c1bcf166db0e94c3bb012103951bbeb5b73e530b6849fca68e470118f4b379ad9126015caf1355dc2a9e8480",
                "sequence": 4294967295,
                "pkscript": "76a9149c8ab044348d826b9ae88d698d575a45a6e8fc6988ac",
                "value": 207730,
                "address": "1FGiZB7K757EUixGcyeyME6Jp8qQZEiUUk",
                "witness": [],
            },
            {
                "coinbase": False,
                "txid": "3c2dc36fd0bebc46062362aff0c4f307d1c99900c5f358fdd37b436a15d37a5f",
                "output": 0,
                "sigscript": "4730440220322e489e971b2c651224c2e03bea408df8c67a0a1c18ddfd20e940d90a8e61990220707ba2431bde31500ebe6a2b3c4a7974b87c4b9ee33849e1453c0831318bed14012103951bbeb5b73e530b6849fca68e470118f4b379ad9126015caf1355dc2a9e8480",
                "sequence": 4294967295,
                "pkscript": "76a9149c8ab044348d826b9ae88d698d575a45a6e8fc6988ac",
                "value": 231716,
                "address": "1FGiZB7K757EUixGcyeyME6Jp8qQZEiUUk",
                "witness": [],
            },
        ],
        "outputs": [
            {
                "address": "1FGiZB7K757EUixGcyeyME6Jp8qQZEiUUk",
                "pkscript": "76a9149c8ab044348d826b9ae88d698d575a45a6e8fc6988ac",
                "value": 269276,
                "spent": True,
                "spender": {
                    "txid": "c7ed715e9f73b2792957af94d3143750525a29f6a62fd6f68d470e56e4bbef7b",
                    "input": 0,
                },
            },
            {
                "address": None,
                "pkscript": "6a208627c703aeac41df8acad1c643d9ee9c2370f9cace1af05a0ac41219116b5e0b",
                "value": 0,
                "spent": False,
                "spender": None,
            },
        ],
        "block": {"height": 756449, "position": 2},
        "deleted": False,
        "time": 1664582470,
        "rbf": False,
        "weight": 1524,
    },
    "rate": 16526.38,
    "latestBlock": 769853,
    "id": "43eebdc59c6c5ce948ccd9cf514b6c2ece9f1289f136c2e3d9d69dcd29304142",
    "description": False,
    "fiat": "USD",
    "labels": {},
}

To get the time:

print(datetime.fromtimestamp(data["transaction"]["time"]))

Prints:

2022-10-01 02:01:10

Answered By: Andrej Kesely

How can I scrape the "Time" and other data in the advanced details section using Beautiful Soup

Question:

Answers: