Webscraping with div ng scope class

Question

I am new to web scraping. I am trying to scrape the Event History Table on this site (for example) https://portal.assessor.lacounty.gov/parceldetail/2004001003

However, the class of this "table" is ng-scope and none of the tutorial I googled mentioned how to tackle this class. Can someone provide me some guidance on how to scrape this into a dataframe?

My problem is when I run this it returns nothing

table1 = soup.find('div', attrs={"class": "panel-body"})

Thank you!

Asked By: sucksatmath

||

Source

Answer 1

The HTML element "panel-body" is empty because the JavaScript has not loaded the API response when your scraper requests the page.

Rather than scraping the page, you can query the API directly. I found the API endpoint by inspecting the page and viewing the GET requests when the page loaded.

import requests

# GET request to API
url = 'https://portal.assessor.lacounty.gov/api/parcel_assessmenthistory?ain=2004001003'

# request url content
r = requests.get(url)

# read JSON from response
data = r.json()

# print first element of JSON
print(data['Parcel_AssessmentHistory'][0])

Output:

{'TaxYear': '2023', 'BillNumber': '', 'BillType': '', 'BillTypeDesc': '', 'BillStatusCode': '', 'BillStatusCodeDesc': '', 'OwnershipCode1': '3', 'OwnershipCode1Desc': 'Review Not Required', 'OwnershipCode2': '7', 'OwnershipCode2Desc': 'Non-Reappraisable', 'DocumentType': 'B', 'DocumentTypeDesc': 'File Correction                                             ', 'DocumentReasonCode': 'U', 'DocumentReasonDesc': 'Non-reappraisable Trust Transfer (SEC. 62D)', 'DateCreated': '', 'DateToAuditor': '', 'RecordingDate': '05/29/2009', 'SequenceNumber': '50', 'DocumentNumber': '0798640', 'PercentInterest': '00%-0', 'LandValue': '697735', 'ImpValue': '295271', 'LandReasonCode': 'T', 'LandReasonCodeDesc': 'Change in Ownership (Transfer)', 'ImpReasonCode': 'T', 'ImpReasonCodeDesc': 'Change in Ownership (Transfer)', 'LandBaseYear': '2006', 'ImpBaseYear': '2006', 'LandValuationDate': '09/06/2005', 'ImpValuationDate': '09/06/2005', 'LandAppraisalDate': '10/31/2005', 'ImpAppraisalDate': '10/31/2005', 'LandAddedValue': '', 'ImpAddedValue': '', 'NewConstructionDate': '', 'NewConstructionType': '', 'NewConstructionTypeDesc': '', 'FixtureValue': '', 'PersonalPropertyValue': '', 'HomeOwnerExemption': '0', 'RealEstateExemption': '', 'FixtureExemption': '', 'PersonalPropertyExemption': '', 'TaxOriginCode': '', 'TaxOriginCodeDesc': '', 'TaxReasonCode': '', 'TaxReasonCodeDesc': '', 'TaxRateArea': '00016', 'ProcessingCode': '', 'ProcessingCodeDesc': '', 'CPI': '', 'PSEGYear': '2023', 'ExemptionKey': ' ', 'ExemptionStatus': 'None'}

Inspect page for API endpoint URLs:

Answered By: Captain Caveman

Webscraping with div ng scope class

Question:

Answers: