How to scrape string that does not have unique ID for data extraction?
Question:
Here in the image there is a text called for sale in 63702 Kolaram.
Please tell how to extract that string using BeautifulSoup Python.
https://www.magicbricks.com/property-for-sale-in-namakkal-pppfs
Answers:
You can simply do this with the find_all() function of beautifulsoap
import requests
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")
result_text = soup.find_all(text=your_text_which_you_want_to_find)
This will return a list with text which you want to find using bs4
If only texts are needed, selector pointing to the span elements can be used:
from bs4 import BeautifulSoup
import requests
import re
url = 'https://www.magicbricks.com/property-for-sale-in-namakkal-pppfs'
request = requests.get(url)
soup = BeautifulSoup(request.content, 'html5lib')
spans = [re.sub('[s]+', ' ', re.sub('[nrt]*', '', span.text)).strip() for span in soup.select('div.m-srp-card__container > div.m-srp-card__desc > div.m-srp-card__heading > h2 > span.m-srp-card__title')]
print(spans)
That will return list of texts:
['3 BHK Villa for Sale in 637201 Kolaram', '1 BHK House for Sale in Rasipuram', '2 BHK House for Sale in Rasipuram', 'Plot/Land for Sale in Thiruchengode Rd', 'Plot/Land for Sale in Mathiyampatty', 'Plot/Land for Sale in Namagiripetai', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in Ladhuvaadi', 'Plot/Land for Sale in TGP Star City', 'Plot/Land for Sale in Palpakki', 'Plot/Land for Sale in Pavai International school back side', 'Plot/Land for Sale in Rasipuram', '3 BHK Villa for Sale in Kondichettipatti', '2 BHK House for Sale in Thiruchengode Rd', '1 BHK House for Sale in Ganesapuram namakkal', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in Pon nagar bus stop', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in Rasipuram', '2 BHK House for Sale in Periyapatti', '2 BHK House for Sale in Tiruchengode', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in L.Kanavaipatti Road', 'Plot/Land for Sale in Muthugapatti', 'Plot/Land for Sale in Kondichettipatti', 'Plot/Land for Sale in Rasipuram', '2 BHK House for Sale in Sai Brindhavan nagar Neare', 'Plot/Land for Sale in sri sowrnabairavar nagar', '2 BHK House for Sale in Namagiripettai', '3 BHK House for Sale in Pachal']
If some other information will be needed from each card, reduce the selector and iterate over div elements, extracting information needed from their children.
Here in the image there is a text called for sale in 63702 Kolaram.
Please tell how to extract that string using BeautifulSoup Python.
https://www.magicbricks.com/property-for-sale-in-namakkal-pppfs
You can simply do this with the find_all() function of beautifulsoap
import requests
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")
result_text = soup.find_all(text=your_text_which_you_want_to_find)
This will return a list with text which you want to find using bs4
If only texts are needed, selector pointing to the span elements can be used:
from bs4 import BeautifulSoup
import requests
import re
url = 'https://www.magicbricks.com/property-for-sale-in-namakkal-pppfs'
request = requests.get(url)
soup = BeautifulSoup(request.content, 'html5lib')
spans = [re.sub('[s]+', ' ', re.sub('[nrt]*', '', span.text)).strip() for span in soup.select('div.m-srp-card__container > div.m-srp-card__desc > div.m-srp-card__heading > h2 > span.m-srp-card__title')]
print(spans)
That will return list of texts:
['3 BHK Villa for Sale in 637201 Kolaram', '1 BHK House for Sale in Rasipuram', '2 BHK House for Sale in Rasipuram', 'Plot/Land for Sale in Thiruchengode Rd', 'Plot/Land for Sale in Mathiyampatty', 'Plot/Land for Sale in Namagiripetai', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in Ladhuvaadi', 'Plot/Land for Sale in TGP Star City', 'Plot/Land for Sale in Palpakki', 'Plot/Land for Sale in Pavai International school back side', 'Plot/Land for Sale in Rasipuram', '3 BHK Villa for Sale in Kondichettipatti', '2 BHK House for Sale in Thiruchengode Rd', '1 BHK House for Sale in Ganesapuram namakkal', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in Pon nagar bus stop', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in Rasipuram', '2 BHK House for Sale in Periyapatti', '2 BHK House for Sale in Tiruchengode', 'Plot/Land for Sale in Rasipuram', 'Plot/Land for Sale in L.Kanavaipatti Road', 'Plot/Land for Sale in Muthugapatti', 'Plot/Land for Sale in Kondichettipatti', 'Plot/Land for Sale in Rasipuram', '2 BHK House for Sale in Sai Brindhavan nagar Neare', 'Plot/Land for Sale in sri sowrnabairavar nagar', '2 BHK House for Sale in Namagiripettai', '3 BHK House for Sale in Pachal']
If some other information will be needed from each card, reduce the selector and iterate over div elements, extracting information needed from their children.