Python Selenium – how to get all urls on a page that only load the link after clicking on the div?


I’m trying to scrap the results from this page using Selenium, but I got stuck on obtaining the url of each result. It seems safe to say that each card’s url is not stored on a <a> element and apparently not stored at all at any point of the inner html of each div.

The only way to obtain the address is by clicking on the div, which opens a new tab.
Currently, I’m using selenium to click on each one, copying the address and then closing the tab, but not only this is a much more complex and time consuming process but also could trigger some captcha by doing that many requests to the website.

Is there a way to obtain the urls of all offers on this page without this clicking process? I tried using the inspect tool on chrome but couldn’t figure out what is the js or wtv resposible for this behavior.


Asked By: Jn Coe



I checked out the site and it looks like each card-container has a data-id that can be used to access the listing.
The link for this card:

<div data-id="2593637292" class="card-container js-listing-card">{THE HTML FOR THAT CARD}</div>

would be

Answered By: Runinho
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.