How do I get all divs inside of this div using webdriver and selenium?

Question

I am new to webscraping and am trying to scrape a website and would like to get all the elements that have data-feature-id=homepage/story with the ultimate goal of getting a subelement of each of the divs inside of the parent element. I am having trouble getting all of the elements with this parameter and am unsure how to go about doing so.

I’ve tried the outer div of all the elements that have data-feature-id=homepage/story; however that results in there being only one element in the list returned.

    def getLinks(self):
        container = self.driver.find_element(By.XPATH, '//*[@id="__next"]/div[3]/div/main/article/div')
        for item in container:
            headline = container.find_element(By.XPATH, '//*[@id="__next"]/div[3]/div/main/article/div/div[1]/div/div[1]/a/h3')
            print(headline)

The above returns

Type Error:'WebElement' object is not iterable

Can anyone please give some guidance on how to go about this problem?

Image of the html

Asked By: LoganN

||

Source

Answer 1

Instead of using XPath in this case, which may be prone to changing frequently based on new additions to the page (like new articles) or third party content (like ads), try using CSS Selectors.

Try entering the following CSS Selector in your browser’s inspector: div[data-feature-id='homepage/story']

For your Selenium script, try using the following line instead of your XPath Selector:

driver.find_element(By.CSS_SELECTOR, "div[data-feature-id='homepage/story']")

Answered By: Zach Bellay

How do I get all divs inside of this div using webdriver and selenium?

Question:

Answers: