Selenium finding and updating class with web app

Question

Some time ago I set up a Python script utilizing selenium to crawl across a website. I have tried to adopt the code to get data from a new webpage but have run into trouble. The issue seems to be that the page is an app rather than a more standard page (and my understanding is not ideal).

The element I seem to be trying to update is "ag-paging-number". I tried click() instead of page and this did not work either.

Ulitmately, the script should cycle through the 1496 pages and save down the data (Any tips on how to set the directory would be great). Any help would be appreciated.
Obviously once the script is working I will switch to headless

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
import codecs


options = Options()
options.add_argument("--window-size=1920,1200")

driver = webdriver.Chrome()

url = "https://app.climatevaluation.com/apps/projections/table/index.html"
driver.get(url)
for page in range(1,1496,1):
        time.sleep(10)
        try: 
            driver.find_element_by_class_name("ag-paging-number").click()
        except:
            break
with codecs.open('session'+str(page)+'.htm', 'w','utf-8') as out:
         out.write(driver.page_source)

Asked By: Matthew Oldham

||

Source

Answer 1

I think you are willing to click the next page button?

Use driver.find_element_by_xpath("//div[@ref='btNext']/span").click()

You might also want to add driver.implicitly_wait(10) before the loop

Answered By: levangode

Selenium finding and updating class with web app

Question:

Answers: