How to scrap the website by finding a class element using selenium driver in python?

Question

I am unable to find an element when try to inspect element on each class, basically want to scrap the website courses using selenium driver. But i am unable to get a data back when running scripting its terminating. How can improve my logic to get data back when using selenium driver to scrap website?

//Steps to inspect the element using browser.
1). Right click on the course content
2). Look for div class has a unique name, but some name are not relevant for selinium driver to receive data of that url link(this could be my problem and dont know ways around this).
3. Copy the class name by property element, copy as xpath then it will give me what is given on the parameters indicated from the code.

scrapping of the class-central.com website links

this application uses selinium driver to access the web-pages

`from selenium import webdriver
import time
url = "https://www.classcentral.com/collection/top-free-online-courses"

driver = webdriver.Chrome()

driver.get(url)
time.sleep(5)

search_results = driver.find_element('search-results__blanket')

for context in search_results:
    home_subjects = context.find_elements_by_xpath('//*[@id="page-collection"]/div[1]/div[2]/div[5]/ol/li[1]/div[1]/div[1]/div[2]/a[1]/h2').text
    home_discover = context.find_elements_by_xpath('//*[@id="page-collection"]/div[1]/div[2]/div[5]/ol/li[3]/div[1]/div[1]/div[2]/a[1]/h2').text
    print(home_subjects, home_discover)

driver.quit()

// Error from the terminal
 File "C:UsersZuxPycharmProjectsClassCentralmain.py", line 13, in <module>
    search_results = driver.find_element('search-results__blanket')
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:UsersZuxPycharmProjectsClassCentralClassCentralLibsite-packagesseleniumwebdriverremotewebdriver.py", line 830, in find_element
    return self.execute(Command.FIND_ELEMENT, {"using": by, "value": value})["value"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:UsersZuxPycharmProjectsClassCentralClassCentralLibsite-packagesseleniumwebdriverremotewebdriver.py", line 440, in execute
    self.error_handler.check_response(response)
  File "C:UsersZuxPycharmProjectsClassCentralClassCentralLibsite-packagesseleniumwebdriverremoteerrorhandler.py", line 245, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: invalid locator
  (Session info: chrome=110.0.5481.178)`

Asked By: Hemna Lucas

||

Source

Answer 1

you may find some insights from the code below:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time

url = "https://www.classcentral.com/collection/top-free-online-courses"
driver = webdriver.Chrome()
driver.get(url)
time.sleep(1)

all_courses = driver.find_element(by=By.CLASS_NAME, value='catalog-grid__results')
course_titles = all_courses .find_elements(by=By.CSS_SELECTOR, value='[class="color-charcoal course-name"]')

for title in course_titles:
    print(title.text)

Answered By: Ajeet Verma

How to scrap the website by finding a class element using selenium driver in python?

Question:

scrapping of the class-central.com website links

this application uses selinium driver to access the web-pages

Answers: