clicking a tab on a page to alow selenium to scrape


from selenium import webdriver
from import Options
from import WebDriverWait
from import expected_conditions as EC
from import By
from time import sleep
from datetime import datetime
import pandas as pd
import warnings
import os


url = ''

option = Options()
option.headless = True
driver = webdriver.Chrome("C:/Users/paulc/Documents/Medium Football/chromedriver.exe",options=option)

# Scraping the data

HomeTeam = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[1]/div[1]/p").text
AwayTeam = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[3]/div[2]").text
Result = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[2]").text

elem = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div1/main/div/div[3]/div/div1/div/div[4]")))

enter image description here

Hi Guys, I am looking to scrape the world cup data, I have managed to start it easily by getting the team names and scores. The in game statistics are housed in the stats tab in the image. So to start scraping them I need to be able to click on it with selenium and make the page active.

Am i missing something obvious and that this cannot be done through an xpath?

help is appreciated.

Asked By: Paul Corcoran



In case you want to click on "LINE UP" tab this can be done with the following XPath: "//div[text()='LINE UP']". To click on "STATS" tab you can use this XPath: "//div[text()='STATS']". So, Selenium code line can be as following:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='LINE UP']"))).click()

For the LINE UP tab and

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='STATS']"))).click()

for the "STATS" tab.
Also, you need to improve your locators. Very long absolute XPaths are very breakable.

Answered By: Prophet