clicking a tab on a page to alow selenium to scrape

Question

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from time import sleep
from datetime import datetime
import pandas as pd
import warnings
import os

warnings.filterwarnings('ignore')


url = 'https://www.fifa.com/fifaplus/en/match-centre/match/17/255711/285063/400128082?country=IE&wtw-filter=ALL'

option = Options()
option.headless = True
driver = webdriver.Chrome("C:/Users/paulc/Documents/Medium Football/chromedriver.exe",options=option)

# Scraping the data

HomeTeam = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[1]/div[1]/p").text
AwayTeam = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[3]/div[2]").text
Result = driver.find_element(By.XPATH, "/html/body/div[1]/main/div/div[1]/div/section/div[1]/div[1]/div[2]/div[3]/div/div[1]/div/div/div[2]").text

elem = WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div1/main/div/div[3]/div/div1/div/div[4]")))
elem.click()

Hi Guys, I am looking to scrape the world cup data, I have managed to start it easily by getting the team names and scores. The in game statistics are housed in the stats tab in the image. So to start scraping them I need to be able to click on it with selenium and make the page active.

Am i missing something obvious and that this cannot be done through an xpath?

help is appreciated.

Asked By: Paul Corcoran

||

Source

Answer 1

In case you want to click on "LINE UP" tab this can be done with the following XPath: "//div[text()='LINE UP']". To click on "STATS" tab you can use this XPath: "//div[text()='STATS']". So, Selenium code line can be as following:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='LINE UP']"))).click()

For the LINE UP tab and

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()='STATS']"))).click()

for the "STATS" tab.
Also, you need to improve your locators. Very long absolute XPaths are very breakable.

Answered By: Prophet

clicking a tab on a page to alow selenium to scrape

Question:

Answers: