Not getting values from cells after scraping a table from a dynamic webpage using Selenium

Question

I have the following code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome('C:chromedriver_win32chromedriver.exe')
global_dynamicUrl = "https://draft.shgn.com/nfc/public/dp/788/grid"
driver.get(global_dynamicUrl)
wait = WebDriverWait(driver, 15)
table_entries = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table tr")))
print(len(table_entries))
print(table_entries)
html = driver.page_source
open('site.txt', 'wt').write(html)

with open('site.txt', encoding='utf-8') as f:
    lines = f.read()
driver.close()

The resulting scrape is saved in a text file for further analysis. The problem is that when checking or parsing what was scraped, there is no trace of the names that appear in the rendered webpage (you can check the pic). What could I be doing wrong?

Asked By: Carlos Marcano

||

Source

Answer 1

A minor adjustment would make your program perfecto. Instead of (By.CSS_SELECTOR, "table tr") induce WebDriverWait for visibility_of_all_elements_located() for the (By.CSS_SELECTOR, "table tr span.lname.ng-binding.medium") elements as follows:

Code block:

driver.get('https://draft.shgn.com/nfc/public/dp/788/grid')
table_entries = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table tr span.lname.ng-binding.medium")))
print(len(table_entries))
print(table_entries)
html = driver.page_source
open('site.txt', 'wt').write(html)
driver.quit()

Text file contents:

<span class="lname ng-binding medium" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Acuna Jr.</span>
.
<span class="lname ng-binding" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Trout</span>
.
<span class="team ng-binding">NYM<!-- ngIf: ::(dbs.sport === 'NFL') --></span></span> <span class="lname ng-binding" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Marte</span>
.
<span class="lname ng-binding" ng-class="::{'medium': (dbs.player.l.length >= 7), 'mediumlong': (dbs.player.l.length >= 10), 'long': (dbs.player.l.length >= 12), 'verylong': (dbs.player.l.length >= 14)}" ng-bind="::dbs.player.l">Doval</span>

Answered By: undetected Selenium

Not getting values from cells after scraping a table from a dynamic webpage using Selenium

Question:

Answers: