How to find the attribute and element id by selenium.webdriver?


I am learning web scrapping since I need it for my work. I wrote the following code:

from selenium import webdriver    
driver = webdriver.Chrome(chromedriver)
df = pd.read_html(driver.find_element_by_id("table.example.display.datatable").get_attribute('example'))[0]

However, it is showing the following error:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="table.example.display.datatable"]"}
  (Session info: chrome=103.0.5060.134)

Then I inspect the table that I wanna scrape this table from this page
enter image description here

what is the attribute that needs to be included in get_attribute() function in the following line?

df = pd.read_html(driver.find_element_by_id("table.example.display.datatable").get_attribute('example'))[0]

what I should write in the driver.find_element_by_id?

Some tables have lots of records in multi-pages.
For example, this page has 2,246 entries, which shows 100 entries on each page. Once I tried to web-scrape it, there were only 320 entries in df and the record ID is from 1232-1713, which means it took entries from the next few pages and it is not starting from the first page to the end at the last page.

What we can do in such cases?

Asked By: S.EB



If you want to select table by @id you need






If you want to extract @id value you need


Since there is not much sense in searching by @id to extract that exact @id you might use other attribute of table node:

Answered By: JaSON

I personally suggest you to use explicit waits instead of implicit ones.
Anyway it’s not clear what you’re trying to do and what you’re looking for. So I will just stick to the question and show you how I would find an element ID:

from selenium import webdriver
from import Service
from import ChromeDriverManager
from import By
from import WebDriverWait
from import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
df = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.XPATH, "<XPATH_OF_THE_ELEMENT_YOU_WANT>"))).get_attribute("id")

By the way I suggest you to read the documentation that explains in detail how to locate items.

Answered By: Carapace

You need to get the outerHTML property of the table first, then call the table element from pandas.

You need to wait for element to be visible. Use explicit wait like WebdriverWait()

df = pd.read_html(tableRows)[0]

Import below libraries.

from import expected_conditions as EC
from import WebDriverWait
from import By
import pandas as pd


     ID      PMID  YEAR  ...                                 DSSP Natural Structure Final Structure
0   1643  16137634  2005  ...                     CCCCCCCCCCCSCCCC               NaN             NaN
1   1644  16137634  2005  ...                        CCTTSCCSSCCCC               NaN             NaN
2   1645  16137634  2005  ...                   CTTTCGGGHHHHHHHHCC               NaN             NaN
3   1646  16137634  2005  ...                   CGGGTTTHHHHHHHGGGC               NaN             NaN
4   1647  16137634  2005  ...                CCSCCCSSCHHHHHHHHHTTC               NaN             NaN
5   1910  16730859  2006  ...  CCCCCCCSSCCSHHHHHHHHTTHHHHHHHHSSCCC               NaN             NaN
6   1911  16730859  2006  ...                                CCSCC               NaN             NaN
7   1912  16730859  2006  ...                            CCSSSCSCC               NaN             NaN
8   1913  16730859  2006  ...       CCCSSCCSSCCSHHHHHTTHHHHTTTCSCC               NaN             NaN
9   1914  16730859  2006  ...                 CCSHHHHHHHHHHHHHCCCC               NaN             NaN
10  2110  11226440  2001  ...              CCCSSCCCBTTBTSSSSSSCSCC               NaN             NaN
11  3799   9204560  1997  ...                               CCSSCC               NaN             NaN
12  4149  16137634  2005  ...                       CCHHHHHHHHHHHC               NaN             NaN

[13 rows x 17 columns]
Answered By: KunduK