Selenium returns an empty html body

Question

I am running selenium from AWS lambda, to automate a process, I was getting an error for element not found, and when I printed the page source it seems like it is actually empty, I have tried sleep for 30 seconds, without success, this works as expected locally

from time import sleep
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.chrome.options import Options
    from secretsx import get_secret_values



def main():
 chrome_options = Options()  
chrome_options.binary_location = '/opt/headless-chromium'
chrome_options.add_argument("--headless")  
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--single-process')
chrome_options.add_argument('--disable-dev-shm-usage')

driver = webdriver.Chrome('/opt/chromedriver',options=chrome_options)

driver.get(mypage)
driver.maximize_window()
driver.implicitly_wait(30) 

print(driver.page_source) #this returns <html xmlns="http://www.w3.org/1999/xhtml"><head></head><body></body></html>

Asked By: Javier Perez

||

Source

Answer 1

the issue was actually related to SSL certs, you can setup selenium to ignore it

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities


desired_capabilities = DesiredCapabilities.CHROME.copy()
desired_capabilities['acceptInsecureCerts'] = True

driver = webdriver.Chrome('/opt/chromedriver',options=chrome_options,desired_capabilities=desired_capabilities)

I hope this is helpful for somebody haha since I spent 6 hours looking for an answer

Answered By: Javier Perez

Answer 2

You saved me from spending a few hours on this problem. My code looked like this:

    options = Options()
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-gpu')
    options.add_argument('--disable-dev-shm-usage')    
    desired_capabilities = DesiredCapabilities.CHROME.copy() 
    desired_capabilities['acceptInsecureCerts'] = True
    browser = webdriver.Chrome('chromedriver', options=options, desired_capabilities=desired_capabilities)

Answered By: Bruno

Selenium returns an empty html body

Question:

Answers: