How can I get the page source, without showing the page opened, using selenium with chromedriver and python?
Question:
I’m using selenium with Chrome driver; How can I get the page source, without showing the page opened? What I should specify in webdriver.ChromeOptions()?
Here the code:
from selenium.common.exceptions import WebDriverException
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("???")
bowser = webdriver.Chrome(chrome_options=chrome_options)
browser = webdriver.Chrome()
try:
browser.get("www.google.com")
html_content = browser.page_source
#do stuff
browser.quit()
except WebDriverException:
print "Invalid URL"
Answers:
You should not use ChromeDriver
but some headless Webdriver like HtmlUnitDriver
, explained here
If you are adamant to use selenium, then you can use any of the headless browsers such as htmlunit driver.
Else you can can just send a get request on the URL and get the response text.
Selenium / Chrome has a headless option, which allows you to load webpages from code:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
browser = Chrome(options=chrome_options, executable_path='path_to_chromedriver')
browser.get('https://wwww.mywebsite.com')
I’m using selenium with Chrome driver; How can I get the page source, without showing the page opened? What I should specify in webdriver.ChromeOptions()?
Here the code:
from selenium.common.exceptions import WebDriverException
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("???")
bowser = webdriver.Chrome(chrome_options=chrome_options)
browser = webdriver.Chrome()
try:
browser.get("www.google.com")
html_content = browser.page_source
#do stuff
browser.quit()
except WebDriverException:
print "Invalid URL"
You should not use ChromeDriver
but some headless Webdriver like HtmlUnitDriver
, explained here
If you are adamant to use selenium, then you can use any of the headless browsers such as htmlunit driver.
Else you can can just send a get request on the URL and get the response text.
Selenium / Chrome has a headless option, which allows you to load webpages from code:
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
browser = Chrome(options=chrome_options, executable_path='path_to_chromedriver')
browser.get('https://wwww.mywebsite.com')