How to run Headless Chrome with Selenium in Python?
Question:
I’m trying some stuff out with selenium, and I really want my script to run quickly.
I thought that running my script with headless Chrome would make it faster.
First, is that assumption correct, or does it not matter if I run my script with a headless driver?
I want headless Chrome to work, but somehow it isn’t working correctly. I tried different things, and most suggested that it would work as said here in the October update:
How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
But when I tried that, I saw weird console output, and it still doesn’t seem to work.
Any tips appreciated.
Answers:
To run chrome-headless just add --headless via chrome_options.add_argument, e.g.:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
#chrome_options.add_argument("--disable-extensions")
#chrome_options.add_argument("--disable-gpu")
#chrome_options.add_argument("--no-sandbox") # linux only
chrome_options.add_argument("--headless")
# chrome_options.headless = True # also works
driver = webdriver.Chrome(options=chrome_options)
start_url = "https://duckgo.com"
driver.get(start_url)
print(driver.page_source.encode("utf-8"))
# b'<!DOCTYPE html><html rel="nofollow noreferrer">headless-chrome
If you are using Linux environment, may be you have to add --no-sandbox as well and also specific window size settings. The --no-sandbox flag is no needed on Windows if you set user container properly.
Use --disable-gpu only on Windows. Other platforms no longer require it. The --disable-gpu flag is a temporary work around for a few bugs.
//Headless chrome browser and configure
WebDriverManager.chromedriver().setup();
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--no-sandbox");
chromeOptions.addArguments("--headless");
chromeOptions.addArguments("disable-gpu");
// chromeOptions.addArguments("window-size=1400,2100"); // Linux should be activate
driver = new ChromeDriver(chromeOptions);
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path="./chromedriver", options=chrome_options)
url = "https://stackoverflow.com/questions/53657215/running-selenium-with-headless-chrome-webdriver"
driver.get(url)
sleep(5)
h1 = driver.find_element_by_xpath("//h1[@itemprop='name']").text
print(h1)
Then I run script on our local machine
➜ python script.py
Running Selenium with Headless Chrome Webdriver
It is working and it is with headless Chrome.
Todo (tested on headless server Debian Linux 9.4):
-
Do this:
# install chrome
curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list
apt-get -y update
apt-get -y install google-chrome-stable
# install chrome driver
wget https://chromedriver.storage.googleapis.com/77.0.3865.40/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
mv chromedriver /usr/bin/chromedriver
chown root:root /usr/bin/chromedriver
chmod +x /usr/bin/chromedriver
-
Install selenium:
pip install selenium
and run this Python code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("no-sandbox")
options.add_argument("headless")
options.add_argument("start-maximized")
options.add_argument("window-size=1900,1080");
driver = webdriver.Chrome(chrome_options=options, executable_path="/usr/bin/chromedriver")
driver.get("https://www.example.com")
html = driver.page_source
print(html)
Install & run containerized Chrome:
docker pull selenium/standalone-chrome
docker run --rm -d -p 4444:4444 --shm-size=2g selenium/standalone-chrome
Connect using webdriver.Remote:
driver = webdriver.Remote('http://localhost:4444/wd/hub', webdriver.DesiredCapabilities.CHROME)
driver.set_window_size(1280, 1024)
driver.get('https://www.google.com')
Once you have selenium and web driver installed. Below worked for me with headless Chrome on linux cluster :
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-extensions")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")
options.add_experimental_option("prefs",{"download.default_directory":"/databricks/driver"})
driver = webdriver.Chrome(chrome_options=options)
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=r"C:Program
FilesGoogleChromeApplicationchromedriver.exe", options=chrome_options)
This is ok for me.
As stated by the accepted answer:
options.add_argument("--headless")
These tips might help to speed things up especially for headless:
There are quite a few things you can do in headless that you cant do in non headless
Since you will be using Chrome Headless, I've found adding this reduces the CPU usage by about 20% for me (I found this to be a CPU and memory hog when looking at htop)
--disable-crash-reporter
This will only disable when you are running in headless This might speed things up for you!!!
My settings are currently as follows and I reduce the CPU (but only a marginal time saving) by about 20%:
options.add_argument("--no-sandbox");
options.add_argument("--disable-dev-shm-usage");
options.add_argument("--disable-renderer-backgrounding");
options.add_argument("--disable-background-timer-throttling");
options.add_argument("--disable-backgrounding-occluded-windows");
options.add_argument("--disable-client-side-phishing-detection");
options.add_argument("--disable-crash-reporter");
options.add_argument("--disable-oopr-debug-crash-dump");
options.add_argument("--no-crash-upload");
options.add_argument("--disable-gpu");
options.add_argument("--disable-extensions");
options.add_argument("--disable-low-res-tiling");
options.add_argument("--log-level=3");
options.add_argument("--silent");
I found this to be a pretty good list (full list I think) of command line switches with explanations: https://peter.sh/experiments/chromium-command-line-switches/
Some additional things you can turn off are also mentioned here: https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md
I hope this helps someone
Recently there is an update performed on headless mode of Chrome.
The flag --headless is now modified and can be used as below
- For Chrome version 109 and above,
--headless=new flag allows us to explore full functionality Chrome browser in headless mode.
- For Chrome version 108 and below (till Version 96),
--headless=chrome option will provide us the headless chrome browser.
So, let's add
options.add_argument("--headless=new")
for newer version of Chrome in headless mode as mentioned above.
You can run Headless Chrome with Selenium in Python as shown below. *--headless=new is better because--headless uses old headless mode according Headless is Going Away!:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless=new") # Here
driver = webdriver.Chrome(options=options)
Or:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless=new") # Here
driver = webdriver.Chrome(options=options)
In addition, the examples below can test Django Admin with Headless Chrome, Selenium, pytest-django and Django. *My answer explains how to test Django Admin with multiple Headless browsers(Chrome, Microsoft Edge and Firefox), Selenium, pytest-django and Django:
# "tests/test.py"
import pytest
from selenium import webdriver
from django.test import LiveServerTestCase
@pytest.fixture(scope="class")
def chrome_driver_init(request):
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
chrome_driver = webdriver.Chrome(options=options)
request.cls.driver = chrome_driver
yield
chrome_driver.close()
@pytest.mark.usefixtures("chrome_driver_init")
class Test_URL_Chrome(LiveServerTestCase):
def test_open_url(self):
self.driver.get(("%s%s" % (self.live_server_url, "/admin/")))
assert "Log in | Django site admin" in self.driver.title
Or:
# "tests/conftest.py"
import pytest
from selenium import webdriver
@pytest.fixture(scope="class")
def chrome_driver_init(request):
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
chrome_driver = webdriver.Chrome(options=options)
request.cls.driver = chrome_driver
yield
chrome_driver.close()
# "tests/test.py"
import pytest
from django.test import LiveServerTestCase
@pytest.mark.usefixtures("chrome_driver_init")
class Test_URL_Chrome(LiveServerTestCase):
def test_open_url(self):
self.driver.get(("%s%s" % (self.live_server_url, "/admin/")))
assert "Log in | Django site admin" in self.driver.title
There are different ways of running Chrome in headless environments. (You'll find more details in this answer: https://stackoverflow.com/a/73840130/7058266)
One, the standard headless mode: (Faster than headed mode, but you may experience compatibility issues.)
options.add_argument("--headless")
Then there's the new Chrome headless mode as of Chrome 109: (It runs at the same speed as headed mode, as the two are virtually identical.)
options.add_argument("--headless=new")
(Between Chrome 96 and 108, that new mode used to be --headless=chrome, but it was renamed.)
You can also run regular Chrome in a headless environment if using a headless display, such as Xvfb with a Python program that controls it, such as pyvirtualdisplay. (See https://stackoverflow.com/a/6300672/7058266 and https://stackoverflow.com/a/23447450/7058266)
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
driver = webdriver.Chrome()
driver.get('http://www.google.com')
driver.quit()
display.stop()
For more compatibility, you can try combining the above together with new Chrome headless mode:
options.add_argument("--headless=new")
I’m trying some stuff out with selenium, and I really want my script to run quickly.
I thought that running my script with headless Chrome would make it faster.
First, is that assumption correct, or does it not matter if I run my script with a headless driver?
I want headless Chrome to work, but somehow it isn’t working correctly. I tried different things, and most suggested that it would work as said here in the October update:
How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
But when I tried that, I saw weird console output, and it still doesn’t seem to work.
Any tips appreciated.
To run chrome-headless just add --headless via chrome_options.add_argument, e.g.:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
#chrome_options.add_argument("--disable-extensions")
#chrome_options.add_argument("--disable-gpu")
#chrome_options.add_argument("--no-sandbox") # linux only
chrome_options.add_argument("--headless")
# chrome_options.headless = True # also works
driver = webdriver.Chrome(options=chrome_options)
start_url = "https://duckgo.com"
driver.get(start_url)
print(driver.page_source.encode("utf-8"))
# b'<!DOCTYPE html><html rel="nofollow noreferrer">headless-chromeIf you are using Linux environment, may be you have to add --no-sandbox as well and also specific window size settings. The --no-sandbox flag is no needed on Windows if you set user container properly.
Use --disable-gpu only on Windows. Other platforms no longer require it. The --disable-gpu flag is a temporary work around for a few bugs.
//Headless chrome browser and configure
WebDriverManager.chromedriver().setup();
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--no-sandbox");
chromeOptions.addArguments("--headless");
chromeOptions.addArguments("disable-gpu");
// chromeOptions.addArguments("window-size=1400,2100"); // Linux should be activate
driver = new ChromeDriver(chromeOptions);
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path="./chromedriver", options=chrome_options)
url = "https://stackoverflow.com/questions/53657215/running-selenium-with-headless-chrome-webdriver"
driver.get(url)
sleep(5)
h1 = driver.find_element_by_xpath("//h1[@itemprop='name']").text
print(h1)
Then I run script on our local machine
➜ python script.py
Running Selenium with Headless Chrome Webdriver
It is working and it is with headless Chrome.
Todo (tested on headless server Debian Linux 9.4):
-
Do this:
# install chrome curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list apt-get -y update apt-get -y install google-chrome-stable # install chrome driver wget https://chromedriver.storage.googleapis.com/77.0.3865.40/chromedriver_linux64.zip unzip chromedriver_linux64.zip mv chromedriver /usr/bin/chromedriver chown root:root /usr/bin/chromedriver chmod +x /usr/bin/chromedriver -
Install selenium:
pip install seleniumand run this Python code:
from selenium import webdriver from selenium.webdriver.chrome.options import Options options = Options() options.add_argument("no-sandbox") options.add_argument("headless") options.add_argument("start-maximized") options.add_argument("window-size=1900,1080"); driver = webdriver.Chrome(chrome_options=options, executable_path="/usr/bin/chromedriver") driver.get("https://www.example.com") html = driver.page_source print(html)
Install & run containerized Chrome:
docker pull selenium/standalone-chrome
docker run --rm -d -p 4444:4444 --shm-size=2g selenium/standalone-chrome
Connect using webdriver.Remote:
driver = webdriver.Remote('http://localhost:4444/wd/hub', webdriver.DesiredCapabilities.CHROME)
driver.set_window_size(1280, 1024)
driver.get('https://www.google.com')
Once you have selenium and web driver installed. Below worked for me with headless Chrome on linux cluster :
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-extensions")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")
options.add_experimental_option("prefs",{"download.default_directory":"/databricks/driver"})
driver = webdriver.Chrome(chrome_options=options)
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=r"C:Program
FilesGoogleChromeApplicationchromedriver.exe", options=chrome_options)
This is ok for me.
As stated by the accepted answer:
options.add_argument("--headless")
These tips might help to speed things up especially for headless:
There are quite a few things you can do in headless that you cant do in non headless
Since you will be using Chrome Headless, I've found adding this reduces the CPU usage by about 20% for me (I found this to be a CPU and memory hog when looking at htop)
--disable-crash-reporter
This will only disable when you are running in headless This might speed things up for you!!!
My settings are currently as follows and I reduce the CPU (but only a marginal time saving) by about 20%:
options.add_argument("--no-sandbox");
options.add_argument("--disable-dev-shm-usage");
options.add_argument("--disable-renderer-backgrounding");
options.add_argument("--disable-background-timer-throttling");
options.add_argument("--disable-backgrounding-occluded-windows");
options.add_argument("--disable-client-side-phishing-detection");
options.add_argument("--disable-crash-reporter");
options.add_argument("--disable-oopr-debug-crash-dump");
options.add_argument("--no-crash-upload");
options.add_argument("--disable-gpu");
options.add_argument("--disable-extensions");
options.add_argument("--disable-low-res-tiling");
options.add_argument("--log-level=3");
options.add_argument("--silent");
I found this to be a pretty good list (full list I think) of command line switches with explanations: https://peter.sh/experiments/chromium-command-line-switches/
Some additional things you can turn off are also mentioned here: https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md
I hope this helps someone
Recently there is an update performed on headless mode of Chrome.
The flag --headless is now modified and can be used as below
- For Chrome version 109 and above,
--headless=newflag allows us to explore full functionality Chrome browser in headless mode. - For Chrome version 108 and below (till Version 96),
--headless=chromeoption will provide us the headless chrome browser.
So, let's add
options.add_argument("--headless=new")
for newer version of Chrome in headless mode as mentioned above.
You can run Headless Chrome with Selenium in Python as shown below. *--headless=new is better because--headless uses old headless mode according Headless is Going Away!:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless=new") # Here
driver = webdriver.Chrome(options=options)
Or:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("--headless=new") # Here
driver = webdriver.Chrome(options=options)
In addition, the examples below can test Django Admin with Headless Chrome, Selenium, pytest-django and Django. *My answer explains how to test Django Admin with multiple Headless browsers(Chrome, Microsoft Edge and Firefox), Selenium, pytest-django and Django:
# "tests/test.py"
import pytest
from selenium import webdriver
from django.test import LiveServerTestCase
@pytest.fixture(scope="class")
def chrome_driver_init(request):
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
chrome_driver = webdriver.Chrome(options=options)
request.cls.driver = chrome_driver
yield
chrome_driver.close()
@pytest.mark.usefixtures("chrome_driver_init")
class Test_URL_Chrome(LiveServerTestCase):
def test_open_url(self):
self.driver.get(("%s%s" % (self.live_server_url, "/admin/")))
assert "Log in | Django site admin" in self.driver.title
Or:
# "tests/conftest.py"
import pytest
from selenium import webdriver
@pytest.fixture(scope="class")
def chrome_driver_init(request):
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
chrome_driver = webdriver.Chrome(options=options)
request.cls.driver = chrome_driver
yield
chrome_driver.close()
# "tests/test.py"
import pytest
from django.test import LiveServerTestCase
@pytest.mark.usefixtures("chrome_driver_init")
class Test_URL_Chrome(LiveServerTestCase):
def test_open_url(self):
self.driver.get(("%s%s" % (self.live_server_url, "/admin/")))
assert "Log in | Django site admin" in self.driver.title
There are different ways of running Chrome in headless environments. (You'll find more details in this answer: https://stackoverflow.com/a/73840130/7058266)
One, the standard headless mode: (Faster than headed mode, but you may experience compatibility issues.)
options.add_argument("--headless")
Then there's the new Chrome headless mode as of Chrome 109: (It runs at the same speed as headed mode, as the two are virtually identical.)
options.add_argument("--headless=new")
(Between Chrome 96 and 108, that new mode used to be --headless=chrome, but it was renamed.)
You can also run regular Chrome in a headless environment if using a headless display, such as Xvfb with a Python program that controls it, such as pyvirtualdisplay. (See https://stackoverflow.com/a/6300672/7058266 and https://stackoverflow.com/a/23447450/7058266)
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
driver = webdriver.Chrome()
driver.get('http://www.google.com')
driver.quit()
display.stop()
For more compatibility, you can try combining the above together with new Chrome headless mode:
options.add_argument("--headless=new")