Selenium Headless chrome runs much slower

Question

I have a class for Selenium parser:

class DynamicParser(Parser):
    """Selenium Parser with processing JS"""
    driver: Chrome = None

    def __init__(self, driver_path='./chromedriver', headless=True):
        chrome_options = Options()
        if headless:
            chrome_options.add_argument("--headless")
            chrome_options.add_argument("window-size=1920,1080")

        # bypass OS security
        chrome_options.add_argument('--no-sandbox')
        # overcome limited resources
        chrome_options.add_argument('--disable-dev-shm-usage')
        # don't tell chrome that it is automated
        chrome_options.add_experimental_option(
            "excludeSwitches", ["enable-automation"])
        chrome_options.add_experimental_option('useAutomationExtension', False)
        # disable images
        prefs = {"profile.managed_default_content_settings.images": 2}
        chrome_options.add_experimental_option("prefs", prefs)

        # Setting Capabilities
        capabilities = DesiredCapabilities.CHROME.copy()
        capabilities['acceptSslCerts'] = True
        capabilities['acceptInsecureCerts'] = True

        self.driver = Chrome(chrome_options=chrome_options,
                             executable_path=driver_path, desired_capabilities=capabilities)

    def goto(self, url: str):
        """Goes to specified URL"""
        self.driver.get(url)

    def get_seller_name(self) -> str:
        """Returns seller's name"""
        offer_actions_tag = self.driver.find_element_by_class_name(
            'offer-user__actions')
        profile_link_tag = offer_actions_tag.find_element_by_tag_name('a')
        return profile_link_tag.text.strip()

Also I have a test script, which creates DynamicParser, goes to some page and calls .get_seller_name().

I noticed that when I run Chromedriver headless, it runs much slower, so i tested it using time python3 test.py.

Output for headless chrome:

python3 test.py  2,98s user 0,94s system 3% cpu 2:04,65 total

Output for non-headless chrome:

python3 test.py  1,48s user 0,33s system 47% cpu 3,790 total

As we can see, headless chrome runs almost 33 times slower!

Chrome version: 83.0.4103.116

Chromedriver version: 83.0.4103.39

I don’t really understand what’s the problem. When I developed my previous application, headless chrome worked fast enough.

Asked By: user8531441

||

Source

Answer 1

Just found the problem. It was

chrome_options.add_argument('--disable-dev-shm-usage')

I supposed that it should have unlimit chrome resources, but it definitely doesn’t work in this case.

Answered By: user8531441

Answer 2

When running headless driver you can also use those settings for the performance improvement.

browser_options = webdriver.ChromeOptions()
browser_options.headless = True
image_preferences = {"profile.managed_default_content_settings.images": 2}
browser_options.add_experimental_option("prefs", image_preferences)

Answered By: Michael Krezalek

Answer 3

I found that none of these worked for me.

However, changing options.add_argument('--headless') to options.add_argument('--headless=new') made a huge difference and seemed to have entirely fixed the problem.

Answered By: no1no2no3no4

Selenium Headless chrome runs much slower

Question:

Answers: