Can't run Chrome in headless mode using Selenium

Question

So here’s my code first:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time
from fake_useragent import UserAgent
import random

ua = UserAgent()

options = Options()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--blink-settings=imagesEnabled=false')
chrome_options.add_argument('--headless')
chrome_options.add_argument(f'user-agent={ua.random}')
driver = webdriver.Chrome(options=options, chrome_options=chrome_options)
driver.maximize_window()
url = "https://magiceden.io/marketplace/hasuki"
driver.get(url)
element = driver.find_element(By.CSS_SELECTOR, "#content > div.tw-w-full.tw-py-0.sm:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md:tw-justify-between.tw-gap-2.md:tw-gap-4.md:tw-sticky.tw-top-[133px].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)")
print(f"The current instant sell price is {element.text}")

When I run it, I get weird long error, that ends with:

Backtrace:
        (No symbol) [0x00806643]
        (No symbol) [0x0079BE21]
        (No symbol) [0x0069DA9D]
        (No symbol) [0x006D1342]
        (No symbol) [0x006D147B]
        (No symbol) [0x00708DC2]
        (No symbol) [0x006EFDC4]
        (No symbol) [0x00706B09]
        (No symbol) [0x006EFB76]
        (No symbol) [0x006C49C1]
        (No symbol) [0x006C5E5D]
        GetHandleVerifier [0x00A7A142+2497106]
        GetHandleVerifier [0x00AA85D3+2686691]
        GetHandleVerifier [0x00AABB9C+2700460]
        GetHandleVerifier [0x008B3B10+635936]
        (No symbol) [0x007A4A1F]
        (No symbol) [0x007AA418]
        (No symbol) [0x007AA505]
        (No symbol) [0x007B508B]
        BaseThreadInitThunk [0x75EB00F9+25]
        RtlGetAppContainerNamedObjectPath [0x77A27BBE+286]
        RtlGetAppContainerNamedObjectPath [0x77A27B8E+238]

BUT if I comment out "chrome_options.add_argument(‘–headless’)", my code works perfectly fine. What’s the issue here? I suppose the problem is that website doesn’t let me use headless mode, how can I solve this?

I want my program to run in headless mode, but I get restricted either by the website or chrome browser.

Asked By: syrok

||

Source

Answer 1

Basically, you are seeing a NoSuchElementException

Full stacktrace:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#content > div.tw-w-full.tw-py-0.sm:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md:tw-justify-between.tw-gap-2.md:tw-gap-4.md:tw-sticky.tw-top-[133px].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)"}
  (Session info: headless chrome=109.0.5414.75)
Stacktrace:
Backtrace:
    (No symbol) [0x00E04AD3]
    (No symbol) [0x00D99211]
    (No symbol) [0x00C8D9CD]
    (No symbol) [0x00CC1102]
    (No symbol) [0x00CC123B]
    (No symbol) [0x00CF8A72]
    (No symbol) [0x00CDFB84]
    (No symbol) [0x00CF67B9]
    (No symbol) [0x00CDF936]
    (No symbol) [0x00CB4788]
    (No symbol) [0x00CB5C1D]
    GetHandleVerifier [0x01079342+2502274]
    GetHandleVerifier [0x010A7959+2692249]
    GetHandleVerifier [0x010AABDC+2705180]
    GetHandleVerifier [0x00EB3480+643008]
    (No symbol) [0x00DA208F]
    (No symbol) [0x00DA7AB8]
    (No symbol) [0x00DA7BA5]
    (No symbol) [0x00DB273B]
    BaseThreadInitThunk [0x76D9FA29+25]
    RtlGetAppContainerNamedObjectPath [0x77B37A4E+286]
    RtlGetAppContainerNamedObjectPath [0x77B37A1E+238]

Solution

The desired element is a dynamic element, so ideally to print the desired text from the element you need to induce WebDriverWait for the visibility_of_element_located() and you can use the following Locator Strategy:

Using CSS_SELECTOR:

driver.get("https://magiceden.io/marketplace/hasuki")
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#content > div.tw-w-full.tw-py-0.sm:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md:tw-justify-between.tw-gap-2.md:tw-gap-4.md:tw-sticky.tw-top-[133px].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)"))).text)
driver.quit()

Console Output:
```
0.3664
```

Note: You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Answered By: undetected Selenium

Can't run Chrome in headless mode using Selenium

Question:

Answers:

Solution