How to run Selenium headless Chrome with Python effectively?

General Discussions
, ,
#1

How to run Selenium headless Chrome using Python?

I’m working on speeding up my script using Selenium headless Chrome, which would help. Is running the script with a headless driver faster, or does it not make a difference?

I want to use Selenium headless Chrome, but I’m having issues with it. Despite following several suggestions, including those in the October update on configuring ChromeDriver for headless mode, it still doesn’t work correctly. I see strange console output, and the script isn’t running as expected.

Any tips would be appreciated.

#2

Hey Saanvi,

To run Selenium Chrome in headless mode, you can add --headless using chrome_options.add_argument, as shown below:

from selenium import webdriver 
from selenium.webdriver.chrome.options import Options

chrome_options = Options()
# chrome_options.add_argument("--disable-extensions")
# chrome_options.add_argument("--disable-gpu")
# chrome_options.add_argument("--no-sandbox") # linux only
chrome_options.add_argument("--headless=new") # for Chrome >= 109
# chrome_options.add_argument("--headless")
# chrome_options.headless = True # also works
driver = webdriver.Chrome(options=chrome_options)
start_url = "https://duckduckgo.com"
driver.get(start_url)
print(driver.page_source.encode("utf-8"))
# b'<!DOCTYPE html><html xmlns="http://www....
driver.quit()

I believe running the script with headless Chrome might speed it up. You can also try using additional Chrome options like --disable-extensions or --disable-gpu and benchmark the performance. However, don’t expect a substantial improvement.

#3

Hey Saanvi,

Recently, there has been an update to Chrome’s headless mode in Selenium. The --headless flag has been modified and can now be used as follows:

For Chrome version 109 and above, the --headless=new flag allows full functionality in headless mode. For Chrome version 108 and below (up to version 96), use the --headless=chrome option for headless mode. So, for newer versions of Selenium, add:

options.add_argument("--headless=new")

to enable Selenium headless Chrome mode as mentioned above.