How to perform web scraping with Selenium?

Product Support
1

How to perform web scraping with Selenium?

2

Hi Brett,

You can perform scraping with Selenium as shown below:

The results are in the iframe so, switch to it and then get the .page_source for that add the commands given below:

iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)
I also added a wait for the table to be loaded:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 10)

#locate and switch to the iframe
iframe = driver.find_element_by_css_selector("#mainContent iframe")
driver.switch_to.frame(iframe)

#wait for the table to load
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,'companyName')))

print(driver.page_source)