03 Jul 2020

Python - Web Scraping with Selenium

  • python
  • web_scraping
  • final_url = f'{url}{name}001&termArray=f_20_2310/'
    driver = webdriver.Chrome()
    driver.get(final_url)
    response = driver.execute_script("return document.documentElement.outerHTML")
    driver.quit()
    soup = BeautifulSoup(response, "lxml")
    panel = soup.find('div', attrs={'id':'classScheduleBody'})
    

    how to web scrape without opening the browser

    from selenium.webdriver.chrome.options import Options
    
    options = Options()
    options.add_argument('--headless')