23 Jun 2020

Django - how to fetch data from external website

  • concepts
  • web
  • server-side
  • Fetch

    Fetch is a process of bringing or displaying data to an alternate location.

    Fetch data from web

    Web scraping

    def fetchFilteredClasses(dpt):
        classnames=[]
        url = 'https://www.lsa.umich.edu/cg/cg_results.aspx?termArray=f_20_2310&cgtype=ug&department=&allsections=true'
        response = requests.get(url)
        soup = BeautifulSoup(response.content,"html.parser")
        for a in soup.findAll('div', attrs = {'class' : 'row toppadding_main bottompadding_interior'}):
            name = a.find('div', attrs={'class':'col-sm-12'})
            classnames.append(name.text)
    
        df = pd.DataFrame({'Class Name':classnames}) 
        df.to_csv('products.csv', index=False, encoding='utf-8')