How to grab data from a website and save to a file - lunawyh/covid19viz GitHub Wiki

There are 3 ways to grab data from a website and save the content to a file:

1. urllib

    import urllib
   
    # save to a local file
   
    urllib.urlretrieve(url_of_website, file_to_be_saved)
   

2. requests

    import requests
   
    r = requests.get(url_of_website)
   
    with open(file_to_be_saved, 'wb') as f:
   
        f.write(r.content)
   

3. selenium

    from selenium import webdriver
   
    from selenium.webdriver.common.keys import Keys
   
    from selenium.webdriver.common.action_chains import ActionChains
   
    driver = webdriver.Chrome()
   
    driver.get(url_of_website)
   
    time.sleep(5)
   
    page_text = driver.page_source
   
    with open(file_to_be_saved, "w") as fp:
   
        fp.write(page_text.encode('utf8'))