Attempting to download a .csv via a link on within a page w/ python

Question:

I’m attempting to download and use data contained in a .csv file. I’m pretty amatuer at scraping or most things coding related and would appreciate any help. Is it possible I’m being blocked from pulling the link from the website? I asked the AI gods a bunch of different ways also and they aren’t any help either. Thanks for reading and I appreciate any help 🙂

from bs4 import BeautifulSoup
import requests 
import csv 

user_input = input("Enter a zipcode: ")

url = f"https://www.redfin.com/zipcode/{user_input}/filter/include=sold-3mo"

response = requests.get(url)

# Parse the HTML content using Beautiful Soup
soup = BeautifulSoup(response.content, "html.parser")

# Find the download link
download_link = soup.find("a", string='download-and-save')

if download_link:
    # Extract the href attribute from the link
    href = download_link["href"]
    # Download the CSV file
    response = requests.get(href)
    with open("data.csv", "wb") as f:
        f.write(response.content)
else:
    print("Error: Unable to find link to CSV file")```

I've tried switching writing the code in different ways such as trying to get the the code to print url and can't even get it to do that. I've also tried removing the inputs for zip code but I don't think that is a big deal. I was expecting to download the .csv but I guess not. I only get Unable to find link to CSV file as output 
Asked By: Dakota Hutchinson

||

Answers:

You can use:

import pandas as pd
import requests
import bs4
import io

user_input = '91605'  # input("Enter a zipcode: ")

url = 'https://www.redfin.com'
endpoint = f'{url}/zipcode/{user_input}/filter/include=sold-3mo'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0'}

response = requests.get(endpoint, headers=headers)
soup = bs4.BeautifulSoup(response.content, 'html.parser')

download_link = soup.find('a', {'class': 'downloadLink'})['href']

csvfile = requests.get(f'{url}{download_link}', headers=headers)
buf = io.BytesIO(csvfile.content)
df = pd.read_csv(buf)

Output:

>>> df
    SALE TYPE         SOLD DATE              PROPERTY TYPE                          ADDRESS             CITY STATE OR PROVINCE  ...   SOURCE        MLS#  FAVORITE  INTERESTED   LATITUDE   LONGITUDE
0   PAST SALE   February-8-2023  Single Family Residential                  7749 Beeman Ave  North Hollywood                CA  ...    CRMLS  BB22257833         N           Y  34.211741 -118.409218
1   PAST SALE               NaN                Condo/Co-op       7804 Laurel Canyon Blvd #4  North Hollywood                CA  ...      NaN         NaN         N           Y  34.212322 -118.396181
2   PAST SALE   February-6-2023     Multi-Family (5+ Unit)                   6901 Hinds Ave  North Hollywood                CA  ...    CRMLS  BB23021008         N           Y  34.196008 -118.391276
3   PAST SALE               NaN  Single Family Residential                  8034 Gentry Ave  North Hollywood                CA  ...      NaN         NaN         N           Y  34.216784 -118.392647
4   PAST SALE     March-15-2023                Condo/Co-op                6861 Hinds Ave #5  North Hollywood                CA  ...  CRISNet  SR23029018         N           Y  34.195651 -118.391344
5   PAST SALE               NaN  Single Family Residential                12224 Milldale Ct  North Hollywood                CA  ...      NaN         NaN         N           Y  34.211048 -118.399583
6   PAST SALE     March-21-2023                Condo/Co-op              6904 Radford Ave #8  North Hollywood                CA  ...    CRMLS    P1-12258         N           Y  34.196028 -118.391814
7   PAST SALE  February-13-2023  Single Family Residential                    7841 Beck Ave  North Hollywood                CA  ...  CRISNet  SR22210989         N           Y  34.213260 -118.383656
8   PAST SALE     March-16-2023  Single Family Residential               11949 Strathern St  North Hollywood                CA  ...  CRISNet  SR22257978         N           Y  34.215976 -118.393718
9   PAST SALE      March-8-2023                  Townhouse          7420 Laurelgrove Ave #1  North Hollywood                CA  ...    CRMLS  GD22228601         N           Y  34.205464 -118.399913
10  PAST SALE   January-25-2023                Condo/Co-op  7131 N Coldwater Canyon Ave #10  North Hollywood                CA  ...   TheMLS   22-225607         N           Y  34.200340 -118.415193
11  PAST SALE      March-8-2023  Single Family Residential               7606 Van Noord Ave  North Hollywood                CA  ...    CRMLS  CV23006173         N           Y  34.208269 -118.414910
12  PAST SALE  December-30-2022     Multi-Family (5+ Unit)                 12755 Vanowen St  North Hollywood                CA  ...  CRISNet  SR22242878         N           Y  34.194225 -118.411473
13  PAST SALE   January-19-2023  Single Family Residential                  7848 Melita Ave  North Hollywood                CA  ...  CRISNet  SR22247835         N           Y  34.212691 -118.419827
14  PAST SALE   January-31-2023     Multi-Family (5+ Unit)                8241 Whitsett Ave  North Hollywood                CA  ...   TheMLS   22-220253         N           Y  34.221112 -118.405676
15  PAST SALE   January-30-2023    Multi-Family (2-4 Unit)                  6860 Gentry Ave  North Hollywood                CA  ...    CRMLS  DW22227164         N           Y  34.195664 -118.392965
16  PAST SALE   January-18-2023  Single Family Residential                 12561 Willard St  North Hollywood                CA  ...  CRISNet  SR22229727         N           Y  34.219215 -118.407428
17  PAST SALE   February-3-2023  Single Family Residential               7029 Longridge Ave  North Hollywood                CA  ...  CRISNet  SR22235657         N           Y  34.198397 -118.421764
18  PAST SALE   January-30-2023  Single Family Residential                  11737 Blythe St  North Hollywood                CA  ...  CRISNet  SR22214841         N           Y  34.214993 -118.389130
19  PAST SALE      March-6-2023  Single Family Residential                   12201 Crewe St  North Hollywood                CA  ...   TheMLS   22-224627         N           Y  34.196816 -118.398849
20  PAST SALE   February-3-2023  Single Family Residential                  11912 Burton St  North Hollywood                CA  ...  CRISNet  SR22238841         N           Y  34.219658 -118.392965
21  PAST SALE    January-3-2023  Single Family Residential                 12762 Cantara St  North Hollywood                CA  ...  CRISNet  SR22200303         N           Y  34.219651 -118.411907
22  PAST SALE    January-6-2023    Multi-Family (2-4 Unit)                 7343 Tujunga Ave  North Hollywood                CA  ...    CRMLS  BB22206682         N           Y  34.204287 -118.379464
23  PAST SALE   January-31-2023  Single Family Residential                   7604 Lilly Way  North Hollywood                CA  ...    CRMLS  WS22224124         N           Y  34.208676 -118.398021
24  PAST SALE   January-10-2023   Mobile/Manufactured Home                 7337 Ethel Ave W  North Hollywood                CA  ...  CRISNet  SR22185471         N           Y  34.204002 -118.419327
25  PAST SALE      March-3-2023  Single Family Residential                 11556 Valerio St  North Hollywood                CA  ...   TheMLS   22-180731         N           Y  34.204637 -118.384480
26  PAST SALE               NaN  Single Family Residential                7263 Farmdale Ave  North Hollywood                CA  ...      NaN         NaN         N           Y  34.203048 -118.381499
27  PAST SALE               NaN  Single Family Residential               7952 Longridge Ave  North Hollywood                CA  ...      NaN         NaN         N           Y  34.215341 -118.421399
28  PAST SALE               NaN  Single Family Residential                 7936 Vantage Ave  North Hollywood                CA  ...      NaN         NaN         N           Y  34.214909 -118.397452
29  PAST SALE               NaN  Single Family Residential              8067 Shadyglade Ave  North Hollywood                CA  ...      NaN         NaN         N           Y  34.217734 -118.402247
30  PAST SALE               NaN  Single Family Residential                   13044 Lorne St  North Hollywood                CA  ...      NaN         NaN         N           Y  34.217240 -118.417976

[31 rows x 27 columns]
Answered By: Corralien