Attempting to download a .csv via a link on within a page w/ python
Question:
I’m attempting to download and use data contained in a .csv file. I’m pretty amatuer at scraping or most things coding related and would appreciate any help. Is it possible I’m being blocked from pulling the link from the website? I asked the AI gods a bunch of different ways also and they aren’t any help either. Thanks for reading and I appreciate any help 🙂
from bs4 import BeautifulSoup
import requests
import csv
user_input = input("Enter a zipcode: ")
url = f"https://www.redfin.com/zipcode/{user_input}/filter/include=sold-3mo"
response = requests.get(url)
# Parse the HTML content using Beautiful Soup
soup = BeautifulSoup(response.content, "html.parser")
# Find the download link
download_link = soup.find("a", string='download-and-save')
if download_link:
# Extract the href attribute from the link
href = download_link["href"]
# Download the CSV file
response = requests.get(href)
with open("data.csv", "wb") as f:
f.write(response.content)
else:
print("Error: Unable to find link to CSV file")```
I've tried switching writing the code in different ways such as trying to get the the code to print url and can't even get it to do that. I've also tried removing the inputs for zip code but I don't think that is a big deal. I was expecting to download the .csv but I guess not. I only get Unable to find link to CSV file as output
Answers:
You can use:
import pandas as pd
import requests
import bs4
import io
user_input = '91605' # input("Enter a zipcode: ")
url = 'https://www.redfin.com'
endpoint = f'{url}/zipcode/{user_input}/filter/include=sold-3mo'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0'}
response = requests.get(endpoint, headers=headers)
soup = bs4.BeautifulSoup(response.content, 'html.parser')
download_link = soup.find('a', {'class': 'downloadLink'})['href']
csvfile = requests.get(f'{url}{download_link}', headers=headers)
buf = io.BytesIO(csvfile.content)
df = pd.read_csv(buf)
Output:
>>> df
SALE TYPE SOLD DATE PROPERTY TYPE ADDRESS CITY STATE OR PROVINCE ... SOURCE MLS# FAVORITE INTERESTED LATITUDE LONGITUDE
0 PAST SALE February-8-2023 Single Family Residential 7749 Beeman Ave North Hollywood CA ... CRMLS BB22257833 N Y 34.211741 -118.409218
1 PAST SALE NaN Condo/Co-op 7804 Laurel Canyon Blvd #4 North Hollywood CA ... NaN NaN N Y 34.212322 -118.396181
2 PAST SALE February-6-2023 Multi-Family (5+ Unit) 6901 Hinds Ave North Hollywood CA ... CRMLS BB23021008 N Y 34.196008 -118.391276
3 PAST SALE NaN Single Family Residential 8034 Gentry Ave North Hollywood CA ... NaN NaN N Y 34.216784 -118.392647
4 PAST SALE March-15-2023 Condo/Co-op 6861 Hinds Ave #5 North Hollywood CA ... CRISNet SR23029018 N Y 34.195651 -118.391344
5 PAST SALE NaN Single Family Residential 12224 Milldale Ct North Hollywood CA ... NaN NaN N Y 34.211048 -118.399583
6 PAST SALE March-21-2023 Condo/Co-op 6904 Radford Ave #8 North Hollywood CA ... CRMLS P1-12258 N Y 34.196028 -118.391814
7 PAST SALE February-13-2023 Single Family Residential 7841 Beck Ave North Hollywood CA ... CRISNet SR22210989 N Y 34.213260 -118.383656
8 PAST SALE March-16-2023 Single Family Residential 11949 Strathern St North Hollywood CA ... CRISNet SR22257978 N Y 34.215976 -118.393718
9 PAST SALE March-8-2023 Townhouse 7420 Laurelgrove Ave #1 North Hollywood CA ... CRMLS GD22228601 N Y 34.205464 -118.399913
10 PAST SALE January-25-2023 Condo/Co-op 7131 N Coldwater Canyon Ave #10 North Hollywood CA ... TheMLS 22-225607 N Y 34.200340 -118.415193
11 PAST SALE March-8-2023 Single Family Residential 7606 Van Noord Ave North Hollywood CA ... CRMLS CV23006173 N Y 34.208269 -118.414910
12 PAST SALE December-30-2022 Multi-Family (5+ Unit) 12755 Vanowen St North Hollywood CA ... CRISNet SR22242878 N Y 34.194225 -118.411473
13 PAST SALE January-19-2023 Single Family Residential 7848 Melita Ave North Hollywood CA ... CRISNet SR22247835 N Y 34.212691 -118.419827
14 PAST SALE January-31-2023 Multi-Family (5+ Unit) 8241 Whitsett Ave North Hollywood CA ... TheMLS 22-220253 N Y 34.221112 -118.405676
15 PAST SALE January-30-2023 Multi-Family (2-4 Unit) 6860 Gentry Ave North Hollywood CA ... CRMLS DW22227164 N Y 34.195664 -118.392965
16 PAST SALE January-18-2023 Single Family Residential 12561 Willard St North Hollywood CA ... CRISNet SR22229727 N Y 34.219215 -118.407428
17 PAST SALE February-3-2023 Single Family Residential 7029 Longridge Ave North Hollywood CA ... CRISNet SR22235657 N Y 34.198397 -118.421764
18 PAST SALE January-30-2023 Single Family Residential 11737 Blythe St North Hollywood CA ... CRISNet SR22214841 N Y 34.214993 -118.389130
19 PAST SALE March-6-2023 Single Family Residential 12201 Crewe St North Hollywood CA ... TheMLS 22-224627 N Y 34.196816 -118.398849
20 PAST SALE February-3-2023 Single Family Residential 11912 Burton St North Hollywood CA ... CRISNet SR22238841 N Y 34.219658 -118.392965
21 PAST SALE January-3-2023 Single Family Residential 12762 Cantara St North Hollywood CA ... CRISNet SR22200303 N Y 34.219651 -118.411907
22 PAST SALE January-6-2023 Multi-Family (2-4 Unit) 7343 Tujunga Ave North Hollywood CA ... CRMLS BB22206682 N Y 34.204287 -118.379464
23 PAST SALE January-31-2023 Single Family Residential 7604 Lilly Way North Hollywood CA ... CRMLS WS22224124 N Y 34.208676 -118.398021
24 PAST SALE January-10-2023 Mobile/Manufactured Home 7337 Ethel Ave W North Hollywood CA ... CRISNet SR22185471 N Y 34.204002 -118.419327
25 PAST SALE March-3-2023 Single Family Residential 11556 Valerio St North Hollywood CA ... TheMLS 22-180731 N Y 34.204637 -118.384480
26 PAST SALE NaN Single Family Residential 7263 Farmdale Ave North Hollywood CA ... NaN NaN N Y 34.203048 -118.381499
27 PAST SALE NaN Single Family Residential 7952 Longridge Ave North Hollywood CA ... NaN NaN N Y 34.215341 -118.421399
28 PAST SALE NaN Single Family Residential 7936 Vantage Ave North Hollywood CA ... NaN NaN N Y 34.214909 -118.397452
29 PAST SALE NaN Single Family Residential 8067 Shadyglade Ave North Hollywood CA ... NaN NaN N Y 34.217734 -118.402247
30 PAST SALE NaN Single Family Residential 13044 Lorne St North Hollywood CA ... NaN NaN N Y 34.217240 -118.417976
[31 rows x 27 columns]
I’m attempting to download and use data contained in a .csv file. I’m pretty amatuer at scraping or most things coding related and would appreciate any help. Is it possible I’m being blocked from pulling the link from the website? I asked the AI gods a bunch of different ways also and they aren’t any help either. Thanks for reading and I appreciate any help 🙂
from bs4 import BeautifulSoup
import requests
import csv
user_input = input("Enter a zipcode: ")
url = f"https://www.redfin.com/zipcode/{user_input}/filter/include=sold-3mo"
response = requests.get(url)
# Parse the HTML content using Beautiful Soup
soup = BeautifulSoup(response.content, "html.parser")
# Find the download link
download_link = soup.find("a", string='download-and-save')
if download_link:
# Extract the href attribute from the link
href = download_link["href"]
# Download the CSV file
response = requests.get(href)
with open("data.csv", "wb") as f:
f.write(response.content)
else:
print("Error: Unable to find link to CSV file")```
I've tried switching writing the code in different ways such as trying to get the the code to print url and can't even get it to do that. I've also tried removing the inputs for zip code but I don't think that is a big deal. I was expecting to download the .csv but I guess not. I only get Unable to find link to CSV file as output
You can use:
import pandas as pd
import requests
import bs4
import io
user_input = '91605' # input("Enter a zipcode: ")
url = 'https://www.redfin.com'
endpoint = f'{url}/zipcode/{user_input}/filter/include=sold-3mo'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/111.0'}
response = requests.get(endpoint, headers=headers)
soup = bs4.BeautifulSoup(response.content, 'html.parser')
download_link = soup.find('a', {'class': 'downloadLink'})['href']
csvfile = requests.get(f'{url}{download_link}', headers=headers)
buf = io.BytesIO(csvfile.content)
df = pd.read_csv(buf)
Output:
>>> df
SALE TYPE SOLD DATE PROPERTY TYPE ADDRESS CITY STATE OR PROVINCE ... SOURCE MLS# FAVORITE INTERESTED LATITUDE LONGITUDE
0 PAST SALE February-8-2023 Single Family Residential 7749 Beeman Ave North Hollywood CA ... CRMLS BB22257833 N Y 34.211741 -118.409218
1 PAST SALE NaN Condo/Co-op 7804 Laurel Canyon Blvd #4 North Hollywood CA ... NaN NaN N Y 34.212322 -118.396181
2 PAST SALE February-6-2023 Multi-Family (5+ Unit) 6901 Hinds Ave North Hollywood CA ... CRMLS BB23021008 N Y 34.196008 -118.391276
3 PAST SALE NaN Single Family Residential 8034 Gentry Ave North Hollywood CA ... NaN NaN N Y 34.216784 -118.392647
4 PAST SALE March-15-2023 Condo/Co-op 6861 Hinds Ave #5 North Hollywood CA ... CRISNet SR23029018 N Y 34.195651 -118.391344
5 PAST SALE NaN Single Family Residential 12224 Milldale Ct North Hollywood CA ... NaN NaN N Y 34.211048 -118.399583
6 PAST SALE March-21-2023 Condo/Co-op 6904 Radford Ave #8 North Hollywood CA ... CRMLS P1-12258 N Y 34.196028 -118.391814
7 PAST SALE February-13-2023 Single Family Residential 7841 Beck Ave North Hollywood CA ... CRISNet SR22210989 N Y 34.213260 -118.383656
8 PAST SALE March-16-2023 Single Family Residential 11949 Strathern St North Hollywood CA ... CRISNet SR22257978 N Y 34.215976 -118.393718
9 PAST SALE March-8-2023 Townhouse 7420 Laurelgrove Ave #1 North Hollywood CA ... CRMLS GD22228601 N Y 34.205464 -118.399913
10 PAST SALE January-25-2023 Condo/Co-op 7131 N Coldwater Canyon Ave #10 North Hollywood CA ... TheMLS 22-225607 N Y 34.200340 -118.415193
11 PAST SALE March-8-2023 Single Family Residential 7606 Van Noord Ave North Hollywood CA ... CRMLS CV23006173 N Y 34.208269 -118.414910
12 PAST SALE December-30-2022 Multi-Family (5+ Unit) 12755 Vanowen St North Hollywood CA ... CRISNet SR22242878 N Y 34.194225 -118.411473
13 PAST SALE January-19-2023 Single Family Residential 7848 Melita Ave North Hollywood CA ... CRISNet SR22247835 N Y 34.212691 -118.419827
14 PAST SALE January-31-2023 Multi-Family (5+ Unit) 8241 Whitsett Ave North Hollywood CA ... TheMLS 22-220253 N Y 34.221112 -118.405676
15 PAST SALE January-30-2023 Multi-Family (2-4 Unit) 6860 Gentry Ave North Hollywood CA ... CRMLS DW22227164 N Y 34.195664 -118.392965
16 PAST SALE January-18-2023 Single Family Residential 12561 Willard St North Hollywood CA ... CRISNet SR22229727 N Y 34.219215 -118.407428
17 PAST SALE February-3-2023 Single Family Residential 7029 Longridge Ave North Hollywood CA ... CRISNet SR22235657 N Y 34.198397 -118.421764
18 PAST SALE January-30-2023 Single Family Residential 11737 Blythe St North Hollywood CA ... CRISNet SR22214841 N Y 34.214993 -118.389130
19 PAST SALE March-6-2023 Single Family Residential 12201 Crewe St North Hollywood CA ... TheMLS 22-224627 N Y 34.196816 -118.398849
20 PAST SALE February-3-2023 Single Family Residential 11912 Burton St North Hollywood CA ... CRISNet SR22238841 N Y 34.219658 -118.392965
21 PAST SALE January-3-2023 Single Family Residential 12762 Cantara St North Hollywood CA ... CRISNet SR22200303 N Y 34.219651 -118.411907
22 PAST SALE January-6-2023 Multi-Family (2-4 Unit) 7343 Tujunga Ave North Hollywood CA ... CRMLS BB22206682 N Y 34.204287 -118.379464
23 PAST SALE January-31-2023 Single Family Residential 7604 Lilly Way North Hollywood CA ... CRMLS WS22224124 N Y 34.208676 -118.398021
24 PAST SALE January-10-2023 Mobile/Manufactured Home 7337 Ethel Ave W North Hollywood CA ... CRISNet SR22185471 N Y 34.204002 -118.419327
25 PAST SALE March-3-2023 Single Family Residential 11556 Valerio St North Hollywood CA ... TheMLS 22-180731 N Y 34.204637 -118.384480
26 PAST SALE NaN Single Family Residential 7263 Farmdale Ave North Hollywood CA ... NaN NaN N Y 34.203048 -118.381499
27 PAST SALE NaN Single Family Residential 7952 Longridge Ave North Hollywood CA ... NaN NaN N Y 34.215341 -118.421399
28 PAST SALE NaN Single Family Residential 7936 Vantage Ave North Hollywood CA ... NaN NaN N Y 34.214909 -118.397452
29 PAST SALE NaN Single Family Residential 8067 Shadyglade Ave North Hollywood CA ... NaN NaN N Y 34.217734 -118.402247
30 PAST SALE NaN Single Family Residential 13044 Lorne St North Hollywood CA ... NaN NaN N Y 34.217240 -118.417976
[31 rows x 27 columns]