Use python requests to download CSV

From a little search, that I understand the file should be opened in universal newline mode, which you cannot directly do with a response content (I guess).

To finish the task, you can either save the downloaded content to a temporary file, or process it in memory.

Save as file:

import requests
import csv
import os

temp_file_name = 'temp_csv.csv'
url = 'http://url.to/file.csv'
download = requests.get(url)

with open(temp_file_name, 'w') as temp_file:
    temp_file.writelines(download.content)

with open(temp_file_name, 'rU') as temp_file:
    csv_reader = csv.reader(temp_file, dialect=csv.excel_tab)
    for line in csv_reader:
        print line

# delete the temp file after process
os.remove(temp_file_name)

In memory:

(To be updated)

You can update the accepted answer with the iter_lines method of requests if the file is very large

import csv
import requests

CSV_URL = 'http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv'

with requests.Session() as s:
    download = s.get(CSV_URL)

    line_iterator = (x.decode('utf-8') for x in download.iter_lines(decode_unicode=True))

    cr = csv.reader(line_iterator, delimiter=',')
    my_list = list(cr)
    for row in my_list:
        print(row)

To simplify these answers, and increase performance when downloading a large file, the below may work a bit more efficiently.

import requests
from contextlib import closing
import csv
from codecs import iterdecode

url = "http://download-and-process-csv-efficiently/python.csv"

with closing(requests.get(url, stream=True)) as r:
    reader = iterdecode(csv.reader(r.iter_lines(), 'utf-8'), 
                        delimiter=',', 
                        quotechar='"')
    for row in reader:
        print(row)

By setting stream=True in the GET request, when we pass r.iter_lines() to csv.reader(), we are passing a generator to csv.reader(). By doing so, we enable csv.reader() to lazily iterate over each line in the response with for row in reader.

This avoids loading the entire file into memory before we start processing it, drastically reducing memory overhead for large files.

You can also use the DictReader to iterate dictionaries of {'columnname': 'value', ...}

import csv
import requests

response = requests.get('http://example.test/foo.csv')
reader = csv.DictReader(response.iter_lines())
for record in reader:
    print(record)

I like the answers from The Aelfinn and aheld. I can improve them only by shortening a bit more, removing superfluous pieces, using a real data source, making it 2.x & 3.x-compatible, and maintaining the high-level of memory-efficiency seen elsewhere:

import csv
import requests

CSV_URL = 'http://web.cs.wpi.edu/~cs1004/a16/Resources/SacramentoRealEstateTransactions.csv'

with requests.get(CSV_URL, stream=True) as r:
    lines = (line.decode('utf-8') for line in r.iter_lines())
    for row in csv.reader(lines):
        print(row)

Too bad 3.x is less flexible CSV-wise because the iterator must emit Unicode strings (while requests does bytes) while the 2.x-only version—for row in csv.reader(r.iter_lines()):—is more Pythonic (shorter and easier-to-read). Anyhow, note the 2.x/3.x solution above won’t handle the situation described by the OP where a NEWLINE is found unquoted in the data read.

For the part of the OP’s question regarding downloading (vs. processing) the actual CSV file, here’s another script that does that, 2.x & 3.x-compatible, minimal, readable, and memory-efficient:

import os
import requests

CSV_URL = 'http://web.cs.wpi.edu/~cs1004/a16/Resources/SacramentoRealEstateTransactions.csv'

with open(os.path.split(CSV_URL)[1], 'wb') as f, 
        requests.get(CSV_URL, stream=True) as r:
    for line in r.iter_lines():
        f.write(line+'n'.encode())

The following approach worked well for me. I also did not need to use csv.reader() or csv.writer() functions, which I feel makes the code cleaner. The code is compatible with Python2 and Python 3.

from six.moves import urllib

DOWNLOAD_URL = "https://raw.githubusercontent.com/gjreda/gregreda.com/master/content/notebooks/data/city-of-chicago-salaries.csv"
DOWNLOAD_PATH ="datasetscity-of-chicago-salaries.csv" 
urllib.request.urlretrieve(URL,DOWNLOAD_PATH)

Note – six is a package that helps in writing code that is compatible with both Python 2 and Python 3. For additional details regarding six see – What does from six.moves import urllib do in Python?

I use this code (I use Python 3):

import csv
import io
import requests

url = "http://samplecsvs.s3.amazonaws.com/Sacramentorealestatetransactions.csv"
r = requests.get(url)
r.encoding = 'utf-8'  # useful if encoding is not sent (or not sent properly) by the server
csvio = io.StringIO(r.text, newline="")
data = []
for row in csv.DictReader(csvio):
    data.append(row)

Python3 Supported Code

    with closing(requests.get(PHISHTANK_URL, stream=True})) as r:
        reader = csv.reader(codecs.iterdecode(r.iter_lines(), 'utf-8'), delimiter=',', quotechar='"')
        for record in reader:
           print (record)

this worked nicely for me:

from csv import DictReader

f = requests.get('https://somedomain.com/file').content.decode('utf-8')
reader = DictReader(f.split('n'))
csv_dict_list = list(reader)

To convert to Pandas DataFrame:

from io import StringIO
text=StringIO(download.content.decode('utf-8'))
df=pd.read_csv(text)

I used below solution (Unfortunately others did not work for me):

import pandas as pd 
df = pd.read_csv('http://.../file.csv') 

Python 3.x:

For anyone trying the solutions above, if you are getting a ufeff at every new line, just change the encoding from utf-8 to utf-8-sig.

If you only want the CSV in string format, simply access the .text property of the request, like this:

req = requests.get("https: ...")
req.encoding = 'utf-8-sig'
csv_data = req.text