Convert text data from requests object to dataframe with pandas

Question

Using requests I am creating an object which is in .csv format. How can I then write that object to a DataFrame with pandas?

To get the requests object in text format:

import requests
import pandas as pd
url = r'http://test.url' 
r = requests.get(url)
r.text  #this will return the data as text in csv format

I tried (doesn’t work):

pd.read_csv(r.text)
pd.DataFrame.from_csv(r.text)

Asked By: sparrow

||

Source

Answer 1

I think you can use read_csv with url:

pd.read_csv(url)

filepath_or_buffer : str, pathlib.Path, py._path.local.LocalPath or any object with a read() method (such as a file handle or StringIO)

The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. For instance, a local file could be file ://localhost/path/to/table.csv

import pandas as pd
import io
import requests

url = r'http://...' 
r = requests.get(url)  
df = pd.read_csv(io.StringIO(r))

If it doesnt work, try update last line:

import pandas as pd
import io
import requests

url = r'http://...' 
r = requests.get(url)  
df = pd.read_csv(io.StringIO(r.text))

Answered By: jezrael

Answer 2

if the url has no authentication then you can directly use read_csv(url)

if you have authentication you can use request to get it un-pickel and print the csv and make sure the result is CSV and use panda.

You can directly use importing
import csv

Answered By: rkoots

Answer 3

Try this

import requests
import pandas as pd
import io

urlData = requests.get(url).content
rawData = pd.read_csv(io.StringIO(urlData.decode('utf-8')))

Answered By: Merlin

Answer 4

Using “read_csv with url” worked:

import requests, csv
import pandas as pd
url = 'https://arte.folha.uol.com.br/ciencia/2020/coronavirus/csv/mundo/dados-bra.csv'
corona_bra = pd.read_csv(url)
print(corona_bra.head())

Answered By: Bento

Convert text data from requests object to dataframe with pandas

Question:

Answers: