Request returns bytes and I'm failing to decode them
Question:
Essentially I made a request to a website and got a byte response back: b'[{"geonameId:"703448"}..........'. I’m confused because although it is of type byte, it is very human readable and appears like a list of json. I do know that the response is encoded in latin1 from running r.encoding which returned ISO-859-1 and I have tried to decode it, but it just returns an empty string. Here’s what I have so far:
r = response.content
string = r.decode("ISO-8859-1")
print (string)
and this is where it prints a blank line.
However when I run
len(string)
I get: back 31023
How can I decode these bytes without getting back an empty string?
Answers:
Did you try to parse it with the json module?
import json
parsed = json.loads(response.content)
Another solution is to use response.text, which returns the content in unicode
Type: property
String form: <property object at 0x7f76f8c79db8>
Docstring:
Content of the response, in unicode.
If Response.encoding is None, encoding will be guessed using
``chardet``.
The encoding of the response content is determined based solely on HTTP
headers, following RFC 2616 to the letter. If you can take advantage of
non-HTTP knowledge to make a better guess at the encoding, you should
set ``r.encoding`` appropriately before accessing this property.
There is r.text and r.content. The first one is a string, the second one is bytes.
You want
import json
data = json.loads(r.text)
I faced a similar issue using beautifulsoup4 and requests while scraping webpages, however both response.text and response.content looked like it was bytes.
The response headers included 'Content-Type': 'text/html; charset=UTF-8' encoding in the headers, also had this in the response headers – 'Content-Encoding': 'br'. It turns out I hadn’t installed brotlipy in the environment and running pip install brotlipy fixed my issues. I thought chardet or cchardet would be enough, but the data needed to be correctly decompressed.
A similar issue was solved here in the same way, and linking to this answer since it didn’t come up until I explicitly searched for brotli compression.
Essentially I made a request to a website and got a byte response back: b'[{"geonameId:"703448"}..........'. I’m confused because although it is of type byte, it is very human readable and appears like a list of json. I do know that the response is encoded in latin1 from running r.encoding which returned ISO-859-1 and I have tried to decode it, but it just returns an empty string. Here’s what I have so far:
r = response.content
string = r.decode("ISO-8859-1")
print (string)
and this is where it prints a blank line.
However when I run
len(string)
I get: back 31023
How can I decode these bytes without getting back an empty string?
Did you try to parse it with the json module?
import json
parsed = json.loads(response.content)
Another solution is to use response.text, which returns the content in unicode
Type: property
String form: <property object at 0x7f76f8c79db8>
Docstring:
Content of the response, in unicode.
If Response.encoding is None, encoding will be guessed using
``chardet``.
The encoding of the response content is determined based solely on HTTP
headers, following RFC 2616 to the letter. If you can take advantage of
non-HTTP knowledge to make a better guess at the encoding, you should
set ``r.encoding`` appropriately before accessing this property.
There is r.text and r.content. The first one is a string, the second one is bytes.
You want
import json
data = json.loads(r.text)
I faced a similar issue using beautifulsoup4 and requests while scraping webpages, however both response.text and response.content looked like it was bytes.
The response headers included 'Content-Type': 'text/html; charset=UTF-8' encoding in the headers, also had this in the response headers – 'Content-Encoding': 'br'. It turns out I hadn’t installed brotlipy in the environment and running pip install brotlipy fixed my issues. I thought chardet or cchardet would be enough, but the data needed to be correctly decompressed.
A similar issue was solved here in the same way, and linking to this answer since it didn’t come up until I explicitly searched for brotli compression.