Why do I get a ChunkedEncodingError in Python when using requests module?

Question

I try to query this API https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi
with these params ?db=mesh&id=68016019
So the whole URL is https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=mesh&id=68016019
When executing this in Postman everything’s fine and the output is as expected:

1: Survival Analysis
A class of statistical procedures for estimating the survival function (function 
of time, starting with a population 100% well at a given time and providing the
percentage of the population still well at later times). The survival analysis is
then used for making inferences about the effects of treatments, prognostic
etc.
*
*
*

BUT when I try to query this via requests module in Python I receive this error:

("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

So what am I doing exactly? I just fire this command:

response2 = requests.get(
        "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=mesh&id=" + str(id))

(The id is converted to a string as I got some issues when I don’t do it.) For 99.9% of the cases this approach is absolutely fine but then there comes this 0.1% cases I am lost. I do not get what this ChunkedEncodingError is about.

Is this error coming from the API or from my script?
Can please someone help?

I tried to query this in Postman -> it works
I tried to execute the same query from my script in a complete new script -> it works (???)

but why not in my original script?

Asked By: Hannes

||

Source

Answer 1

If you are occasionally getting a ChunkedEncodingError, you can implement some retry logic in that case:

Something like this would retry a given URL 3 times (with a 1 second pause between each retry):

import requests
import time

article_id=68016019

for attempt in range(3):
    try:
        res = requests.get(
            f"https://eutils.ncbi.nlm.nih.gov/entre/eutils/efetch.fcgi?db=mesh&id={article_id}")
        break
    except requests.exceptions.ChunkedEncodingError:
        time.sleep(1)
else:
    print("Failed to retrieve url")

You would obviously want to implement better error handling than simply printing a messsage and continuing.

Answered By: larsks

Why do I get a ChunkedEncodingError in Python when using requests module?

Question:

Answers: