Python error with request post : 'Connection aborted, timeout('The write operation timed out')
Question:
I am using requests post to send some data, now I have set the timeout value of requests.post to something like 60. Other similar questions where related to specific applications so I wanted to ask it as a generic python error.
this is the error that I get:
failed to connect ('Connection aborted.', timeout('The write operation timed out'))
I assume this is because for some reason my thread couldn’t send all the data in time, but I am not sure. So I have the following questions:
-
Is this error happening because the thread couldn’t send out all the data in the timeout time?
-
If so, is there anyway I can set the timeout in a way that only starts after I send out all the data out? I don’t want to wait for ever for the server to respond, but obviously I want the timer to start when I send all the data out, and not at the start of function call. (and the data I send varies in size)
Answers:
If you are using the python-request library, you can specify the ‘timeout’ parameter as a tuple. The first element is the "connection timeout" and the second is the "read timeout".
requests.get('http://google.com', timeout=(10,200) # give it 10 seconds to connect to the server, and timeout if server does not send any data back for 200+ seconds
requests uses the urrlib3.util.Timeout under the hood.
Per the urllib3 docs, setting the read timeout to None will wait forever for the server to respond. But, I agree waiting forever is a bad idea.
How much data are you sending in your request? I read through some of the python-requests code and couldnt find an easy way to detect when the request is done sending.
As a proof of concept I wrote this code based on urequests (micro requests)
It allows you to pass a callback function to the requests.get(). This callback will be triggered after all the request data is sent, but before any data is received by the server.
import time
def sent_callback():
print('The request has finished sending at', time.time())
resp = get('http://google.com', send_callback=sent_callback)
print('response received at', time.time())
print('resp.text=', resp.text)
This could be because of a network issue, should the project be taken to a production server with high internet speed, should fix this problem (I’m still to check with my project).
But for development you can give this a try:
# ('Connection aborted.', TimeoutError('The write operation timed out')): Increase chunksize
storage.blob._DEFAULT_CHUNKSIZE = 5 * 1024 * 1024
storage.blob._MAX_MULTIPART_SIZE = 5 * 1024 * 1024
I am using requests post to send some data, now I have set the timeout value of requests.post to something like 60. Other similar questions where related to specific applications so I wanted to ask it as a generic python error.
this is the error that I get:
failed to connect ('Connection aborted.', timeout('The write operation timed out'))
I assume this is because for some reason my thread couldn’t send all the data in time, but I am not sure. So I have the following questions:
-
Is this error happening because the thread couldn’t send out all the data in the timeout time?
-
If so, is there anyway I can set the timeout in a way that only starts after I send out all the data out? I don’t want to wait for ever for the server to respond, but obviously I want the timer to start when I send all the data out, and not at the start of function call. (and the data I send varies in size)
If you are using the python-request library, you can specify the ‘timeout’ parameter as a tuple. The first element is the "connection timeout" and the second is the "read timeout".
requests.get('http://google.com', timeout=(10,200) # give it 10 seconds to connect to the server, and timeout if server does not send any data back for 200+ seconds
requests uses the urrlib3.util.Timeout under the hood.
Per the urllib3 docs, setting the read timeout to None will wait forever for the server to respond. But, I agree waiting forever is a bad idea.
How much data are you sending in your request? I read through some of the python-requests code and couldnt find an easy way to detect when the request is done sending.
As a proof of concept I wrote this code based on urequests (micro requests)
It allows you to pass a callback function to the requests.get(). This callback will be triggered after all the request data is sent, but before any data is received by the server.
import time
def sent_callback():
print('The request has finished sending at', time.time())
resp = get('http://google.com', send_callback=sent_callback)
print('response received at', time.time())
print('resp.text=', resp.text)
This could be because of a network issue, should the project be taken to a production server with high internet speed, should fix this problem (I’m still to check with my project).
But for development you can give this a try:
# ('Connection aborted.', TimeoutError('The write operation timed out')): Increase chunksize
storage.blob._DEFAULT_CHUNKSIZE = 5 * 1024 * 1024
storage.blob._MAX_MULTIPART_SIZE = 5 * 1024 * 1024