urllib3 connectionpool – Connection pool is full, discarding connection
Question:
Does seeing the
urllib3.connectionpool WARNING - Connection pool is full, discarding connection
mean that I am effectively loosing data (because of lost connection)
OR
Does it mean that connection is dropped (because pool is full); however, the same connection will be re-tried later on when connection pool becomes available?
Answers:
No data is being lost!
The connection is being discarded after the request is completed (because the pool is full, as mentioned). This means that this particular connection is not going to be re-used in the future.
Because a urllib3 PoolManager reuses connections, it will limit how many connections are retained per hos to avoid accumulating too many unused sockets. The PoolManager can be configured to avoid creating excess sockets when the pool doesn’t have any idle sockets available with PoolManager(..., block=True).
If you’re relying on concurrency, it could be a good idea to increase the size of the pool (maxsize) to be at least as large as the number of threads you’re using, so that each thread effectively gets its own connection.
More details here: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#customizing-pool-behavior
According to the documentation on Customizing Pool Behavior, neither of your interpretations are correct:
By default, if a new request is made and there is no free connection in the pool then a new connection will be created. However, this connection will not be saved if more than maxsize connections exist. This means that maxsize does not determine the maximum number of connections that can be open to a particular host, just the maximum number of connections to keep in the pool.
(my emphasis)
So connections were not aborted to be retried later. They were made immediately, as requested, and results returned. Then, after they have completed, those "extra" connections were discarded, i.e., they were not kept in the pool for later reuse.
For example, if your maxsize is 10 (the default when using urllib3 via requests), and you launch 50 requests in parallel, those 50 connections will be performed at once, and after completion only 10 will remain in the pool while 40 will be discarded (and issue that warning).
Does seeing the
urllib3.connectionpool WARNING - Connection pool is full, discarding connection
mean that I am effectively loosing data (because of lost connection)
OR
Does it mean that connection is dropped (because pool is full); however, the same connection will be re-tried later on when connection pool becomes available?
No data is being lost!
The connection is being discarded after the request is completed (because the pool is full, as mentioned). This means that this particular connection is not going to be re-used in the future.
Because a urllib3 PoolManager reuses connections, it will limit how many connections are retained per hos to avoid accumulating too many unused sockets. The PoolManager can be configured to avoid creating excess sockets when the pool doesn’t have any idle sockets available with PoolManager(..., block=True).
If you’re relying on concurrency, it could be a good idea to increase the size of the pool (maxsize) to be at least as large as the number of threads you’re using, so that each thread effectively gets its own connection.
More details here: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#customizing-pool-behavior
According to the documentation on Customizing Pool Behavior, neither of your interpretations are correct:
By default, if a new request is made and there is no free connection in the pool then a new connection will be created. However, this connection will not be saved if more than maxsize connections exist. This means that maxsize does not determine the maximum number of connections that can be open to a particular host, just the maximum number of connections to keep in the pool.
(my emphasis)
So connections were not aborted to be retried later. They were made immediately, as requested, and results returned. Then, after they have completed, those "extra" connections were discarded, i.e., they were not kept in the pool for later reuse.
For example, if your maxsize is 10 (the default when using urllib3 via requests), and you launch 50 requests in parallel, those 50 connections will be performed at once, and after completion only 10 will remain in the pool while 40 will be discarded (and issue that warning).