MariaDBConnector/Python Connection Pools get slower to get connections with each connection in the pool
Question:
I’m using the MariaDB/Python connector on Python 3.11, Linux Mint, latest version of MariaDB, etc and I can’t for the life of me figure this out. I have a simple function called get_connection()
that returns a Connection
from mariadb.ConnectionPool
, and with pool_size=1
, the response time in Postman is 75ms, but with pool_size=5
, the response time is 400ms. There is nothing else going on in this request except for a FastAPI Middleware that opens and closes this connection. There comes a threshold with this issue where it’s simply faster to open a new connection per request, as opposed to using a ConnectionPool
.
I expect get_connection()
to have the same response time whether I have pool_size=5
or pool_size=64
.
Here is the code:
POOL = mariadb.ConnectionPool(
host=os.environ.get("DATABASE_IP"),
port=3306,
user=os.environ.get("DATABASE_USER"),
password=os.environ.get("DATABASE_PASSWORD"),
pool_name="poolname",
pool_size=5,
pool_reset_connection=True
)
def get_connection() -> Connection:
"""
Returns a DatabaseHandler object
:return:
"""
before = datetime.datetime.now()
connection = POOL.get_connection()
after = datetime.datetime.now()
print((after - before).microseconds)
return connection
Answers:
We experienced the same problem when running internal benchmarks with connection pool instead of a single connection.
The problem is caused by unnecessary health check (sending MYSQL_PING) when iterating over unused connections.
Fix is planned for 1.1.6 (scheduled for mid Feb)
For tracking status of this issue, please check CONPY-245
Update (Feb 1): Issue has been fixed and will be available in 1.1.6 release:
$ python3.11 conpy245.py
Acquiring 100000 connections via Pool
Pool size: 64
Average time per connection: 19 ms
Acquiring 100000 connections without Pool
Average time per connection: 170 ms
I’m using the MariaDB/Python connector on Python 3.11, Linux Mint, latest version of MariaDB, etc and I can’t for the life of me figure this out. I have a simple function called get_connection()
that returns a Connection
from mariadb.ConnectionPool
, and with pool_size=1
, the response time in Postman is 75ms, but with pool_size=5
, the response time is 400ms. There is nothing else going on in this request except for a FastAPI Middleware that opens and closes this connection. There comes a threshold with this issue where it’s simply faster to open a new connection per request, as opposed to using a ConnectionPool
.
I expect get_connection()
to have the same response time whether I have pool_size=5
or pool_size=64
.
Here is the code:
POOL = mariadb.ConnectionPool(
host=os.environ.get("DATABASE_IP"),
port=3306,
user=os.environ.get("DATABASE_USER"),
password=os.environ.get("DATABASE_PASSWORD"),
pool_name="poolname",
pool_size=5,
pool_reset_connection=True
)
def get_connection() -> Connection:
"""
Returns a DatabaseHandler object
:return:
"""
before = datetime.datetime.now()
connection = POOL.get_connection()
after = datetime.datetime.now()
print((after - before).microseconds)
return connection
We experienced the same problem when running internal benchmarks with connection pool instead of a single connection.
The problem is caused by unnecessary health check (sending MYSQL_PING) when iterating over unused connections.
Fix is planned for 1.1.6 (scheduled for mid Feb)
For tracking status of this issue, please check CONPY-245
Update (Feb 1): Issue has been fixed and will be available in 1.1.6 release:
$ python3.11 conpy245.py
Acquiring 100000 connections via Pool
Pool size: 64
Average time per connection: 19 ms
Acquiring 100000 connections without Pool
Average time per connection: 170 ms