Django not reconnecting when PostgreSQL dies, custom backend needed?
Question:
I have been doing some testing and have been able to confirm that using Django with PostgreSQL and PGBouncer that it does not automatically reconnect on losing it’s connection. To be honest, I am not sure if this is a bug or if this is by design. If it is a bug I will happily report it, if not I would like some explanation of why and how to get around it other than another custom back end.
I did these tests fairly easily by doing the following on Django 1.8.4 and on Django 1.8.6:
>>>Model.objects.all().count()
24
# Restart postgres with `sudo service postgres restart`
>>>Model.objects.all().count()
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 318, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 466, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 447, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
It does not matter how long I wait after restarting PostgreSQL to run the query I still get the same results. I see that in Django there is code to run a Select 1 to check for a connection but that doesn’t seem to be working. Back in Django 1.5.x we had written a custom PostgreSQL back end to do this same thing but removed it since it seemed Django had it built in when we upgraded. Is this a bug?
Edit 11/6/2015: In Django, the PostgreSQL backend implements the is_usable function that does a SELECT 1 on the database. However, I am not able to find any usages of is_usable other than in close_if_unusable_or_obsolete but I couldn’t find any usages of that anywhere. I would think it would be used within some database wrapper that catches exceptions and retries/reconnects based on is_usable.
The code mentioned above is found at django/db/backends/postgresql_psychopg2/base.py in Django 1.8.
Edit 2 11/6/2015: Ok, I wrote my own custom wrapper for the database and just overrode ensure_connection method to try and reconnect to the database when the connection is lost. However, on the first attempt to hit the database for a session I get this other gnarly traceback. But if I immediately query again it works. If I wrap what I am doing in a try/except: pass block it all seems to be working fine but cannot really tell if it will cause problems later on. Here is the traceback and code.
>>> c = Model.objects.all().count()
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 318, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 466, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 447, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 838, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
cursor = self.make_cursor(self._cursor())
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
self.ensure_connection()
File "/usr/lib/python2.7/dist-packages/custom/db/backends/postgresql_psycopg2/base.py", line 73, in ensure_connection
self._reconnect()
File "/usr/lib/python2.7/dist-packages/custom/db/backends/postgresql_psycopg2/base.py", line 63, in _reconnect
self.connect()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 120, in connect
self.set_autocommit(self.settings_dict['AUTOCOMMIT'])
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 295, in set_autocommit
self._set_autocommit(autocommit)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
self.connection.autocommit = autocommit
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
self.connection.autocommit = autocommit
ProgrammingError: autocommit cannot be used inside a transaction
Now for the code:
from django.db.backends.postgresql_psycopg2.base import DatabaseError,
IntegrityError, DatabaseWrapper as PostgresWrapper
class DatabaseWrapper(PostgresWrapper):
def _reconnect(self):
try:
self.connect()
except (DatabaseError, OperationalError):
pass
def ensure_connection(self):
"""
Guarantees that a connection to the database is established.
"""
if self.connection is None:
with self.wrap_database_errors:
self._reconnect()
else:
try:
self.connection.cursor().execute('SELECT 1')
except (DatabaseError, OperationalError):
self._reconnect()
Answers:
I think I figured it out…finally. Not entirely sure what was wrong with my first approach but this one seems to be working a ton better.
class DatabaseWrapper(PostgresWrapper):
def _cursor(self):
if self.connection is not None:
if not self.is_usable():
self.connection.close()
self.connection = None
return super(DatabaseWrapper, self)._cursor()
Edit: Ended up open sourcing this. I am not sure if it is 100% needed, but it is working correctly after restarting the Postgres service on the server. You can find it on pypi as django-postgreconnect, and on GitHub: https://github.com/EccoTheFlintstone/django-postgreconnect.
I have been doing some testing and have been able to confirm that using Django with PostgreSQL and PGBouncer that it does not automatically reconnect on losing it’s connection. To be honest, I am not sure if this is a bug or if this is by design. If it is a bug I will happily report it, if not I would like some explanation of why and how to get around it other than another custom back end.
I did these tests fairly easily by doing the following on Django 1.8.4 and on Django 1.8.6:
>>>Model.objects.all().count()
24
# Restart postgres with `sudo service postgres restart`
>>>Model.objects.all().count()
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 318, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 466, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 447, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
It does not matter how long I wait after restarting PostgreSQL to run the query I still get the same results. I see that in Django there is code to run a Select 1 to check for a connection but that doesn’t seem to be working. Back in Django 1.5.x we had written a custom PostgreSQL back end to do this same thing but removed it since it seemed Django had it built in when we upgraded. Is this a bug?
Edit 11/6/2015: In Django, the PostgreSQL backend implements the is_usable function that does a SELECT 1 on the database. However, I am not able to find any usages of is_usable other than in close_if_unusable_or_obsolete but I couldn’t find any usages of that anywhere. I would think it would be used within some database wrapper that catches exceptions and retries/reconnects based on is_usable.
The code mentioned above is found at django/db/backends/postgresql_psychopg2/base.py in Django 1.8.
Edit 2 11/6/2015: Ok, I wrote my own custom wrapper for the database and just overrode ensure_connection method to try and reconnect to the database when the connection is lost. However, on the first attempt to hit the database for a session I get this other gnarly traceback. But if I immediately query again it works. If I wrap what I am doing in a try/except: pass block it all seems to be working fine but cannot really tell if it will cause problems later on. Here is the traceback and code.
>>> c = Model.objects.all().count()
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 318, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 466, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py", line 447, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 838, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
cursor = self.make_cursor(self._cursor())
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
self.ensure_connection()
File "/usr/lib/python2.7/dist-packages/custom/db/backends/postgresql_psycopg2/base.py", line 73, in ensure_connection
self._reconnect()
File "/usr/lib/python2.7/dist-packages/custom/db/backends/postgresql_psycopg2/base.py", line 63, in _reconnect
self.connect()
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 120, in connect
self.set_autocommit(self.settings_dict['AUTOCOMMIT'])
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 295, in set_autocommit
self._set_autocommit(autocommit)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
self.connection.autocommit = autocommit
File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 97, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/dist-packages/django/db/backends/postgresql_psycopg2/base.py", line 218, in _set_autocommit
self.connection.autocommit = autocommit
ProgrammingError: autocommit cannot be used inside a transaction
Now for the code:
from django.db.backends.postgresql_psycopg2.base import DatabaseError,
IntegrityError, DatabaseWrapper as PostgresWrapper
class DatabaseWrapper(PostgresWrapper):
def _reconnect(self):
try:
self.connect()
except (DatabaseError, OperationalError):
pass
def ensure_connection(self):
"""
Guarantees that a connection to the database is established.
"""
if self.connection is None:
with self.wrap_database_errors:
self._reconnect()
else:
try:
self.connection.cursor().execute('SELECT 1')
except (DatabaseError, OperationalError):
self._reconnect()
I think I figured it out…finally. Not entirely sure what was wrong with my first approach but this one seems to be working a ton better.
class DatabaseWrapper(PostgresWrapper):
def _cursor(self):
if self.connection is not None:
if not self.is_usable():
self.connection.close()
self.connection = None
return super(DatabaseWrapper, self)._cursor()
Edit: Ended up open sourcing this. I am not sure if it is 100% needed, but it is working correctly after restarting the Postgres service on the server. You can find it on pypi as django-postgreconnect, and on GitHub: https://github.com/EccoTheFlintstone/django-postgreconnect.