How to create a new database using SQLAlchemy?
Question:
Using SQLAlchemy, an Engine object is created like this:
from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost/mydb")
Accessing engine fails if the database specified in the argument to create_engine (in this case, mydb) does not exist. Is it possible to tell SQLAlchemy to create a new database if the specified database doesn’t exist?
Answers:
On postgres, three databases are normally present by default. If you are able to connect as a superuser (eg, the postgres role), then you can connect to the postgres or template1 databases. The default pg_hba.conf permits only the unix user named postgres to use the postgres role, so the simplest thing is to just become that user. At any rate, create an engine as usual with a user that has the permissions to create a database:
>>> engine = sqlalchemy.create_engine("postgres://postgres@/postgres")
You cannot use engine.execute() however, because postgres does not allow you to create databases inside transactions, and sqlalchemy always tries to run queries in a transaction. To get around this, get the underlying connection from the engine:
>>> conn = engine.connect()
But the connection will still be inside a transaction, so you have to end the open transaction with a commit:
>>> conn.execute("commit")
And you can then proceed to create the database using the proper PostgreSQL command for it.
>>> conn.execute("create database test")
>>> conn.close()
It’s possible to avoid manual transaction management while creating database by providing isolation_level='AUTOCOMMIT' to create_engine function:
import sqlalchemy
with sqlalchemy.create_engine(
'postgresql:///postgres',
isolation_level='AUTOCOMMIT'
).connect() as connection:
connection.execute('CREATE DATABASE my_database')
Also if you are not sure that database doesn’t exist there is a way to ignore database creation error due to existence by suppressing sqlalchemy.exc.ProgrammingError exception:
import contextlib
import sqlalchemy.exc
with contextlib.suppress(sqlalchemy.exc.ProgrammingError):
# creating database as above
SQLAlchemy-Utils provides custom data types and various utility functions for SQLAlchemy. You can install the most recent official version using pip:
pip install sqlalchemy-utils
The database helpers include a create_database function:
from sqlalchemy import create_engine
from sqlalchemy_utils import database_exists, create_database
engine = create_engine("postgres://localhost/mydb")
if not database_exists(engine.url):
create_database(engine.url)
print(database_exists(engine.url))
Please note that I couldn’t get the above suggestions with database_exists because whenever I check if the database exists using if not database_exists(engine.url): I get this error:
InterfaceError(‘(pyodbc.InterfaceError) (‘28000′, u'[28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for
user \’myUser\’. (18456) (SQLDriverConnect); [28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open
database “MY_DATABASE” requested by the login. The login failed.
(4060); [28000] [Microsoft][SQL Server Native Client 11.0][SQL
Server]Login failed for user \’myUser\’. (18456); [28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open
database “MY_DATABASE” requested by the login. The login failed.
(4060)’)’,)
Also contextlib/suppress was not working and I’m not using postgres so I ended up doing this to ignore the exception if the database happens to already exist with SQL Server:
import logging
import sqlalchemy
logging.basicConfig(filename='app.log', format='%(asctime)s-%(levelname)s-%(message)s', level=logging.DEBUG)
engine = create_engine('mssql+pyodbc://myUser:mypwd@localhost:1234/MY_DATABASE?driver=SQL+Server+Native+Client+11.0?trusted_connection=yes', isolation_level = "AUTOCOMMIT")
try:
engine.execute('CREATE DATABASE ' + a_database_name)
except Exception as db_exc:
logging.exception("Exception creating database: " + str(db_exc))
Extending the accepted answer using with yields:
from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost")
NEW_DB_NAME = 'database_name'
with engine.connect() as conn:
conn.execute("commit")
# Do not substitute user-supplied database names here.
conn.execute(f"CREATE DATABASE {NEW_DB_NAME}")
If someone like me don’t want to take whole sqlalchemy_utils to your project just for database creation, you can use script like this. I’ve come with it, based on SingleNegationElimination‘s answer. I’m using pydantic here (it’s FastAPI project) and my imported settings for reference, but you can easily change this:
from sqlalchemy import create_engine
from sqlalchemy.exc import OperationalError
from pydantic import PostgresDsn
def build_db_connection_url(custom_db: Optional[str] = None):
db_name = f"/{settings.POSTGRES_DB or ''}" if custom_db is None else "/" + custom_db
return PostgresDsn.build(
scheme='postgresql+psycopg2',
user=settings.POSTGRES_USER,
password=settings.POSTGRES_PASSWORD,
host=settings.POSTGRES_HOST,
path=db_name,
)
def create_database(db_name: str):
try:
eng = create_engine(build_db_connection_url(custom_db=db_name))
conn = eng.connect()
conn.close()
del eng
except OperationalError as exc:
if "does not exist" in exc.__str__():
eng = create_engine(build_db_connection_url(custom_db="postgres"))
conn = eng.connect()
conn.execute("commit")
conn.execute(f"create database {db_name}")
conn.close()
# TODO: log about db creation
print(f"Database {db_name} created")
else:
raise exc
create_database("test_database")
Using SQLAlchemy, an Engine object is created like this:
from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost/mydb")
Accessing engine fails if the database specified in the argument to create_engine (in this case, mydb) does not exist. Is it possible to tell SQLAlchemy to create a new database if the specified database doesn’t exist?
On postgres, three databases are normally present by default. If you are able to connect as a superuser (eg, the postgres role), then you can connect to the postgres or template1 databases. The default pg_hba.conf permits only the unix user named postgres to use the postgres role, so the simplest thing is to just become that user. At any rate, create an engine as usual with a user that has the permissions to create a database:
>>> engine = sqlalchemy.create_engine("postgres://postgres@/postgres")
You cannot use engine.execute() however, because postgres does not allow you to create databases inside transactions, and sqlalchemy always tries to run queries in a transaction. To get around this, get the underlying connection from the engine:
>>> conn = engine.connect()
But the connection will still be inside a transaction, so you have to end the open transaction with a commit:
>>> conn.execute("commit")
And you can then proceed to create the database using the proper PostgreSQL command for it.
>>> conn.execute("create database test")
>>> conn.close()
It’s possible to avoid manual transaction management while creating database by providing isolation_level='AUTOCOMMIT' to create_engine function:
import sqlalchemy
with sqlalchemy.create_engine(
'postgresql:///postgres',
isolation_level='AUTOCOMMIT'
).connect() as connection:
connection.execute('CREATE DATABASE my_database')
Also if you are not sure that database doesn’t exist there is a way to ignore database creation error due to existence by suppressing sqlalchemy.exc.ProgrammingError exception:
import contextlib
import sqlalchemy.exc
with contextlib.suppress(sqlalchemy.exc.ProgrammingError):
# creating database as above
SQLAlchemy-Utils provides custom data types and various utility functions for SQLAlchemy. You can install the most recent official version using pip:
pip install sqlalchemy-utils
The database helpers include a create_database function:
from sqlalchemy import create_engine
from sqlalchemy_utils import database_exists, create_database
engine = create_engine("postgres://localhost/mydb")
if not database_exists(engine.url):
create_database(engine.url)
print(database_exists(engine.url))
Please note that I couldn’t get the above suggestions with database_exists because whenever I check if the database exists using if not database_exists(engine.url): I get this error:
InterfaceError(‘(pyodbc.InterfaceError) (‘28000′, u'[28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for
user \’myUser\’. (18456) (SQLDriverConnect); [28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open
database “MY_DATABASE” requested by the login. The login failed.
(4060); [28000] [Microsoft][SQL Server Native Client 11.0][SQL
Server]Login failed for user \’myUser\’. (18456); [28000]
[Microsoft][SQL Server Native Client 11.0][SQL Server]Cannot open
database “MY_DATABASE” requested by the login. The login failed.
(4060)’)’,)
Also contextlib/suppress was not working and I’m not using postgres so I ended up doing this to ignore the exception if the database happens to already exist with SQL Server:
import logging
import sqlalchemy
logging.basicConfig(filename='app.log', format='%(asctime)s-%(levelname)s-%(message)s', level=logging.DEBUG)
engine = create_engine('mssql+pyodbc://myUser:mypwd@localhost:1234/MY_DATABASE?driver=SQL+Server+Native+Client+11.0?trusted_connection=yes', isolation_level = "AUTOCOMMIT")
try:
engine.execute('CREATE DATABASE ' + a_database_name)
except Exception as db_exc:
logging.exception("Exception creating database: " + str(db_exc))
Extending the accepted answer using with yields:
from sqlalchemy import create_engine
engine = create_engine("postgresql://localhost")
NEW_DB_NAME = 'database_name'
with engine.connect() as conn:
conn.execute("commit")
# Do not substitute user-supplied database names here.
conn.execute(f"CREATE DATABASE {NEW_DB_NAME}")
If someone like me don’t want to take whole sqlalchemy_utils to your project just for database creation, you can use script like this. I’ve come with it, based on SingleNegationElimination‘s answer. I’m using pydantic here (it’s FastAPI project) and my imported settings for reference, but you can easily change this:
from sqlalchemy import create_engine
from sqlalchemy.exc import OperationalError
from pydantic import PostgresDsn
def build_db_connection_url(custom_db: Optional[str] = None):
db_name = f"/{settings.POSTGRES_DB or ''}" if custom_db is None else "/" + custom_db
return PostgresDsn.build(
scheme='postgresql+psycopg2',
user=settings.POSTGRES_USER,
password=settings.POSTGRES_PASSWORD,
host=settings.POSTGRES_HOST,
path=db_name,
)
def create_database(db_name: str):
try:
eng = create_engine(build_db_connection_url(custom_db=db_name))
conn = eng.connect()
conn.close()
del eng
except OperationalError as exc:
if "does not exist" in exc.__str__():
eng = create_engine(build_db_connection_url(custom_db="postgres"))
conn = eng.connect()
conn.execute("commit")
conn.execute(f"create database {db_name}")
conn.close()
# TODO: log about db creation
print(f"Database {db_name} created")
else:
raise exc
create_database("test_database")