How to build a flask application around an already existing database?
Question:
I already have an existing Database that has a lot of tables and a lot of data in MySQL. I intend to create a Flask app and use sqlalchemy along with it. Now I asked out on irc and looked around on google and tried the following ideas:
First I used sqlacodegen to generate the models from my DB. But then I was confused about it a little and looked some more. And I found this.
This looked like an elegant solution.
So Second, I rewrote my models.py according to the solution there and now I am even more confused. I am looking for the best approach to build this flask app along with the already existing DB.
I looked into the flask documentation but didnt really get any help for a project with an already existing db. There is a lot of good stuff for creating something from scratch, creating the db and all. But I am really confused.
Please Note that its my first day with Flask, but I have experience with Django, so the basic concepts are not a hurdle. I need some guidance in choosing the best approach for this usecase. A detailed explanation would be greatly appreciated. By detailed I definitely do not expect someone to write all the code and spoon feed me on this, but just enough to get me started, that is integrate this db seamlessly into flask via sqlalchemy. Note my DB is in MySQL.
Answers:
I’d say your question has nothing to do with flask at all. For example, you don’t have a problem with the templates, routes, views or logon decorators.
Where you struggle at is at SQLAlchemy.
So my suggestion is to ignore Flask for a while and get used to SQLAlchemy first. You need to get used to your existing database and how to access it from SQLAlchemy. Use some MySQL documentation tool to find your way around this. The start with something like this (note that it has nothing to do with Flask ask all … yet):
#!/usr/bin/python
# -*- mode: python -*-
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///webmgmt.db', convert_unicode=True, echo=False)
Base = declarative_base()
Base.metadata.reflect(engine)
from sqlalchemy.orm import relationship, backref
class Users(Base):
__table__ = Base.metadata.tables['users']
if __name__ == '__main__':
from sqlalchemy.orm import scoped_session, sessionmaker, Query
db_session = scoped_session(sessionmaker(bind=engine))
for item in db_session.query(Users.id, Users.name):
print item
In the line “engine =” you need to provide your path to your MySQL database, so that SQLAlchemy finds it. In my case I used a pre-existing sqlite3 database.
In the line “class Users(Base)” you need to use one of existing tables in your MySQL database. I knew that my sqlite3 database had a table named “users”.
After this point, SQLalchemy knows how to connect to your MySQL database and it knows about one of the tables. You need now to add all the other tables that you care for. Finally, you need to specify relationships to SQLalchemy. Here I mean things like one-to-one, one-to-many, many-to-many, parent-child and so on. The SQLAlchemy web site contains a rather lenghty section about this.
After the line “if __name__ == '__main__'” just comes some test code. It will be executed if I don’t import my python script, but run. Here you see that I create a DB session and is that for a very simply query.
My suggestion is that you first read about the important parts of SQLAlchemy’s documentation, for example the descriptive table definition, the relationship model and how to query. Once you know this, you can change the last part of my example into a controller (e.g. using Python’s yield method) and write a view that uses that controller.
I recently went through the same thing, with the additional challenge of linking the models across two databases.
I used Flask-SQLAlchemy and all I had to do was define my models in the same way as my database tables looked. What I found difficult was figuring out exactly what my project structure should look like.
My project was a Restful API, and this is what I ended up with:
conf/
__init__.py
local.py
dev.py
stage.py
live.py
deploy/
#nginx, uwsgi config, etc
middleware/
authentication.py
app_name/
blueprints/
__init__.py
model_name.py #routes for model_name
...
models/
__init.py
model_name.py
__init__.py
database.py
tests/
unit/
test_etc.py
...
run.py
Files of note:
conf/xxx.py
This is how we tell Flask-SQLAlchemy what to connect to, plus you can put any other config items in here (like log location, debugging config, etc).
SQLALCHEMY_DATABASE_URI = 'mysql://username:password@host:port/db_name'
app_name/__init__.py
This is where I create my app and initialise the db. This db object will be imported and used across the entire app (i.e., in the models, tests, etc). I also set my logger, initialise my APIs and blueprints and attach my middleware in here (not shown).
from app_name.database import db
from flask import Flask
def create_app(*args, **kwargs):
env = kwargs['env']
app = Flask(__name__)
app.config.from_object('conf.%s' % env)
db.init_app(app)
return app
app_name/database.py
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
app_name/models/model_name.py
from services.database import db
class Bar(db.Model):
__tablename__ = 'your_MySQL_table_name'
id = db.Column('YourMySQLColumnName', db.Integer, primary_key=True)
name = db.Column('WhateverName', db.String(100))
foo = db.Column(db.ForeignKey('another_MySQLTableName.id'))
class Foo(db.Model):
__tablename__ = 'another_MySQLTableName'
id = db.Column('FooId', db.Integer, primary_key=True)
...
run.py
#! /usr/bin/env python
from app_name import create_app
app = create_app(env='local')
if __name__ == '__main__':
app.run()
I use run.py to run the app locally, but I use nginx + uWSGI to run the app in the dev/stage/live environments.
I’m guessing you’ll have a views/ directory in there in addition to this though.
The key to connecting Holger’s answer to a flask context is that db.Model is a declarative_base object like Base. Took me a while to notice this important sentence in flask-sqlalchemy’s documentation
Below are the steps I used for my app:
-
initiate a db object in the usual flask-alchemy manner:db = SQLAlchemy(app). Note you’ll need to set app.config['SQLALCHEMY_DATABASE_URI'] = 'connection_string' before that.
-
bind the declarative base to an engine: db.Model.metadata.reflect(db.engine)
-
Then you can use existing tables easily (eg. I have a table called BUILDINGS):
class Buildings(db.Model):
__table__ = db.Model.metadata.tables['BUILDING']
def __repr__(self):
return self.DISTRICT
Now your Buildings class will follow the existing schema. You can try dir(Buildings) in a Python shell and see all the columns already listed.
This is an alternative way to set up the engine path described in Holger’s answer. Convenient if there are special characters in your user name or password.
from sqlalchemy.engine.url import URL
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
engine_URL = URL('mssql+pymssql',
username='DOMAIN\USERNAME',
password="""p@ssword'!""",
host='host.com',
database='database_name')
engine = create_engine(engine_URL)
Base = declarative_base()
Base.metadata.reflect(engine)
I think the easiest way to use an existing database with sqlalchemy is to use AutomapBase class . A sample code from docs is as follows:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
from sqlalchemy import create_engine
Base = automap_base()
# engine, suppose it has two tables 'user' and 'address' set up
engine = create_engine("sqlite:///mydatabase.db")
# reflect the tables
Base.prepare(engine, reflect=True)
# mapped classes are now created with names by default
# matching that of the table name.
User = Base.classes.user
Address = Base.classes.address
session = Session(engine)
# rudimentary relationships are produced
session.add(Address(email_address="[email protected]", user=User(name="foo")))
session.commit()
# collection-based relationships are by default named
# "<classname>_collection"
print (u1.address_collection)
Refer SqlAlchemy-Automap for details and more complicated usages
This solution worked for me
"""Example for reflecting database tables to ORM objects
This script creates classes for each table reflected
from the database.
Note: The class names are imported to the global namespace using
the same name as the tables. This is useful for quick utility scripts.
A better solution for production code would be to return a dict
of reflected ORM objects.
"""
from sqlalchemy import create_engine, MetaData
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
def reflect_all_tables_to_declarative(uri):
"""Reflects all tables to declaratives
Given a valid engine URI and declarative_base base class
reflects all tables and imports them to the global namespace.
Returns a session object bound to the engine created.
"""
# create an unbound base our objects will inherit from
Base = declarative_base()
engine = create_engine(uri)
metadata = MetaData(bind=engine)
Base.metadata = metadata
g = globals()
metadata.reflect()
for tablename, tableobj in metadata.tables.items():
g[tablename] = type(str(tablename), (Base,), {'__table__' : tableobj })
print("Reflecting {0}".format(tablename))
Session = sessionmaker(bind=engine)
return Session()
# set to database credentials/host
CONNECTION_URI = "postgres://..."
session = reflect_all_tables_to_declarative(CONNECTION_URI)
# do something with the session and the orm objects
results = session.query(some_table_name).all()
I try to use autogenerated but nothing works or I couldn’t run it. When I searching generate code using sqlacodegen I find https://github.com/ksindi/flask-sqlacodegen, you can generate the code just
flask-sqlacodegen mysql://username:password@host:port/db_name --schema yourschema --tables table1,table2 --flask
I tried and it works perfectly
alembic (the tool behind flask-sqlalchemy) can be configured to ignore tables. The configuration isn’t too hard to setup. see: https://gist.github.com/utek/6163250
I’d suggest using dynamic class
from flask import Flask
from sqlalchemy import Table, MetaData
from flask_sqlalchemy import SQLAlchemy
import os
app = Flask(__name__)
class Config(Object):
basedir = os.path.abspath(os.path.dirname(__file__))
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + + os.path.join(basedir, 'db.sqlite')
SQLALCHEMY_TRACK_MODIFICATIONS = False
app.config.from_object(config)
db = SQLAlchemy(app)
metadata = MetaData()
table_reflection = Table("table_name", metadata, autoload=True, autoload_with=db.engine)
attrs = {"__table__": table_reflection}
TableModel = type("table_name", (db.Model,), attrs)
Now, TableModel can be used as models class
# Querying
TableModel.query.all()
TableModel.query.get(1)
TableModel.query.filter_by(id=2).first()
# insertion
new_row = TableModel()
db.session.add(new_row)
db.session.commit()
If your application is simple you can use db.reflect(app) on your model file:
def configure(app):
db.init_app(app)
db.reflect(app=app)
app.db = db
I already have an existing Database that has a lot of tables and a lot of data in MySQL. I intend to create a Flask app and use sqlalchemy along with it. Now I asked out on irc and looked around on google and tried the following ideas:
First I used sqlacodegen to generate the models from my DB. But then I was confused about it a little and looked some more. And I found this.
This looked like an elegant solution.
So Second, I rewrote my models.py according to the solution there and now I am even more confused. I am looking for the best approach to build this flask app along with the already existing DB.
I looked into the flask documentation but didnt really get any help for a project with an already existing db. There is a lot of good stuff for creating something from scratch, creating the db and all. But I am really confused.
Please Note that its my first day with Flask, but I have experience with Django, so the basic concepts are not a hurdle. I need some guidance in choosing the best approach for this usecase. A detailed explanation would be greatly appreciated. By detailed I definitely do not expect someone to write all the code and spoon feed me on this, but just enough to get me started, that is integrate this db seamlessly into flask via sqlalchemy. Note my DB is in MySQL.
I’d say your question has nothing to do with flask at all. For example, you don’t have a problem with the templates, routes, views or logon decorators.
Where you struggle at is at SQLAlchemy.
So my suggestion is to ignore Flask for a while and get used to SQLAlchemy first. You need to get used to your existing database and how to access it from SQLAlchemy. Use some MySQL documentation tool to find your way around this. The start with something like this (note that it has nothing to do with Flask ask all … yet):
#!/usr/bin/python
# -*- mode: python -*-
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine('sqlite:///webmgmt.db', convert_unicode=True, echo=False)
Base = declarative_base()
Base.metadata.reflect(engine)
from sqlalchemy.orm import relationship, backref
class Users(Base):
__table__ = Base.metadata.tables['users']
if __name__ == '__main__':
from sqlalchemy.orm import scoped_session, sessionmaker, Query
db_session = scoped_session(sessionmaker(bind=engine))
for item in db_session.query(Users.id, Users.name):
print item
In the line “engine =” you need to provide your path to your MySQL database, so that SQLAlchemy finds it. In my case I used a pre-existing sqlite3 database.
In the line “class Users(Base)” you need to use one of existing tables in your MySQL database. I knew that my sqlite3 database had a table named “users”.
After this point, SQLalchemy knows how to connect to your MySQL database and it knows about one of the tables. You need now to add all the other tables that you care for. Finally, you need to specify relationships to SQLalchemy. Here I mean things like one-to-one, one-to-many, many-to-many, parent-child and so on. The SQLAlchemy web site contains a rather lenghty section about this.
After the line “if __name__ == '__main__'” just comes some test code. It will be executed if I don’t import my python script, but run. Here you see that I create a DB session and is that for a very simply query.
My suggestion is that you first read about the important parts of SQLAlchemy’s documentation, for example the descriptive table definition, the relationship model and how to query. Once you know this, you can change the last part of my example into a controller (e.g. using Python’s yield method) and write a view that uses that controller.
I recently went through the same thing, with the additional challenge of linking the models across two databases.
I used Flask-SQLAlchemy and all I had to do was define my models in the same way as my database tables looked. What I found difficult was figuring out exactly what my project structure should look like.
My project was a Restful API, and this is what I ended up with:
conf/
__init__.py
local.py
dev.py
stage.py
live.py
deploy/
#nginx, uwsgi config, etc
middleware/
authentication.py
app_name/
blueprints/
__init__.py
model_name.py #routes for model_name
...
models/
__init.py
model_name.py
__init__.py
database.py
tests/
unit/
test_etc.py
...
run.py
Files of note:
conf/xxx.py
This is how we tell Flask-SQLAlchemy what to connect to, plus you can put any other config items in here (like log location, debugging config, etc).
SQLALCHEMY_DATABASE_URI = 'mysql://username:password@host:port/db_name'
app_name/__init__.py
This is where I create my app and initialise the db. This db object will be imported and used across the entire app (i.e., in the models, tests, etc). I also set my logger, initialise my APIs and blueprints and attach my middleware in here (not shown).
from app_name.database import db
from flask import Flask
def create_app(*args, **kwargs):
env = kwargs['env']
app = Flask(__name__)
app.config.from_object('conf.%s' % env)
db.init_app(app)
return app
app_name/database.py
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
app_name/models/model_name.py
from services.database import db
class Bar(db.Model):
__tablename__ = 'your_MySQL_table_name'
id = db.Column('YourMySQLColumnName', db.Integer, primary_key=True)
name = db.Column('WhateverName', db.String(100))
foo = db.Column(db.ForeignKey('another_MySQLTableName.id'))
class Foo(db.Model):
__tablename__ = 'another_MySQLTableName'
id = db.Column('FooId', db.Integer, primary_key=True)
...
run.py
#! /usr/bin/env python
from app_name import create_app
app = create_app(env='local')
if __name__ == '__main__':
app.run()
I use run.py to run the app locally, but I use nginx + uWSGI to run the app in the dev/stage/live environments.
I’m guessing you’ll have a views/ directory in there in addition to this though.
The key to connecting Holger’s answer to a flask context is that db.Model is a declarative_base object like Base. Took me a while to notice this important sentence in flask-sqlalchemy’s documentation
Below are the steps I used for my app:
-
initiate a
dbobject in the usual flask-alchemy manner:db = SQLAlchemy(app). Note you’ll need to setapp.config['SQLALCHEMY_DATABASE_URI'] = 'connection_string'before that. -
bind the declarative base to an engine:
db.Model.metadata.reflect(db.engine) -
Then you can use existing tables easily (eg. I have a table called BUILDINGS):
class Buildings(db.Model): __table__ = db.Model.metadata.tables['BUILDING'] def __repr__(self): return self.DISTRICT
Now your Buildings class will follow the existing schema. You can try dir(Buildings) in a Python shell and see all the columns already listed.
This is an alternative way to set up the engine path described in Holger’s answer. Convenient if there are special characters in your user name or password.
from sqlalchemy.engine.url import URL
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
engine_URL = URL('mssql+pymssql',
username='DOMAIN\USERNAME',
password="""p@ssword'!""",
host='host.com',
database='database_name')
engine = create_engine(engine_URL)
Base = declarative_base()
Base.metadata.reflect(engine)
I think the easiest way to use an existing database with sqlalchemy is to use AutomapBase class . A sample code from docs is as follows:
from sqlalchemy.ext.automap import automap_base
from sqlalchemy.orm import Session
from sqlalchemy import create_engine
Base = automap_base()
# engine, suppose it has two tables 'user' and 'address' set up
engine = create_engine("sqlite:///mydatabase.db")
# reflect the tables
Base.prepare(engine, reflect=True)
# mapped classes are now created with names by default
# matching that of the table name.
User = Base.classes.user
Address = Base.classes.address
session = Session(engine)
# rudimentary relationships are produced
session.add(Address(email_address="[email protected]", user=User(name="foo")))
session.commit()
# collection-based relationships are by default named
# "<classname>_collection"
print (u1.address_collection)
Refer SqlAlchemy-Automap for details and more complicated usages
This solution worked for me
"""Example for reflecting database tables to ORM objects
This script creates classes for each table reflected
from the database.
Note: The class names are imported to the global namespace using
the same name as the tables. This is useful for quick utility scripts.
A better solution for production code would be to return a dict
of reflected ORM objects.
"""
from sqlalchemy import create_engine, MetaData
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
def reflect_all_tables_to_declarative(uri):
"""Reflects all tables to declaratives
Given a valid engine URI and declarative_base base class
reflects all tables and imports them to the global namespace.
Returns a session object bound to the engine created.
"""
# create an unbound base our objects will inherit from
Base = declarative_base()
engine = create_engine(uri)
metadata = MetaData(bind=engine)
Base.metadata = metadata
g = globals()
metadata.reflect()
for tablename, tableobj in metadata.tables.items():
g[tablename] = type(str(tablename), (Base,), {'__table__' : tableobj })
print("Reflecting {0}".format(tablename))
Session = sessionmaker(bind=engine)
return Session()
# set to database credentials/host
CONNECTION_URI = "postgres://..."
session = reflect_all_tables_to_declarative(CONNECTION_URI)
# do something with the session and the orm objects
results = session.query(some_table_name).all()
I try to use autogenerated but nothing works or I couldn’t run it. When I searching generate code using sqlacodegen I find https://github.com/ksindi/flask-sqlacodegen, you can generate the code just
flask-sqlacodegen mysql://username:password@host:port/db_name --schema yourschema --tables table1,table2 --flask
I tried and it works perfectly
alembic (the tool behind flask-sqlalchemy) can be configured to ignore tables. The configuration isn’t too hard to setup. see: https://gist.github.com/utek/6163250
I’d suggest using dynamic class
from flask import Flask
from sqlalchemy import Table, MetaData
from flask_sqlalchemy import SQLAlchemy
import os
app = Flask(__name__)
class Config(Object):
basedir = os.path.abspath(os.path.dirname(__file__))
SQLALCHEMY_DATABASE_URI = 'sqlite:///' + + os.path.join(basedir, 'db.sqlite')
SQLALCHEMY_TRACK_MODIFICATIONS = False
app.config.from_object(config)
db = SQLAlchemy(app)
metadata = MetaData()
table_reflection = Table("table_name", metadata, autoload=True, autoload_with=db.engine)
attrs = {"__table__": table_reflection}
TableModel = type("table_name", (db.Model,), attrs)
Now, TableModel can be used as models class
# Querying
TableModel.query.all()
TableModel.query.get(1)
TableModel.query.filter_by(id=2).first()
# insertion
new_row = TableModel()
db.session.add(new_row)
db.session.commit()
If your application is simple you can use db.reflect(app) on your model file:
def configure(app):
db.init_app(app)
db.reflect(app=app)
app.db = db