dynamically loading django apps at runtime
Question:
Is it possible to dynamically load Django apps at runtime? Usually, apps are loaded at initialization, using the INSTALLED_APPS tuple in settings.py. However, is it possible to load additional apps at runtime? I am encountering this issue in different situations. One situation, for example, arises during testing, when I would like to dynamically load or unload apps.
In order to make the problem more concrete, imagine I have a directory called apps where I put my apps and I would like to automatically install any new app that goes in there without manually editing the settings.py.
This is easy enough. Following the example code in
Django: Dynamically add apps as plugin, building urls and other settings automatically
we put the following code in settings.py to could loop over the names of all sub-directories in the app directory and increment the INSTALLED_APPS tuple in settings.py like this:
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in INSTALLED_APPS:
INSTALLED_APPS += (app_name, )
After that, if I was in a Django shell, I could something like
from django.conf import settings
and the apps would be listed in settings.INSTALLED_APPS. And if I did
from django.core import management
management.call_command('syncdb', interactive=False)
that would create the necessary DB tables for the apps.
However, if I were to now add some more apps to the apps/ directory, without re-starting, these would not be listed in settings.INSTALLED_APPS, and so a subsequent call to the syncdb would have no effect.
What I would like to know is if there is something I could do — without restarting — to reload the settings and load/install new apps.
I have tried to directly import my settings.py, i.e.
from myproject import settings
and then reload that settings using the python builtin after any app directory changes. Although settings.INSTALLED_APPS is now changed to include the newly added apps, this ultimately makes no difference. For example,
from django.db import models
models.get_apps()
shows only the original apps in apps and not the newly added ones and likewise
management.call_command('syncdb', interactive=False)
will not see the newly added apps.
As I stated above, I am thinking about this situation particularly in the context of testings where I dynamically would add or remove apps.
Ps. I working with Django 1.6, but on the advice of @RickyA, I see that there are some substantial changes to Django’s treatment of applications in 1.7
https://docs.djangoproject.com/en/1.7/ref/applications/
I’m still not sure what this might mean for the problem I am facing.
Answers:
To answer my own question…
While I do not have a completely general solution to this problem, I do have one that is sufficient for the purposes of dynamically loading apps during testing.
The basic solution is simple, and I found it at a wee little bixly blog.
Continuing with my example above, if I was in a django shell and wanted to add and load some new apps that were added to my apps directory, I could do
import os
from django.conf import settings
from django.db.models import loading
from django.core import management
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in settings.INSTALLED_APPS:
settings.INSTALLED_APPS += (app_name, )
and then
loading.cache.loaded = False
management.call_command('syncdb', interactive=False)
Update for Django 1.8 on how to load an app that is not loaded yet
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
# To load the new app let's reset app_configs, the dictionary
# with the configuration of loaded apps
apps.app_configs = OrderedDict()
# set ready to false so that populate will work
apps.ready = False
# re-initialize them all; is there a way to add just one without reloading them all?
apps.populate(settings.INSTALLED_APPS)
# now I can generate the migrations for the new app
management.call_command('makemigrations', new_app_name, interactive=False)
# and migrate it
management.call_command('migrate', new_app_name, interactive=False)
Yes! Anything (or almost everything) in Python is possible. You should use os.walk() in order to get all folders, subfolders and files within your apps path in order to get all of your apps including the nested ones.
def get_installed_apps():
from os import walk, chdir, getcwd
previous_path = getcwd()
master = []
APPS_ROOT_PATH = '/my/project/apps/folder'
chdir(APPS_ROOT_PATH)
for root, directories, files in walk(top=getcwd(), topdown=False):
for file in files:
if 'apps.py' in file:
app_path = f"{root.replace(BASE_DIR + '/', '').replace('/', '.')}.apps"
print(app_path)
master.append(app_path)
chdir(previous_path)
return master
print(get_installed_apps())
With Django 2.2 this work for me
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
apps.app_configs = OrderedDict()
apps.apps_ready = apps.models_ready = apps.loading = apps.ready = False
apps.clear_cache()
apps.populate(settings.INSTALLED_APPS)
management.call_command('makemigrations', new_app_name, interactive=False)
management.call_command('migrate', new_app_name, interactive=False)
It is possible to dynamically load and unload applications in tests in Django >= 1.7 (and also in the current 4.1) by a override_settings() decorator:
@override_settings(INSTALLED_APPS=[...]) # added or removed some apps
class MyTest(TestCase):
# some tests with these apps
def test_foo(self):
pass
It has been possible since September 2014, not before the question.
Many other topics from the question are solved by django.apps also in Django >= 1.7. Generally: Dynamic configuration at startup is easy in the current Django. Dynamic loading and unloading after startup can’t be recommended in production, even that it could work eventually.
Is it possible to dynamically load Django apps at runtime? Usually, apps are loaded at initialization, using the INSTALLED_APPS tuple in settings.py. However, is it possible to load additional apps at runtime? I am encountering this issue in different situations. One situation, for example, arises during testing, when I would like to dynamically load or unload apps.
In order to make the problem more concrete, imagine I have a directory called apps where I put my apps and I would like to automatically install any new app that goes in there without manually editing the settings.py.
This is easy enough. Following the example code in
Django: Dynamically add apps as plugin, building urls and other settings automatically
we put the following code in settings.py to could loop over the names of all sub-directories in the app directory and increment the INSTALLED_APPS tuple in settings.py like this:
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in INSTALLED_APPS:
INSTALLED_APPS += (app_name, )
After that, if I was in a Django shell, I could something like
from django.conf import settings
and the apps would be listed in settings.INSTALLED_APPS. And if I did
from django.core import management
management.call_command('syncdb', interactive=False)
that would create the necessary DB tables for the apps.
However, if I were to now add some more apps to the apps/ directory, without re-starting, these would not be listed in settings.INSTALLED_APPS, and so a subsequent call to the syncdb would have no effect.
What I would like to know is if there is something I could do — without restarting — to reload the settings and load/install new apps.
I have tried to directly import my settings.py, i.e.
from myproject import settings
and then reload that settings using the python builtin after any app directory changes. Although settings.INSTALLED_APPS is now changed to include the newly added apps, this ultimately makes no difference. For example,
from django.db import models
models.get_apps()
shows only the original apps in apps and not the newly added ones and likewise
management.call_command('syncdb', interactive=False)
will not see the newly added apps.
As I stated above, I am thinking about this situation particularly in the context of testings where I dynamically would add or remove apps.
Ps. I working with Django 1.6, but on the advice of @RickyA, I see that there are some substantial changes to Django’s treatment of applications in 1.7
https://docs.djangoproject.com/en/1.7/ref/applications/
I’m still not sure what this might mean for the problem I am facing.
To answer my own question…
While I do not have a completely general solution to this problem, I do have one that is sufficient for the purposes of dynamically loading apps during testing.
The basic solution is simple, and I found it at a wee little bixly blog.
Continuing with my example above, if I was in a django shell and wanted to add and load some new apps that were added to my apps directory, I could do
import os
from django.conf import settings
from django.db.models import loading
from django.core import management
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in settings.INSTALLED_APPS:
settings.INSTALLED_APPS += (app_name, )
and then
loading.cache.loaded = False
management.call_command('syncdb', interactive=False)
Update for Django 1.8 on how to load an app that is not loaded yet
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
# To load the new app let's reset app_configs, the dictionary
# with the configuration of loaded apps
apps.app_configs = OrderedDict()
# set ready to false so that populate will work
apps.ready = False
# re-initialize them all; is there a way to add just one without reloading them all?
apps.populate(settings.INSTALLED_APPS)
# now I can generate the migrations for the new app
management.call_command('makemigrations', new_app_name, interactive=False)
# and migrate it
management.call_command('migrate', new_app_name, interactive=False)
Yes! Anything (or almost everything) in Python is possible. You should use os.walk() in order to get all folders, subfolders and files within your apps path in order to get all of your apps including the nested ones.
def get_installed_apps():
from os import walk, chdir, getcwd
previous_path = getcwd()
master = []
APPS_ROOT_PATH = '/my/project/apps/folder'
chdir(APPS_ROOT_PATH)
for root, directories, files in walk(top=getcwd(), topdown=False):
for file in files:
if 'apps.py' in file:
app_path = f"{root.replace(BASE_DIR + '/', '').replace('/', '.')}.apps"
print(app_path)
master.append(app_path)
chdir(previous_path)
return master
print(get_installed_apps())
With Django 2.2 this work for me
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
apps.app_configs = OrderedDict()
apps.apps_ready = apps.models_ready = apps.loading = apps.ready = False
apps.clear_cache()
apps.populate(settings.INSTALLED_APPS)
management.call_command('makemigrations', new_app_name, interactive=False)
management.call_command('migrate', new_app_name, interactive=False)
It is possible to dynamically load and unload applications in tests in Django >= 1.7 (and also in the current 4.1) by a override_settings() decorator:
@override_settings(INSTALLED_APPS=[...]) # added or removed some apps
class MyTest(TestCase):
# some tests with these apps
def test_foo(self):
pass
It has been possible since September 2014, not before the question.
Many other topics from the question are solved by django.apps also in Django >= 1.7. Generally: Dynamic configuration at startup is easy in the current Django. Dynamic loading and unloading after startup can’t be recommended in production, even that it could work eventually.