How to dynamically load a Python class
Question:
Given a string of a Python class, e.g. my_package.my_module.MyClass
, what is the best possible way to load it?
In other words I am looking for a equivalent Class.forName()
in Java, function in Python. It needs to work on Google App Engine.
Preferably this would be a function that accepts the FQN of the class as a string, and returns a reference to the class:
my_class = load_class('my_package.my_module.MyClass')
my_instance = my_class()
Answers:
module = __import__("my_package/my_module")
the_class = getattr(module, "MyClass")
obj = the_class()
From the python documentation, here’s the function you want:
def my_import(name):
components = name.split('.')
mod = __import__(components[0])
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
The reason a simple __import__
won’t work is because any import of anything past the first dot in a package string is an attribute of the module you’re importing. Thus, something like this won’t work:
__import__('foo.bar.baz.qux')
You’d have to call the above function like so:
my_import('foo.bar.baz.qux')
Or in the case of your example:
klass = my_import('my_package.my_module.my_class')
some_object = klass()
EDIT: I was a bit off on this. What you’re basically wanting to do is this:
from my_package.my_module import my_class
The above function is only necessary if you have a empty fromlist. Thus, the appropriate call would be like this:
mod = __import__('my_package.my_module', fromlist=['my_class'])
klass = getattr(mod, 'my_class')
def import_class(cl):
d = cl.rfind(".")
classname = cl[d+1:len(cl)]
m = __import__(cl[0:d], globals(), locals(), [classname])
return getattr(m, classname)
import importlib
module = importlib.import_module('my_package.my_module')
my_class = getattr(module, 'MyClass')
my_instance = my_class()
If you don’t want to roll your own, there is a function available in the pydoc
module that does exactly this:
from pydoc import locate
my_class = locate('my_package.my_module.MyClass')
The advantage of this approach over the others listed here is that locate
will find any python object at the provided dotted path, not just an object directly within a module. e.g. my_package.my_module.MyClass.attr
.
If you’re curious what their recipe is, here’s the function:
def locate(path, forceload=0):
"""Locate an object by name or dotted path, importing as necessary."""
parts = [part for part in split(path, '.') if part]
module, n = None, 0
while n < len(parts):
nextmodule = safeimport(join(parts[:n+1], '.'), forceload)
if nextmodule: module, n = nextmodule, n + 1
else: break
if module:
object = module
else:
object = __builtin__
for part in parts[n:]:
try:
object = getattr(object, part)
except AttributeError:
return None
return object
It relies on pydoc.safeimport
function. Here are the docs for that:
"""Import a module; handle errors; return None if the module isn't found.
If the module *is* found but an exception occurs, it's wrapped in an
ErrorDuringImport exception and reraised. Unlike __import__, if a
package path is specified, the module at the end of the path is returned,
not the package at the beginning. If the optional 'forceload' argument
is 1, we reload the module from disk (unless it's a dynamic extension)."""
In Google App Engine there is a webapp2
function called import_string
. For more info see here:https://webapp-improved.appspot.com/api/webapp2.html
So,
import webapp2
my_class = webapp2.import_string('my_package.my_module.MyClass')
For example this is used in the webapp2.Route
where you can either use a handler or a string.
OK, for me that is the way it worked (I am using Python 2.7):
a = __import__('file_to_import', globals(), locals(), ['*'], -1)
b = a.MyClass()
Then, b is an instance of class ‘MyClass’
If you happen to already have an instance of your desired class, you can use the ‘type’ function to extract its class type and use this to construct a new instance:
class Something(object):
def __init__(self, name):
self.name = name
def display(self):
print(self.name)
one = Something("one")
one.display()
cls = type(one)
two = cls("two")
two.display()
If you’re using Django you can use import_string
.
Yes i’m aware OP did not ask for django, but i ran across this question looking for a Django solution, didn’t find one, and put it here for the next boy/gal that looks for it.
# It's available for v1.7+
# https://github.com/django/django/blob/stable/1.7.x/django/utils/module_loading.py
from django.utils.module_loading import import_string
Klass = import_string('path.to.module.Klass')
func = import_string('path.to.module.func')
var = import_string('path.to.module.var')
Keep in mind, if you want to import something that doesn’t have a .
, like re
or argparse
use:
re = __import__('re')
Here is to share something I found on __import__
and importlib
while trying to solve this problem.
I am using Python 3.7.3.
When I try to get to the class d
in module a.b.c
,
mod = __import__('a.b.c')
The mod
variable refer to the top namespace a
.
So to get to the class d
, I need to
mod = getattr(mod, 'b') #mod is now module b
mod = getattr(mod, 'c') #mod is now module c
mod = getattr(mod, 'd') #mod is now class d
If we try to do
mod = __import__('a.b.c')
d = getattr(mod, 'd')
we are actually trying to look for a.d
.
When using importlib
, I suppose the library has done the recursive getattr
for us. So, when we use importlib.import_module
, we actually get a handle on the deepest module.
mod = importlib.import_module('a.b.c') #mod is module c
d = getattr(mod, 'd') #this is a.b.c.d
Python has an inbuilt library importlib to get the job done. :, How to access module method and class method dynamically bypassing package name as a param. An example is given below.
Module 1:
def get_scenario_data():
return "module1 scenario data"
class Module1:
def module1_function1(self):
return "module1_function"
def module1_function2(self):
return "module2_function"
Module 2:
def get_scenario_data():
return "module2 scenario data"
class Module2:
def module2_function1(self):
return "module2_function1"
def module2_function2(self):
return "module2_function2"
ModuleTest:
- Will access the module methods dynamically based on the package name as param
- Will access the class methods dynamically based on the package name as param.
ModuleTest
import importlib
module = importlib.import_module('pack1.nestedpack1.module1')
print(module.get_scenario_data())
modul1_cls_obj = getattr(module, 'Module1')()
print(modul1_cls_obj.module1_function1())
print(modul1_cls_obj.module1_function2())
module = importlib.import_module('pack1.nestedpack1.module2')
modul2_cls_obj = getattr(module, 'Module2')()
print(modul2_cls_obj.module2_function1())
print(modul2_cls_obj.module2_function2())
print(module.get_scenario_data())
Results
module1 scenario data
module1_function
module2_function
module2_function1
module2_function2
module2 scenario data
PyPI module autoloader & import
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)
Adding a bit of sophistication to the existing answers….
Depending on the use case, it may be somewhat inconvenient to have to explicitly specify the full path (E.g. package.subpackage.module...
) of the class/method you want to import. On top of importlib, we can leverage __init__.py
to make things even cleaner.
Let’s say I have a python package, like so:
├── modes
│ ├── __init__.py
│ ├── bar.py
│ ├── foo.py
│ ├── modes.py
foo.py
, say, have some class/functions we’d like to use somewhere else in our program:
from modes.modes import Mode
class Foo(Mode):
def __init__(self, *arg, **kwargs):
super(Foo, self).__init__(*arg, **kwargs)
def run(self):
self.LOG.info(f"This is FOO!")
With a command line argument, I can pass an argument that corresponds to a mode that I want to run. I’d like to be able to so something like this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes.foo')
getattr(module, mode)().run()
which outputs:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
That works fine, however what we’d REALLY want to get at is this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes') # only import the package, not modules explicitely
getattr(module, mode)().run()
Which raises an error:
>> set_mode("Foo")
>> AttributeError: module 'modes' has no attribute 'Foo'
However, we can add the following to /modes/__init__.py
:
from .foo import Foo
from .bar import Bar
Then, we can do:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
>> set_mode("Bar")
>> engine_logger:INFO - This is BAR!
In other worlds, all sub modules/functions/classes we import in init.py will be found directly with importlib.import_module(…), without having to specify the full path from outside.
Given a string of a Python class, e.g. my_package.my_module.MyClass
, what is the best possible way to load it?
In other words I am looking for a equivalent Class.forName()
in Java, function in Python. It needs to work on Google App Engine.
Preferably this would be a function that accepts the FQN of the class as a string, and returns a reference to the class:
my_class = load_class('my_package.my_module.MyClass')
my_instance = my_class()
module = __import__("my_package/my_module")
the_class = getattr(module, "MyClass")
obj = the_class()
From the python documentation, here’s the function you want:
def my_import(name):
components = name.split('.')
mod = __import__(components[0])
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
The reason a simple __import__
won’t work is because any import of anything past the first dot in a package string is an attribute of the module you’re importing. Thus, something like this won’t work:
__import__('foo.bar.baz.qux')
You’d have to call the above function like so:
my_import('foo.bar.baz.qux')
Or in the case of your example:
klass = my_import('my_package.my_module.my_class')
some_object = klass()
EDIT: I was a bit off on this. What you’re basically wanting to do is this:
from my_package.my_module import my_class
The above function is only necessary if you have a empty fromlist. Thus, the appropriate call would be like this:
mod = __import__('my_package.my_module', fromlist=['my_class'])
klass = getattr(mod, 'my_class')
def import_class(cl):
d = cl.rfind(".")
classname = cl[d+1:len(cl)]
m = __import__(cl[0:d], globals(), locals(), [classname])
return getattr(m, classname)
import importlib
module = importlib.import_module('my_package.my_module')
my_class = getattr(module, 'MyClass')
my_instance = my_class()
If you don’t want to roll your own, there is a function available in the pydoc
module that does exactly this:
from pydoc import locate
my_class = locate('my_package.my_module.MyClass')
The advantage of this approach over the others listed here is that locate
will find any python object at the provided dotted path, not just an object directly within a module. e.g. my_package.my_module.MyClass.attr
.
If you’re curious what their recipe is, here’s the function:
def locate(path, forceload=0):
"""Locate an object by name or dotted path, importing as necessary."""
parts = [part for part in split(path, '.') if part]
module, n = None, 0
while n < len(parts):
nextmodule = safeimport(join(parts[:n+1], '.'), forceload)
if nextmodule: module, n = nextmodule, n + 1
else: break
if module:
object = module
else:
object = __builtin__
for part in parts[n:]:
try:
object = getattr(object, part)
except AttributeError:
return None
return object
It relies on pydoc.safeimport
function. Here are the docs for that:
"""Import a module; handle errors; return None if the module isn't found.
If the module *is* found but an exception occurs, it's wrapped in an
ErrorDuringImport exception and reraised. Unlike __import__, if a
package path is specified, the module at the end of the path is returned,
not the package at the beginning. If the optional 'forceload' argument
is 1, we reload the module from disk (unless it's a dynamic extension)."""
In Google App Engine there is a webapp2
function called import_string
. For more info see here:https://webapp-improved.appspot.com/api/webapp2.html
So,
import webapp2
my_class = webapp2.import_string('my_package.my_module.MyClass')
For example this is used in the webapp2.Route
where you can either use a handler or a string.
OK, for me that is the way it worked (I am using Python 2.7):
a = __import__('file_to_import', globals(), locals(), ['*'], -1)
b = a.MyClass()
Then, b is an instance of class ‘MyClass’
If you happen to already have an instance of your desired class, you can use the ‘type’ function to extract its class type and use this to construct a new instance:
class Something(object):
def __init__(self, name):
self.name = name
def display(self):
print(self.name)
one = Something("one")
one.display()
cls = type(one)
two = cls("two")
two.display()
If you’re using Django you can use import_string
.
Yes i’m aware OP did not ask for django, but i ran across this question looking for a Django solution, didn’t find one, and put it here for the next boy/gal that looks for it.
# It's available for v1.7+
# https://github.com/django/django/blob/stable/1.7.x/django/utils/module_loading.py
from django.utils.module_loading import import_string
Klass = import_string('path.to.module.Klass')
func = import_string('path.to.module.func')
var = import_string('path.to.module.var')
Keep in mind, if you want to import something that doesn’t have a .
, like re
or argparse
use:
re = __import__('re')
Here is to share something I found on __import__
and importlib
while trying to solve this problem.
I am using Python 3.7.3.
When I try to get to the class d
in module a.b.c
,
mod = __import__('a.b.c')
The mod
variable refer to the top namespace a
.
So to get to the class d
, I need to
mod = getattr(mod, 'b') #mod is now module b
mod = getattr(mod, 'c') #mod is now module c
mod = getattr(mod, 'd') #mod is now class d
If we try to do
mod = __import__('a.b.c')
d = getattr(mod, 'd')
we are actually trying to look for a.d
.
When using importlib
, I suppose the library has done the recursive getattr
for us. So, when we use importlib.import_module
, we actually get a handle on the deepest module.
mod = importlib.import_module('a.b.c') #mod is module c
d = getattr(mod, 'd') #this is a.b.c.d
Python has an inbuilt library importlib to get the job done. :, How to access module method and class method dynamically bypassing package name as a param. An example is given below.
Module 1:
def get_scenario_data():
return "module1 scenario data"
class Module1:
def module1_function1(self):
return "module1_function"
def module1_function2(self):
return "module2_function"
Module 2:
def get_scenario_data():
return "module2 scenario data"
class Module2:
def module2_function1(self):
return "module2_function1"
def module2_function2(self):
return "module2_function2"
ModuleTest:
- Will access the module methods dynamically based on the package name as param
- Will access the class methods dynamically based on the package name as param.
ModuleTest
import importlib
module = importlib.import_module('pack1.nestedpack1.module1')
print(module.get_scenario_data())
modul1_cls_obj = getattr(module, 'Module1')()
print(modul1_cls_obj.module1_function1())
print(modul1_cls_obj.module1_function2())
module = importlib.import_module('pack1.nestedpack1.module2')
modul2_cls_obj = getattr(module, 'Module2')()
print(modul2_cls_obj.module2_function1())
print(modul2_cls_obj.module2_function2())
print(module.get_scenario_data())
Results
module1 scenario data
module1_function
module2_function
module2_function1
module2_function2
module2 scenario data
PyPI module autoloader & import
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)
Adding a bit of sophistication to the existing answers….
Depending on the use case, it may be somewhat inconvenient to have to explicitly specify the full path (E.g. package.subpackage.module...
) of the class/method you want to import. On top of importlib, we can leverage __init__.py
to make things even cleaner.
Let’s say I have a python package, like so:
├── modes
│ ├── __init__.py
│ ├── bar.py
│ ├── foo.py
│ ├── modes.py
foo.py
, say, have some class/functions we’d like to use somewhere else in our program:
from modes.modes import Mode
class Foo(Mode):
def __init__(self, *arg, **kwargs):
super(Foo, self).__init__(*arg, **kwargs)
def run(self):
self.LOG.info(f"This is FOO!")
With a command line argument, I can pass an argument that corresponds to a mode that I want to run. I’d like to be able to so something like this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes.foo')
getattr(module, mode)().run()
which outputs:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
That works fine, however what we’d REALLY want to get at is this:
def set_mode(mode):
""" """
import importlib
module = importlib.import_module('modes') # only import the package, not modules explicitely
getattr(module, mode)().run()
Which raises an error:
>> set_mode("Foo")
>> AttributeError: module 'modes' has no attribute 'Foo'
However, we can add the following to /modes/__init__.py
:
from .foo import Foo
from .bar import Bar
Then, we can do:
>> set_mode("Foo")
>> engine_logger:INFO - This is FOO!
>> set_mode("Bar")
>> engine_logger:INFO - This is BAR!
In other worlds, all sub modules/functions/classes we import in init.py will be found directly with importlib.import_module(…), without having to specify the full path from outside.