import module from string variable
Question:
I’m working on a documentation (personal) for nested matplotlib (MPL) library, which differs from MPL own provided, by interested submodule packages. I’m writing Python script which I hope will automate document generation from future MPL releases.
I selected interested submodules/packages and want to list their main classes from which I’ll generate list and process it with pydoc
Problem is that I can’t find a way to instruct Python to load submodule from string. Here is example of what I tried:
import matplotlib.text as text
x = dir(text)
.
i = __import__('matplotlib.text')
y = dir(i)
.
j = __import__('matplotlib')
z = dir(j)
And here is 3 way comparison of above lists through pprint:
I don’t understand what’s loaded in y
object – it’s base matplotlib
plus something else, but it lack information that I wanted and that is main classes from matplotlib.text
package. It’s top blue coloured part on screenshot (x
list)
Please don’t suggest Sphinx as different approach.
Answers:
importlib.import_module
is what you are looking for. It returns the imported module.
import importlib
mymodule = importlib.import_module('matplotlib.text')
You can thereafter access anything in the module as mymodule.myclass
, mymodule.myfunction
, etc.
The __import__
function can be a bit hard to understand.
If you change
i = __import__('matplotlib.text')
to
i = __import__('matplotlib.text', fromlist=[''])
then i
will refer to matplotlib.text
.
In Python 2.7 and Python 3.1 or later, you can use importlib
:
import importlib
i = importlib.import_module("matplotlib.text")
Some notes
-
If you’re trying to import something from a sub-folder e.g. ./feature/email.py
, the code will look like importlib.import_module("feature.email")
-
You can’t import anything if there is no __init__.py
in the folder with file you are trying to import
spent some time trying to import modules from a list, and this is the thread that got me most of the way there – but I didnt grasp the use of ___import____ –
so here’s how to import a module from a string, and get the same behavior as just import. And try/except the error case, too. 🙂
pipmodules = ['pycurl', 'ansible', 'bad_module_no_beer']
for module in pipmodules:
try:
# because we want to import using a variable, do it this way
module_obj = __import__(module)
# create a global object containging our module
globals()[module] = module_obj
except ImportError:
sys.stderr.write("ERROR: missing python module: " + module + "n")
sys.exit(1)
and yes, for python 2.7> you have other options – but for 2.6<, this works.
I developed these 3 useful functions:
def loadModule(moduleName):
module = None
try:
import sys
del sys.modules[moduleName]
except BaseException as err:
pass
try:
import importlib
module = importlib.import_module(moduleName)
except BaseException as err:
serr = str(err)
print("Error to load the module '" + moduleName + "': " + serr)
return module
def reloadModule(moduleName):
module = loadModule(moduleName)
moduleName, modulePath = str(module).replace("' from '", "||").replace("<module '", '').replace("'>", '').split("||")
if (modulePath.endswith(".pyc")):
import os
os.remove(modulePath)
module = loadModule(moduleName)
return module
def getInstance(moduleName, param1, param2, param3):
module = reloadModule(moduleName)
instance = eval("module." + moduleName + "(param1, param2, param3)")
return instance
And everytime I want to reload a new instance I just have to call getInstance() like this:
myInstance = getInstance("MyModule", myParam1, myParam2, myParam3)
Finally I can call all the functions inside the new Instance:
myInstance.aFunction()
The only specificity here is to customize the params list (param1, param2, param3) of your instance.
Apart from using the importlib
one can also use exec
method to import a module from a string variable.
Here I am showing an example of importing the combinations
method from itertools
package using the exec
method:
MODULES = [
['itertools','combinations'],
]
for ITEM in MODULES:
import_str = "from {0} import {1}".format(ITEM[0],', '.join(str(i) for i in ITEM[1:]))
exec(import_str)
ar = list(combinations([1, 2, 3, 4], 2))
for elements in ar:
print(elements)
Output:
(1, 2)
(1, 3)
(1, 4)
(2, 3)
(2, 4)
(3, 4)
Module auto-install & import from list
Below script works fine with both submodules and pseudo submodules.
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)
I’m working on a documentation (personal) for nested matplotlib (MPL) library, which differs from MPL own provided, by interested submodule packages. I’m writing Python script which I hope will automate document generation from future MPL releases.
I selected interested submodules/packages and want to list their main classes from which I’ll generate list and process it with pydoc
Problem is that I can’t find a way to instruct Python to load submodule from string. Here is example of what I tried:
import matplotlib.text as text
x = dir(text)
.
i = __import__('matplotlib.text')
y = dir(i)
.
j = __import__('matplotlib')
z = dir(j)
And here is 3 way comparison of above lists through pprint:
I don’t understand what’s loaded in y
object – it’s base matplotlib
plus something else, but it lack information that I wanted and that is main classes from matplotlib.text
package. It’s top blue coloured part on screenshot (x
list)
Please don’t suggest Sphinx as different approach.
importlib.import_module
is what you are looking for. It returns the imported module.
import importlib
mymodule = importlib.import_module('matplotlib.text')
You can thereafter access anything in the module as mymodule.myclass
, mymodule.myfunction
, etc.
The __import__
function can be a bit hard to understand.
If you change
i = __import__('matplotlib.text')
to
i = __import__('matplotlib.text', fromlist=[''])
then i
will refer to matplotlib.text
.
In Python 2.7 and Python 3.1 or later, you can use importlib
:
import importlib
i = importlib.import_module("matplotlib.text")
Some notes
-
If you’re trying to import something from a sub-folder e.g.
./feature/email.py
, the code will look likeimportlib.import_module("feature.email")
-
You can’t import anything if there is no
__init__.py
in the folder with file you are trying to import
spent some time trying to import modules from a list, and this is the thread that got me most of the way there – but I didnt grasp the use of ___import____ –
so here’s how to import a module from a string, and get the same behavior as just import. And try/except the error case, too. 🙂
pipmodules = ['pycurl', 'ansible', 'bad_module_no_beer']
for module in pipmodules:
try:
# because we want to import using a variable, do it this way
module_obj = __import__(module)
# create a global object containging our module
globals()[module] = module_obj
except ImportError:
sys.stderr.write("ERROR: missing python module: " + module + "n")
sys.exit(1)
and yes, for python 2.7> you have other options – but for 2.6<, this works.
I developed these 3 useful functions:
def loadModule(moduleName):
module = None
try:
import sys
del sys.modules[moduleName]
except BaseException as err:
pass
try:
import importlib
module = importlib.import_module(moduleName)
except BaseException as err:
serr = str(err)
print("Error to load the module '" + moduleName + "': " + serr)
return module
def reloadModule(moduleName):
module = loadModule(moduleName)
moduleName, modulePath = str(module).replace("' from '", "||").replace("<module '", '').replace("'>", '').split("||")
if (modulePath.endswith(".pyc")):
import os
os.remove(modulePath)
module = loadModule(moduleName)
return module
def getInstance(moduleName, param1, param2, param3):
module = reloadModule(moduleName)
instance = eval("module." + moduleName + "(param1, param2, param3)")
return instance
And everytime I want to reload a new instance I just have to call getInstance() like this:
myInstance = getInstance("MyModule", myParam1, myParam2, myParam3)
Finally I can call all the functions inside the new Instance:
myInstance.aFunction()
The only specificity here is to customize the params list (param1, param2, param3) of your instance.
Apart from using the importlib
one can also use exec
method to import a module from a string variable.
Here I am showing an example of importing the combinations
method from itertools
package using the exec
method:
MODULES = [
['itertools','combinations'],
]
for ITEM in MODULES:
import_str = "from {0} import {1}".format(ITEM[0],', '.join(str(i) for i in ITEM[1:]))
exec(import_str)
ar = list(combinations([1, 2, 3, 4], 2))
for elements in ar:
print(elements)
Output:
(1, 2)
(1, 3)
(1, 4)
(2, 3)
(2, 4)
(3, 4)
Module auto-install & import from list
Below script works fine with both submodules and pseudo submodules.
# PyPI imports
import pkg_resources, subprocess, sys
modules = {'lxml.etree', 'pandas', 'screeninfo'}
required = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing = required - installed
if missing:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])
for module in set.union(required, modules):
globals()[module] = __import__(module)
Tests:
print(pandas.__version__)
print(lxml.etree.LXML_VERSION)