When using importlib to load a module, can I put it in a package without an __init__.py file?
Question:
My Python application loads plugins from a user-specified path (which is not part of sys.path
), according to the importlib documentation:
import importlib.util
import sys
def load_module(module_name, file_path):
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
return module
plugin_module = load_module("some_plugin", "/path/to/plugins/some_plugin.py")
To make it possible for the user to factor out common functionality between multiple plugins, I want to allow relative imports in the plugins:
from . import plugin_common
def plugin_function(x):
return plugin_common.something(x)
When implemented like this, I get an ImportError in the plugin:
ImportError: attempted relative import with no known parent package
To my understanding, this is because the some_plugin
module is not considered part of a package, and relative imports can therefore not be used (inside some_plugin.py
, __name__
is 'some_plugin'
and __package__
is empty).
I can solve this by first loading the surrounding package and then putting the imported module into that package:
load_module("plugins_package", "/path/to/plugins/__init__.py")
plugin_module = load_module("plugins_package.some_plugin", "/path/to/plugins/some_plugin.py")
Now, __name__
is 'plugins_package.some_plugin'
, __package__
is 'plugins_package'
, and I can use relative imports.
However, this requires the user to put an (empty) __init__.py
file in the plugins directory, which I would like to avoid. Since normal packages don’t require an __init__.py
file (they will be treated as a namespace packages), it seems like this should be possible.
It seems like it should be possible to create a namespace package dynamically (using importlib
) for plugins_package
and using that as package for the imported plugin_module
. But I haven’t found a way to do this.
So:
- Can I create a namespace package (where I can put
plugin_module
in) dynamically?
- Can I dynamically create a normal package without the need for an
__init__.py
file?
- Am I on the wrong track and there is a better way to achieve what I want (relative imports in a module loaded dynamically from outside
sys.path
)?
Answers:
I’ve created an import library that allows you to create a namespace package dynamically: ultraimport.
Check out quickstart example #10:
import ultraimport
plugin_package = ultraimport.create_ns_package('plugins_package.some_plugin', '/path/to/plugins')
With ultraimport you can also load the plugin file and create the namespace package implicitly on the fly like this:
import ultraimport
plugin_module = ultraimport("/path/to/plugins/some_plugin.py", package=1)
This is explained in more detail quickstart example #7.
My Python application loads plugins from a user-specified path (which is not part of sys.path
), according to the importlib documentation:
import importlib.util
import sys
def load_module(module_name, file_path):
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)
return module
plugin_module = load_module("some_plugin", "/path/to/plugins/some_plugin.py")
To make it possible for the user to factor out common functionality between multiple plugins, I want to allow relative imports in the plugins:
from . import plugin_common
def plugin_function(x):
return plugin_common.something(x)
When implemented like this, I get an ImportError in the plugin:
ImportError: attempted relative import with no known parent package
To my understanding, this is because the some_plugin
module is not considered part of a package, and relative imports can therefore not be used (inside some_plugin.py
, __name__
is 'some_plugin'
and __package__
is empty).
I can solve this by first loading the surrounding package and then putting the imported module into that package:
load_module("plugins_package", "/path/to/plugins/__init__.py")
plugin_module = load_module("plugins_package.some_plugin", "/path/to/plugins/some_plugin.py")
Now, __name__
is 'plugins_package.some_plugin'
, __package__
is 'plugins_package'
, and I can use relative imports.
However, this requires the user to put an (empty) __init__.py
file in the plugins directory, which I would like to avoid. Since normal packages don’t require an __init__.py
file (they will be treated as a namespace packages), it seems like this should be possible.
It seems like it should be possible to create a namespace package dynamically (using importlib
) for plugins_package
and using that as package for the imported plugin_module
. But I haven’t found a way to do this.
So:
- Can I create a namespace package (where I can put
plugin_module
in) dynamically? - Can I dynamically create a normal package without the need for an
__init__.py
file? - Am I on the wrong track and there is a better way to achieve what I want (relative imports in a module loaded dynamically from outside
sys.path
)?
I’ve created an import library that allows you to create a namespace package dynamically: ultraimport.
Check out quickstart example #10:
import ultraimport
plugin_package = ultraimport.create_ns_package('plugins_package.some_plugin', '/path/to/plugins')
With ultraimport you can also load the plugin file and create the namespace package implicitly on the fly like this:
import ultraimport
plugin_module = ultraimport("/path/to/plugins/some_plugin.py", package=1)
This is explained in more detail quickstart example #7.