Loading environment modules within a python script

Question

Is there a way for a python script to load and use environment modules? os.system('module load xxx') doesn’t work since it executes them in a subshell (at least, I think that’s what’s happening).

Asked By: marshall.ward

||

Source

Answer 1

Not directly, but here’s one possible workaround, depending on your environment. Assuming you can preface your system command with ENVVAR=value, you can do something along these lines:

import os
os.environ['EDITOR'] = 'vi'
cmd = "EDITOR=%(EDITOR)s $EDITOR" % os.environ
os.system(cmd)

The code assigns vi to your EDITOR environment variable, then passes it on the command line and runs the command, which (in this case) is EDITOR.

Answered By: Jeff Bauer

Answer 2

One of our admins was able to solve the problem for me using os.popen() calls to modulecmd:

cmd = os.popen('/path/to/modulecmd python load my-module')
exec(cmd)

Answered By: marshall.ward

Answer 3

I know this question’s kind of old but it’s still relevant enough that I was looking for the answer, so I’m posting what I found that works as well:

At least in the 3.2.9+ sources, you can include the python “init” file to get a python function version of module:

>>> exec(open('/usr/local/Modules/default/init/python.py').read())
>>> module('list')
No Modulefiles Currently Loaded.
>>> module('load','foo')
>>> module('list')
Currently Loaded Modulefiles:
  1) foo/1.0

I’ve been told earlier versions can do the same without the .py extension, but that’s second hand, so ymmv.

Alternative “init” file location (from comment by @lib): /usr/share/Modules/init/python.py

To use with Python 3, version 4.0 or later of Environment Modules is required, as that is the first version to have a bug-free Python3-compliant version of the Python init file.

Answered By: jnewman

Answer 4

While the accepted solution works, I found it to be easier to write:

import sys
sys.path.insert(0, '/path/to/environment/modules')
# Environment modules become available by loading python.py (the name choice could be better here)
from python import module

# ...
module('use', '/some/dir')
module('load', 'program/1.2.3')

This looks more pythonic to me and also it allows IDEs to offer auto-completion etc.

Answered By: andreee

Answer 5

I found the answers from jnewman, andreee and ian quite helpful. However, in my case the $MODULESHOME/init/python.py is written in Python 2 and is incompatible with Python 3 (it uses exec output instead of exec(output)). Here is my modified version that supports Python 3:

import os
import subprocess

def load_module(module_name, *, capture_output=True, check=True, **kw):
    module_home = os.environ["MODULESHOME"]
    modulecmd = os.path.join(module_home, "bin/modulecmd")
    process = subprocess.run(
        [modulecmd, "python", "load", module_name],
        capture_output=capture_output,
        check=check,
        **kw,
    )
    return process

Example use case:

CONDA_MODULE = "Anaconda3/2022.05"
load_module(CONDA_MODULE)
print(subprocess.check_output(["conda", "--help"], stderr=subprocess.STDOUT).decode("utf8"))

Answered By: Silverstone

Loading environment modules within a python script

Question:

Answers: