How to make a copy of a python module at runtime?

Question

I need to make a copy of a socket module to be able to use it and to have one more socket module monkey-patched and use it differently.

Is this possible?

I mean to really copy a module, namely to get the same result at runtime as if I’ve copied socketmodule.c, changed the initsocket() function to initmy_socket(), and installed it as my_socket extension.

Asked By: Dmitry Trofimov

||

Source

Answer 1

Physically copy the socket module to socket_monkey and go from there? I don’t feel you need any “clever” work-around… but I might well be over simplifying!

Answered By: Jon Clements

Answer 2

Here’s some code that creates a new module with the functions and variables of the old:

def copymodule(old):
    new = type(old)(old.__name__, old.__doc__)
    new.__dict__.update(old.__dict__)
    return new

Note that this does a fairly shallow copy of the module. The dictionary is newly created, so basic monkey patching will work, but any mutables in the original module will be shared between the two.

Edit: According to the comment, a deep copy is needed. I tried messing around with monkey-patching the copy module to support deep copies of modules, but that didn’t work. Next I tried importing the module twice, but since modules are cached in sys.modules, that gave me the same module twice. Finally, the solution I hit upon was removing the modules from sys.modules after importing it the first time, then importing it again.

from imp import find_module, load_module
from sys import modules

def loadtwice(name, path=None):
    """Import two copies of a module.

    The name and path arguments are as for `find_module` in the `imp` module.
    Note that future imports of the module will return the same object as
    the second of the two returned by this function.
    """
    startingmods = modules.copy()
    foundmod = find_module(name, path)
    mod1 = load_module(name, *foundmod)
    newmods = set(modules) - set(startingmods)
    for m in newmods:
        del modules[m]
    mod2 = load_module(name, *foundmod)
    return mod1, mod2

Answered By: Abe Karplus

Answer 3

This is pretty disgusting, but this might suffice:

import sys

# if socket was already imported, get rid of it and save a copy
save = sys.modules.pop('socket', None)

# import socket again (it's not in sys.modules, so it will be reimported)
import socket as mysock

if save is None:
    # if we didn't have a saved copy, remove my version of 'socket'
    del sys.modules['socket']
else:
    # if we did have a saved copy overwrite my socket with the original
    sys.modules['socket'] = save

Answered By: Nathan Binkert

Answer 4

You can always do tricks like importing a module then deleting it from sys.modules or trying to copy a module. However, Python already provides what you want in its Standard Library.

import imp # Standard module to do such things you want to.

# We can import any module including standard ones:
os1=imp.load_module('os1', *imp.find_module('os'))

# Here is another one:
os2=imp.load_module('os2', *imp.find_module('os'))

# This returns True:
id(os1)!=id(os2)

Python3.3+

imp.load_module is deprecated in python3.3+, and recommends the use of importlib

#!/usr/bin/env python3

import sys
import importlib.util

SPEC_OS = importlib.util.find_spec('os')
os1 = importlib.util.module_from_spec(SPEC_OS)
SPEC_OS.loader.exec_module(os1)
sys.modules['os1'] = os1

os2 = importlib.util.module_from_spec(SPEC_OS)
SPEC_OS.loader.exec_module(os2)
sys.modules['os2'] = os2
del SPEC_OS

assert os1 is not os2, 
    "Module `os` instancing failed"

Here, we import the same module twice but as completely different module objects. If you check sys.modules, you can see two names you entered as first parameters to load_module calls. Take a look at the documentation for details.

UPDATE:

To make the main difference of this approach obvious, I want to make this clearer: When you import the same module this way, you will have both versions globally accessible for every other module you import in runtime, which is exactly what the questioner needs as I understood.

Below is another example to emphasize this point.

These two statements do exactly the same thing:

import my_socket_module as socket_imported

socket_imported = imp.load_module('my_socket_module',
    *imp.find_module('my_socket_module')
)

On second line, we repeat ‘my_socket_module’ string twice and that is how import statement works; but these two strings are, in fact, used for two different reasons.

Second occurrence as we passed it to find_module is used as the file name that will be found on the system. The first occurrence of the string as we passed it to load_module method is used as system-wide identifier of the loaded module.

So, we can use different names for these which means we can make it work exactly like we copied the python source file for the module and loaded it.

socket = imp.load_module('socket_original', *imp.find_module('my_socket_module'))
socket_monkey = imp.load_module('socket_patched',*imp.find_module('my_socket_module'))

def alternative_implementation(blah, blah):
    return 'Happiness'

socket_monkey.original_function = alternative_implementation

import my_sub_module

Then in my_sub_module, I can import ‘socket_patched’ which does not exist on system! Here we are in my_sub_module.py.

import socket_patched
socket_patched.original_function('foo', 'bar')
# This call brings us 'Happiness'

Answered By: hasanyasin

How to make a copy of a python module at runtime?

Question:

Answers:

Python3.3+