Unload a module in Python

(You should try to write more concise questions; I’ve only read the beginning and skimmed the rest.) I see a simple problem at the start:

sm = sys.modules.copy()

You made a copy of sys.modules, so now your copy has a reference to the module–so of course it won’t be collected. You can see what’s referring to it with gc.get_referrers.

This works fine:

# module1.py
class test(object):
    def __del__(self):
        print "unloaded module1"
a = test()

print "loaded module1"

.

# testing.py
def run():
    print "importing module1"
    import module1
    print "finished importing module1"

def main():
    run()
    import sys
    del sys.modules["module1"]
    print "finished"

if __name__ == '__main__':
    main()

module1 is unloaded as soon as we remove it from sys.modules, because there are no remaining references to the module. (Doing module1 = None after importing would work, too–I just put the import in another function for clarity. All you have to do is drop your references to it.)

Now, it’s a little tricky to do this in practice, because of two issues:

• In order to collect a module, all references to the module must be unreachable (as with collecting any object). That means that any other modules that imported it need to be dereferenced and reloaded, too.
• If you remove a module from sys.modules when it’s still referenced somewhere else, you’ve created an unusual situation: the module is still loaded and used by code, but the module loader doesn’t know about it anymore. The next time you import the module, you won’t get a reference to the existing one (since you deleted the record of that), so it’ll load a second coexisting copy of the module. This can cause serious consistency problems. So, make sure that there are no remaining references to the module before finally removing it from sys.modules.

There are some tricky problems to use this in general: detecting which modules depend on the module you’re unloading; knowing whether it’s okay to unload those too (depends heavily on your use case); handling threading while examining all this (take a look at imp.acquire_lock), and so on.

I could contrive a case where doing this might be useful, but most of the time I’d recommend just restarting the app if its code changes. You’ll probably just give yourself headaches.

I’m not sure about Python, but in other languages, calling the equivalent of gc.collect() does not release unused memory – it will only release that memory if/when the memory is actually needed.

Otherwise, it makes sense for Python to keep the modules in memory for the time being, in case they need to be loaded again.

Python's small object manager rarely returns memory back to the Operating System. From here and here. So, stricktly speaking, python has (by design) a kind of memory leak, even when objects are “gc collected”.

I can not find an authoritative perspective on this in python3 (10 years later) (now python3.8). However, we can do better percentage-wise now.

import gc
import sys

the_objs = gc.get_objects()
print(len(gc.get_objects())) # 5754 objects in memory
origin_modules = set(sys.modules.keys())
import http.client # it was renamed ;)

print(len(gc.get_objects())) # 9564 objects in memory
for new_mod in set(sys.modules.keys()) - origin_modules:
    del sys.modules[new_mod]
    try:
        del globals()[new_mod]
    except KeyError:
        pass
    try:
        del locals()[new_mod]
    except KeyError:
        pass
del origin_modules
# importlib.invalidate_caches()  happens to not do anything
gc.collect()
print(len(gc.get_objects())) # 6528 objects in memory 

only increasing 13%. If you look at the kind of objects that get loaded in the new gc.get_objects, some of them are builtins, source code, random.* utilities, datetime utilities, etc. I am mostly leaving this here as an update to start the conversation for @shuttle and will delete if we can make more progress.