Limit RAM usage to python program

Question

I’m trying to limit the RAM usage from a Python program to half so it doesn’t totally freezes when all the RAM is used, for this I’m using the following code which is not working and my laptop is still freezing:

import sys
import resource

def memory_limit():
    rsrc = resource.RLIMIT_DATA
    soft, hard = resource.getrlimit(rsrc)
    soft /= 2
    resource.setrlimit(rsrc, (soft, hard))

if __name__ == '__main__':
    memory_limit() # Limitates maximun memory usage to half
    try:
        main()
    except MemoryError:
        sys.stderr.write('MAXIMUM MEMORY EXCEEDED')
        sys.exit(-1)

I’m using other functions which I call from the main function.

What am I doing wrong?

Thanks in advance.

PD: I already searched about this and found the code I’ve put but it’s still not working…

Asked By: Ulises CT

||

Source

Answer 1

I’ve done some research and found a function to get the memory from Linux systems here: Determine free RAM in Python and I modified it a bit to set the memory hard limit to half of the free memory available.

import resource
import sys

def memory_limit():
    """Limit max memory usage to half."""
    soft, hard = resource.getrlimit(resource.RLIMIT_AS)
    # Convert KiB to bytes, and divide in two to half
    resource.setrlimit(resource.RLIMIT_AS, (get_memory() * 1024 / 2, hard))

def get_memory():
    with open('/proc/meminfo', 'r') as mem:
        free_memory = 0
        for i in mem:
            sline = i.split()
            if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
                free_memory += int(sline[1])
    return free_memory  # KiB

if __name__ == '__main__':
    memory_limit_half()
    try:
        main()
    except MemoryError:
        sys.stderr.write('nnERROR: Memory Exceptionn')
        sys.exit(1)

Answered By: Ulises CT

Answer 2

due to https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773

it is better to use :

if str(sline[0]) == 'MemAvailable:':
            free_memory = int(sline[1])
            break

the answer code provided me with 8 GB of available mem on a machine with 1TB of RAM

Answered By: The_Pingu

Answer 3

I modify the answer of @Ulises CT. Because I think to change too much original function is not so good, so I turn it to a decorator. I hope it helps.

import resource
import platform
import sys

def memory_limit(percentage: float):
    """
    只在linux操作系统起作用
    """
    if platform.system() != "Linux":
        print('Only works on linux!')
        return
    soft, hard = resource.getrlimit(resource.RLIMIT_AS)
    resource.setrlimit(resource.RLIMIT_AS, (get_memory() * 1024 * percentage, hard))

def get_memory():
    with open('/proc/meminfo', 'r') as mem:
        free_memory = 0
        for i in mem:
            sline = i.split()
            if str(sline[0]) in ('MemFree:', 'Buffers:', 'Cached:'):
                free_memory += int(sline[1])
    return free_memory

def memory(percentage=0.8):
    def decorator(function):
        def wrapper(*args, **kwargs):
            memory_limit(percentage)
            try:
                return function(*args, **kwargs)
            except MemoryError:
                mem = get_memory() / 1024 /1024
                print('Remain: %.2f GB' % mem)
                sys.stderr.write('nnERROR: Memory Exceptionn')
                sys.exit(1)
        return wrapper
    return decorator

@memory(percentage=0.8)
def main():
    print('My memory is limited to 80%.')

Answered By: qiuyuan

Limit RAM usage to python program

Question:

Answers: