How can I check the memory usage of objects in iPython?

Question

I am using iPython to run my code. I wonder if there is any module or command which would allow me to check the memory usage of an object. For instance:

In [1]: a = range(10000)
In [2]: %memusage a
Out[2]: 1MB

Something like %memusage <object> and return the memory used by the object.

Duplicate

Find out how much memory is being used by an object in Python

Asked By: Ross

||

Source

Answer 1

UPDATE: Here is another, maybe more thorough recipe for estimating the size of a python object.

Here is a thread addressing a similar question

The solution proposed is to write your own… using some estimates of the known size of primitives, python’s object overhead, and the sizes of built in container types.

Since the code is not that long, here is a direct copy of it:

def sizeof(obj):
    """APPROXIMATE memory taken by some Python objects in 
    the current 32-bit CPython implementation.

    Excludes the space used by items in containers; does not
    take into account overhead of memory allocation from the
    operating system, or over-allocation by lists and dicts.
    """
    T = type(obj)
    if T is int:
        kind = "fixed"
        container = False
        size = 4
    elif T is list or T is tuple:
        kind = "variable"
        container = True
        size = 4*len(obj)
    elif T is dict:
        kind = "variable"
        container = True
        size = 144
        if len(obj) > 8:
            size += 12*(len(obj)-8)
    elif T is str:
        kind = "variable"
        container = False
        size = len(obj) + 1
    else:
        raise TypeError("don't know about this kind of object")
    if kind == "fixed":
        overhead = 8
    else: # "variable"
        overhead = 12
    if container:
        garbage_collector = 8
    else:
        garbage_collector = 0
    malloc = 8 # in most cases
    size = size + overhead + garbage_collector + malloc
    # Round to nearest multiple of 8 bytes
    x = size % 8
    if x != 0:
        size += 8-x
        size = (size + 8)
    return size

Answered By: John Mulder

Answer 2

Unfortunately this is not possible, but there are a number of ways of approximating the answer:

for very simple objects (e.g. ints, strings, floats, doubles) which are represented more or less as simple C-language types you can simply calculate the number of bytes as with John Mulder’s solution.
For more complex objects a good approximation is to serialize the object to a string using cPickle.dumps. The length of the string is a good approximation of the amount of memory required to store an object.

There is one big snag with solution 2, which is that objects usually contain references to other objects. For example a dict contains string-keys and other objects as values. Those other objects might be shared. Since pickle always tries to do a complete serialization of the object it will always over-estimate the amount of memory required to store an object.

Answered By: Salim Fadhley

Answer 3

If you are using a numpy array, then you can use the attribute ndarray.nbytes to evaluate its size in memory:

from pylab import *   
d = array([2,3,4,5])   
d.nbytes
#Output: 32

Answered By: HoWil

Answer 4

I was trying to figure out how to do just that for myself. I tried several solutions on this page and other pages. I then did some search and came across https://ipython-books.github.io/44-profiling-the-memory-usage-of-your-code-with-memory_profiler/ which seems to give an alternative solution. The gist of the solution: Use %mprun in ipython.

First, install memory_profiler: pip install memory_profiler
Start ipython and load memory_profiler: %load_ext memory_profiler
Create a function in a physical file, say, myfunc.py (important: %mprun can only be used on functions defined in physical files). Create the object in question in a function, e.g.:

# myfunc.py
def myfunc():
    # create the object, e.g.
    a = [*range(10000)]

Run

from myfunc import myfunc
%mprun -T mprof -f myfunc myfunc()

which generates the file mprof. The content is also displayed:

Line #    Mem usage    Increment   Line Contents
================================================
     1     49.1 MiB     49.1 MiB   def myfunc():
     2                                 # create the object, e.g.
     3     49.4 MiB      0.3 MiB       a = [*range(10000)]

From the inrement in line 3, we know the memory used by a is 0.3 MiB.

Lets try a = [*range(100000)]:

# myfunc1.py
def myfunc1():
    # create the object, e.g.
    a = [*range(100000)]

Run

from myfunc1 import myfunc1
%mprun -T mprof1 -f myfunc1 myfunc1()

Line #    Mem usage    Increment   Line Contents
================================================
     1     49.2 MiB     49.2 MiB   def myfunc1():
     2                                 # create the object, e.g.
     3     52.3 MiB      3.0 MiB       a = [*range(100000)]

seems to be in line with our expectation.

Answered By: mikey

How can I check the memory usage of objects in iPython?

Question:

Answers: