Are Python inner functions compiled?

Question

So AFAIK in CPython, function definitions are compiled into function objects when executed at parse time. But what about inner functions? Do they get compiled into function objects at parse time or do they get compiled (or interpreted) every single time the function is called? Do inner functions incur any performance penalty at all?

Asked By: Y.H Wong

||

Source

Answer 1

>>> import dis
>>> def foo():
...     def bar():
...             print "stuff"
...     return bar
... 
>>> b = foo()
>>> dis.dis(foo)
  2           0 LOAD_CONST               1 (<code object bar at 0x20bf738, file "<stdin>", line 2>)
              3 MAKE_FUNCTION            0
              6 STORE_FAST               0 (bar)

  4           9 LOAD_FAST                0 (bar)
             12 RETURN_VALUE        
>>> dis.dis(b)
  3           0 LOAD_CONST               1 ('stuff')
              3 PRINT_ITEM          
              4 PRINT_NEWLINE       
              5 LOAD_CONST               0 (None)
              8 RETURN_VALUE

I suspect this is heavily implementation dependent, but that was CPython 2.6.6, and the inner function looks like it was compiled. Here’s another example:

>>> def foo():
...     def bar():
...             return 1
...     return dis.dis(bar)
... 
>>> foo()
  3           0 LOAD_CONST               1 (1)
              3 RETURN_VALUE

So we can conclude that they are compiled. As for their performance characteristics, use them. If you start having performance issues, profile. I know it’s not really an answer, but it almost never matters and when it does, general answers don’t cut it. Function calls incur some overhead and it looks like inner functions are just like functions.

Answered By: nmichaels

Answer 2

Easy test: the default arguments to a function are called once, at define time.

>>> def foo():
...     def bar(arg=count()):
...             pass
...     pass
...
>>> def count():
...     print "defined"
...
>>> foo()
defined
>>> foo()
defined

So yes: this is a minor (very very! minor) performance hit.

Answered By: Katriel

Answer 3

To extend nmichaels answer inner function are compiled in compile time as he guessed and there byte code is saved in the foo.func_code.co_consts and they are accessed using the opcode LOAD_CONST as you can see in the disassembly of the function.

Example:

>>> def foo():
...     def inner():
...         pass
>>> print foo.func_code.co_consts
(None, <code object inner at 0x249c6c0, file "<ipython console>", line 2>)

Answered By: mouad

Answer 4

To give a general explaination – assuming you have the following code in a module:

def outer(x=1):
    def inner(y=2):
        return x+y

When the file is parsed by python via compile(), the above text is turned into bytecode for how to execute the module. In the module bytecode, there are two “code objects”, one for the bytecode of outer() and one for the bytecode inner(). Note that I said code objects, not functions – the code objects contain little more than the bytecode used by the function, and any information that could be known at compile time – such as the bytecode for outer() containing a ref to the bytecode for inner().

When the module actually loads, by evaluating the code object associated with the module, one thing which happens is an actual “function object” is created for outer(), and stored in the module’s outer attribute. The function object acts as a collection of the bytecode and all context-relavant things that are needed to call the function (eg which globals dict it should pull from, etc) that can’t be known at compile time. In a way, a code object is a template for a function, which is a template for execution of the actual bytecode with all variables filled in.

None of this involved inner()-as-a-function yet – Each time you actually get around to calling outer(), that’s when a new inner() function object is created for that invocation of outer, which binds the already-created inner bytecode object to a list of globals, including the value of x as passed into that call to outer. As you can imagine, this is pretty fast, since no parsing is needed, just filling in a quick struct with some pointers to other already-existing objects.

Answered By: Eli Collins

Answer 5

I’m late on this, but as a little experimental complement to these thorough answers: you may use the builtin function id to verify whether a new object is created or not:

In []: # inner version
       def foo():
           def bar():
               return id(bar)
           return bar()

       foo(), foo()

Out[]: (4352951432, 4352952752)

The actual numbers may differ, but their difference indicates that two distinct instances of bar are indeed created.

In []: # outer version
       def bar():
           return id(bar)

       def foo():
           return bar()

       foo(), foo()

Out[]: (4352950952, 4352950952)

This time, as expected, the two ids are the same.

Now for some timeit measurements. Inner first, outer second:

100000 loops, best of 3: 1.93 µs per loop
1000000 loops, best of 3: 1.25 µs per loop

So, on my machine, it seems that the inner version is 50% slower (Python 2.7, IPython Notebook).

Answered By: Aristide

Are Python inner functions compiled?

Question:

Answers: