Why program functionally in Python?

I program in Python everyday, and I have to say that too much ‘bandwagoning’ toward OO or functional could lead toward missing elegant solutions. I believe that both paradigms have their advantages to certain problems – and I think that’s when you know what approach to use. Use a functional approach when it leaves you with a clean, readable, and efficient solution. Same goes for OO.

And that’s one of the reasons I love Python – the fact that it is multi-paradigm and lets the developer choose how to solve his/her problem.

The standard functions filter(), map() and reduce() are used for various operations on a list and all of the three functions expect two arguments: A function and a list

We could define a separate function and use it as an argument to filter() etc., and its probably a good idea if that function is used several times, or if the function is too complex to be written in a single line. However, if it’s needed only once and it’s quite simple, it’s more convenient to use a lambda construct to generate a (temporary) anonymous function and pass it to filter().

This helps in readability and compact code.

Using these function, would also turn out to be efficient, because the looping on the elements of the list is done in C, which is a little bit faster than looping in python.

And object oriented way is forcibly needed when states are to be maintained, apart from abstraction, grouping, etc., If the requirement is pretty simple, I would stick with functional than to Object Oriented programming.

This answer is completely re-worked. It incorporates a lot of observations from the other answers.

As you can see, there is a lot of strong feelings surrounding the use of functional programming constructs in Python. There are three major groups of ideas here.

First, almost everybody but the people who are most wedded to the purest expression of the functional paradigm agree that list and generator comprehensions are better and clearer than using map or filter. Your colleagues should be avoiding the use of map and filter if you are targeting a version of Python new enough to support list comprehensions. And you should be avoiding itertools.imap and itertools.ifilter if your version of Python is new enough for generator comprehensions.

Secondly, there is a lot of ambivalence in the community as a whole about lambda. A lot of people are really annoyed by a syntax in addition to def for declaring functions, especially one that involves a keyword like lambda that has a rather strange name. And people are also annoyed that these small anonymous functions are missing any of the nice meta-data that describes any other kind of function. It makes debugging harder. Lastly the small functions declared by lambda are often not terribly efficient as they require the overhead of a Python function call each time they are invoked, which is frequently in an inner loop.

Lastly, most (meaning > 50%, but most likely not 90%) people think that reduce is a little strange and obscure. I myself admit to having print reduce.__doc__ whenever I want to use it, which isn’t all that often. Though when I see it used, the nature of the arguments (i.e. function, list or iterator, scalar) speak for themselves.

As for myself, I fall in the camp of people who think the functional style is often very useful. But balancing that thought is the fact that Python is not at heart a functional language. And overuse of functional constructs can make programs seem strangely contorted and difficult for people to understand.

To understand when and where the functional style is very helpful and improves readability, consider this function in C++:

unsigned int factorial(unsigned int x)
{
    int fact = 1;
    for (int i = 2; i <= n; ++i) {
        fact *= i;
    }
    return fact
 }

This loop seems very simple and easy to understand. And in this case it is. But its seeming simplicity is a trap for the unwary. Consider this alternate means of writing the loop:

unsigned int factorial(unsigned int n)
{
    int fact = 1;
    for (int i = 2; i <= n; i += 2) {
        fact *= i--;
    }
     return fact;
 }

Suddenly, the loop control variable no longer varies in an obvious way. You are reduced to looking through the code and reasoning carefully about what happens with the loop control variable. Now this example is a bit pathological, but there are real-world examples that are not. And the problem is with the fact that the idea is repeated assignment to an existing variable. You can’t trust the variable’s value is the same throughout the entire body of the loop.

This is a long recognized problem, and in Python writing a loop like this is fairly unnatural. You have to use a while loop, and it just looks wrong. Instead, in Python you would write something like this:

def factorial(n):
    fact = 1
    for i in xrange(2, n):
        fact = fact * i;
    return fact

As you can see, the way you talk about the loop control variable in Python is not amenable to fooling with it inside the loop. This eliminates a lot of the problems with ‘clever’ loops in other imperative languages. Unfortunately, it’s an idea that’s semi-borrowed from functional languages.

Even this lends itself to strange fiddling. For example, this loop:

c = 1
for i in xrange(0, min(len(a), len(b))):
    c = c * (a[i] + b[i])
    if i < len(a):
        a[i + 1] = a[a + 1] + 1

Oops, we again have a loop that is difficult to understand. It superficially resembles a really simple and obvious loop, and you have to read it carefully to realize that one of the variables used in the loop’s computation is being messed with in a way that will effect future runs of the loop.

Again, a more functional approach to the rescue:

from itertools import izip
c = 1
for ai, bi in izip(a, b):
   c = c * (ai + bi)

Now by looking at the code we have some strong indication (partly by the fact that the person is using this functional style) that the lists a and b are not modified during the execution of the loop. One less thing to think about.

The last thing to be worried about is c being modified in strange ways. Perhaps it is a global variable and is being modified by some roundabout function call. To rescue us from this mental worry, here is a purely function approach:

from itertools import izip
c = reduce(lambda x, ab: x * (ab[0] + ab[1]), izip(a, b), 1)

Very concise, and the structure tells us that x is purely an accumulator. It is a local variable everywhere it appear. The final result is unambiguously assigned to c. Now there is much less to worry about. The structure of the code removes several classes of possible error.

That is why people might choose a functional style. It is concise and clear, at least if you understand what reduce and lambda do. There are large classes of problems that could afflict a program written in a more imperative style that you know won’t afflict your functional style program.

In the case of factorial, there is a very simple and clear way to write this function in Python in a functional style:

import operator
def factorial(n):
    return reduce(operator.mul, xrange(2, n+1), 1)

Map and Filter have their place in OO programming. Right next to list comprehensions and generator functions.

Reduce less so. The algorithm for reduce can rapidly suck down more time than it deserves; with a tiny bit of thinking, a manually-written reduce-loop will be more efficient than a reduce which applies a poorly-thought-out looping function to a sequence.

Lambda never. Lambda is useless. One can make the argument that it actually does something, so it’s not completely useless. First: Lambda is not syntactic “sugar”; it makes things bigger and uglier. Second: the one time in 10,000 lines of code that think you need an “anonymous” function turns into two times in 20,000 lines of code, which removes the value of anonymity, making it into a maintenance liability.

However.

The functional style of no-object-state-change programming is still OO in nature. You just do more object creation and fewer object updates. Once you start using generator functions, much OO programming drifts in a functional direction.

Each state change appears to translate into a generator function that builds a new object in the new state from old object(s). It’s an interesting world view because reasoning about the algorithm is much, much simpler.

But that’s no call to use reduce or lambda.

FP is important not only for concurrency; in fact, there’s virtually no concurrency in the canonical Python implementation (maybe 3.x changes that?). in any case, FP lends itself well to concurrency because it leads to programs with no or fewer (explicit) states. states are troublesome for a few reasons. one is that they make distributing the computation hard(er) (that’s the concurrency argument), another, far more important in most cases, is the tendency to inflict bugs. the biggest source of bugs in contemporary software is variables (there’s a close relationship between variables and states). FP may reduce the number of variables in a program: bugs squashed!

see how many bugs can you introduce by mixing the variables up in these versions:

def imperative(seq):
    p = 1
    for x in seq:
        p *= x
    return p

versus (warning, my.reduce‘s parameter list differs from that of python’s reduce; rationale given later)

import operator as ops

def functional(seq):
    return my.reduce(ops.mul, 1, seq)

as you can see, it’s a matter of fact that FP gives you fewer opportunities to shoot yourself in the foot with a variables-related bug.

also, readability: it may take a bit of training, but functional is way easier to read than imperative: you see reduce (“ok, it’s reducing a sequence to a single value”), mul (“by multiplication”). wherease imperative has the generic form of a for cycle, peppered with variables and assignments. these for cycles all look the same, so to get an idea of what’s going on in imperative, you need to read almost all of it.

then there’s succintness and flexibility. you give me imperative and I tell you I like it, but want something to sum sequences as well. no problem, you say, and off you go, copy-pasting:

def imperative(seq):
    p = 1
    for x in seq:
        p *= x
    return p

def imperative2(seq):
    p = 0
    for x in seq:
        p += x
    return p

what can you do to reduce the duplication? well, if operators were values, you could do something like

def reduce(op, seq, init):
    rv = init
    for x in seq:
        rv = op(rv, x)
    return rv

def imperative(seq):
    return reduce(*, 1, seq)

def imperative2(seq):
    return reduce(+, 0, seq)

oh wait! operators provides operators that are values! but.. Alex Martelli condemned reduce already… looks like if you want to stay within the boundaries he suggests, you’re doomed to copy-pasting plumbing code.

is the FP version any better? surely you’d need to copy-paste as well?

import operator as ops

def functional(seq):
    return my.reduce(ops.mul, 1, seq)

def functional2(seq):
    return my.reduce(ops.add, 0, seq)

well, that’s just an artifact of the half-assed approach! abandoning the imperative def, you can contract both versions to

import functools as func, operator as ops

functional  = func.partial(my.reduce, ops.mul, 1)
functional2 = func.partial(my.reduce, ops.add, 0)

or even

import functools as func, operator as ops

reducer = func.partial(func.partial, my.reduce)
functional  = reducer(ops.mul, 1)
functional2 = reducer(ops.add, 0)

(func.partial is the reason for my.reduce)

what about runtime speed? yes, using FP in a language like Python will incur some overhead. here i’ll just parrot what a few professors have to say about this:

• premature optimization is the root of all evil.
• most programs spend 80% of their runtime in 20% percent of their code.
• profile, don’t speculate!

I’m not very good at explaining things. Don’t let me muddy the water too much, read the first half of the speech John Backus gave on the occasion of receiving the Turing Award in 1977. Quote:

5.1 A von Neumann Program for Inner Product

c := 0
for i := I step 1 until n do
   c := c + a[i] * b[i]

Several properties of this program are
worth noting:

1. Its statements operate on an invisible “state” according to complex
   rules.
2. It is not hierarchical. Except for the right side of the assignment
   statement, it does not construct
   complex entities from simpler ones.
   (Larger programs, however, often do.)
3. It is dynamic and repetitive. One must mentally execute it to
   understand it.
4. It computes word-at-a-time by repetition (of the assignment) and by
   modification (of variable i).
5. Part of the data, n, is in the program; thus it lacks generality and
   works only for vectors of length n.
6. It names its arguments; it can only be used for vectors a and b.
   To become general, it requires a
   procedure declaration. These involve
   complex issues (e.g., call-by-name
   versus call-by-value).
7. Its “housekeeping” operations are represented by symbols in
   scattered places (in the for statement
   and the subscripts in the assignment).
   This makes it impossible to
   consolidate housekeeping operations,
   the most common of all, into single,
   powerful, widely useful operators.
   Thus in programming those operations
   one must always start again at square
   one, writing “for i := ...” and
   “for j := ...” followed by
   assignment statements sprinkled with
   i‘s and j‘s.

Here’s a short summary of positive answers when/why to program functionally.

• List comprehensions were imported from Haskell, a FP language. They are Pythonic. I’d prefer to write

y = [i*2 for i in k if i % 3 == 0]

than to use an imperative construct (loop).

• I’d use lambda when giving a complicated key to sort, like list.sort(key=lambda x: x.value.estimate())

• It’s cleaner to use higher-order functions than to write code using OOP’s design patterns like visitor or abstract factory

• People say that you should program Python in Python, C++ in C++ etc. That’s true, but certainly you should be able to think in different ways at the same thing. If while writing a loop you know that you’re really doing reducing (folding), then you’ll be able to think on a higher level. That cleans your mind and helps to organize. Of course lower-level thinking is important too.

You should NOT overuse those features – there are many traps, see Alex Martelli’s post. I’d subjectively say the most serious danger is that excessive use of those features will destroy readability of your code, which is a core attribute of Python.

The question, which seems to be mostly ignored here:

does programming Python functionally really help with concurrency?

No. The value FP brings to concurrency is in eliminating state in computation, which is ultimately responsible for the hard-to-grasp nastiness of unintended errors in concurrent computation. But it depends on the concurrent programming idioms not themselves being stateful, something that doesn’t apply to Twisted. If there are concurrency idioms for Python that leverage stateless programming, I don’t know of them.