Is Python's "with" monadic?
Question:
Like many a foolhardy pioneer before me, I’m endeavoring to cross the trackless wasteland that is Understanding Monads.
I’m still staggering through, but I can’t help noticing a certain monad-like quality about Python’s with statement. Consider this fragment:
with open(input_filename, 'r') as f:
for line in f:
process(line)
Consider the open() call as the “unit” and the block itself as the “bind”. The actual monad isn’t exposed (uh, unless f is the monad), but the pattern is there. Isn’t it? Or am I just mistaking all of FP for monadry? Or is it just 3 in the morning and anything seems plausible?
A related question: if we have monads, do we need exceptions?
In the above fragment, any failure in the I/O can be hidden from the code. Disk corruption, the absence of the named file, and an empty file can all be treated the same. So no need for a visible IO Exception.
Certainly, Scala’s Option typeclass has eliminated the dreaded Null Pointer Exception. If you rethought numbers as Monads (with NaN and DivideByZero as the special cases)…
Like I said, 3 in the morning.
Answers:
Yes.
Right below the definition, Wikipedia says:
In object-oriented programming terms, the type construction would correspond to the declaration of the monadic type, the unit function takes the role of a constructor method, and the binding operation contains the logic necessary to execute its registered callbacks (the monadic functions).
This sounds to me exactly like the context manager protocol, the implementation of the context manager protocol by the object, and the with statement.
From @Owen in a comment on this post:
Monads, at their most basic level, are more or less a cool way to use continuation-passing style: >>= takes a “producer” and a “callback”; this is also basically what with is: a producer like open(…) and a block of code to be called once it’s created.
The full Wikipedia definition:
A type construction that defines, for every underlying type, how to obtain a corresponding monadic type. In Haskell’s notation, the name of the monad represents the type constructor. If M is the name of the monad and t is a data type, then “M t” is the corresponding type in the monad.
This sounds like the context manager protocol to me.
A unit function that maps a value in an underlying type to a value in the corresponding monadic type. The result is the “simplest” value in the corresponding type that completely preserves the original value (simplicity being understood appropriately to the monad). In Haskell, this function is called return due to the way it is used in the do-notation described later. The unit function has the polymorphic type t→M t.
The actual implementation of the context manager protocol by the object.
A binding operation of polymorphic type (M t)→(t→M u)→(M u), which Haskell represents by the infix operator >>=. Its first argument is a value in a monadic type, its second argument is a function that maps from the underlying type of the first argument to another monadic type, and its result is in that other monadic type.
This corresponds to the with statement and its suite.
So yes, I’d say with is a monad. I searched PEP 343 and all the related rejected and withdrawn PEPs, and none of them mentioned the word “monad”. It certainly applies, but it seems the goal of the with statement was resource management, and a monad is just a useful way to get it.
It’s almost too trivial to mention, but the first problem is that with isn’t a function and doesn’t take a function as an argument. You can easily get around this by writing a function wrapper for with:
def withf(context, f):
with context as x:
f(x)
Since this is so trivial, you could not bother to distinguish withf and with.
The second problem with with being a monad is that, as a statement rather than an expression, it doesn’t have a value. If you could give it a type, it would be M a -> (a -> None) -> None (this is actually the type of withf above). Speaking practically, you can use Python’s _ to get a value for the with statement. In Python 3.1:
class DoNothing (object):
def __init__(self, other):
self.other = other
def __enter__(self):
print("enter")
return self.other
def __exit__(self, type, value, traceback):
print("exit %s %s" % (type, value))
with DoNothing([1,2,3]) as l:
len(l)
print(_ + 1)
Since withf uses a function rather than a code block, an alternative to _ is to return the value of the function:
def withf(context, f):
with context as x:
return f(x)
There is another thing preventing with (and withf) from being a monadic bind. The value of the block would have to be a monadic type with the same type constructor as the with item. As it is, with is more generic. Considering agf’s note that every interface is a type constructor, I peg the type of with as M a -> (a -> b) -> b, where M is the context manager interface (the __enter__ and __exit__ methods). In between the types of bind and with is the type M a -> (a -> N b) -> N b. To be a monad, with would have to fail at runtime when b wasn’t M a. Moreover, while you could use with monadically as a bind operation, it would rarely make sense to do so.
The reason you need to make these subtle distinctions is that if you mistakenly consider with to be monadic, you’ll wind up misusing it and writing programs that will fail due to type errors. In other words, you’ll write garbage. What you need to do is distinguish a construct that is a particular thing (e.g. a monad) from one that can be used in the manner of that thing (e.g. again, a monad). The latter requires discipline on the part of a programmer, or the definition of additional constructs to enforce the discipline. Here’s a nearly monadic version of with (the type is M a -> (a -> b) -> M b):
def withm(context, f):
with context as x:
return type(context)(f(x))
In the final analysis, you could consider with to be like a combinator, but a more general one than the combinator required by monads (which is bind). There can be more functions using monads than the two required (the list monad also has cons, append and length, for example), so if you defined the appropriate bind operator for context managers (such as withm) then with could be monadic in the sense of involving monads.
I have thought unnecessarily long about this and I believe the answer is “yes, when it’s used a certain way” (thanks outis :),
but not for the reason I thought before.
I mentioned in a comment to agf’s answer, that >>= is just continuation passing style —
give it a producer and a callback and it “runs” the producer and feeds it to the
callback. But that’s not quite true. Also important is that >>= has to run
some interaction between the producer and the result of the callback.
In the case of the List monad, this would be concatenating lists. This
interaction is what makes monads special.
But I believe that Python’s with does do this interaction, just not in the
way you might expect.
Here’s an example python program employing two with statements:
class A:
def __enter__(self):
print 'Enter A'
def __exit__(self, *stuff):
print 'Exit A'
class B:
def __enter__(self):
print 'Enter B'
def __exit__(self, *stuff):
print 'Exit B'
def foo(a):
with B() as b:
print 'Inside'
def bar():
with A() as a:
foo(a)
bar()
When run the output is:
Enter A
Enter B
Inside
Exit B
Exit A
Now, Python is an imperative language, so instead of merely producing data, it
produces side-effects. But you can think of those side-effects as being data
(like IO ()) — you can’t combine them in all the cool ways you could combine
IO (), but they’re getting at the same goal.
So what you should focus on is the sequencing of those operations — that is,
the order of the print statements.
Now compare the same program in Haskell:
data Context a = Context [String] a [String]
deriving (Show)
a = Context ["Enter A"] () ["Exit A"]
b = Context ["Enter B"] () ["Exit B"]
instance Monad Context where
return x = Context [] x []
(Context x1 p y1) >>= f =
let
Context x2 q y2 = f p
in
Context (x1 ++ x2) q (y2 ++ y1)
foo :: a -> Context String
foo _ = b >> (return "Inside")
bar :: () -> Context String
bar () = a >>= foo
main = do
print $ bar ()
Which produces:
Context ["Enter A","Enter B"] "Inside" ["Exit B","Exit A"]
And the ordering is the same.
The analogy between the two programs is very direct: a Context has some
“entering” bits, a “body”, and some “exiting” bits. I used String instead of
IO actions because it’s easier — I think it should be similar with IO actions (correct me if it’s not).
And >>= for Context does exactly what with in Python does: it runs the
entering statements, feeds the value to the body, and runs the exiting
statements.
(There’s another huge difference which is that the body should depend on the
entering statements. Again I think that should be fixible).
Haskell has an equivalent of with for files, it’s called withFile. This:
with open("file1", "w") as f:
with open("file2", "r") as g:
k = g.readline()
f.write(k)
is equivalent to:
withFile "file1" WriteMode $ f ->
withFile "file2" ReadMode $ g ->
do k <- hGetLine g
hPutStr f k
Now, withFile might look like something monadic. Its type is:
withFile :: FilePath -> IOMode -> (Handle -> IO r) -> IO r
right side looks like (a -> m b) -> m b.
Another similarity: In Python you can skip as, and in Haskell you can use >> instead of >>= (or, a do block without <- arrow).
So I’ll answer this question: is withFile monadic?
You could think that it can be written like this:
do f <- withFile "file1" WriteMode
g <- withFile "file2" ReadMode
k <- hGetLine g
hPutStr f k
But this doesn’t type check. And it cannot.
It’s because in Haskell the IO monad is sequential: if you write
do x <- a
y <- b
c
after a is executed, b is executed and then c. There is no “backtrack”
to clean a at the end or something like that. withFile, on the other hand,
has to close the handle after the block is executed.
There is another monad, called continuation monad, which allows to do such
things. However, you have now two monads, IO and continuations, and using effects of two monads at once requires using monad transformers.
import System.IO
import Control.Monad.Cont
k :: ContT r IO ()
k = do f <- ContT $ withFile "file1" WriteMode
g <- ContT $ withFile "file2" ReadMode
lift $ hGetLine g >>= hPutStr f
main = runContT k return
That’s ugly. So the answer is: somewhat, but that requires dealing with a lot of subtleties that make the issue rather opaque.
Python’s with can simulate only a limited bit of what monads can do – add entering and finalization code. I don’t think you can simulate e.g.
do x <- [2,3,4]
y <- [0,1]
return (x+y)
using with (it might be possible with some dirty hacks). Instead, use for:
for x in [2,3,4]:
for y in [0,1]:
print x+y
And there’s a Haskell function for this – forM:
forM [2,3,4] $ x ->
forM [0,1] $ y ->
print (x+y)
I recommed reading about yield which bears more resemblance to monads than with:
http://www.valuedlessons.com/2008/01/monads-in-python-with-nice-syntax.html
A related question: if we have monads, do we need exceptions?
Basically no, instead of a function that throws A or returns B you can make a function that returns Either A B. The monad for Either A will then behave just like exceptions – if one line of code will return an error, the whole block will.
However, that would mean that division would have type Integer -> Integer -> Either Error Integer and so on, to catch division by zero. You would have to detect errors (explicitly pattern match or use bind) in any code that uses division or has even slightest possibility of going wrong. Haskell uses exceptions to avoid doing this.
Like many a foolhardy pioneer before me, I’m endeavoring to cross the trackless wasteland that is Understanding Monads.
I’m still staggering through, but I can’t help noticing a certain monad-like quality about Python’s with statement. Consider this fragment:
with open(input_filename, 'r') as f:
for line in f:
process(line)
Consider the open() call as the “unit” and the block itself as the “bind”. The actual monad isn’t exposed (uh, unless f is the monad), but the pattern is there. Isn’t it? Or am I just mistaking all of FP for monadry? Or is it just 3 in the morning and anything seems plausible?
A related question: if we have monads, do we need exceptions?
In the above fragment, any failure in the I/O can be hidden from the code. Disk corruption, the absence of the named file, and an empty file can all be treated the same. So no need for a visible IO Exception.
Certainly, Scala’s Option typeclass has eliminated the dreaded Null Pointer Exception. If you rethought numbers as Monads (with NaN and DivideByZero as the special cases)…
Like I said, 3 in the morning.
Yes.
Right below the definition, Wikipedia says:
In object-oriented programming terms, the type construction would correspond to the declaration of the monadic type, the unit function takes the role of a constructor method, and the binding operation contains the logic necessary to execute its registered callbacks (the monadic functions).
This sounds to me exactly like the context manager protocol, the implementation of the context manager protocol by the object, and the with statement.
From @Owen in a comment on this post:
Monads, at their most basic level, are more or less a cool way to use continuation-passing style: >>= takes a “producer” and a “callback”; this is also basically what with is: a producer like open(…) and a block of code to be called once it’s created.
The full Wikipedia definition:
A type construction that defines, for every underlying type, how to obtain a corresponding monadic type. In Haskell’s notation, the name of the monad represents the type constructor. If M is the name of the monad and t is a data type, then “M t” is the corresponding type in the monad.
This sounds like the context manager protocol to me.
A unit function that maps a value in an underlying type to a value in the corresponding monadic type. The result is the “simplest” value in the corresponding type that completely preserves the original value (simplicity being understood appropriately to the monad). In Haskell, this function is called return due to the way it is used in the do-notation described later. The unit function has the polymorphic type t→M t.
The actual implementation of the context manager protocol by the object.
A binding operation of polymorphic type (M t)→(t→M u)→(M u), which Haskell represents by the infix operator >>=. Its first argument is a value in a monadic type, its second argument is a function that maps from the underlying type of the first argument to another monadic type, and its result is in that other monadic type.
This corresponds to the with statement and its suite.
So yes, I’d say with is a monad. I searched PEP 343 and all the related rejected and withdrawn PEPs, and none of them mentioned the word “monad”. It certainly applies, but it seems the goal of the with statement was resource management, and a monad is just a useful way to get it.
It’s almost too trivial to mention, but the first problem is that with isn’t a function and doesn’t take a function as an argument. You can easily get around this by writing a function wrapper for with:
def withf(context, f):
with context as x:
f(x)
Since this is so trivial, you could not bother to distinguish withf and with.
The second problem with with being a monad is that, as a statement rather than an expression, it doesn’t have a value. If you could give it a type, it would be M a -> (a -> None) -> None (this is actually the type of withf above). Speaking practically, you can use Python’s _ to get a value for the with statement. In Python 3.1:
class DoNothing (object):
def __init__(self, other):
self.other = other
def __enter__(self):
print("enter")
return self.other
def __exit__(self, type, value, traceback):
print("exit %s %s" % (type, value))
with DoNothing([1,2,3]) as l:
len(l)
print(_ + 1)
Since withf uses a function rather than a code block, an alternative to _ is to return the value of the function:
def withf(context, f):
with context as x:
return f(x)
There is another thing preventing with (and withf) from being a monadic bind. The value of the block would have to be a monadic type with the same type constructor as the with item. As it is, with is more generic. Considering agf’s note that every interface is a type constructor, I peg the type of with as M a -> (a -> b) -> b, where M is the context manager interface (the __enter__ and __exit__ methods). In between the types of bind and with is the type M a -> (a -> N b) -> N b. To be a monad, with would have to fail at runtime when b wasn’t M a. Moreover, while you could use with monadically as a bind operation, it would rarely make sense to do so.
The reason you need to make these subtle distinctions is that if you mistakenly consider with to be monadic, you’ll wind up misusing it and writing programs that will fail due to type errors. In other words, you’ll write garbage. What you need to do is distinguish a construct that is a particular thing (e.g. a monad) from one that can be used in the manner of that thing (e.g. again, a monad). The latter requires discipline on the part of a programmer, or the definition of additional constructs to enforce the discipline. Here’s a nearly monadic version of with (the type is M a -> (a -> b) -> M b):
def withm(context, f):
with context as x:
return type(context)(f(x))
In the final analysis, you could consider with to be like a combinator, but a more general one than the combinator required by monads (which is bind). There can be more functions using monads than the two required (the list monad also has cons, append and length, for example), so if you defined the appropriate bind operator for context managers (such as withm) then with could be monadic in the sense of involving monads.
I have thought unnecessarily long about this and I believe the answer is “yes, when it’s used a certain way” (thanks outis :),
but not for the reason I thought before.
I mentioned in a comment to agf’s answer, that >>= is just continuation passing style —
give it a producer and a callback and it “runs” the producer and feeds it to the
callback. But that’s not quite true. Also important is that >>= has to run
some interaction between the producer and the result of the callback.
In the case of the List monad, this would be concatenating lists. This
interaction is what makes monads special.
But I believe that Python’s with does do this interaction, just not in the
way you might expect.
Here’s an example python program employing two with statements:
class A:
def __enter__(self):
print 'Enter A'
def __exit__(self, *stuff):
print 'Exit A'
class B:
def __enter__(self):
print 'Enter B'
def __exit__(self, *stuff):
print 'Exit B'
def foo(a):
with B() as b:
print 'Inside'
def bar():
with A() as a:
foo(a)
bar()
When run the output is:
Enter A
Enter B
Inside
Exit B
Exit A
Now, Python is an imperative language, so instead of merely producing data, it
produces side-effects. But you can think of those side-effects as being data
(like IO ()) — you can’t combine them in all the cool ways you could combine
IO (), but they’re getting at the same goal.
So what you should focus on is the sequencing of those operations — that is,
the order of the print statements.
Now compare the same program in Haskell:
data Context a = Context [String] a [String]
deriving (Show)
a = Context ["Enter A"] () ["Exit A"]
b = Context ["Enter B"] () ["Exit B"]
instance Monad Context where
return x = Context [] x []
(Context x1 p y1) >>= f =
let
Context x2 q y2 = f p
in
Context (x1 ++ x2) q (y2 ++ y1)
foo :: a -> Context String
foo _ = b >> (return "Inside")
bar :: () -> Context String
bar () = a >>= foo
main = do
print $ bar ()
Which produces:
Context ["Enter A","Enter B"] "Inside" ["Exit B","Exit A"]
And the ordering is the same.
The analogy between the two programs is very direct: a Context has some
“entering” bits, a “body”, and some “exiting” bits. I used String instead of
IO actions because it’s easier — I think it should be similar with IO actions (correct me if it’s not).
And >>= for Context does exactly what with in Python does: it runs the
entering statements, feeds the value to the body, and runs the exiting
statements.
(There’s another huge difference which is that the body should depend on the
entering statements. Again I think that should be fixible).
Haskell has an equivalent of with for files, it’s called withFile. This:
with open("file1", "w") as f:
with open("file2", "r") as g:
k = g.readline()
f.write(k)
is equivalent to:
withFile "file1" WriteMode $ f ->
withFile "file2" ReadMode $ g ->
do k <- hGetLine g
hPutStr f k
Now, withFile might look like something monadic. Its type is:
withFile :: FilePath -> IOMode -> (Handle -> IO r) -> IO r
right side looks like (a -> m b) -> m b.
Another similarity: In Python you can skip as, and in Haskell you can use >> instead of >>= (or, a do block without <- arrow).
So I’ll answer this question: is withFile monadic?
You could think that it can be written like this:
do f <- withFile "file1" WriteMode
g <- withFile "file2" ReadMode
k <- hGetLine g
hPutStr f k
But this doesn’t type check. And it cannot.
It’s because in Haskell the IO monad is sequential: if you write
do x <- a
y <- b
c
after a is executed, b is executed and then c. There is no “backtrack”
to clean a at the end or something like that. withFile, on the other hand,
has to close the handle after the block is executed.
There is another monad, called continuation monad, which allows to do such
things. However, you have now two monads, IO and continuations, and using effects of two monads at once requires using monad transformers.
import System.IO
import Control.Monad.Cont
k :: ContT r IO ()
k = do f <- ContT $ withFile "file1" WriteMode
g <- ContT $ withFile "file2" ReadMode
lift $ hGetLine g >>= hPutStr f
main = runContT k return
That’s ugly. So the answer is: somewhat, but that requires dealing with a lot of subtleties that make the issue rather opaque.
Python’s with can simulate only a limited bit of what monads can do – add entering and finalization code. I don’t think you can simulate e.g.
do x <- [2,3,4]
y <- [0,1]
return (x+y)
using with (it might be possible with some dirty hacks). Instead, use for:
for x in [2,3,4]:
for y in [0,1]:
print x+y
And there’s a Haskell function for this – forM:
forM [2,3,4] $ x ->
forM [0,1] $ y ->
print (x+y)
I recommed reading about yield which bears more resemblance to monads than with:
http://www.valuedlessons.com/2008/01/monads-in-python-with-nice-syntax.html
A related question: if we have monads, do we need exceptions?
Basically no, instead of a function that throws A or returns B you can make a function that returns Either A B. The monad for Either A will then behave just like exceptions – if one line of code will return an error, the whole block will.
However, that would mean that division would have type Integer -> Integer -> Either Error Integer and so on, to catch division by zero. You would have to detect errors (explicitly pattern match or use bind) in any code that uses division or has even slightest possibility of going wrong. Haskell uses exceptions to avoid doing this.