How is Lisp's read-eval-print loop different than Python's?
Question:
I’ve encounter a following statement by Richard Stallman:
‘When you start a Lisp system, it enters a read-eval-print loop. Most other languages have nothing comparable to read, nothing comparable to eval, and nothing comparable to print. What gaping deficiencies! ‘
Now, I did very little programming in Lisp, but I’ve wrote considerable amount of code in Python and recently a little in Erlang. My impression was that these languages also offer read-eval-print loop, but Stallman disagrees (at least about Python):
‘I skimmed documentation of Python after people told me it was fundamentally similar to Lisp. My conclusion is that that is not so. When you start Lisp, it does ‘read’, ‘eval’, and ‘print’, all of which are missing in Python.’
Is there really a fundamental technical difference between Lisp’s and Python’s read-eval-print loops? Can you give examples of things that Lisp REPL makes easy and that are difficult to do in Python?
Answers:
In support of Stallman’s position, Python does not do the same thing as typical Lisp systems in the following areas:
-
The read function in Lisp reads an S-expression, which represents an arbitrary data structure that can either be treated as data, or evaluated as code. The closest thing in Python reads a single string, which you would have to parse yourself if you want it to mean anything.
-
The eval function in Lisp can execute any Lisp code. The eval function in Python evaluates only expressions, and needs the exec statement to run statements. But both these work with Python source code represented as text, and you have to jump through a bunch of hoops to “eval” a Python AST.
-
The print function in Lisp writes out an S-expression in exactly the same form that read accepts. print in Python prints out something defined by the data you’re trying to print, which is certainly not always reversible.
Stallman’s statement is a bit disingenuous, because clearly Python does have functions named exactly eval and print, but they do something different (and inferior) to what he expects.
In my opinion, Python does have some aspects similar to Lisp, and I can understand why people might have recommended that Stallman look into Python. However, as Paul Graham argues in What Made Lisp Different, any programming language that includes all the capabilities of Lisp, must also be Lisp.
Stallman’s point is that not implementing an explicit “reader” makes Python’s REPL appear crippled compared to Lisps because it removes a crucial step from the REPL process. Reader is the component that transforms a textual input stream into the memory — think of something like an XML parser built into the language and used for both source code and for data. This is useful not only for writing macros (which would in theory be possible in Python with the ast module), but also for debugging and introspection.
Say you’re interested in how the incf special form is implemented. You can test it like this:
[4]> (macroexpand '(incf a))
(SETQ A (+ A 1)) ;
But incf can do much more than incrementing symbol values. What exactly does it do when asked to increment a hash table entry? Let’s see:
[2]> (macroexpand '(incf (gethash htable key)))
(LET* ((#:G3069 HTABLE) (#:G3070 KEY) (#:G3071 (+ (GETHASH #:G3069 #:G3070) 1)))
(SYSTEM::PUTHASH #:G3069 #:G3070 #:G3071)) ;
Here we learn that incf calls a system-specific puthash function, which is an implementation detail of this Common Lisp system. Note how the “printer” is making use of features known to the “reader”, such as introducing anonymous symbols with the #: syntax, and referring to the same symbols within the scope of the expanded expression. Emulating this kind of inspection in Python would be much more verbose and less accessible.
In addition to the obvious uses at the REPL, experienced Lispers use print and read in the code as a simple and readily available serialization tool, comparable to XML or json. While Python has the str function, equivalent to Lisp’s print, it lacks the equivalent of read, the closest equivalent being eval. eval of course conflates two different concepts, parsing and evaluation, which leads to problems like this and solutions like this and is a recurring topic on Python forums. This would not be an issue in Lisp precisely because the reader and the evaluator are cleanly separated.
Finally, advanced features of the reader facility enable the programmer to extend the language in ways that even macros could not otherwise provide. A perfect example of such making hard things possible is the infix package by Mark Kantrowitz, implementing a full-featured infix syntax as a reader macro.
In a Lisp-based system one typically develops the program while it is running from the REPL (read eval print loop). So it integrates a bunch of tools: completion, editor, command-line-interpreter, debugger, … The default is to have that. Type an expression with an error – you are in another REPL level with some debugging commands enabled. You actually have to do something to get rid of this behavior.
You can have two different meanings of the REPL concept:
-
the Read Eval Print Loop like in Lisp (or a few other similar languages). It reads programs and data, it evaluates and prints the result data. Python does not work this way. Lisp’s REPL allows you to work directly in a meta-programming way, writing code which generates (code), check the expansions, transform actual code, etc.. Lisp has read/eval/print as the top loop. Python has something like readstring/evaluate/printstring as the top-loop.
-
the Command Line Interface. An interactive shell. See for example for IPython. Compare that to Common Lisp’s SLIME.
The default shell of Python in default mode is not really that powerful for interactive use:
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a+2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>>
You get an error message and that’s it.
Compare that to the CLISP REPL:
rjmba:~ joswig$ clisp
i i i i i i i ooooo o ooooooo ooooo ooooo
I I I I I I I 8 8 8 8 8 o 8 8
I `+' / I 8 8 8 8 8 8
`-+-' / 8 8 8 ooooo 8oooo
`-__|__-' 8 8 8 8 8
| 8 o 8 8 o 8 8
------+------ ooooo 8oooooo ooo8ooo ooooo 8
Welcome to GNU CLISP 2.49 (2010-07-07) <http://clisp.cons.org/>
Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2010
Type :h and hit Enter for context help.
[1]> (+ a 2)
*** - SYSTEM::READ-EVAL-PRINT: variable A has no value
The following restarts are available:
USE-VALUE :R1 Input a value to be used instead of A.
STORE-VALUE :R2 Input a new value for A.
ABORT :R3 Abort main loop
Break 1 [2]>
CLISP uses Lisp’s condition system to break into a debugger REPL. It presents some restarts. Within the error context, the new REPL provides extended commands.
Let’s use the :R1 restart:
Break 1 [2]> :r1
Use instead of A> 2
4
[3]>
Thus you get interactive repair of programs and execution runs…
Python’s interactive mode differs from Python’s “read code from file” mode in several, small, crucial ways, probably inherent in the textual representation of the language. Python is also not homoiconic, something that makes me call it “interactive mode” rather than “read-eval-print loop”. That aside, I’d say that it is more a difference of grade than a difference in kind.
Now, something tahtactually comes close to “difference in kind”, in a Python code file, you can easily insert blank lines:
def foo(n):
m = n + 1
return m
If you try to paste the identical code into the interpreter, it will consider the function to be “closed” and complain that you have a naked return statement at the wrong indentation. This does not happen in (Common) Lisp.
Furthermore, there are some rather handy convenience variables in Common Lisp (CL) that are not available (at least as far as I know) in Python. Both CL and Python have “value of last expression” (* in CL, _ in Python), but CL also has ** (value of expression before last) and *** (the value of the one before that) and +, ++ and +++ (the expressions themselves). CL also doesn’t distinguish between expressions and statements (in essence, everything is an expression) and all of that does help build a much richer REPL experience.
As I said at the beginning, it is more a difference in grade than difference in kind. But had the gap been only a smidgen wider between them, it would probably be a difference in kind, as well.
I’ve encounter a following statement by Richard Stallman:
‘When you start a Lisp system, it enters a read-eval-print loop. Most other languages have nothing comparable to read, nothing comparable to eval, and nothing comparable to print. What gaping deficiencies! ‘
Now, I did very little programming in Lisp, but I’ve wrote considerable amount of code in Python and recently a little in Erlang. My impression was that these languages also offer read-eval-print loop, but Stallman disagrees (at least about Python):
‘I skimmed documentation of Python after people told me it was fundamentally similar to Lisp. My conclusion is that that is not so. When you start Lisp, it does ‘read’, ‘eval’, and ‘print’, all of which are missing in Python.’
Is there really a fundamental technical difference between Lisp’s and Python’s read-eval-print loops? Can you give examples of things that Lisp REPL makes easy and that are difficult to do in Python?
In support of Stallman’s position, Python does not do the same thing as typical Lisp systems in the following areas:
-
The
readfunction in Lisp reads an S-expression, which represents an arbitrary data structure that can either be treated as data, or evaluated as code. The closest thing in Python reads a single string, which you would have to parse yourself if you want it to mean anything. -
The
evalfunction in Lisp can execute any Lisp code. Theevalfunction in Python evaluates only expressions, and needs theexecstatement to run statements. But both these work with Python source code represented as text, and you have to jump through a bunch of hoops to “eval” a Python AST. -
The
printfunction in Lisp writes out an S-expression in exactly the same form thatreadaccepts.printin Python prints out something defined by the data you’re trying to print, which is certainly not always reversible.
Stallman’s statement is a bit disingenuous, because clearly Python does have functions named exactly eval and print, but they do something different (and inferior) to what he expects.
In my opinion, Python does have some aspects similar to Lisp, and I can understand why people might have recommended that Stallman look into Python. However, as Paul Graham argues in What Made Lisp Different, any programming language that includes all the capabilities of Lisp, must also be Lisp.
Stallman’s point is that not implementing an explicit “reader” makes Python’s REPL appear crippled compared to Lisps because it removes a crucial step from the REPL process. Reader is the component that transforms a textual input stream into the memory — think of something like an XML parser built into the language and used for both source code and for data. This is useful not only for writing macros (which would in theory be possible in Python with the ast module), but also for debugging and introspection.
Say you’re interested in how the incf special form is implemented. You can test it like this:
[4]> (macroexpand '(incf a))
(SETQ A (+ A 1)) ;
But incf can do much more than incrementing symbol values. What exactly does it do when asked to increment a hash table entry? Let’s see:
[2]> (macroexpand '(incf (gethash htable key)))
(LET* ((#:G3069 HTABLE) (#:G3070 KEY) (#:G3071 (+ (GETHASH #:G3069 #:G3070) 1)))
(SYSTEM::PUTHASH #:G3069 #:G3070 #:G3071)) ;
Here we learn that incf calls a system-specific puthash function, which is an implementation detail of this Common Lisp system. Note how the “printer” is making use of features known to the “reader”, such as introducing anonymous symbols with the #: syntax, and referring to the same symbols within the scope of the expanded expression. Emulating this kind of inspection in Python would be much more verbose and less accessible.
In addition to the obvious uses at the REPL, experienced Lispers use print and read in the code as a simple and readily available serialization tool, comparable to XML or json. While Python has the str function, equivalent to Lisp’s print, it lacks the equivalent of read, the closest equivalent being eval. eval of course conflates two different concepts, parsing and evaluation, which leads to problems like this and solutions like this and is a recurring topic on Python forums. This would not be an issue in Lisp precisely because the reader and the evaluator are cleanly separated.
Finally, advanced features of the reader facility enable the programmer to extend the language in ways that even macros could not otherwise provide. A perfect example of such making hard things possible is the infix package by Mark Kantrowitz, implementing a full-featured infix syntax as a reader macro.
In a Lisp-based system one typically develops the program while it is running from the REPL (read eval print loop). So it integrates a bunch of tools: completion, editor, command-line-interpreter, debugger, … The default is to have that. Type an expression with an error – you are in another REPL level with some debugging commands enabled. You actually have to do something to get rid of this behavior.
You can have two different meanings of the REPL concept:
-
the Read Eval Print Loop like in Lisp (or a few other similar languages). It reads programs and data, it evaluates and prints the result data. Python does not work this way. Lisp’s REPL allows you to work directly in a meta-programming way, writing code which generates (code), check the expansions, transform actual code, etc.. Lisp has read/eval/print as the top loop. Python has something like readstring/evaluate/printstring as the top-loop.
-
the Command Line Interface. An interactive shell. See for example for IPython. Compare that to Common Lisp’s SLIME.
The default shell of Python in default mode is not really that powerful for interactive use:
Python 2.7.2 (default, Jun 20 2012, 16:23:33)
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a+2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'a' is not defined
>>>
You get an error message and that’s it.
Compare that to the CLISP REPL:
rjmba:~ joswig$ clisp
i i i i i i i ooooo o ooooooo ooooo ooooo
I I I I I I I 8 8 8 8 8 o 8 8
I `+' / I 8 8 8 8 8 8
`-+-' / 8 8 8 ooooo 8oooo
`-__|__-' 8 8 8 8 8
| 8 o 8 8 o 8 8
------+------ ooooo 8oooooo ooo8ooo ooooo 8
Welcome to GNU CLISP 2.49 (2010-07-07) <http://clisp.cons.org/>
Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2010
Type :h and hit Enter for context help.
[1]> (+ a 2)
*** - SYSTEM::READ-EVAL-PRINT: variable A has no value
The following restarts are available:
USE-VALUE :R1 Input a value to be used instead of A.
STORE-VALUE :R2 Input a new value for A.
ABORT :R3 Abort main loop
Break 1 [2]>
CLISP uses Lisp’s condition system to break into a debugger REPL. It presents some restarts. Within the error context, the new REPL provides extended commands.
Let’s use the :R1 restart:
Break 1 [2]> :r1
Use instead of A> 2
4
[3]>
Thus you get interactive repair of programs and execution runs…
Python’s interactive mode differs from Python’s “read code from file” mode in several, small, crucial ways, probably inherent in the textual representation of the language. Python is also not homoiconic, something that makes me call it “interactive mode” rather than “read-eval-print loop”. That aside, I’d say that it is more a difference of grade than a difference in kind.
Now, something tahtactually comes close to “difference in kind”, in a Python code file, you can easily insert blank lines:
def foo(n):
m = n + 1
return m
If you try to paste the identical code into the interpreter, it will consider the function to be “closed” and complain that you have a naked return statement at the wrong indentation. This does not happen in (Common) Lisp.
Furthermore, there are some rather handy convenience variables in Common Lisp (CL) that are not available (at least as far as I know) in Python. Both CL and Python have “value of last expression” (* in CL, _ in Python), but CL also has ** (value of expression before last) and *** (the value of the one before that) and +, ++ and +++ (the expressions themselves). CL also doesn’t distinguish between expressions and statements (in essence, everything is an expression) and all of that does help build a much richer REPL experience.
As I said at the beginning, it is more a difference in grade than difference in kind. But had the gap been only a smidgen wider between them, it would probably be a difference in kind, as well.