What can you do with Lisp macros that you can't do with first-class functions?
Question:
I think I understand Lisp macros and their role in the compilation phase.
But in Python, you can pass a function into another function
def f(filename, g):
try:
fh = open(filename, "rb")
g(fh)
finally:
close(fh)
So, we get lazy evaluation here. What can I do with macros and not with functions as first class objects?
Answers:
Macros are expanded in a compile-time. Closures are constructed in runtime. With macros you can implement highly efficient compilers of embedded domain specific languages, and with high order functions you can only implement inefficient interpreters. That eDSL compilers may do all kinds of static checks, do whatever expensive optimisations you fancy implementing, but when you’ve got only runtime you can’t do anything expensive.
And needless to mention that macros allows much more flexible syntax (literally, any syntax) for your eDSLs and language extensions.
See the answers to this question for more details: Collection of Great Applications and Programs using Macros
Here’s Matthias Felleisen’s answer from 2002 (via http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg01539.html):
I’d like to propose that there are
three disciplined uses of macros:
-
data sublanguages: I can write simple looking expressions and create
complex nested lists/arrays/tables
with quote, unquote etc neatly dressed
up with macros.
-
binding constructs: I can introduce new binding constructs with macros.
That helps me get rid of lambda’s and
with placing things closer together
that belong together. For example, one
of our teachpacks contains a form
(web-query ([last-name
(string-append “Hello ” first-name ” what’s your last
name?”])
… last-name … first-name …) with the obvious interaction between a
program and a Web consumer implied.
[Note: In ML you could write
web-query(fn last-name => …)string_append(…) but by golly
that’s a pain and an unnecessary
pattern.]
-
evaluation reordering: I can introduce constructs that
delay/postpone the evaluation of
expressions as needed. Think of loops,
new conditionals, delay/force, etc.
[Note: In Haskell, you don’t need that
one.]
I understand that Lispers use macros
for other reasons. In all honesty, I
believe that this is partly due to
compiler deficiencies, and partly due
to “semantic” irregularities in the
target language.
I challenge people to address all
three issues when they say language X
can do what macros can do.
— Matthias
Felleisen is one of the most influential macro researchers in the field. (I don’t know whether he would still agree with this message, though.)
More reading: Paul Graham’s On Lisp (http://www.paulgraham.com/onlisp.html; Graham definitely doesn’t agree with Felleisen that these are the only useful uses of macros), and Shriram Krishnamurthi’s paper “Automata via Macros” (http://www.cs.brown.edu/~sk/Publications/Papers/Published/sk-automata-macros/).
First of all Lisp has first-class functions too, so you could as well ask: “Why do I need macros in Lisp if I already have first-class functions”. The answer to that is that first-class functions don’t allow you to play with syntax.
On a cosmetic level, first-class functions allow you to write f(filename, some_function) or f(filename, lambda fh: fh.whatever(x)), but not f(filename, fh, fh.whatever(x)). Though arguably that’s a good thing because in that last case it is a lot less clear where fh suddenly comes from.
More importantly functions can only contain code that is valid. So you can’t write a higher-order function reverse_function that takes a function as an argument and executes it “in reverse”, so that reverse_function(lambda: "hello world" print) would execute print "hello world". With a macro you can do this. Of course this particular example is quite silly, but this ability is enormously useful when embedding domain specific languages.
For example you couldn’t implement common lisp’s loop construct in python. Hell, you couldn’t even implement python’s for ... in construct in python if it wasn’t really built-in – at least not with that syntax. Sure you could implement something like for(collection, function), but that’s a lot less pretty.
Macros do code transformations
The macro transforms source code. A lazy evaluation does not. Imagine that you can now write functions which transform arbitrary code to arbitrary different code.
Very simple code transformations
The creation of simple language constructs is also only a very simple example. Consider your example of opening a file:
(with-open-file (stream file :direction :input)
(do-something stream))
vs.
(call-with-stream (function do-something)
file
:direction :input)
What the macro gives me is a slightly different syntax and code structure.
Embedded language: advanced iteration constructs
Next consider a slightly different example:
(loop for i from 10 below 20 collect (sqr i))
vs.
(collect-for 10 20 (function sqr))
We can define a function COLLECT-FOR which does the same for a simple loop and has variables for start, end and a step function.
But LOOP provides a new language. The LOOP macro is a compiler for this language. This compiler can do LOOP specific optimizations and can also check the syntax at compile time for this new language. An even more powerful loop macro is ITERATE. These powerful tools on the language level now can be written as libraries without any special compiler support.
Walking the code tree in a macro and making changes
Next another simple example:
(with-slots (age name) some-person
(print name)
(princ " "
(princ age))
vs. something similar:
(flet ((age (person) (slot-value person 'age))
(name (person) (slot-value person 'name)))
(print (name))
(princ " ")
(princ (age)))
The WITH-SLOTS macro causes the complete walk of the enclosed source tree and replaces the variable name with a call to (SLOT-VALUE SOME-PERSON 'name):
(progn
(print (slot-value some-person 'name))
(princ " "
(princ (slot-value some-person 'age)))
In this case the macro can rewrite selected parts of the code. It understands the structure of the Lisp language and knows that name and age are variables. It also understands that in some situations name and age might not be variables and should not be rewritten. This is an application of a so-called Code Walker, a tool that can walk code trees and make changes to the code tree.
Macros can modify the compile-time environment
Another simple example, the contents of a small file:
(defmacro oneplus (x)
(print (list 'expanding 'oneplus 'with x))
`(1+ ,x))
(defun example (a b)
(+ (oneplus a) (oneplus (* a b))))
In this example we are not interested in the macro ONEPLUS, but in the macro DEFMACRO itself.
What is interesting about it? In Lisp you can have a file with above contents and use the file compiler to compile that file.
;;; Compiling file /private/tmp/test.lisp ...
;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 1
;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; ONEPLUS
(EXPANDING ONEPLUS SOURCE A)
(EXPANDING ONEPLUS SOURCE (* A B))
; EXAMPLE
;; Processing Cross Reference Information
So we see, that the file compiler expands the use of the ONEPLUS macro.
What is special about that? There is a macro definition in the file and in the next form we already use that new macro ONEPLUS. We have never loaded the macro definition into Lisp. Somehow the compiler knows and registers the defined macro ONEPLUS and is then able to use it.
So the macro DEFMACRO registers the newly defined macro ONEPLUS in the compile-time environment, so that the compiler knows about this macro – without loading the code. The macro then can be executed at compile-time during macro expansion.
With a function we can’t do that. The compiler creates code for function calls, but does not run them. But a macro can be run at compile time and add ‘knowledge’ to the compiler. This knowledge then is valid during the run of the compiler and partially forgotten later. DEFMACRO is a macro which executes at compile time and then informs the compile-time environment of a new macro.
Note also that the macro ONEPLUS is also run twice, since it is used twice in the file. The side effect is that it prints something. But ONEPLUS could have also other arbitrary side effects. For example it could check the enclosed source against a rule base and alert you if for example the enclosed code violates some rules (think of a style checker).
That means, that a macro, here DEFMACRO, can change the language and its environment during compilation of a file. In other languages the compiler might provide special compiler directives which will be recognized during compilation. There are many examples for such defining macros influencing the compiler: DEFUN, DEFCLASS, DEFMETHOD, …
Macros can make the user code shorter
A typical example is the DEFSTRUCT macro for defining record-like data structures.
(defstruct person name age salary)
Above defstruct macro creates code for
- a new structure type
person with three slots
- slot accessors for reading and writing the values
- a predicate to check if some object is of class
person
- a
make-person function to create structure objects
- a printed representation
Additionally it may:
- record the source code
- record the origin of the source code (file, editor buffer, REPL, …)
- cross-reference the source code
The original code to define the structure is a short line. The expanded code is much longer.
The DEFSTRUCT macro does not need access to a meta-level of the language to create these various things. It just transforms a compact piece of descriptive code into the, typically longer, defining code using the typical language constructs.
Instead of a high-level answer, here’s a concrete suggestion: read Shriram’s The Swine Before Perl. It shows how to develop a macro that is is doing several different things — a specific control flow, a binding, and a data language. (Plus, you’ll see how to actually do this kind of stuff.)
Macros are most useful in languages which use the same form for data and code, because a macro treats the code as data and produces new code. A macro expansion is a just-in-time code generation, which is performed during a compilation phase before the evaluation starts. This makes it very easy to design DSLs.
In Python you can not take some code, pass it to a function, which generates new code, in order to execute the new code. In order to achieve something like macros in Python you have to generate an AST from your Python code, modify the AST and evaluate the modified AST.
This means that you can not write an if statement in Python. You can use only the existing one, but you can not modify it or write you own statements. But Lisp macros allow you to write your own statements. For example you can write a fi statement which behaves like if but takes the else part as its first argument and the then part as the second.
The following article describes the difference between macros and procedures more detailed:
ftp://ftp.cs.utexas.edu/pub/garbage/cs345/schintro-v13/schintro_130.html
In an example other than lisp, for example elixir, the if control flow statement is in fact a macro. The if is implemented as a function. But in order to have a cleaner more memorable syntax, it was also implemented as a macro.
if true do 1+2 end
if true, do: ( 1+2 )
if(true, do: 1+2)
if(true, [do: (1+2)])
if(true, [{:do, 1+2}])
All of the above are equivalent. But the first line is the macro implementation of if, which presumably gets expanded into the if function below.
By making if a function and accessible as a macro gives you this cool ability to put if control flows inside a parameter of another function, while preserving familiarity with other languages.
is_number(if true do 1+2 end)
is_number(if true, do: (1+2))
So I think Macros allow you to control syntax better, thus allowing you to create DSLs that standard functions cannot.
Firstly, what macros can do that functions cannot is all those actions that special operators can do. A macro call can expand into special operators, whereas a function call cannot. For instance, if we have a lexical variable x, then (mac x) can plausibly update its value. But (fun x) cannot.
Macros are expanded and thereby disappear, after which there are only function calls and special operators. Therefore, anything that macros can do not only can be done without macros but is ultimately done without them, every time. But that’s exactly like saying that everything you can do in a higher level language can be coded in assembler. In pure theory yes (I mean , look, the compiler is in fact coding it all assembler, supplying an instant existence proof), but it’s not practical.
I think I understand Lisp macros and their role in the compilation phase.
But in Python, you can pass a function into another function
def f(filename, g):
try:
fh = open(filename, "rb")
g(fh)
finally:
close(fh)
So, we get lazy evaluation here. What can I do with macros and not with functions as first class objects?
Macros are expanded in a compile-time. Closures are constructed in runtime. With macros you can implement highly efficient compilers of embedded domain specific languages, and with high order functions you can only implement inefficient interpreters. That eDSL compilers may do all kinds of static checks, do whatever expensive optimisations you fancy implementing, but when you’ve got only runtime you can’t do anything expensive.
And needless to mention that macros allows much more flexible syntax (literally, any syntax) for your eDSLs and language extensions.
See the answers to this question for more details: Collection of Great Applications and Programs using Macros
Here’s Matthias Felleisen’s answer from 2002 (via http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg01539.html):
I’d like to propose that there are
three disciplined uses of macros:
data sublanguages: I can write simple looking expressions and create
complex nested lists/arrays/tables
with quote, unquote etc neatly dressed
up with macros.binding constructs: I can introduce new binding constructs with macros.
That helps me get rid of lambda’s and
with placing things closer together
that belong together. For example, one
of our teachpacks contains a form
(web-query ([last-name
(string-append “Hello ” first-name ” what’s your last
name?”])
… last-name … first-name …) with the obvious interaction between a
program and a Web consumer implied.
[Note: In ML you could write
web-query(fn last-name => …)string_append(…) but by golly
that’s a pain and an unnecessary
pattern.]evaluation reordering: I can introduce constructs that
delay/postpone the evaluation of
expressions as needed. Think of loops,
new conditionals, delay/force, etc.
[Note: In Haskell, you don’t need that
one.]I understand that Lispers use macros
for other reasons. In all honesty, I
believe that this is partly due to
compiler deficiencies, and partly due
to “semantic” irregularities in the
target language.I challenge people to address all
three issues when they say language X
can do what macros can do.— Matthias
Felleisen is one of the most influential macro researchers in the field. (I don’t know whether he would still agree with this message, though.)
More reading: Paul Graham’s On Lisp (http://www.paulgraham.com/onlisp.html; Graham definitely doesn’t agree with Felleisen that these are the only useful uses of macros), and Shriram Krishnamurthi’s paper “Automata via Macros” (http://www.cs.brown.edu/~sk/Publications/Papers/Published/sk-automata-macros/).
First of all Lisp has first-class functions too, so you could as well ask: “Why do I need macros in Lisp if I already have first-class functions”. The answer to that is that first-class functions don’t allow you to play with syntax.
On a cosmetic level, first-class functions allow you to write f(filename, some_function) or f(filename, lambda fh: fh.whatever(x)), but not f(filename, fh, fh.whatever(x)). Though arguably that’s a good thing because in that last case it is a lot less clear where fh suddenly comes from.
More importantly functions can only contain code that is valid. So you can’t write a higher-order function reverse_function that takes a function as an argument and executes it “in reverse”, so that reverse_function(lambda: "hello world" print) would execute print "hello world". With a macro you can do this. Of course this particular example is quite silly, but this ability is enormously useful when embedding domain specific languages.
For example you couldn’t implement common lisp’s loop construct in python. Hell, you couldn’t even implement python’s for ... in construct in python if it wasn’t really built-in – at least not with that syntax. Sure you could implement something like for(collection, function), but that’s a lot less pretty.
Macros do code transformations
The macro transforms source code. A lazy evaluation does not. Imagine that you can now write functions which transform arbitrary code to arbitrary different code.
Very simple code transformations
The creation of simple language constructs is also only a very simple example. Consider your example of opening a file:
(with-open-file (stream file :direction :input)
(do-something stream))
vs.
(call-with-stream (function do-something)
file
:direction :input)
What the macro gives me is a slightly different syntax and code structure.
Embedded language: advanced iteration constructs
Next consider a slightly different example:
(loop for i from 10 below 20 collect (sqr i))
vs.
(collect-for 10 20 (function sqr))
We can define a function COLLECT-FOR which does the same for a simple loop and has variables for start, end and a step function.
But LOOP provides a new language. The LOOP macro is a compiler for this language. This compiler can do LOOP specific optimizations and can also check the syntax at compile time for this new language. An even more powerful loop macro is ITERATE. These powerful tools on the language level now can be written as libraries without any special compiler support.
Walking the code tree in a macro and making changes
Next another simple example:
(with-slots (age name) some-person
(print name)
(princ " "
(princ age))
vs. something similar:
(flet ((age (person) (slot-value person 'age))
(name (person) (slot-value person 'name)))
(print (name))
(princ " ")
(princ (age)))
The WITH-SLOTS macro causes the complete walk of the enclosed source tree and replaces the variable name with a call to (SLOT-VALUE SOME-PERSON 'name):
(progn
(print (slot-value some-person 'name))
(princ " "
(princ (slot-value some-person 'age)))
In this case the macro can rewrite selected parts of the code. It understands the structure of the Lisp language and knows that name and age are variables. It also understands that in some situations name and age might not be variables and should not be rewritten. This is an application of a so-called Code Walker, a tool that can walk code trees and make changes to the code tree.
Macros can modify the compile-time environment
Another simple example, the contents of a small file:
(defmacro oneplus (x)
(print (list 'expanding 'oneplus 'with x))
`(1+ ,x))
(defun example (a b)
(+ (oneplus a) (oneplus (* a b))))
In this example we are not interested in the macro ONEPLUS, but in the macro DEFMACRO itself.
What is interesting about it? In Lisp you can have a file with above contents and use the file compiler to compile that file.
;;; Compiling file /private/tmp/test.lisp ...
;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 1
;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; ONEPLUS
(EXPANDING ONEPLUS SOURCE A)
(EXPANDING ONEPLUS SOURCE (* A B))
; EXAMPLE
;; Processing Cross Reference Information
So we see, that the file compiler expands the use of the ONEPLUS macro.
What is special about that? There is a macro definition in the file and in the next form we already use that new macro ONEPLUS. We have never loaded the macro definition into Lisp. Somehow the compiler knows and registers the defined macro ONEPLUS and is then able to use it.
So the macro DEFMACRO registers the newly defined macro ONEPLUS in the compile-time environment, so that the compiler knows about this macro – without loading the code. The macro then can be executed at compile-time during macro expansion.
With a function we can’t do that. The compiler creates code for function calls, but does not run them. But a macro can be run at compile time and add ‘knowledge’ to the compiler. This knowledge then is valid during the run of the compiler and partially forgotten later. DEFMACRO is a macro which executes at compile time and then informs the compile-time environment of a new macro.
Note also that the macro ONEPLUS is also run twice, since it is used twice in the file. The side effect is that it prints something. But ONEPLUS could have also other arbitrary side effects. For example it could check the enclosed source against a rule base and alert you if for example the enclosed code violates some rules (think of a style checker).
That means, that a macro, here DEFMACRO, can change the language and its environment during compilation of a file. In other languages the compiler might provide special compiler directives which will be recognized during compilation. There are many examples for such defining macros influencing the compiler: DEFUN, DEFCLASS, DEFMETHOD, …
Macros can make the user code shorter
A typical example is the DEFSTRUCT macro for defining record-like data structures.
(defstruct person name age salary)
Above defstruct macro creates code for
- a new structure type
personwith three slots - slot accessors for reading and writing the values
- a predicate to check if some object is of class
person - a
make-personfunction to create structure objects - a printed representation
Additionally it may:
- record the source code
- record the origin of the source code (file, editor buffer, REPL, …)
- cross-reference the source code
The original code to define the structure is a short line. The expanded code is much longer.
The DEFSTRUCT macro does not need access to a meta-level of the language to create these various things. It just transforms a compact piece of descriptive code into the, typically longer, defining code using the typical language constructs.
Instead of a high-level answer, here’s a concrete suggestion: read Shriram’s The Swine Before Perl. It shows how to develop a macro that is is doing several different things — a specific control flow, a binding, and a data language. (Plus, you’ll see how to actually do this kind of stuff.)
Macros are most useful in languages which use the same form for data and code, because a macro treats the code as data and produces new code. A macro expansion is a just-in-time code generation, which is performed during a compilation phase before the evaluation starts. This makes it very easy to design DSLs.
In Python you can not take some code, pass it to a function, which generates new code, in order to execute the new code. In order to achieve something like macros in Python you have to generate an AST from your Python code, modify the AST and evaluate the modified AST.
This means that you can not write an if statement in Python. You can use only the existing one, but you can not modify it or write you own statements. But Lisp macros allow you to write your own statements. For example you can write a fi statement which behaves like if but takes the else part as its first argument and the then part as the second.
The following article describes the difference between macros and procedures more detailed:
ftp://ftp.cs.utexas.edu/pub/garbage/cs345/schintro-v13/schintro_130.html
In an example other than lisp, for example elixir, the if control flow statement is in fact a macro. The if is implemented as a function. But in order to have a cleaner more memorable syntax, it was also implemented as a macro.
if true do 1+2 end
if true, do: ( 1+2 )
if(true, do: 1+2)
if(true, [do: (1+2)])
if(true, [{:do, 1+2}])
All of the above are equivalent. But the first line is the macro implementation of if, which presumably gets expanded into the if function below.
By making if a function and accessible as a macro gives you this cool ability to put if control flows inside a parameter of another function, while preserving familiarity with other languages.
is_number(if true do 1+2 end)
is_number(if true, do: (1+2))
So I think Macros allow you to control syntax better, thus allowing you to create DSLs that standard functions cannot.
Firstly, what macros can do that functions cannot is all those actions that special operators can do. A macro call can expand into special operators, whereas a function call cannot. For instance, if we have a lexical variable x, then (mac x) can plausibly update its value. But (fun x) cannot.
Macros are expanded and thereby disappear, after which there are only function calls and special operators. Therefore, anything that macros can do not only can be done without macros but is ultimately done without them, every time. But that’s exactly like saying that everything you can do in a higher level language can be coded in assembler. In pure theory yes (I mean , look, the compiler is in fact coding it all assembler, supplying an instant existence proof), but it’s not practical.