These are a few of my favourite macros

Much of this post seems familiar to me, as if I've seen it somewhere else, perhaps on LL1-discuss or comp.lang.*. But I can't find the post I remember, so maybe I'm imagining someone else saying what I'm thinking.

Macros are flexible, and unfamiliar to most programmers, so they inspire a lot of confusion (more, in my opinion, than they deserve, but that's a topic for another day). Sometimes people try to make sense of this confusion by classifying them into a few categories. These classifications typically include:

1. Macros that evaluate some arguments lazily, like if and and, or repeatedly, like while.
2. Macros that pass some arguments by reference rather than by value, like the setf family.
3. Binding macros that simply save a lambda: with-open-file. In languages with very terse lambda (like Smalltalk) these are not very useful, but in languages that require something like (lambda (x) ...), they're useful and common.
4. Macros that quote some arguments (i.e. treat them as data, not expressions).
5. Defining macros like defstruct.
6. Unhygienic binding macros: op, aif.

The reasons for the classifications vary. Sometimes the point is that all of the categories are either trivial or controversial. (The people making this argument usually say the trivial ones should be expressed functionally, and the controversial ones should not be expressed at all.) Sometimes, as in this case, the point is that some of the categories are hard to express in any other way. Sometimes the point is that some categories are common enough that they should be built in to the language (e.g. laziness) or supported in some other way (e.g. terse lambda) rather than requiring macros.

These classifications aren't wrong, but they are misleading, because the most valuable macros don't fit any of these categories. Instead they do what any good abstraction does: they hide irrelevant details. Here are some of my favourites.

Lazy cons

If you want to use lazy streams in an eager language, you can build them out of delay and eager lists. But this is easy to get wrong. Do you cons an item onto a stream with (delay (cons a b))? (cons (delay a) (delay b))? (delay (cons (delay a) b)? Something else?

This is hard enough that there's a paper about which one is best and why. Even if you know (and regardless of whether you disagree with that paper), it's easy to make mistakes when writing the delays by hand. But the exact place where laziness is introduced is an implementation detail; code producing streams doesn't usually care about it. A lazy-cons macro can hide that detail, so you can use lazy streams without worrying about how they work. That's what any good abstraction should do.

Sequencing actions

Haskell's do is not, officially, a macro, but this is only because standard Haskell doesn't have macros; in any case do is defined and implemented by macroexpansion. Its purpose is to allow stateful code to be written sequentially, in imperative style. Its expansion is a hideous chain of nested >>= and lambdas, which no one wants to write by hand (or read). Without this macro, IO actions would be much more awkward to use. Some of this awkwardness could be recovered through functions like sequence, but the use of actions to write in imperative style would be impractical. do hides the irrelevant functional plumbing and relieves the pain of something necessary but very un-Haskell-like. Really, would you want to use Haskell without it?

List comprehensions

Haskell's list comprehensions, like its do, express something that could be done with functions, but less readably. List comprehensions combine the functionality of map, mapcat, and filter in a binding construct that looks a lot like set comprehensions. They save having to mention those list functions or write any lambdas.

I sometimes wish there was a way to get a fold in there too, but it's a good macro as it is.

Haskell list comprehensions wear a pretty syntactic skin over their macro structure, but this is not essential. Clojure's for demonstrates that a bare macro works as well.

Partial application

Goo's op (and its descendants like Arc's [... _ ...] and Clojure's #(... % ...)) is an unhygienic binding macro that abbreviates partial application and other simple lambdas by making the argument list implicit. It hides the irrelevant detail of naming arguments, which makes it much terser than lambda, and makes high-order functions easier to use.

Language embedding

There is a class of macros that embed other languages, with semantics different from the host. The composition macro from my earlier posts is one such. A lazily macro that embeds a language with implicit laziness is another. The embedded languages can be very different from the host: macros for defining parsers, for example, often look nothing like the host language. Instead of function call, their important forms are concatenation, alternatives, and repetition. Macros for embedding Prolog look like the host language, but have very different semantics, which would be awkward to express otherwise.

Like do, these macros replace ugly, repetitive code (typically with a lot of explicit lambdas) with something simpler and much closer to pseudocode.

The usual tricks

Most macros do fall into the simple categories: binding, laziness and other calling conventions, quotation, defining, etc. It's easy to think, of each of these uses, that it ought to be built into the language so you don't have to “fake” it using macros.

Fake? There's nothing wrong with using a language's expressive power to supply features it doesn't have! That's what abstraction is for!

The C preprocessor is a very useful thing, but of course it has given macros a bad name. I suspect this colors the thinking even of people who do know real (i.e. tree) macros, leading them to prefer a “proper” built-in feature to its macro implementation.

From my point of view, a macro is much better than a built-in feature. A language feature complicates the language's kernel, making it harder to implement, and in particular harder to analyze. Macros cover all of them, plus others the designers haven't thought of, in a single feature — and they don't even complicate analysis, because they disappear when expanded, so the analysis phase never sees them.

(To be fair, macros do require the language's runtime to be present at compile-time, and create the possibility of phasing bugs. But either interactive compilation or self-hosting requires the former anyway, and the latter only interferes with macros, so at worst it's equivalent to not having them. Neither is remotely as bad as being unable to express things the language designer didn't think of.)

So I see macros not as a weird, overpowered feature but as an abstractive tool nearly as important as functions and classes. Every language that aims for expressive power should have them.

An anaphoric-macro-defining macro

Attention conservation notice: this post is about a special-purpose macro-defining macro. I doubt it will be of much interest to anyone but macro fans.

A year ago, Vladimir Sedach asked for arguments about why macros are and aren't useful. (He didn't get many; most of the people who can make such arguments are tired of doing so and not confident they're right.) In the same post, he mentioned that he uses the Anaphora macro library. Anaphora itself provides an example of how macros are useful, not only because it defines some handy ones, but because many of their definitions are repetitive:

(defmacro awhen (test &body body)
  "Like WHEN, except binds the result of the test to IT (via LET) for the scope
of the body."
  `(anaphoric when ,test ,@body))

(defmacro swhen (test &body body)
  "Like WHEN, except binds the test form to IT (via SYMBOL-MACROLET) for the
scope of the body. IT can be set with SETF."
  `(symbolic when ,test ,@body))

(defmacro acase (keyform &body cases)
  "Like CASE, except binds the result of the keyform to IT (via LET) for the
scope of the cases."
  `(anaphoric case ,keyform ,@cases))

(defmacro aetypecase (keyform &body cases)
  "Like ETYPECASE, except binds the result of the keyform to IT (via LET) for
the scope of the cases."
  `(anaphoric etypecase ,keyform ,@cases))

This is the sort of repetition that could be addressed with a macro. We can generate these definitions automatically, docstrings and all:

(defmacro defanaphoric (name (firstarg &rest moreargs) original &key settable)
  "Define NAME as a macro like ORIGINAL, but binding IT to the result of FIRSTARG.
If SETTABLE is true, IT will be a symbol-macro which can be set with SETF."
  (let ((whole (gensym "WHOLE")))
    `(defmacro ,name (&whole ,whole ,firstarg ,@moreargs)
       ,(format nil "Like ~S, except binds the result of ~S to IT (via ~S) for the scope of the ~A.~A"
          original
          firstarg
          (if settable 'let 'symbol-macrolet)
          (if (and moreargs (member (car moreargs) '(&body &rest))) (cadr moreargs) "body")
          (if settable " IT can be set with SETF." ""))
       ,@(mapcan (lambda (a) (unless (member a lambda-list-keywords)
                               `((declare (ignore ,a)))))
                 (cons firstarg moreargs))
       `(,',(if settable 'symbolic 'anaphoric) ,',original ,@(cdr ,whole)))))

Then it's easy to define individual anaphoric macros:

(defanaphoric awhen (test &body body) when)
(defanaphoric swhen (test &body body) when :symbolic t)
(defanaphoric acase (keyform &body cases) case)
(defanaphoric aetypecase (keyform &body cases) etypecase)

Note that almost half the code about generating docstrings. Anaphoric macros, like many other things, would be much simpler to define if they didn't need documentation.

The argument list (firstarg &rest moreargs) also exists purely for documentation purposes. We don't need it to define the macros — a simple &rest would do — but we need to pass it to defmacro so tools like SLIME and describe can display it. But repeating it as a parameter is ugly; it would be better to get the original's argument list automatically. There's no standard way to do this, but every implementation supports it; if we have a portability wrapper like swank-backend:arglist (or some non-Swank-dependent equivalent), we can do away with the explicit argument list:

(defmacro defanaphoric (name original &key settable)
  "Define NAME as a macro like ORIGINAL, but binding IT to the result of the first argument.
If SETTABLE is true, IT will be a symbol-macro which can be set with SETF."
  (let ((whole (gensym "WHOLE"))
        (arglist (swank-backend:arglist original)))
    `(defmacro ,name (&whole ,whole ,@arglist)
       ,(format nil "Like ~S, except binds the result of ~S to IT (via ~S) for the scope of the ~A.~A"
          original
          (car arglist)
          (if settable 'let 'symbol-macrolet)
          (if (and (cdr arglist) (member (cadr arglist) '(&body &rest)))
       (caddr arglist)
       "body")
          (if settable " IT can be set with SETF." ""))
       ,@(mapcan (lambda (a) (unless (member a lambda-list-keywords)
                               `((declare (ignore ,a)))))
                 arglist)
       `(,',(if settable 'symbolic 'anaphoric) ,',original ,@(cdr ,whole)))))

This makes the definitions of anaphoric macros transparently simple:

(defanaphoric awhen when)
(defanaphoric swhen when :settable t)
(defanaphoric acase case)
(defanaphoric aetypecase etypecase)

This might be useful for simplifying Anaphora, or even as a tool to expose to its users, but it's not much good as an answer to Vladimir's request for arguments for and against macros — macro-defining macros do not look useful to anyone who doesn't already think macros are useful.

Doing without modify macros

Speaking of setf-like macros, there's something that bothers me about Common Lisp's define-modify-macro: most of its uses are very predictable. Alexandria, for example, contains at least five of them: minf, maxf, appendf, removef, and deletef. Except for argument order, all five are identical macroifications of standard functions. From their names, you can predict not only their meanings but their whole implementations.

This is not a good sign. Rather than trying to define a modify macro for each function, wouldn't it be better to just have one modify macro which takes a function as a parameter?

(defmacro modify (fn place &rest args &environment e)
  (expand-setf (v r w) place e
    `(let ((,v (,fn ,r ,@args)))
       ,w)))

So instead of (define-modify-macro foof (args ...) foo) and (foof place args ...), you'd write (modify foo place args ...). This could make most modify macros unnecessary, although it would need a shorter name, perhaps even one as short as !. However, its usefulness depends heavily on having functions available with the right argument order. If you have to fiddle with the function to fix its argument order, modify loses its brevity, and you might as well write out the setf.

Simplifying setf-like macros

Attention conservation notice: I sometimes post things that are of general interest to language designers, but this is not one of those times. If you don't speak Common Lisp, don't like macrology, or don't care about generalized assignment, this post is not for you.

As the rplacd&pop macro illustrates, defining new setf-like macros is quite awkward in Common Lisp. I'm not talking about defining new setf-able places, but about defining macros that operate on places, like push and swapf. They're so awkward to write that the Hyperspec doesn't directly define most of them, but instead resorts to simplified versions with disclaimers:

The effect of (push item place) is equivalent to

(setf place (cons item place))

except that the subforms of place are evaluated only once, and item is evaluated before place.

There is actually a sample implementation of pop at get-setf-expansion, but it's not linked from pop, probably because it's too complex to be of much help in explaining what pop does. Handling places properly requires calling get-setf-expansion and doing nonobvious things with its five return values, which is a lot of trouble that has little to do with the desired macro. Consider the definition of rplacd&pop:

(defmacro rplacd&pop (x &optional y &environment e)
  (multiple-value-bind (temps forms svars writeform readform)
                       (get-setf-expansion x e)
    `(let* (,@(mapcar #'list temps forms)
            (,(car svars) (cdr ,readform)))
      (prog1 ,readform
        (setf (cdr ,readform) ,y)
        ,writeform))))

There is a lot of predictable code here. This macro differs from pop in only two sections (italicized above), and from push in those two plus argument order. Most of the code is the same. Can we abstract it out with a macro?

(defmacro expand-setf ((valuevar readvar writevar) place env &body body)
  "Generate code for a SETF-like operation: evaluate BODY with the three
vars bound to the corresponding parts of PLACE's SETF expansion, then
wrap the result in a LET to bind the SETF temporaries.
BODY should normally evaluate to a form that binds VALUEVAR to the
desired new value of PLACE and then does WRITEVAR."
  (with-gensyms (temps forms storevars)
    `(multiple-value-bind (,temps ,forms ,storevars ,writevar ,readvar)
                          (get-setf-expansion ,place ,env)
       (let ((,valuevar (car ,storevars)))
         `(let (,@(mapcar #'list ,temps ,forms))
            ,,@body)))))

I confess that I had some trouble writing this: I got confused about nested splicing. I'd feel bad about this, but I know everyone else has trouble with it too. Is there any other part of Lisp that so reliably earns its difficult reputation?

Anyway, having spent an hour writing expand-setf, I tried it out and was disappointed: it doesn't save as much code as I hoped. I forgot one of the basic rules of making difficult abstractions: write an example call before you write the macro.

(defmacro rplacd&pop (place &optional newcdr &environment e)
  (expand-setf (val readform writeform) place e
    `(let ((,val (cdr ,readform)))
       (prog1 ,readform
         (setf (cdr ,readform) ,newcdr)
         ,writeform)))

This is, unfortunately, only two lines shorter than the get-setf-expansion version, because expand-setf still requires the hassle of binding the store variable and passing the environment around. It scales nicely to more complicated macros, though:

(defmacro swapf (p1 p2 &environment e)
  "Two-argument ROTATEF."
  (expand-setf (v1 r1 w1) p1 e
    (expand-setf (v2 r2 w2) p2 e
      `(let ((,v1 ,r2) (,v2 ,r1))
         ,w1
         ,w2))))

We could get rid of the magical storevar by packaging writeform as a function, so you do (,writefn exp) instead of (let ((,storeform exp)) ,writeform). Unfortunately this doesn't necessarily make the callers any shorter, because they often have to store the value in a variable anyway:

(defmacro rplacd&pop (place &optional y &environment e)
  (expand-setf (r w) place e
    (with-gensyms (newval)
      `(let ((,newval ,r))
         (prog1 ,r
           (rplacd (cdr ,r) ,y)
           (,w ,newval)))))

It's also annoying that expand-setf wraps a form around the result of its body. That's a surprising thing for a macro to do, and I can imagine it confusing people and causing bugs. We can get rid of that - and of the whole macro - in a clean way, by packaging the expansion as a function:

(defun place-accessor (place env)
  "Return a form evaluating to an accessor for PLACE: a function
that returns the value of PLACE when called with no arguments, and
sets PLACE when called with one argument."
  (expand-setf (v r w env)
    (with-gensyms (set?)
      `(lambda (&optional (,v nil ,set?))
         (if ,set?    ;CASE-LAMBDA would be ideal here
           ,r
           (progn ,w ,v))))))

This turns places into first-class values, which, not surprisingly, makes them easier to work with. (I think this has been done before to emulate pass-by-reference, but I've never seen it done to simplify writing other macros. Update January 2016: Strachey did it in 1967 in Fundamental Concepts in Programming Languages, under the name “load-update pair”.) Distinguishing reads from writes by arity this way isn't entirely clean, but it beats packaging them as two separate functions, which would be prohibitively awkward for the caller. Here's how it works in practice:

(defmacro swapf (place1 place2 &environment e)
  (with-gensyms (a1 a2 v)
    `(let ((,a1 ,(place-accessor place1 e))
           (,a2 ,(place-accessor place2 e)))
       (let ((,v (funcall ,a1)))
         (funcall ,a1 (funcall ,a2))
         (funcall ,a2 ,v)))))

This is longer than the expand-setf version, because it doesn't handle the gensyms and variables itself. (Ignore the funcalls; they wouldn't be there in a Lisp-1.) It's less magical, but I'm not sure it's any easier to read. place-accessor would be best used as the foundation of a setf system, since it's conceptually simpler than get-setf-expansion, but it still needs a with-place-accessor wrapper macro. And I've lost sight of the original problem. None of these solutions make defining rplacd&pop and push as simple as it ought to be. There must be a better way.

Maybe I'm coming at this from the wrong direction. Rather than simplify the general case, what about generalizing the simple case? define-modify-macro already covers the most common kind of one-place macros: those that simply apply some function to the value in the place, and store (and return) the result:

(setf place (some-function place args ...))

For example:

CL-USER> (define-modify-macro pop* () cdr
           "Like POP, but doesn't return the popped value.")
POP*
CL-USER> (macroexpand-1 '(pop* place))
(LET* ((#:G4467 (CDR PLACE)))
  (SETQ PLACE #:G4467))

The only reason define-modify-macro can't express pop and rplacd&pop is that the macros it defines must return the same value they store. We could fix this by adding another argument: in addition to a function to determine the stored value, we can supply one to determine the returned value:

(defmacro define-modify-macro* (name arglist transformfn resultfn
                                &optional docstring)
  "Like DEFINE-MODIFY-MACRO, but takes an additional function, to
define a macro that returns a different value than it stores.
Subforms of the place are evaluated first, then any additional arguments,
then TRANSFORMFN is called, then RESULTFN, then the value is stored, and
finally the result of RESULTFN is returned.
ARGLIST may contain &optional arguments but not &rest or &key."
  (let* ((args (arg-names arglist))
         (gensyms (mapcar (lambda (a) (gensym (symbol-name a))) args)))
         ;GENSYMS is a list of names (in the intermediate expansion) of
         ; names (in the final expansion) of the values of the ARGS. They
         ; have to be gensymmed to protect TRANSFORMFM and RESULTFN.
    (with-gensyms (place env)
      `(defmacro ,name (,place ,@arglist &environment ,env)
         ,@(when docstring (list docstring))
         (expand-setf (v r w) ,place ,env
           (with-gensyms ,gensyms
             `(let (,,@(mapcar (lambda (gen arg) ``(,,gen ,,arg))
                               gensyms args))
                (let ((,v (,',transformfn ,r ,,@gensyms)))
                  (prog1 (,',resultfn ,r ,,@gensyms)
                         ,w)))))))))

(defun arg-names (arglist)
  "Return the names of the arguments in a lambda-list."
  (loop for arg in arglist
        unless (member arg lambda-list-keywords)
        collect (if (consp arg) (car arg) arg)))

(I had trouble writing this macro too - not just the multiple levels of splicing and gensyms, but also the awkwardness of extracting the argument names. Common Lisp has complex argument-lists, but no tools to parse them, so it took me a few tries before I realized I needed to do it myself, and then a few more before I gave up trying to handle &rest and &key. And I almost forgot to prevent multiple evaluation of argument forms!)

With this macro, pop-like macros can be defined more conveniently:

(define-modify-macro* rplacd&pop (&optional newcdr)
  (lambda (x y) (declare (ignore y)) (cdr x))
  rplacd
  "I am not explaining this one again.")

Except for the hassle of lambda and ignore to give cdr the right argument, this is not too painful. push requires a wrapper to fix its argument order, but pop is easy:

(define-modify-macro %push (val)
  (lambda (list val) (cons val list))
  (lambda (list val) (declare (ignore list)) val)
  "Helper for PUSH")

(defmacro push (val place)
  `(%push ,place ,val))

(define-modify-macro* pop () cdr car)

Correction: push returns the same value it stores, so it doesn't require define-modify-macro*.

I started this post intending to learn something about how to design a better setf-like generalized assignment system, and succeeded only in writing a few macros to make dealing with Common Lisp's setf less painful. This is, I suppose, still useful, but it's disappointing.

The cost of macros

I had a tool problem while reading obfuscated Lisp: I wanted automatic refactoring. In particular I wanted to be able to α-rename the obnoxious variables to something that didn't look so much like brackets. But I had to do it by hand, because there are no good refactoring tools for Lisp. This is partly because Lisp culture values tools for expression more than tools for maintenance, but partly because automating most refactorings is hard in the presence of macros.

Most Lisps have macros in their purest form: they're arbitrary functions that transform new forms to old ones. That means there's no general way to walk the arguments, because neither the function nor the expansion will tell you what parts of the call are forms, let alone what environment they belong in. There is no way to be sure the macro doesn't implement a different language, which makes analysis nearly impossible.

You can almost do it by observation: if part of a macro call appears as a form in the expansion, you can treat it as a form in the call — and you even know its environment. (Note that this requires eq, because you want to detect that it's the same value, not another identical one.) Unfortunately this fails when the same form appears more than once in the original — and this is normal for symbols, so this technique doesn't get you very far. Even if the representation of code were different, so variable references appeared as (ref x) rather than being abbreviated to x, it would still break when a form appears more than once in the same tree. And in the presence of macros, partial sharing is actually rather common, because multiple calls to the same macro often share part of their expansions. So reliably walking macro calls requires having more information about a macro than just how to expand it.

This is an advantage of more restrictive macro systems: they're easier to analyze. In a strict template-filling system like syntax-rules, you can always determine the role of a macro argument. DrScheme takes advantage of this for its fancy (but not very useful IME) syntax-highlighting. It doesn't work for procedural macros (Update: yes it does; see comments) but it could, if there were a way for macro definitions to supply the analysis along with the expander. Of course it would still be necessary to support unanalyzable mystery macros, because some macros are too hard to analyze, and because many authors won't bother.

I don't think procedural macros are a bad feature — on the contrary, I think they are the best and purest form of one of the four most important abstraction methods in any language (the other three are variables, functions, and user-defined datatypes). But they do have a cost. And I think the cost is mostly in what other tools they interfere with, not in any difficulty humans have with them.