Macro of the day: build-list

Some operations are rare in “real” programs but common in throwaways and toys. One of these is creating lists of random data. Toy programs frequently use random input, simply because it's easier than getting real data. Usually this takes the form of evaluating some (random-thing) expression over and over and returning a list of the results:

(map (λ (x) (random-thing)) (range n))

The unused x here offends me. The randomly generated things usually don't depend on where they end up in the list, so why should the function that generates them receive the index as an argument?

I only find the unused argument annoying if it's in an explicit λ. Doing the same thing with Clojure's for doesn't feel nearly as sinful, since it doesn't explicitly say the index is being passed to a function:

(for [x (range n)] (random-thing))

That's acceptably simple. But what I really want to write is something like this:

(build-list n (random-thing))

...which evaluates (random-thing) n times and returns the results as a list. This is a straightforward macro:

(defmacro build-list
  "Return a seq of the results of evaluating EXPR N times."
  [n expr]
  `(for [i# (range ~n)] ~expr))

Or, in CL (which, even using loop, doesn't look good in comparison):

(defmacro build-list (n body)
  (with-gensyms (i)
    `(loop for ,i from 1 to ,n collect ,body)))

Clojure's built-in repeatedly is the functional equivalent of build-list: it's map over range, but without passing the argument. With # as a one-character λ, this approaches the ideal:

(repeatedly n #(random-thing))

These tools are simple and frequently useful, especially in the throwaway programs that most benefit from library support, but few standard libraries offer them. Some come close, but can't resist including the index as an argument. Racket, for instance, has a build-list function which is equivalent to map over a range, but it's barely an improvement for this use:

(build-list n (λ (x) (random-thing)))

I suspect this is because of cultural dissonance. Calling something repeatedly without passing it an argument is obviously useful only if it's not purely functional, and functional programmers tend to be suspicious of that. We (well, except for the Haskellers) accept impurities like randomness, but not so enthusiastically that we're comfortable including additional library functions to help with them. We expect our libraries to do the Right Thing, and some of our trusted heuristics tell us that making lists of random data is not the Right Thing. So, useful or not, we don't really try to make it easy.

What's so cool about APL?

Why does APL have such a devoted following, despite its strange appearance? What have its users, since the 1960s, seen in it that made them embrace such an unpopular language?

I'm not one of those fanatical APLers, but I think I understand why. Imagine the year is 1965. All computing is on mainframes, and the only high-level languages you've ever seen are Fortran and Algol-60. And then one day you meet APL, and discover:

• It has a read-eval-print loop: you can type expressions in and see the results immediately, without running a compiler. It's a remarkably powerful calculator, in the days before calculators were common. (This may account for its popularity in finance.)
• It's mostly functional: most operations return a result rather than requiring you to specify a destination to modify, so you can easily combine many operations in one expression.
• It has built-in collections — specifically multidimensional arrays, but any decent collection support would do as well. We take collections for granted nowadays, at least in languages higher-level than C, but this wasn't always so. There's a reason many early languages (not just APL and Lisp) were built around a collection type.
• It has high-order operations: map is (often) implicit, and reduce, scan, and Cartesian product are single characters.
• It's terse, and not just because of its one-character names. You really can say in a line of APL what would take a page of Fortran.

Under these circumstances, wouldn't you be amazed by the powerful new language? Wouldn't you become a faithful user, and for decades wonder why all other languages were so uselessly primitive?

Complaining about defsystem

Xach wrote a tool to create boilerplate for ASDF projects. I winced to read this. It may be useful, sure, but it hurts to see “little projects” require three files and an ASDF registration.

I've never had the good fortune to work on a Common Lisp program large enough to need defsystem, and most other languages get by without it, so to me all this complexity looks not only unfamiliar but unnecessary. As a result, when I think about module systems and library loading and such, I tend to dismiss most of the potential features as esoteric. Separate compilation, compile-time circular dependencies, multiple versions, replaceable implementations, parameterised modules, analyzing dependencies without loading — these all seem less important than the mundane convenience of being able to write a library in a single file.

I suppose it's better to focus on problems I actually have than on ones I've only heard about. But I wonder which of the problems I'm ignoring are actually important.

The worst thing about viruses is...

Today I heard someone praising Linux as a desktop OS, on the grounds that it's not very susceptible to viruses. This, he said, was good because it “saves cycles” and cuts down on reinstalls.

Not because they're a security threat. That, apparently, was not on the radar.

I think the cycles he was talking about were those taken by antivirus software, not by the viruses themselves, but the point is clear: many users see viruses as an annoyance, not a threat. Performance and convenience — those are the things that matter.

The speaker was a programmer who is involved in designing network protocols. Maybe he was only giving the reasons he thought his audience would understand. I didn't ask; I'm afraid of what the answer would be.

Correct ≠ safe

I wrote, too optimistically:

It's easy to write correct string operations, even in C, even for null-terminated strings in fixed-size buffers — and security-conscious C programmers often do.

Barry Kelly objected:

I don't agree that it is easy [...] to write correct code in C for manipulating null-terminated strings in fixed sized buffers, for a simple reason: if you can't let the fixed length follow the string, either via static typing or dynamically, then normal procedural abstraction will tend to hide the length and lead to errors later.

I didn't find that very convincing until a few weeks ago, when I spent a late night tracking down a mysterious buffer overflow: I had used a “safe” string operation with the wrong buffer size.

Shame vs. helplessness

On Monday I got a bug report: I had broken something due to a stupid oversight. I felt ashamed.

On Tuesday I got another bug report: I had broken something else by making a deliberate change based on reasoning that was completely, obviously wrong. I felt even more ashamed.

On Wednesday I got a third bug report: some poorly factored code — not mine, for a change — had, unsurprisingly, turned out to have perverse error behaviour. Over the course of the day, one of its authors made a series of misguided attempts to fix it, introducing at least one new bug in the process. Watching, I felt even worse than I had about my own stupid mistakes.

Of course: this is an effect of (the illusion of) control. A problem is scarier and more frustrating when you don't feel you can do anything about it. I could imagine avoiding my own mistakes by being smarter or more careful or something, but I have no such illusions about my ability to prevent other people from making mistakes. So I feel helpless about them, and that's more unpleasant than feeling ashamed of my own stupidity.

(Both of my stupid mistakes, by the way, were in code I didn't want to write. It's easy to pay attention to code that does something you care about; it's harder when you don't actually want the functionality you're implementing.)

What? No composition?

I think of function composition as a basic operation, one used regularly in any functional language. But if it's really so common, how could I forget Common Lisp doesn't have it? It has identity and complement and plenty of high-order collection operations, so naturally one might expect it to have compose too. I assumed it did, and was recently surprised to hear otherwise. Evidently I've never tried to use it and found it missing.

This says something about how little high-order style is used in Common Lisp. With the annoyance of #', and no convenient partial application, and no lambdaless define in the function namespace, almost all functions are written with a (possibly implicit) lambda; uses of compose are not very common. Common Lisp may be more expressive than most languages, but it's still far from pseudocode.