Research cultures

John Regehr identifies some virtues of the culture of operating systems research, which are lacking in the culture of programming languages research:

The best argument is a working system. The more code, and the more results, the better. Something that is clearly a toy isn’t convincing. It is not necessary to build an abstract model, conduct a user study, prove soundness, prove correctness, or show any kind of asymptotic bound. In fact, if you want to do these things it may be better to do them elsewhere.

Yes. A working system, especially one that can support diverse applications without major pain, shows that you got everything important approximately right. A proof only shows that you got one thing right, and it's easy to be mistaken about whether that thing is important. So a working system can be stronger evidence that you got it right than a proof.

The style of exposition is simple and direct; this follows from the previous point. I have seen cases where a paper from the OS community and a paper from the programming languages community are describing almost exactly the same thing (probably a static analyzer) but the former paper is super clear whereas the latter is incredibly difficult to figure out.

I too find operating systems papers easier to read than language papers — and I'm from the languages community, so this isn't just a matter of familiarity. Systems researchers write as if they're trying to communicate, but language researchers write as if they're trying to make their results look like fancy academic research. This is a common failure mode (since researchers are evaluated partly on how difficult their work looks), and somehow it's become part of the standard style in language research, leading well-meaning writers to produce impenetrable papers.

It wasn't always so. Why has systems research kept these virtues while language research has descended into theory and impenetrability? How can this be reversed?

3 comments:

  1. Thanks for writing about this. That post was found to be offensive by at least a few people, which wasn't what I intended, though I can see how it might read that way.

    As you imply, I think it boils down to the question "What are we doing here?" that every researcher must ask him/herself. For a systems person it's usually "we want stuff to work, but also it needs to be as simple as possible." My guess is that there's a subset of the PL community who doesn't fall in with "as simple as possible" but rather goes for "mathematically impressive" or something along those lines.

    ReplyDelete
  2. To tell people their field is doing something badly may offend them regardless of how you put it. It still needs to be done.

    There's a surprising amount of agreement on what we're doing here. Most PL researchers these days say their goal is “rigor”, which is supposed to mean something like “confidence in program correctness”, but is easily misinterpreted to mean mathematical impressiveness. If it's only a subset who lean that way, it must be an influential subset, because ISTM PL researchers no longer expect other PL researchers to respect work that doesn't contain imposing math.

    ReplyDelete
  3. Yeah. CS finds itself in a position of tension between math and engineering, and subfields tend to lean one way or the other.

    The thing that bugs me isn't the math, which I quite like, if it's done well and serves some purpose. The thing that bugs me is when people pretend to be doing real engineering and then the paper is all math. Or, when a conference solicits papers that solve real-world problems and then evaluate the papers based almost entirely on their rigor.

    I was on the CAV (computer aided verification) PC a couple years ago and found myself only fighting losing battles for papers that I liked. In the end I computed the correlation between my review score for each paper and the average of the other reviewers' scores. The correlation score was something like -0.6, which I found to be impressively low.

    Even at PLDI (where I was on the ERC last year, and am again this year) where I would expect to be pretty much in agreement with people, this turns out not to be the case, though there is a significant subset of people there who basically think just like I do. For example at several PC meetings I've say next to David Gay and our reviews lined up with each other almost point for point.

    ReplyDelete

It's OK to comment on old posts.