Paul's Pontifications

Sunday, August 12, 2007

Responses to 'Silver Bullets Incoming!'

(((
These are the responses originally posted to "Silver Bullets Incoming". Many of these responses are worth reading in their own right, and I'd like to thank the respondants for taking the time for such thoughtful posts.

Please do not post this to reddit, as it has already been discussed there under the original URL.
)))

Stephen Says:
December 9th, 2006 at 4:38 pm e
Paul:

I enjoyed your first article quite a bit - it got me thinking about technical language issues again (always fun).

I’d like to comment on your update to the original article. Specifically, I have some comments regarding C++

C++ is not an “old” language, incorporating many language features of more “modern” languages, including exceptions, automatic memory management (via garbage collection libraries and RIIA techniques), and templates, a language feature that is only available in C++, and that provides support for generic programming and template metaprogramming, two relatively new programming paradigms. Yes, C++ has been around a while, but until I see programmers constantly exhausting the design and implementation possibilities of C++, I won’t call the language “old.”

C++ was not designed to support just OO programming: From “Why C++ Isn’t Just An Object-Oriented Programming Language” (http://www.research.att.com/~bs/oopsla.pdf):

“If I must apply a descriptive label, I use the phrase ‘multiparadigm language’ to describe C++.”

Stroustrup identifies functional, object-oriented, and generic programming as the three paradigms supported by C++, and I would also include metaprogramming (via C++ templates or Lisp macros) as another paradigm, though it is not often used by most developers.

Of course, we should also keep in mind Stroustrup’s statement regarding language comparisons (”The Design and Evolution of C++”, Bjarne Stroustrup, 1994, p.5): “Language comparisons are rarely meaningful and even less often fair.”

Take care, and have a good weekend!

Stephen
pongba Says:
December 12th, 2006 at 11:26 am e
I found it so weird that, on the one hand you argue that haskell is fast( to the extend that it might be even faster than some compiling language such as C++), while on the other hand you said “where correctness matters more than execution speed its fine today”.
Does that sound paradoxical?
Another Paul Says:
December 12th, 2006 at 3:59 pm e
Paul:

“I think that Dijkstra had it right: a line of code is a cost, not an asset. It costs money to write, and then it costs money to maintain. The more lines you have, the more overhead you have when you come to maintain or extend the application”

By that measure, there’s no such thing as an asset. Think about that a moment - someone buys a general ledger or CAD/CAM system and modifies it as companies do. Either system reduces staff, provides more accurate information much more quickly, and renders the company more competitive. Take it away and what happens?

It’s been my experience that while these systems require maintenance (and sometimes a lot) they usually result in a net reduction in staff and the cost of doing business. And some types of systems provide a clear competitive edge as well. I think that makes many systems just as much an asset as a house, factory building, or a lathe.

Interesting article. Thanks.

Another Paul
BobP Says:
December 12th, 2006 at 6:07 pm e
>> An order of magnitude is a factor of 10, no less

> Well, the Wikipedia entry does say about 10. All this stuff is so approximate that anything consistently in excess of 5 is close enough.

0.5 orders of magnitude = power(10.0,0.5) = sqrt(10.0) = 3.1623 (approx)
1.5 orders of magnitude = power(10.0,1.5) = sqrt(1000.0) = 31.623 (approx)

If we are rounding off, a factor of 4 is about one order of magnitude; also, a factor of 30 is about one order of magnitude.
Jeremy Bowers Says:
December 12th, 2006 at 6:36 pm e
You missed my point with Python, or at least failed to address it.

My point wasn’t that Python is also good. My point was that you lept from “10x improvement” to “it must the chewy functional goodness!” But your logic falls down in the face of the fact that Python, Perl, Ruby, and a number of non-functional languages that also have a 10x improvement over C++, therefore it clearly is not a sound argument to just leap to the conclusion that “it must be the chewy functional goodness!” when there are clearly other factors in play.

I’m not criticizing Haskell or functional programming, I’m criticizing your logic, and you’ve still got nothing to back it up.

(This is par for the course for a claim of a silver bullet, though.)
Sam Carter Says:
December 12th, 2006 at 8:14 pm e
“Libraries and languages are complicit: they affect each other in important ways. In the long run the language that makes libraries easier to write will accumulate more of them, and hence become more powerful.”

This argument has a large flaw in it, namely the current state of libraries doesn’t reflect this claim. The largest and most powerful collection of libraries seem to be .NET, CPAN, and the Java libs, certainly not Lisp libraries.

But the advocates of Lisp would argue that it’s the most powerful language, and it’s clearly been around for a long time, yet the Lisp community has not accumulated the most powerful collection of libraries. So unless the next 40 years are going to be different from the previous 40 years, you can’t really assert that language power is going to automatically lead to a rich set of libraries.

I stand by my original comment to the previous article that programming is more about APIs and libraries than about writing their own code, and that if you are focused on measuring code-writing performance, you are just measuring the wrong thing.

I also disagree with the claim that this is unmeasurable because doing a real-world test is too expensive. As long as the project is solvable in a few programmer-weeks, you can test it out with different languages. I took a computer science class (Comp314 at Rice) where we were expected to write a web browser in 2 weeks. It wouldn’t be that hard to have a programming test which incorporated a database, a web or GUI front end, and some kind of client/server architecture, e.g. implementing a small version of Nagios, or an IM client, or some other toy application.

I’m sorry but writing a command line application that parses a CSV file and does some fancy algorithm to simulate monkeys writing Shakespeare is about as relevant to modern software engineering as voodoo is to modern medicine.
Paul Johnson Says:
December 12th, 2006 at 8:27 pm e
pongba:

I’m arguing that Haskell programs are faster to *write*. Execution speed is a much more complicated issue. FP tends to lose in simple benchmarks, but big systems seem to do better in higher level languages because the higher abstraction allows more optimisation.
Paul Johnson Says:
December 12th, 2006 at 10:54 pm e
Another Paul:

The functionality that a package provides is an asset, but the production and maintenance of each line in that package is a cost. If you can provide the same asset with fewer lines of code then you have reduced your liabilities.

Paul.
Paul Johnson Says:
December 12th, 2006 at 11:12 pm e
Jeremy Bowers:

Teasing apart what it is about Haskell and Erlang that gives them such a low line count is tricky, because it is more than the sum of its parts. One part of it is the high level data manipulation and garbage collection that Python shares with functional languages. Another part of it is the chewy functional gooodness. Another part, for Haskell at least, is the purity. OTOH for Erlang it is the clean and simple semantics for concurrency.

What I see in the results from the Prechelt paper is that Python was, on average, about 3 times better than C++ while the average Haskell program (from a sample of 2) was about 4 times better. Actually the longer Haskell program was mine, and I was really embarassed when someone else showed me how much simpler it could have been.

In terms of pure line count I have to conceed that Python and Haskell don’t have a lot to choose between them. A 25% improvement isn’t that much. Its a pity we can’t do a controlled test on a larger problem: I think that Haskell’s type system and monads are major contributors to code that is both reliable and short. Unfortunately I can’t prove it, any more than I could prove that garbage collection was a win back in the days when I was advocating Eiffel over C++.

Paul.
Paul Prescod Says:
December 12th, 2006 at 11:59 pm e
If you cannot “tease apart” what it is about Haskell and Erlang that makes them so productive then you cannot say that any one improvement is a silver bullet. It just feels truthy to you. Furthermore, if you are presented with counter-examples in the form of Python and Ruby then surely you must discard your thesis entirely. The best you can say is that there exist non-functional languages that are 10 times less productive than some functional languages for some projects.

Duh.
Paul Johnson Says:
December 13th, 2006 at 12:26 am e
Sam Carter:

On languages with expressive power gathering libraries; point mostly conceeded, although Perl certainly is a very expressive language, so I don’t think it supports your point, and .NET has Microsoft paying its mongolian hordes, so its not a fair comparison.

There are two sorts of libraries: general purpose ones (e.g. data structures, string manipulation, file management) that get used in many applications, and vertical libraries (HTTP protocol, HTML parsing, SMTP protocol) that are only useful in specific applications. There is no hard dividing line of course, but the usefulness of a language for general purpose programming depends on the language and its general purpose libraries. The vertical libraries have a big impact for applications that use them, but not elsewhere. So I would generally judge a language along with the general purpose libraries that are shipped with it. The special purpose libraries are useful as well, but its a secondary consideration.

Paul.
Paul Johnson Says:
December 13th, 2006 at 12:33 am e
Sam Carter (again):

Sorry, just looked back at your post and realised I’d forgotten the second point.

A worthwhile test is going to take about 10 versions to average out the impact of different developers. So thats 2 weeks times 10 coders is 20 developer-weeks, or almost half a man-year. Say a coder is on $30K per year and total cost of employment is three times that (which is typical). Round numbers $40-50 per language. Ten languages will cost the best part of half a million dollars to evaluate. Not small beer.

Of course you could use students, but on average they will know Java or Javascript better than Python or Haskell, so how do you correct for that?

Paul.
pongba Says:
December 13th, 2006 at 7:10 am e
I’m arguing that Haskell programs are faster to *write*. Execution speed is a much more complicated issue. FP tends to lose in simple benchmarks, but big systems seem to do better in higher level languages because the higher abstraction allows more optimisation.

I always hear people saying that, but I really don’t get it.
I know that *theoretically* abstraction( or non-side-effect, etc) gives more opportunity for optimization, but I have never seen people show some real data that can *really* prove it.
One question constantly annoys me - If higher-level of abstraction allows more optimization, then why .NET put the burden of discriminating value-types and reference-types on us programmers. Shouldn’t the referential-transparency-ness be better at this?
Jonathon Duerig Says:
December 13th, 2006 at 10:58 am e
I have two specific (and one general) criticisms to make about your line of argumentation:

First, I think you do not adequately address the criticisms about lines of code as a metric. The cost of a line of code is the sum of five factors: (a) Difficulty of formulating the operation involved (original coder*1), (b) Difficulty of translating that operation into the target programming language (original coder*1), � Difficulty of parsing the code involved to understand what the line does (maintainer*n), (d) Difficulty of later understanding the purpose of that operation (maintainer*n), and (e) Difficulty of modifying that line while keeping it consistent with the rest of the program (maintainer*n).

(a) and (b) are done only once, but �, (d), and (e) are done many times whenever the program needs to be fixed or modified. Brooks’ argument was specifically that in the general case the time for (a) is more than 1/9 the time for (b), and the time for (d) is more than 1/9 the time for � and (e). This is important because (a) and (d) are both language and tool independent.

When comparing the lines of code from different languages, it is important to realize that the formulation of the operations and the understanding of purpose are spread across those lines. And the verbosity of the language usually doesn’t impede either of these problems (unless it is extreme).

For instance, take the creation of an iterator or enumeration in C++ or Java respectively and compare that to creating a fold function in Scheme. These are roughly equivalent tasks. In C++, an iterator is defined first by defining a class with various access operators like * and -> and ++ and — and then implementing them. This adds a lot of baggage because there are half a dozen or so functions that must be defined and there is a separate class specification. In constrast, a scheme fold function is much simpler from the language perspective. A single function is defined rather than half a dozen. It will almost certainly have fewer lines, possibly by 4 or 5 times.

But let us look at what the creation of the iterator or fold function means from the perspective of items (a) and (d). Both of these are common idioms in their respective languages, so all of the code specifically dealing with iteration/folding is trivial to conceptualize and trivial to understand the purpose of. The difficulty in writing either a custom iterator or a custom fold function lies within the subtleties of the iteration. If it is a tree, what information needs to be maintained and copied to successive iterations (whether that be in the form of state, or in the form of argument passing)? Are there multiple kinds of iterations? How would they be supported? (For example, sometimes a user wants to traverse a tree in pre-order, sometimes in post-order, sometimes in-order, and sometimes level by level in a breadth-first order.) These are the questions which the original coder and the later maintainers will have to contend with. And these are really orthogonal to lines of code counts.

But there is another factor at work here which makes lines of code a faulty cross-language measurement. Every language has a grain to it. If you program with the grain, then any difficulty will be easily solved by the tools in the language. If you program against the grain, then you will run into difficulty after difficulty. This applies to fundamental language properties. You can bypass the type system in C++ and avoid all the type checks, but it is cumbersome and unpredictable if you do it wrong. Ruby allows you to be much more flexible with types and provides a safety net. If you try to enforce a more strict typing in Ruby, then you will have to fight the language every step.

But the grain of the language also includes the question of scale. Some languages provide a lot of flexibility. They allow compact and loose representations of programs which can be customized to the problem domain easily. These languages include Scheme and Ruby and Haskell. These flexible languages are very useful for small projects with one or a few developers because they can be metaphorically molded to fit the hand of the person who wields them. But there is a trade-off because they tend to be more difficult to use in large groups because it is harder for others to undestand what it going on. This is a fundamental trade-off that programming languages must make. And it means that a language which is great at one end of the spectrum will likely be lousy at the other end. And this is reflected in the lines of code required for a particular scale of problem.

My second criticism is in regard to your discussion of state. You point out that Brooks considered managing of state to be a major essential difficulty of programming and you then claim that functional languages obviate this difficulty and hypothesize this as the reason that they can be a silver bullet.

I believe that you have misunderstood the kind of state the Brooks was referring to. He was not talking about run-time state but compile-time state. He was not talking about what variables are changed at run-time. He was talking about the interactions between components of the program. These interactions are still there and just as complex in functional languages as in imperative languages.

Second, even when considering just the run-time state, the referential transparency of functional languages simplifies only the theoretical analysis of a program. As far as a normal programmer who is informally reasoning about what a program does, the programmer must consider how state is transformed in the same way whether or not a modified copy is made or a destructive write is made. This is the same kind of reasoning.

Finally, I have seen many people talk about getting an order of magnitude improvement by finding some incredible programming tool. Functional programming is not unique in that respect. But in my experience this is more likely to be about finding a methodology that suits the persons mindset than about finding the one true language or system. Somebody who thinks about programming in terms of a conceptual universe that changes over time will be an order of magnitude less effective in a functional environment. And somebody who thinks about programming in terms of a conceptual description of the result which is broken up into first class functions will be an order of magnitude less effective in an imperative environment.

I have programmed in both imperative and functional languages. I know and understand the functional idioms and have used them. My mindset tends to the empirical. I am a less effective programmer in such languages. But I have seen programmers who can pull a metaphorical rabbit out of a hat while tapdancing in them. This says to me that evangelism about functional languages or empirical languages is fundamentally misguided regardless of the side.
Paul Johnson Says:
December 13th, 2006 at 7:20 pm e
Jonathon Duerig:

I had decided not to respond to any further comments and instead get on with my next article. But yours is long and carefully argued, so it merits a response regardless. Its also nice to be arguing the point with someone who knows what a fold is.

You make the point that during maintenance the difficulty of later understanding the purpose of an operation is language independent. I’m not so sure. A maintainer may suspect that a C++ iterator is truly orthogonal, but it can’t be taken for granted. There may be a bug hiding in those methods, or perhaps someone fixed a bug or worked around a problem by tweaking the semantics in an irregular way. Also a lot of the understanding of a piece of code comes from context, and it helps a lot to be able to see all the context at once (why else would 3 big monitors be a selling point for coding jobs?). So terse code makes it a lot easier to deduce context because you can see more at once.

(Aside: I remember in my final year project at Uni going into the lab at night because then I could get two VT100s to myself).

You say that Scheme, Ruby and Haskell can be moulded to fit the hand of the user, making them more productive for single person tasks, but less productive for groups because of mutual comprehension difficulties.

This is hard to test because of the lack of statistics, but Haskell is strongly typed and the community has already developed conventions and tools for documentation and testing (Haddock and QuickCheck). I can see that Scheme macros can be used to construct an ideosyncratic personal language, but I really don’t see how this could happen in Haskell. Things that get done with macros in Scheme are usually done with monads in Haskell, but whereas Scheme macros are procedural monads are declaritive and must conform to mathematical laws, making them tractable. My experience with Haskell monads is that you generally build a monadic sub-language in a single module and provide libraries for it in some other modules (e.g. Parsec), and that the end result is intuitive and simple to use. But maybe I’ve only been exposed to well-designed monads.

On the subject of state and informal reasoning: personally I use whatever reasoning forms that will work. In debugging a particularly complex monad I once resorted to writing out the algebraic substitutions long-hand in order to understand how the bind and return operators were interacting. It worked, and I got the monad to do what I wanted. I routinely use informal algebraic reasoning of this sort in simpler cases in order to understand what my program is doing. Any informal reasoning must be a hasty short-hand version of what a full formal proof would do, and it follows that language features that make full formal proof easier will make the informal short-hand mental reasoning easier too.

Pure functions are particularly valuable when trying to understand a large program because you don’t have to worry about the context and history of the system for each call; you just look at what the function does to its arguments. In a real sense this is as big a step forwards as garbage collection, and for the same reason: any time you overwrite a value you are effectively declaring the old value to be garbage. Functional programs (at least notionally) never require you to make this decision, leaving it up to the GC and compiler to figure it out for you based on the global system context. Thus complex design patterns like Memento and Command are rendered trivial or completely obsolete.

Finally you talk about the many over-hyped technologies in this industry. Yes, hype is a common problem. Those of you who think you have a silver bullet are very annoying for those of us who actually do.

Paul.
Paul Johnson Says:
December 13th, 2006 at 7:34 pm e
Jonathon Duerig:

I had decided not to respond to any further comments and instead get on with my next article. But yours is long and carefully argued, so it merits a response regardless. Its also nice to be arguing the point with someone who knows what a fold is.

You make the point that during maintenance the difficulty of later understanding the purpose of an operation is language independent. I’m not so sure. A maintainer may suspect that a C++ iterator is truly orthogonal, but it can’t be taken for granted. There may be a bug hiding in those methods, or perhaps someone fixed a bug or worked around a problem by tweaking the semantics in an irregular way. Also a lot of the understanding of a piece of code comes from context, and it helps a lot to be able to see all the context at once (why else would 3 big monitors be a selling point for coding jobs?). So terse code makes it a lot easier to deduce context because you can see more at once.

(Aside: I remember in my final year project at Uni going into the lab at night because then I could get two VT100s to myself).

You say that Scheme, Ruby and Haskell can be moulded to fit the hand of the user, making them more productive for single person tasks, but less productive for groups because of mutual comprehension difficulties.

This is hard to test because of the lack of statistics, but Haskell is strongly typed and the community has already developed conventions and tools for documentation and testing (Haddock and QuickCheck). I can see that Scheme macros can be used to construct an ideosyncratic personal language, but I really don’t see how this could happen in Haskell. Things that get done with macros in Scheme are usually done with monads in Haskell, but whereas Scheme macros are procedural monads are declaritive and must conform to mathematical laws, making them tractable. My experience with Haskell monads is that you generally build a monadic sub-language in a single module and provide libraries for it in some other modules (e.g. Parsec), and that the end result is intuitive and simple to use. But maybe I’ve only been exposed to well-designed monads.

On the subject of state and informal reasoning: personally I use whatever reasoning forms that will work. In debugging a particularly complex monad I once resorted to writing out the algebraic substitutions long-hand in order to understand how the bind and return operators were interacting. It worked, and I got the monad to do what I wanted. I routinely use informal algebraic reasoning of this sort in simpler cases in order to understand what my program is doing. Any informal reasoning must be a hasty short-hand version of what a full formal proof would do, and it follows that language features that make full formal proof easier will make the informal short-hand mental reasoning easier too.

Pure functions are particularly valuable when trying to understand a large program because you don’t have to worry about the context and history of the system for each call; you just look at what the function does to its arguments. In a real sense this is as big a step forwards as garbage collection, and for the same reason: any time you overwrite a value you are effectively declaring the old value to be garbage. Functional programs (at least notionally) never require you to make this decision, leaving it up to the GC and compiler to figure it out for you based on the global system context. Thus complex design patterns like Memento and Command are rendered trivial or completely obsolete.

Finally you talk about the many over-hyped technologies in this industry. Yes, hype is a common problem. Those of you who think you have a silver bullet are very annoying for those of us who actually do.

Paul.
Toby Says:
December 15th, 2006 at 7:36 am e
Since I happened to stumble upon an actual Dijsktra cite just now, I thought I’d add it here (having read and appreciated your original post a few days ago).

In EWD513, “Trip Report E.W. Dijsktra, Newcastle, 8-12 September 1975,” he writes,

“The willingness to accept what is known to be wrong as if it were right was displayed very explicitly by NN4, who, as said, seems to have made up his mind many years ago. Like so many others, he expressed programmer productivity in terms of ‘number of lines of code produced’. During the discussion I pointed out that a programmer should produce solutions, and that, therefore, we should not talk about the number of lines of code produced, but the number of lines used, and that this number ought to be booked on the other side of the ledger. His answer was ‘Well, I know that it is inadequate, but it is the only thing we can measure.’. As if this undeniable fact also determines the side of the ledger….”

That is the edited version as printed in “Selected Writings on Computing: A Personal Perspective”. The original text can be found in the EWD Archive, at http://www.cs.utexas.edu/users/EWD/transcriptions/EWD05xx/EWD513.html

Original responses to '"No Silver Bullet" and Functional Programming'

(((
These are the comments originally appended to the article. Many of them are thoughtful and worth reading in their own right.

As with the article, please do not submit this to reddit.
)))

Chris Morris Says:
December 7th, 2006 at 6:51 am e
” experts in several different languages were given a simplified spec taken from a real problem and asked to implement it.”

So, was the use of Haskell an integral part of simplifying the original spec given to the experts in the first place? To me, the essential difficulties of programming are in figuring out what the hell to really build in the first place. None of the programming languages in that study helped them do that. They skipped the bulk of the essential difficulties and went right into studying the effects on the accidental portion.
Craig Says:
December 7th, 2006 at 7:41 am e
I really enjoyed that, looking forward to your next blog.
Functional Lover Says:
December 7th, 2006 at 8:21 am e
You know why? Because all the colleges have been brainwashed by Java. Damned be they.
Paul Johnson Says:
December 7th, 2006 at 8:26 am e
There has been some discussion of this at http://discuss.joelonsoftware.com/default.asp?joel.3.424533.12

Paul.
Jonathan Allen Says:
December 7th, 2006 at 9:04 am e
From the study referenced:

> 1. The NSWC experiment was conducted in a very short time-frame with very little direct funding. Thus many corners had to be cut, including significant simplification of the problem itself; the largest of the prototypes was only 1200 lines of code.

> 2. The geo-server specification was by nature ambiguous and imprecise, thus leading to some variation in the functionalities of the developed prototypes. (On the other hand, the specification is probably typical of that found in practice, especially during requirements acquisition.)

> 3. The participants were trusted entirely to report their own development metrics. In addition, not all of the participants attended the first meeting at NSWC; those who did attend were advantaged.

> 4. No guidelines were given as to what exactly should be reported. For example, development times usually included documentation time, but there were major variations in the code-size/documentation-size ratios. In addition, different methods were used to count lines of code for each language, and it is not clear how one line of code for one language relates to that of another.

> 5. The review panel had very little time to conduct a really thorough review. For example, none of the code was actually run by the panel; they relied on the written reports and oral presentations.

The problem was grossly simplified, the code was rushed, the line count and time spent numbers questionable, and the final application never run.

This is your proof?

We don’t even know what those 85 lines of Haskel actually work with good inputs, let alone if they correctly responded to bad data.
Mike Griffiths Says:
December 7th, 2006 at 10:50 am e
I think you re missing the point with regard to the whole process of software development. Even orders of magnitude improvements in the process of writing code have no impact upon the problem of writing the right code - defining right as “fit for the required purpose” once the problem domain exceeds a remarkably small limit.
v Says:
December 7th, 2006 at 1:22 pm e
While the arguments you present in the bulk of your post are convincing, the hard data presented hardly seems so. How does low LOC = high productivity except in the wierd world where CMM makes sense?

What gets put down as code is but the distill of all the knowledge stored in a programmer’s head, and functional languages do demand a higher overhead of that storage space than imperative ones (more for historical reasons, i admit, but there it is).

Moreover, code building tools reduce the effort to actually input those high LOCs with automation that is improving by the day.

Can you present a better quantitative metric than LOC counts for functional languages being an order of magnitude better?
thomas lackner Says:
December 7th, 2006 at 3:07 pm e
You should link to information about the STM implementation in Haskell.. sounds very interesting.

I refuse to comment on the constant debate about programmer efficency, as I’m sure the same program would be 72 characters in K/Q!
hxa7241 Says:
December 7th, 2006 at 3:27 pm e
I have recently translated a minimal global illumination renderer from C++ to OCaml. It is about half the size — similar to the Ruby translation. The difference is in having class interfaces defined separately, and other small things.

The implication that functional is so much more compact and simple than imperative is wrong. Look at the granularity of abstractions: they are practically the same. If you still code with the same sized building blocks it will demand similar effort.

Also, well-made imperative code has many restrictions on state interaction. It is not so far away from functional code.

The only forseeable way to greatly improved productivity is with reuse: more, better libraries/frameworks and ways of using them. (As Brooks says.)
Achilleas Margaritis Says:
December 7th, 2006 at 3:48 pm e
As always, benchmarks are biased.

Getopt is a parser.

You can write a command line parser in a few lines of code in C++ using a library like boost::Spirit. For example (code not tested, just an illustration):

rule letter = range(’a', ‘z’) | range(’A', ‘Z’);
rule digit = range(’0′, ‘9′);
rule alphanumeric = digit | letter;
rule id = *letter << *alphanumeric;
rule num = +digit;
rule cmdLine = *('-' << (id | num));

There are many tasks in which functional programs are shorter and more concise, but these advantages are not due to the functional nature of a program, but due to design choices (better syntax, no semicolons, lambda functions etc). These advantages can happily exist in imperative languages too. The only difference is referential transparency, but, for me, it gets in the way.

Can we please see a MVC application with Haskell where the model is cleanly separated from the view?
Achilleas Margaritis Says:
December 7th, 2006 at 3:59 pm e
As always, results are biased.

The command line parser could be written in C++ with a Spirit-like framework like this:

rule id = letter << *alphanumeric;
rule num = +digit;
rule cmdline = *('-' << (id | num));

The advantages of FP come from concepts that can also be used in imperative languages: lambda functions and closures (Smalltalk/Ruby), simplified syntax (Ruby), etc. On the other hand, in pure FPs many things are unnecessarily complex.
anonymous Says:
December 7th, 2006 at 5:22 pm e
Your argument about the lack of states in functional programs is fundamentally flawed. FP-s appear to have no states simply because they describe what would happen (sort of.) However, in order for a FP to, well, run, someone, somewhere, must pull a lever so that the wheels start turning according to the spec of the FP, and what ticks under the FP ultimately does have state. I am not suggesting that there are no gains from functional programming, but the notion you offer is bogus as it stands.
Edward Ocampo-Gooding Says:
December 7th, 2006 at 5:23 pm e
A popular excuse for why programming folks all don’t immediately jump into functional programming is because it’s not as intuitive a paradigm as the procedural style.

I’d like to see a psychology study that measures programming proficiency vs. time trained with “fresh” students unaware of either paradigm and have two separate groups be trained.
I
You wish Says:
December 7th, 2006 at 5:34 pm e
And yet Lisp, which is considered a functional language, has been around since 1962 (it was spec’ed in 1958), and Brooks doesn’t point to it in his paper. Functional programming isn’t new, and it is certainly not a silver bullet.
Sam Carter Says:
December 7th, 2006 at 8:04 pm e
The reason why the whole world isn’t using functional languages is very simple: shitty library support. The real silver bullet for most programmers is access to large, high quality libraries, with the primary examples being the .NET runtime, or Java’s libraries. Programming tests that require writing 100% of the code from scratch are just flat out inaccurate, because you are measuring the wrong thing. The bulk of modern development time isn’t writing code from scratch. It’s writing code that interfaces with libraries for networking, or XML parsing, or database handling, or whatever (see http://www.joelonsoftware.com/articles/LordPalmerston.html for a fuller treatment of this topic). The functional programming language community is more interested in marginalizing their access to external systems rather than making it easier (research into monads being a great example of that).
Larry O'Brien Says:
December 7th, 2006 at 8:50 pm e
I intended to post the URL of some thoughts on your post, but your comment engine diagnoses dashes in URLs as indicative of spam. (Not so, I think.) Anyhow, http://www.knowing.net/PermaLink,guid,659c9535-6674-48f9-a4f9-8bc34fe724b5.aspx
Paul Says:
December 7th, 2006 at 9:39 pm e
Comparing Haskell to C compares a variety of design parameters at once: strong versus weak-ish typing, garbage collection versus hand collection, high level data types versus low-level ones, different libraries etc.

Based on my experience with functional languages and other high-level languages like Python, Ruby, Smalltalk, Javascript, Perl, C#, REXX etc,, I would guess that those other factors are MUCH MORE likely to be relevant than just the functional programming paradigm.

In addition, comparing lines of code for PROTOTYPING is pretty uninteresting. I’d like to compare lines of code for a functional application used by customers. What if the design decisions in a particular language are focused on robustness and maintainability?
Larry O'Brien Says:
December 7th, 2006 at 10:19 pm e
I’ve posted a reply at my blog. Unfortunately, I can’t paste the exact permalink, which your comment engine wrongly insists are spam.
Indeed You Wish Says:
December 7th, 2006 at 10:19 pm e
Certain problems are more easily solved with certain tools — let’s not fool ourselves, and believe FP solves all problems better.
Jeff Says:
December 7th, 2006 at 10:37 pm e
You can make any language look good by comparing it to C++. If being functional is the magic bullet, then why do stateful languages like Python, Perl, Ruby, Lua, Lisp, etc. always do just as well as Haskell in this sort of omparison?
Stuart Dootson Says:
December 7th, 2006 at 11:29 pm e
Unlike the rest of your commenters (as far as I can tell), I *have* used Haskell for real-world problems, and can confirm that (for the problems I’ve used it for), it is significantly more productive than imperative languages like Java, C++ or Ada.

I don’t think it’s quite ready for the maistream, but it’s definitely got promise.
Not Silver But Still Very Good Says:
December 8th, 2006 at 12:54 am e
I’m very sceptical about LOC as a measure. However, here’s a very recent informal data point, which I’m mentioning partly because it’s notable for the relatively controlled nature of the comparison: Lucian Wischik at Microsoft recently had to rewrite a 6,000 line F# program in C# (C# was required for unrelated reasons). It became 30,000 lines (references to comments by Lucian below). Now, this comparison is with essentially the same .NET libraries (F# adds a few minor libraries), the same (talented) programmer, the same .NET runtime, the same performance profile, the same problem specification, and the C# code has the advantage of being a reimplementation.

See Lucian’s comments on the Google group at microsoft.public.dotnet.languages.csharp at http://groups.google.com/group/microsoft.public.dotnet.languages.csharp/tree/browse_frm/thread/38a144ceaf101030/fb4664a7e4c27cf6?rnum=11&q=F%23&_done=%2Fgroup%2Fmicrosoft.public.dotnet.languages.csharp%2Fbrowse_frm%2Fthread%2F38a144ceaf101030%2F38d9aa4dbda50892%3Flnk%3Dgst%26q%3DF%23%26rnum%3D1%26#doc_814fd26871c2c6b7

More extensive analysis of the differences by Lucian at http://groups.google.com/group/microsoft.public.dotnet.languages.csharp/browse_frm/thread/38a144ceaf101030/38d9aa4dbda50892?lnk=gst&q=f%23&rnum=1#38d9aa4dbda50892
Josh S Says:
December 8th, 2006 at 1:53 am e
“However, in order for a FP to, well, run, someone, somewhere, must pull a lever so that the wheels start turning according to the spec of the FP, and what ticks under the FP ultimately does have state.”

That’s not the point. The point is that an FP utilizes a framework that *safely* translates from an abstract language without a concept of state to a concrete language that is basically nothing but state.

Likewise, object oriented languages do not actually have objects — whether they are detected at compile time or run time, they ultimately map to swaths of memory and functions.

It’s all smoke and mirrors. But it’s the magic of the smoke and mirrors that enhance our productivity. By building the funnel from a “safe”, “imaginary” language to the “dangerous”, “unproductive” one, we eliminate the problems inherent at the lower level.
Joel Says:
December 8th, 2006 at 2:41 am e
Functional programming is not usually adopted because so many real-world systems are almost entirely side-effects. Look at all the things a shell script does; it’s changing system state. What’s that JDBC app doing? Changing database state.

In the world of web applications, you’re always doing I/O. It’s almost all I/O. And that is state (of your I/O streams).

I’ve been looking at Erlang a lot recently. I have a serious jones for its concurrency constructs. They are as awesome as advertised. But man, you can’t do a decent thing with strings in that language. Seriously, it’s not much better than C. Python, Perl, even Lisp can do a lot of manipulation of the strings - things you have to do in parsing protocols and HTML and so on. But Erlang here is almost a non-starter. Ugh!

Then there is the subject of libraries. Some major languages provide many things you don’t have to reimplement. But do the functional languages? Lisp has been around the longest, but try getting a gui library that works on all of (SBCL,CMUCL,CLisp). Or a decent GUI on Erlang at all. Even Paul Graham, champion of Lisp as Solution To Everything, admits that you need massive libraries to be useful - it’s a goal he has for Arc.

At least Haskell has GTK+ bindings. I haven’t explored Haskell enough to comment on it. I just wonder why folks haven’t used it to do Yahoo! Store. If it’s that good, it will make someone money.
fez Says:
December 8th, 2006 at 5:07 am e
I would love to see a shootout between the various web frameworks (including any based on functional languages).

Something like the following:
- well-defined spec
- basic CRUD functionality for the most part
- some integration(s) with a third-party API (thus existing off the shelf libraries will be of use)

Let the best coders from each language duke it out. Record time spent for each segment of app development & log all commits to a Subversion repository.

Have a panel of neutral judges look at not just the time spent but also feature completeness and any other niceties the teams added in the time alotted, and give a score to each team.
Ulf Wiger Says:
December 8th, 2006 at 1:20 pm e
More references were asked for. Here’s one:

“Comparing C++ and Erlang for Motorola Telecoms Software”
carried out by Heriot-Watt University together with Motorola.

http://www.erlang.se/euc/06/proceedings/1600Nystrom.ppt

Two applications were re-written from C++ to Erlang, and one of the applications was benchmarked. The pure Erlang version was 2-3x faster, much more robust, and had 1/3rd to 1/18th the number of lines of code, depending on how you compare the libraries. Detailed analysis suggested:

“- Code for successful case – saves 27%
- Automatic memory management – saves 11%
- High-level communications – saves 23%” (slide 31)

Like all other studies, this one can certainly be debated. Does anyone have a reference to a study concluding that functional programming does NOT lead to a productivity increase?
tndal Says:
December 8th, 2006 at 3:04 pm e
You omitted that ISI Relation Lisp scored the most rapid development time by far: less than half the time required by Haskell.

Although the line count was greater, this version of Lisp cleaned the floor with Haskell as a productivity tool. And Relational Lisp is a mirror of Prolog.
Jason Says:
December 8th, 2006 at 6:42 pm e
Some points:

- LOC does not imply that less effort was involved in crafting the Haskell solution. Just less effort in typing it in.

- Some problems have greater susceptibility to different approaches. One small “real world” problem is completely inadequate.

- Double-blind experiments in this regard are, in fact, impossible, so this will remain a bench-race forever.

—
Jason
Joel Reymont Says:
December 8th, 2006 at 7:51 pm e
Paul, you may be interested in the latest article in my blog. See “Re-birth of a trading platform”. http://wagerlabs.com/2006/12/8/re-birth-of-a-trading-platform

Thanks, Joel
Larry O'Brien Says:
December 8th, 2006 at 7:51 pm e
Jason: “LOC does not imply that less effort was involved in crafting the Haskell solution. Just less effort in typing it in.”

Not so. In industrial systems, lines of code produced per month is essentially constant, regardless of language. Also, defect rates per KLOC is essentially constant, regardless of language. (ref. Capers Jones works on “language levels”) (In small programs, you definitely see greater variation.)

I agree with your other points.
Larry O'Brien Says:
December 8th, 2006 at 7:57 pm e
“Not Silver But Very Good” ref’s Lucian’s claims. The threads have little substance. I’ve been following Lucian’s Website and I feel that a grain of salt is called for. For instance, he wrote an F# program that displays a teapot in Direct3D; fair enough, but he makes it sound like the lighting and manipulation in 3D comes from a few lines of F# when, in fact, the “teapot in a viewport” is canned functionality that can be done in a few lines of _any_ language.
Ben Moseley Says:
December 8th, 2006 at 9:24 pm e
The thrust of your article is very close to the thrust of our “Out of the Tar Pit” paper which investigates FP (along with the relational model) as being very relevant to the Silver Bullet question - http://web.mac.com/ben_moseley/frp/paper-v1_01.pdf .
Not Silver But Still Very Good Says:
December 9th, 2006 at 3:26 am e
Re the teapot - you’ve got the wrong guy: Lucian works at Microsoft, and has never touched a DirectX teapot. I think you’re thinking of Flying Frog consultancy.
Ulf Wiger Says:
December 9th, 2006 at 12:50 pm e
I totally disagree with the idea that LOC wouldn’t matter. It’s not just a matter of the effort of typing in the code. Much of that extra code often represents “unnecessary” detail that distracts the reader, hides the core logic, and which also needs to be maintained. I’ve seen projects that have problems with keeping “boilerplate” code consistent when the system gets complex enough. Some projects resort to using modeling languages that generate the boilerplate. When judged as programming languages, these tools are usually quite crappy, but their supporters defend them based on the opinion that it’s still a lot better than having to code the stuff by hand. But there are good programming langugages that work at the same abstraction level as those modeling tools. We’ve also found that the learning curve for e.g. Erlang is much shorter than for e.g. UML or C++, contrary to the statement that FPLs would be more difficult to learn, or less intuitive.

My own conclusions are based on 10 years of working in and around very large software projects, with code volumes in the order of hundreds of thousand lines, or even millions. I’ve had the opportunity to review several projects using C++, UML, Java and Erlang. I get the feeling that many of the comments above come from very limited comparisons. That’s ok - you have to start somewhere. For me, it took 2-3 months to properly un-learn C++ and OO when I first started with Erlang.
anonymous Says:
December 14th, 2006 at 12:57 pm e
Josh S:

“That’s not the point. The point is that an FP utilizes a framework that *safely* translates from an abstract language without a concept of state to a concrete language that is basically nothing but state.”

Brooks was talking about the complexity arising out of the sheer multitude of possible states (or combinations thereof). In FP, you retain (at least in part) this very complexity in order to be able to produce any useful behavior, and whether this complexity is state-based or not, is totally irrelevant.

What’s left to argue about is whether complexity is significantly reduced by imposing the stateless view, and I for one am still a bit sceptical about any radical claims on that account, especially if well-designed programs/systems from both domains are compared.

Another point that comes to mind is that, paradoxically, FP actually inhibits stateful programming when you need it (and you almost always do when designing and implementing systems,) by making the complexity of combining states explicit. Inherently stateful models, on the other hand, will let you hide some of this complexity, be it at the still-present risk of coming up with something inconsistent.

It pays to note that, compared to the current state of the art in PP/OOP, the FP way of combining state is clunky even with the help of monads and monad transformers, in case you wanted to bring those into the argument.

Silver Bullets Incoming!

((( This was originally posted on December 8th 2006 on Blogthing. It is reposted here because Blogthing seems to be permanently down. The original was posted to Programming Reddit and extensively discussed at the time, so please do not resubmit this version.)))

-----

First, an apology to those who have tuned in to read about the obstacles to technology adoption. I will get to it, but first I wanted to respond to the staggering set of comments to my post about functional programming. In addition to the ones here there is the thread I started at Joel On Software and over 100 posts at programming.reddit.com. Wow!

Some people tried to post here first, but postings are held for moderation even if you don’t get flagged as a spammer. Blogthing is apparently configured to reject URLs with hyphens, even though Blogthing puts hyphens in its own URLs. Anyway, one bit of pond scum posted a bit of JavaScript which automatically redirected a browser to some other site, so be glad you didn’t have to deal with that.

Also I’m sorry I couldn’t moderate more often, but I’m afraid that work and family take priority.

Anyway, on to some responses. I’m going to tackle the more common themes in no particular order:

Lines of code is a lousy metric

I think that Dijkstra had it right: a line of code is a cost, not an asset. It costs money to write, and then it costs money to maintain. The more lines you have, the more overhead you have when you come to maintain or extend the application.

Many people asked about development time, or pointed out that the Relational Lisp version in the US Navy study took even less time than the Haskell. However development time is often not available, when reported it is highly dependent on the programmer in question, and may not always be reported accurately (how do you count the time you spent thinking about the problem when driving to work?).

One study, which I am annoyed with myself for not remembering earlier, did collect coding time and found it highly variable, but on average the terser languages took proportionately less time. Implementations for the specimen problem in that study also exist in Haskell and Lisp. The average Haskell program was 57 lines, the average Lisp program was 119 lines, and the average Python program was about 80 (from the graphs in the article). C, C++ and Java all had averages around 240. The shortest program was in Haskell, with a mere 27 lines. The next shortest appears to be a Python program at about 40 lines.

It was just a toy problem, so not representative

True, but non-toy problems are too expensive to run statistical tests on. The Erlang ATM switch was a non-toy problem, but it was also not a controlled study with a statistically valid number of independent implementations. You can’t have both (unless you are an eccentric millionaire).

As several people pointed out, the Annual ICFP Programming Contest provides a non-toy problem and a tight deadline every year. Its a level playing field: the problems are hard and open ended, and any language may be used. Functional languages always dominate the top ten, and for the last 3 years the winners have used Haskell (although last year it was one of several languages used by the winners).

This kind of problem is obviously suited to functional languages, so they look artificially good

I hope I have cited a sufficiently wide range of problem domains, from text processing to geometry to telecoms, to be able to argue otherwise.

There is nothing in the way that functional programming languages work that necessarily fits them to a niche domain, although they do tend to shine particularly in applications requiring complex algorithms and symbolic rather than numeric computation.

Libraries matter more than languages

Libraries and languages are complicit: they affect each other in important ways. In the long run the language that makes libraries easier to write will accumulate more of them, and hence become more powerful.

There is no sharp dividing line between language and library. For example, Haskell has no loop construct built in to the language. Instead there are functions in the standard library to do the job of loops. Conversely Perl has regular expressions as a built-in type, whereas most languages treat them as a library. Therefore the programming environment as a whole should be judged, with language and libraries combined. I don’t know how well the studies above did this. I do know that the US Navy study did not have STL for C++ because it hadn’t been invented then.

C++ is an old language: you should compare it with modern languages like Python

Fair enough. Haskell is an order of magnitude better than C++, and maybe Java. Against Python it is merely significantly better.

An order of magnitude is a factor of 10, no less

Well, the Wikipedia entry does say about 10. All this stuff is so approximate that anything consistently in excess of 5 is close enough.

You can do functional programming in C++ with Boost

Yes I know. This argument reminds me of the early days of OO, when people argued that OO languages were not necessary because you could do the same thing in C with structures of function pointers. Yes you could, but it was a masochistic exercise. I haven’t played around with Boost, but I suspect the same applies. Trying to retrofit ideas from one paradigm into a language designed for another is generally a bad idea.

Python has functional programming stuff, so why do I need Haskell or Erlang? Anyway, purity just gets in the way.

You want Erlang for big, distributed, highly concurrent systems. There is simply nothing comparable.

If you aren’t persuaded by the scalability argument for purity then try this: the purity of Haskell lets the compiler do lots more optimisations. You could write code in either language to apply a series of maps and folds to a list, but Python would (AFAIK) generate all the intermediate lists, while the Haskell compiler would apply a deforestation transform to eliminate all the intermediate lists and therefore be lots faster. The Haskell compiler can do this because the intermediate functions are guaranteed not to have side effects, and therefore the order of execution can be rearranged at will.

Haskell isn’t ready for prime time yet

True, but its pretty close. The wxHaskell and Hs2GTK GUI libraries are currently being updated, database access works fine, GHCi is getting a debugger, and people are already using it for real work.

I wouldn’t recommend using Haskell on a 50-programmer-year bet-the-company project just yet, but for a small agile project where correctness matters more than execution speed its fine today.

Monday, August 6, 2007

“No Silver Bullet” and Functional Programming

(((This was originally posted on December 6th 2006 on Blogthing. It is reposted here because Blogthing seems to be permanently down. The original was posted to Programming Reddit and extensively discussed at the time, so please do not resubmit this version.)))

-----

This post is about No Silver Bullet by Fred Brooks. If you haven’t already read this paper then trust me, you really should. It says a lot, and it says it far better than I ever could.

In this post I’m going to look at this paper in the context of functional programming. Brooks thesis is that there will be no technology that brings an order-of-magnitude improvement in software productivity “in the next ten years” (i.e. in the years 1987 - 97). This is because most of what makes software difficult to write is “essential” complexity (i.e. inherent to the problem) rather than “accidental” complexity (i.e. bought in by the implementation technology).

Brooks’ career had in fact spanned the adoption of a silver bullet. High level languages had replaced assembler, and led to an order of magnitude improvement in productivity. Brooks argued that assembler has a very high accidental complexity, but high level languages like C and Ada had already disposed of most of this, leaving only the essential complexity. Therefore the big wins had all been achieved, and only small incremental improvements were left.

In fact Brooks has already been proved right. It is now almost 20 years after he wrote, and no order-of-magnitude step has occured in mainstream software. Things have certainly gotten better, and some application domains have improved by an order of magnitude thanks to application frameworks. Ruby On Rails is certainly that much better for writing web apps than raw C on top of sockets. But outside of these niches software is still not much easier to write than in 1987. OO helped, but not by an order of magnitude. Garbage collection also helped, but again not by an order of magnitude.

Brooks argument that only essential complexity was left is founded in the size of the
state space of software. As the data and necessary processing increases, so the number of possible states grows. The data and processing are inherent to the problem, and so are therefore essential
rather than accidental: any technology must grapple with the same complexity.

But now lets look at this argument in the context of functional programming. Back in 1993 the US Navy ran a little test of prototyping languages: experts in several different languages were given a simplified spec
taken from a real problem and asked to implement it. Some results:

Ada 83: 767 lines

Ada 95: 800 lines

C++: 1195 lines

Haskell: 85 lines

So on that particular problem Haskell really was an order of magnitude better than conventional languages. Other studies have confirmed this result, although usually with somewhat lower degrees of improvement. Ulf Wiger wrote a paper called Four-fold Increase in Productivity and Quality about the use of Erlang in an ATM switch, although in the paper Wiger claimed to have seen as much as a 10-fold reduction in line count when C++ code was rewritten in Erlang. The Haskell implementation of getOpt took a mere 13% of the line count of the GNU C version. However you slice it, it looks like functional programming can deliver between 4 and 10-fold reduction in line count.

But is this truly a reduction in complexity, or is it just packing that the same into fewer lines? I believe the former, for two reasons.

First, Wiger reports that defect count fell in proportion to line count (i.e. defects per thousand lines of code was constant, but with fewer lines the Erlang code had proportionately fewer bugs). This strongly suggests
that Erlang was indeed hiding accidental complexity from the developers.

Second, “pure” functional code has no side effects and stores no state. Thus it completely evades two kinds of accidental complexity that are inherent to any conventional “imperative” code:

If the program lacks state, then there is no way that the number of states can become a problem.

The order in which statements are executed, and hence the execution trace of the program, is completely irrelevant. Thus execution order is revealed as a major source of accidental complexity which Brooks mistook for essential complexity, but which is eliminated in pure functional programs.

Of course not everything can be pure functional. Erlang has functions with side effects, and so the order of execution matters for those. But they are only used when necessary e.g. for I/O). If the bulk of the code employs only pure functions then it gains these benefits in proportion.

Haskell goes even further than Erlang. Stateful code is fenced off by the type system using a mathematical construct called a monad, and fragments of monadic code can be passed around and stitched together without the side effects escaping to mess up the pure code. Within a monad most of the usual vices of imperative programming are possible (although an uninitialised variable takes real effort). But like Erlang the bulk of Haskell code does not in fact have side effects, and the resulting lack of state makes Haskell programs easy to understand and safe to modify.

There is one more source of complexity that Brooks did not in fact tackle in his paper: concurrency. In 1987 concurrency was not a major problem in most systems. But today it is becoming a serious headache. Today applications are increasingly distributed and concurrent, and the urrent trend towards multi-core CPUs will make concurrent programming a real imperative for high performance applications. But concurrency is very hard to get right first time, and very hard to debug when you get it wrong. In other words it is a huge source of accidental complexity.

Both Erlang and Haskell tackle concurrency. Erlang was designed for telecom applications, which require both concurrency and reliability in large amounts. Erlang implements a simple message passing model of
concurrency with light weight threads based on Hoare’s CSP formalism. This represented the state of the art when it was designed in the late 1980s and is still far more robust than the Java model of object-as-monitor.

Haskell, on the other hand, has recently introduced “software transactional memory” (STM), which provides transactional semantics for ordinary variables. This was originally tried during the 1990s, but the overhead of tracking every access to every variable made it too slow for practical use. But the Haskell
implementation cleverly exploits both the rarity of state in Haskell programs and the way the type system fences it in. Side effects can occur within a transaction, but only to special STM variables. The type checker guarantees that these side effects cannot escape from the transaction until it is committed. Thus a transaction can be tried, rolled back, and retried as often as necessary without any change in
the behaviour of the program.

Both of these systems are very effective at removing the accidental complexity of concurrency. Erlang
programmers think in terms of concurrent servers and clients exchanging information, where each individual server acts as a transactional machine and no client can ever see the server state half way through a transaction. Haskell provides much more direct transactional semantics, but the overall result is the same: the programmer can ignore the concurrency on their own bit of program without having to worry about the rest of the system.

So: is functional programming an actual bona-fide silver bullet as defined by Brooks? I believe it is.
Functional programming is based on a sound theory of scalable software engineering, and the empirical evidence clearly and consistently supports the theory.

So if its so good, why isn’t the whole world using it? That will be the subject of my next blog.

Sunday, August 5, 2007

Planning to port my old posts

I used to have my blog on Blogthing. However that seems to be more or less permanently down now, so I've retrieved my old posts from the Wayback Machine and will be reposting them here in the next couple of days. I'll also post the comments as separate pages.

Homo Economicus and Other Straw Men

Quite a lot of political arguments seem to depend on whether the economists have got it right. Broadly speaking (and yes, this is a horribly broad generalization with lots of counterexamples) the economists tend to favour more "conservative" policies such as trade liberalisation, low taxation and deregulation, which presents a problem for people who disagree with these policies.

Economics tries to be an academic discipline that is above politics. Whilst it doesn't always succeed, it does favour arguments with intellectual rigour over soundbites and statistical evidence over anecdotes. So if you are a politician or pundit who disagrees with the majority of economists then you have some explaining to do.

The most common starting point for such an attack is the "rational actor" assumption that lies at the foundation of most economic models. This makes a number of assumptions about people:

People only care about money. Actually this is an extra simplifying assumption. The basic rational actor theory supposes that people seek to maximise "utility" (i.e. whatever it is that makes them happy). In practice economists usually have to equate this with money on the assumption that money buys happiness.
They are universally greedy and amoral, and so if someone sees a way to get money then they will automatically go for it regardless of the consequence for others.
They see no difference between options that have the same "expected value". In other words they would see no difference between $10,000 in their pocket and a 10% chance of $100,000.
They are stunningly intelligent and informed, able at the drop of a hat to calculate which of half a dozen cell phone plans (each with its own combination of peak, off-peak, weekend, roaming, subscription charges, bundled handsets and monthly minutes) is going to minimise their phone bill for the next year.

This is not a pretty picture of human beings. It also has some obvious gaps. Why do people give to charity? In some cases they might be buying prestige and status, but thats not a universal explanation. More generally, why do some highly talented people prefer to work as teachers (notoriously underpaid) or go and work helping extremely poor people (some of whom are poor because they are mentally unstable, violent, drunk, or just have very bad personal hygiene). Whatever made Linus Torvalds write Linux it wasn't an expectation of monetary gain. Yes, he's got a job out of it, but there are are less risky and more lucrative ways of exploiting a talent like that than giving something away. For that matter I've written a bit of open source software myself. According to classical economic theory I should have spent that time beavering away at work earning myself a promotion.

The mythical human being who behaves like this is often termed "homo economicus"; economic man. He is such a gross approximation to real human beings that I often think of h. economicus as a stick figure portrait. For those who disagree with the "economic conservative" consensus this is a clear point of attack: demonstrate that economics rests on such shaky foundations and you can cast doubt on the conclusions. However this is very much a straw man attack. Economists use homo economicus because it makes economic questions analytically tractable, not because it leads them to pre-defined answers. In this they closely resemble scientists from other disciplines who will use linear approximations rather than precise models because they are analytically tractable. The key question is whether the analysis is sensitive to the difference between the simplified model and reality.

And the economic models based on h. economicus do seem to match with reality reasonably well. Countries with high levels of taxation, protectionist policies and generous social benefits really do seem to wind up with a lot more slums and shanty towns than countries with low taxation, open trade and limited social support. If protectionist policies were good for people as a whole then an economic embargo ought to be welcomed as an opportunity to build up national industries, when in fact its real impact can be economic destruction.

So I generally take a dim view of political platforms that dismiss economics. Economists are not always right (an old joke says that if you laid all the economists in the world end to end they still wouldn't reach a conclusion). But before I will vote for a politician who disagrees with the mainstream economic consensus I'd want to see a detailed rebuttal explaining exactly why the economists are wrong in this case. And complaining about h. economicus isn't going to cut it.