The next challenge for Linux

I was in the local branch of "Currys" (UK electrical and electronic goods chain) recently and had a look at the line-up of computers. They had some little netbooks, and taped next to each one was a little note saying something to the effect of "This runs Linux, so it won't run Windows software". It was a local version of a wider story about Linux:

• People buy netbooks and then discover that Windows isn't part of the bundle, and they don't like it.

• A teacher found a student handing out Linux CDs and accused him of pirating software.

• A student accidentally bought a Dell laptop with Ubuntu instead of Windows, and complained that she couldn't submit the Word documents required by her college (they say they are happy to accept OpenOffice.org documents) and the installation CD for her broadband service wouldn't install so she couldn't get the Internet (Ubuntu connects to the Internet just fine without it). The local TV station picked up the story as a "consumer rights" case and was amazed to find a virtual lynch mob chasing them for being less than enthusiastic about Ubuntu. They quoted an expert saying that Ubuntu "isn't for everyone" and is more suited to tinkerers than people who just want to get stuff done.

Behind all this is what "everyone knows" about computers (everyone who isn't interested in computers for their own sake, that is):

• There are two sorts of computers: Apple Macs, and PCs.
• Apple Macs run Apple software. PCs run Windows software.
  
• Windows is bundled with PCs. On expensive PCs the bundle might include Office as well.
• If you want extra software, you buy it at a store, take it home and stick the CD in the drive.
  

This is how its been for almost 20 years (25 if you count the MSDOS era). An entire generation has grown up knowing that if its a PC, it runs Windows. They know it in the same way they know the sky is blue: its always been blue.

Of course those of us who use Linux know different. But for most people, Linux is something they might have heard about once or twice, but they didn't pay any attention and couldn't tell you anything about it. Outside its current user base and their immediate circle of friends and family, Linux has zero mindshare.

This is not another lament about Joe Sixpack being too stupid to understand Linux. The problem is not that Linux is too complicated, its that Linux and Windows do things differently. Imagine someone who was raised on Linux; how would they react to Windows? Software installation would seem complicated and geeky, the single desktop would feel claustrophobic, and as for the idea of paying for software...

So I think we need to sort out a message to broadcast to the world and then focus on it. I suggest the following:

• Linux is an alternative to Windows and Apple. It comes in several flavours.
• Linux isn't easier or harder than Windows, but it is different, and it takes a while to get used to those differences.
  
• Linux is bundled with a huge range of downloadable free software, including office, graphics and photographic packages.

So next time you see a story that misrepresents Linux please send a correction with these three points.

Where did all the money go?

I've got a simple question about the Credit Crunch? Where did all the money go?

I've always thought of money as something that is "conserved" in the physics sense, like energy. Money in = money out + money stored. So when someone "loses" money it implies that someone else got it. If I run a company that loses money, what it means is that I'm getting less money in than I'm spending on staff and supplies. But the money hasn't disappeared, its just moved from my pockets to my suppliers (and their suppliers, and so on).

But the financial institutions seem to have had billions of dollars disappear into thin air. They got poorer, sometimes so poor that they went bust, but nobody seems to have become correspondingly richer.

I guess this has something to do with fractional reserve banking, which I know doesn't conserve money. Suppose I start with £1000 and put it in a bank. The bank shows a balance of £1000, but it doesn't just hang on to my money, it loans £500 of it to Joe Bloggs, who spends it on a new TV, and the person who sold the TV also puts the money in the bank. So now the bank has $1,500 on deposit even though our imaginary economy just started with £1,000. So what happens if Joe can't pay the money back?

Can someone enlighten me?

Fundamentalist terrorist assassination plot defeated

Tonight the British people celebrate the defeat of a terrorist assassination plot. A group of disaffected men, followers of a foreign religion and egged on by propaganda from overseas and a radical cleric, planned a spectacular atrocity: they would blow up the entire government in one huge explosion. The plot was foiled by the security services, who, responding to a tip-off, mounted a surveillance operation and arrested one of the plotters red handed as he prepared to detonate the explosives. The rest of the gang were tracked down and shot or arrested when soldiers stormed their hideout. In accordance with an executive order from the head of state the survivors were subsequently imprisoned, tortured, and executed after a show-trial.

This happened over 400 years ago, on November 5th 1605. The plotters were Catholics rather than Muslims, but apart from that the story sounds disconcertingly modern, although we tend not to go in for show trials these days: they went out of fashion with Stalin.

Today the Gunpowder Plot is celebrated as Bonfire Night: Guy Fawkes (the one caught with the explosives) is still burned in effigy, and in some places the Pope gets the same treatment as well. However, over the past 400 years the anti-Catholic content has almost entirely leached away: now its just an excuse to let off some fireworks. A Catholic can join in the celebrations just as I (a Briton) joined in the 4th of July celebrations when I was living in the US.

Maybe one day we will look back on 9/11 the same way. I certainly hope so.

Petition on Broadband Advertising

I've become rather depressed about broadband advertising. I found it very difficult to discover the web page describing my own ISPs traffic policy. Furthermore if you compare it to this version from January on the Wayback Machine you can see that the headline speed on the "Large" package has gone up from 4Mbit/sec to 10Mbit/sec, but the evening download limit has merely increased from 800MB to 1,200MB, while extra restrictions have been added starting at 10am as well. Somehow that 10MBit/sec upgrade doesn't feel so generous now: when I want to install Fedora 10 I'd better make sure I schedule the download to start after 9pm.

Virgin Media, in common with most other ISPs, says that the few people who make heavy use of their broadband link reduce capacity for everyone else, and that providing truly "unlimited" service for these few would mean higher bills for everyone. They do indeed have a point. If I want truly unlimited service then I'm sure I can get it, for a price. And the fact is that I probably don't want to pay the price. The occasions when I do want a multi-gigabyte download are sufficiently rare that I can put up with scheduling around Virgin's traffic limits.

So why am I depressed about it? Its because my problem with finding Virgin Media's traffic management policy is not unusual. A sample taken from Google found:

• TalkTalk have a 40GB monthly limit right up there on the front page, although I'm not sure if you can find out how much of your limit you have used this month. What happens when you use it all? Presumably you are cut off until next month.
  


• O2 claims "unlimited" usage, but if you search the site for the word "unlimited" you find that excessive use at peak times will lead to warnings and then account termination. They don't define "excessive" or "peak time". I couldn't find a link to this information from their list of broadband features; it certainly wasn't obvious.
  


• Tiscali have a similar policy, except that after three warnings they limit your peak time speed. They also don't define what "excessive" and "peak time" mean, and they don't say how long this will last or what the speed limit will be. This page was two not-very-prominent links away from the package features list.


• Fast actually sells a range of monthly capacity limits. They warn you by email when you hit 90%, and when you go over 100% they throttle you down to 100kBit/sec. Full marks!

So out of 5 providers (including Virgin) we have only one that makes all the limits and policies clear up-front while three try to hide behind weasel words and hard-to-find web pages. This makes it difficult for consumers to figure out what they are buying. Is Tiscali's unstated policy better or worse value for money that one of the Fast packages? Even if its better at the moment, will it still be so next month? As a consumer I have no way of knowing. Worse yet, if I were a technically naive consumer I might not even realise that the question needed asking. "Unlimited" sounds much better than "40Gb monthly limit".

I think that something needs doing about this. However the Advertising Standards Authority have wimped out: they have declared that "unlimited" actually means "95% of users don't hit the limit in any given month". So its hardly surprising that people are confused.

Therefore I have started an on-line petition at the Number 10 web site. If you are a British citizen or resident then I urge you to sign it. The petition calls on the Government to require ISPs to make all caps and limits on their services a prominent part of their advertising. Only when they do so will consumers have a clear choice between different packages.

SSDs and the return of the root partition

Back in the days when Unix was new to the world computers came with two sorts of mass storage: fast-small-expensive, and slow-big-cheap. A minicomputer typically had one of each, so that frequently-used files could be kept in the fast unit while everything else was kept on the slow one.

Unix was designed with this in mind, which is why to this day most programs on Unix and Linux are found in one of two directories: /bin holds the frequently used ones, and /usr/bin holds the less frequently used ones. A similar arrangement applies to libraries, which are held in /lib and /usr/lib. The idea was that the "root" partition was put on the fast small device, and the "user" partition went on the big one. This led to some other restrictions as well; /bin and /lib together have to have all the software needed for a minimally functioning system, because until all that stuff is going the system can't mount /usr to get at the rest of it.

All that became irrelevant in the age of the Linux PC because the entire system hung off one physical disk. But maybe that is about to change. Solid State Disks (SSDs) are entering the disk world. They have zero mechanical latency and their read speed is blisteringly fast. Write speed is more variable, but the new Intel product is reported to be very fast. It is also small (80GB) and expensive ($600).

So it looks like the time has once again arrived for keeping the commonly used programs and libraries on a small fast disk and everything else on something big and slow. The original root versus /usr division doesn't work for this of course: people want the system to fire up X windows, KDE or Gnome and a bunch of apps, not just a shell prompt. But all of those things will fit very nicely onto one of these SSDs. Better yet, the user files have now been shifted out from /usr (from which it got its name) and put in /home. So this needs zero surgery to the standard Linux layout.

For the average user $600 is a bit steep though. What is actually needed here is something smaller and cheaper. I run a well-populated version of Fedora, and my root partition with all the Fedora packages is under 10GB. So I while I can't imagine blowing $600 (or whatever that is in £s these days) on 80GB of SSD, I could certainly imagine putting a 10GB version in for $100. That would hold the Linux kernel, X Windows and all the apps, with enough room left over for /etc. That would increase the speed of booting up, logging in and starting applications without greatly increasing the cost of the system, especially since that often involves reading lots of different files scattered around the disk. Even better, most of this stuff is only written rarely, and then mostly in the background by system updates. So the current poor write performance of most SSDs doesn't matter.

(Update: I have of course forgotten that by default Unix and Linux keep track of the access time (atime) of files, which means that every time you read something a little metadata write happens. So write performance does matter here).

Looking around, the nearest product to this specification seems to be the OCZ 32GB unit, which retails for a bit over £100 (say, around $170). Thats bigger and more expensive than what I want, but hey, Moore's Law seems to be working so I'll just wait a bit. I also note that all these products seem to be targetted at the laptop market, which makes sense given SSDs other advantages of low power consumption and physical robustness. Maybe they should start thinking about the market for hybrid desktop systems as well.

Why the banks collapsed, and how a paper on Haskell programming can help stop it happening next time

Trading Risk

The financial system exists to trade three kinds of thing: money, commodities and risk. Money and commodities are the easy bit. Either I have £1,000 or I don't. Similarly with commodities: either I have a barrel of oil or I don't. But risk is a very different matter. In theory risk is easy to quantify: just multiply the probability of something by the cost, and you have the expected loss. But in practice its not so simple because the probability and cost may be difficult to quantify, especially for rare events (like, say, a global credit crunch). Many of the factors that go into a risk model are subjective, so honest people can have genuine disagreements about exactly what the risk is.

The Slippery Slope

Unfortunately risk assessment is not value-neutral. Risk has negative value: you have to pay people to take it off you. The higher the risk, the more you have to pay. And because the amount of risk is always debatable this is a very slippery slope; the people paying others to take the risk away have every incentive to present a lower estimate. Everyone can see that everyone else is doing the same, and so methods of hiding or downplaying risk migrate from dodgy dealing to open secret to standard practice.

Specific examples abound throughout the recent history of the finance industry;

• The retail mortgage houses that originally lent to "sub-prime" clients would hire valuers who were known to be on the generous side with their valuations. So any valuer who wasn't so generous found their income drying up. Background checks on clients were cut back, then eliminated. Eventually borrowers were simply told what income to claim on the forms, regardless of what they actually earned.
• These loans were then bundled up and sold. The idea was that the buyers would each get a share of the incoming loan repayments. Rights to this stream of money were divided into "tranches", the idea being that, for instance, Tranche 1 would get the first 34% of whatever money was due, Tranche 2 would get the next 33%, and Tranche 3 would get the last 33%. When some borrowers defaulted (as some always do), Tranche 3 would lose out first, then Tranche 2. Tranche 1 would only fail to get all their money if the overall repayment rate fell below 34%, which had never happened. The game here was to persuade a credit rating agency that Tranche 1 was so safe that it was worthy of a "Triple A" rating, because that meant that banks, insurance companies and similar big financial institutions could legally buy this debt without having to put cash aside to cover potential losses. The rating agencies earned fees for evaluating securities, so just like the house valuers they found it paid to be on the generous side.
  
• All these institutions had risk management departments who were supposed to watch out for excessively risky behaviour. But in practice they found it very difficult to blow the whistle. Risk managers tell stories of being given two days to review a deal that took ten people a month to negotiate, and of accusations of "not being a team player" when they questioned over-optimistic claims. This story from the New York Times has some details. Look through the comments after the story as well; many of them are by people with their own tales of risk.

None of this is new; similar behaviour has contributed to past financial crises. In theory more regulation can prevent this, and everyone is now planning or demanding lots more regulation (even the banks). But in practice regulation has failed repeatedly because the regulators always concentrate on the way it went wrong last time. Regulations have to be written carefully, and the companies being regulated have to be given time to understand changes and put their compliance mechanisms in place. This prevents regulators from moving as fast as the institutions they regulate.

The regulators also don't have visibility of the information they need to assess systemic risk. Systemic risk arises because financial companies are exposed to each other; if one institution fails, others have to write off any money it owed them, possibly pushing them into bankruptcy as well. Regulators try to make companies insulate themselves by avoiding excessive risk and keeping some cash on hand, but without a clear picture of the risks being run by each company they have no way to tell if this is enough.

The basic problem, I believe, is the food chain of risk management within each institution. At the top are the negotiators and fund managers who design and package the securities. Then the lawyers are bought in to specify the precise terms of the deal. Somewhere along the way the "quants" will be asked to develop mathematical models, and at the bottom coders will be given the job of turning the models into executable code that will actually determine the real price and risks. It is this food chain that needs to be rethought, because its hiding important information.

This 2000 paper by Simon Peyton Jones, Jean-Marc Eber and Julian Seward shows a way forwards. It describes a Domain Specific Language embedded in Haskell for describing the rights and obligations imposed by a contract. Arbitrarily complicated contracts can be built up using a small collection of primitives. Aggregations of these contracts can also be created, as can risks of default and bankruptcy. This created quite a stir in the quantitative analysis world when it was presented, as it was the first time anyone had proposed a formal language for describing contracts. Today the list of commercial Haskell users includes a number of financial institutions using this kind of technique to model their market positions.

But on its own this is just a faster and more efficient way of getting the same wrong answer. It doesn't solve the underlying problem of concealed systemic risk. The solution has to be for big financial companies to reveal their positions to the regulators as formal models of the contracts they have written. At the moment they don't even have to reveal all their contracts, but merely knowing the legal terms of a contract is only the first step. Those terms have to be converted into a mathematical model. That model probably already exists, but only as an internal artefact of the parties to the contract. What ought to be happening is that the contract is specified in a well-defined mathematical language that can be converted into a model automatically. If the regulators have this information about all the contracts entered into by all the finance companies then they can model the impact of, say, a downturn in the housing market or a jump in the price of oil, and if they see systemic risk looming then they can order the companies involved to take corrective action. Unlike the various Risk Management departments they will be able to see the whole picture, and they don't have to worry about being "team players".

ISPs and Bandwidth

This Slate article reminded me of some stuff about how the Internet works that isn't widely appreciated. For a few years now ISPs have been complaining about "bandwidth hogs" while at the same time advertising high bandwidths for fixed prices. Comcast has had its knuckles rapped for discriminating against particular traffic. But this hides the real issues, which are to do with the structure of the market for long-haul infrastructure.

Your ISP has to pay for connection to the Internet in exactly the same way that you do: it pays a big telecom company like AT&T or Qwest. The only exception is if your ISP is one of those companies, but even then the retail ISP business will be a separate division that has to "buy" bandwidth from its parent. These "tier 1 ISPs" don't publish price lists, but the general pricing structure looks a lot like the ones described in the Slate article, and for the same fundamental reason: capacity is limited. So your local ISP may buy, say, 1 terabyte per day upstream and 2 terabytes downstream, with any traffic over those thresholds being charged per megabyte.

Aside: Actually the whole thing is much more complicated: a retail ISP will generally buy connections to more than one upstream ISP, which may be either tier 1 or "tier 2" (with long-haul bandwidth but not global presence). Often it will have multiple connections to each of these ISPs, each of which may have different price plans. Any ISP may also have "peering" agreements with other ISPs of the same size. These are free, but are not allowed to carry "transit" traffic destined for anywhere else. Everyone always tries to offload traffic onto someone else as fast as possible, even if the resulting routes are not ideal. Managing this mess to keep the customers happy at minimum cost is a key skill in the ISP business

The retail ISPs are therefore caught between a rock and a hard place. They are in a commodity business, but the traditional retail price plan of "all you can eat at a given bandwidth" doesn't match their cost structure. Its a general rule that if you are in such a market and have a competitor who's price plan does match their cost structure then you are bound to make a loss, because the customers who find you cheaper are going to be the ones who cost you more than they pay, while the ones who would balance this by paying more than they cost find your competitor cheaper, so they go there instead.

Thus ISPs will gradually converge on pricing plans that are simplified versions of the cost structure of their industry. This will probably be based on a combination of peak-time limitations and traffic caps that give people an incentive to shift their heavy usage off-peak. The winners will be the ones who can innovate. The challenges are:

• At any given point in time the network has a fixed bandwidth. Hence the challenge is not to reduce the total amount of data moved but to even out usage. Similarly heavy users are only a problem when they start pushing out other customers who might collectively pay more.
  
• You can't force users to track their usage in detail. Price plans that suddenly cut a customer off until tomorrow (or next month) are scary and unfriendly. Plans that charge extra for heavy usage are even worse, especially for families with teenagers. Throttling is more user-friendly.
  
• Negotiate with your upstream ISPs to bring your costs into line with your pricing structure. They, too, need to shift usage off-peak and will be prepared to offer pricing plans accordingly. However their idea of "off-peak" may not be the same as a retail ISP with lots of home users.
• Transparency will happen whether you want it to or not. At least some end users are smart enough to detect traffic shaping and other tricks, and their results will be picked up by price-comparison sites for everyone else to read.
  

Application-specific traffic shaping (as tried by Comcast) won't work. Customers and regulators both hate it, but more importantly it gets you into an arms race between stealth P2P protocols and your packet inspection software that you can't win. However there is another option: offer your customers the opportunity to do their own traffic shaping. For instance you could have a metered high-priority service for everyday browsing combined with an unmetered low priority service for bulk downloads. The challenge is to give the customer a simple easy-to-use system that distinguishes between the two and (automatically as far as possible) uses the right one.

In many ways the situation reminds me of 1997, when large ISPs first started to deny peering agreements to smaller competitors and made them pay for transit agreements instead. There were many calls to ban the practice, and grave predictions of the "balkanisation of the Internet". But in practice the economics of Metcalf's Law guaranteed that a well-connected network would have more value than a disconnected network, and after that it was just a matter of how that extra value was distributed. I believe that much the same thing will happen with bandwidth. The Internet has most value when its pipes are full of traffic, and if there is demand for more bandwidth then there will be money to be made by providing it. After that, its just a matter of working out who pays how much for what. As long as the market remains competitive it will converge on the optimum solution, probably quite rapidly.