ISPs and Bandwidth

This Slate article reminded me of some stuff about how the Internet works that isn't widely appreciated. For a few years now ISPs have been complaining about "bandwidth hogs" while at the same time advertising high bandwidths for fixed prices. Comcast has had its knuckles rapped for discriminating against particular traffic. But this hides the real issues, which are to do with the structure of the market for long-haul infrastructure.

Your ISP has to pay for connection to the Internet in exactly the same way that you do: it pays a big telecom company like AT&T or Qwest. The only exception is if your ISP is one of those companies, but even then the retail ISP business will be a separate division that has to "buy" bandwidth from its parent. These "tier 1 ISPs" don't publish price lists, but the general pricing structure looks a lot like the ones described in the Slate article, and for the same fundamental reason: capacity is limited. So your local ISP may buy, say, 1 terabyte per day upstream and 2 terabytes downstream, with any traffic over those thresholds being charged per megabyte.

Aside: Actually the whole thing is much more complicated: a retail ISP will generally buy connections to more than one upstream ISP, which may be either tier 1 or "tier 2" (with long-haul bandwidth but not global presence). Often it will have multiple connections to each of these ISPs, each of which may have different price plans. Any ISP may also have "peering" agreements with other ISPs of the same size. These are free, but are not allowed to carry "transit" traffic destined for anywhere else. Everyone always tries to offload traffic onto someone else as fast as possible, even if the resulting routes are not ideal. Managing this mess to keep the customers happy at minimum cost is a key skill in the ISP business

The retail ISPs are therefore caught between a rock and a hard place. They are in a commodity business, but the traditional retail price plan of "all you can eat at a given bandwidth" doesn't match their cost structure. Its a general rule that if you are in such a market and have a competitor who's price plan does match their cost structure then you are bound to make a loss, because the customers who find you cheaper are going to be the ones who cost you more than they pay, while the ones who would balance this by paying more than they cost find your competitor cheaper, so they go there instead.

Thus ISPs will gradually converge on pricing plans that are simplified versions of the cost structure of their industry. This will probably be based on a combination of peak-time limitations and traffic caps that give people an incentive to shift their heavy usage off-peak. The winners will be the ones who can innovate. The challenges are:

• At any given point in time the network has a fixed bandwidth. Hence the challenge is not to reduce the total amount of data moved but to even out usage. Similarly heavy users are only a problem when they start pushing out other customers who might collectively pay more.
  
• You can't force users to track their usage in detail. Price plans that suddenly cut a customer off until tomorrow (or next month) are scary and unfriendly. Plans that charge extra for heavy usage are even worse, especially for families with teenagers. Throttling is more user-friendly.
  
• Negotiate with your upstream ISPs to bring your costs into line with your pricing structure. They, too, need to shift usage off-peak and will be prepared to offer pricing plans accordingly. However their idea of "off-peak" may not be the same as a retail ISP with lots of home users.
• Transparency will happen whether you want it to or not. At least some end users are smart enough to detect traffic shaping and other tricks, and their results will be picked up by price-comparison sites for everyone else to read.
  

Application-specific traffic shaping (as tried by Comcast) won't work. Customers and regulators both hate it, but more importantly it gets you into an arms race between stealth P2P protocols and your packet inspection software that you can't win. However there is another option: offer your customers the opportunity to do their own traffic shaping. For instance you could have a metered high-priority service for everyday browsing combined with an unmetered low priority service for bulk downloads. The challenge is to give the customer a simple easy-to-use system that distinguishes between the two and (automatically as far as possible) uses the right one.

In many ways the situation reminds me of 1997, when large ISPs first started to deny peering agreements to smaller competitors and made them pay for transit agreements instead. There were many calls to ban the practice, and grave predictions of the "balkanisation of the Internet". But in practice the economics of Metcalf's Law guaranteed that a well-connected network would have more value than a disconnected network, and after that it was just a matter of how that extra value was distributed. I believe that much the same thing will happen with bandwidth. The Internet has most value when its pipes are full of traffic, and if there is demand for more bandwidth then there will be money to be made by providing it. After that, its just a matter of working out who pays how much for what. As long as the market remains competitive it will converge on the optimum solution, probably quite rapidly.

Is Comcast's Packet Spoofing a Federal Crime?

The EFF has gathered evidence showing that Comcast is deliberately disrupting P2P traffic by spoofing RST packets to appear to come from the other end of the connection. See the EFF report for the technical details.

The US Criminal Code Title 18 Part 1 Title 47 Section 1030 covers "Fraud and Related Activity In Connection With Computers". I'm not a lawyer, but here is my understanding of the relevant bits of the statute (quotes from the statute are in italics):

Jursidiction: a "Protected Computer" is defined, amongst other things, as any computer "which is used in interstate or foreign commerce or communication, including a computer located outside the United States that is used in a manner that affects interstate or foreign commerce or communication of the United States". In other words if your computer is on the Internet, even if its outside the US, then its a Protected Computer. That includes anything connected via Comcast, and anything that talks to any computer connected via Comcast.

Offence: there are two things to prove here:

1. That someone employed by Comcast "knowingly causes the transmission of a program, information, code, or command, and as a result of such conduct, intentionally causes damage without authorization, to a protected computer". Damage is defined as "any impairment to the integrity or availability of data, a program, a system, or information". A spoof RST packet instructs the receiving computer to drop a TCP connection, so it is a command that impairs the availability of data. I have no direct evidence that these packets were sent knowingly, but I find it difficult to imagine a scenario in which they were sent by accident.
2. That this action caused "loss to 1 or more persons during any 1-year period [...] aggregating at least $5,000 in value". "Loss" is defined as "any reasonable cost to any victim, including the cost of responding to an offense, conducting a damage assessment, and restoring the data, program, system, or information to its condition prior to the offense, and any revenue lost, cost incurred, or other consequential damages incurred because of interruption of service". This is a bit tricky, but people do value their time. $10 per hour is a pretty low wage, and many professionals charge many times that. If failed P2P connections have cost 500 Comcast users 1 hour each in wasted time then this threshold has been reached. You might also be able to make a case purely on the cost of running a computer and keeping it connected via Comcast. The professional IT people who have taken the time to run tests with packet sniffers could certainly count their time at a professional rate as "responding to an offence" and "conducting a damage assessment". There is also some evidence that Comcast inadvertently disrupted other protocols, including Lotus Notes and Windows Remote Desktop. These are used commercially and their disruption would have real financial impact. So while a detailed accounting remains to be done, it certainly looks likely that the $5,000 threshold has been reached.

Penalty: "a fine under this title or imprisonment for not more than 5 years, or both, [if] the offense was committed for purposes of commercial advantage or private financial gain". Comcast's attempts to block P2P protocols are prompted by their desire to keep costs down while seeming to offer an unrestricted service. That counts as "commercial advantage".

So it certainly looks like a Section 1030 offense has been committed that could see someone put in the pen for five years. Any Comcast customers want to call the FBI?