North American Network Operators Group|
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical
Did we forget that ISP's are businesses?
As Randy pointed out, this conversation has been fairly clue free. Working as a peering coordinator for a large ISP I can tell you that most of the posts in this thread have been so wrong it makes me laugh. ISP's are businesses, and let me tell you that peering is no exception. People seem to like to think that "settlement free interconnections" are "free", but nothing could be further from the truth. You have to buy routers and the cards that go in them, provision transport services to the peering location, and then on to the peer. Provide enough backhaul in your network from where your customers are located to where the peers are located. You have to pay lawyers to review contracts, engineers to configure and troubleshoot sessions, and managers to negotiate the whole deal. The budget for "settlement free interconnections" at a major ISP can run into the millions of dollars. Two major ISP's may have 8xOC-48 between them. That's probably $200,000 in router costs alone. Yes, sometimes you can get a $200 cross connect, but sometimes you also have to have the $6k/month circuit, for each one, creating a $42,000/month cost. That's a half million dollars a year. When you look at the requirements, geographic diversity, volume, ratio, number of routes what is really happening is the companies are trying to make sure there is some balance of costs. It doesn't have to be a 50/50 split...everyone uses their own assets to reduce their costs, but there has to be some equality. Personally, I'm not a fan of the technical requirements to make them equal, but rather like to negotiate equality, but everyone has their own approach. Back to the issue at hand. What I can tell from this situation is that Level 3 thought they were paying a lot of costs for very little return on investment. The idea that Level 3 would take this action because Cogent was selling cheaper seems a bit far fetched to me. Level 3 knows this is going to hurt their customers as well. Indeed, I'll bet this went all the way to the Level 3 CEO for approval first, because they knew their was going to be pain. This isn't some router cowboy going nuts in the middle of the night, this is a business backed into a corner. Why? We'll never know the real story. Maybe L3 is paying for third party circuits because cogent doesn't touch their network and it's costing them a boatload. Maybe L3 wanted to move to GigE peering for cheap high density ports, and Cogent wouldn't budge from using OC-3's because their routers don't have great GigE density. Maybe traffic between the two has dropped to 20 megs, and so L3 doesn't think maintaining ports is a justified cost. Maybe the ratio is 20:1, and that was finally enough for L3 to feel they were carrying too much of the transport cost. Most likely it's a combination of all of these issues. Bottom line is some engineer had to dress up and go over to the land of suits and explain to them that yes, Level 3 was going to totally screw their own customers, but it was also going to save $X, where $X is probably fairly large, and so they really had no choice. Least I seem like I'm on Level 3's side, it may well be that they have high costs due to their own stupidity. Perhaps they cut a deal with Equinix for $5,000/month cross connects. Perhaps they pay list price for their routers. Perhaps they are about to go down the tubes. As for those who want to re-architect the Internet to "fix" this problem, please go away. It's not a technical problem. It's a business problem. Two companies, each responsible for their own bottom line couldn't find an economically feesable way to interconnect. Any attempt to "force" the interconnection (either via regulation, transit through third parties, etc) will RAISE prices for all involved. The key here is that the peering was not economically viable for some reason. It will be interesting to see how this is resolved in the end. As time passes, there will be increased pressure on both companies to fix this problem. Single homed customers are going to think twice about connecting to either one. My own observations? This appears to be happening to Cogent a lot lately. This makes me wonder if part of the reason they have been able to offer lower costs is by finding ways to take advantage of peers and transfer costs to them which is causing the peers to notice and take action. I also find Cogent's practice of offering Level 3 customers free service unseemly. They are partially responsible for the outage, and are trying to use that fact to lure customers. That makes me wonder if they've written off Level 3 entirely already...after all if you're planning on working out a deal with someone you don't rub salt in their wounds as a first step. -- Leo Bicknell - firstname.lastname@example.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - email@example.com, www.tmbg.org