In addition we will have the opportunity for those who have travelled a great distance to introduce themselves to the group, and raise ad hoc issues of interest to Peering Coordinators.

From bill.norton@gmail.com Fri May 13 10:10:09 2005
Date: Mon, 9 May 2005 14:43:31 -0700
From: William B. Norton 
Reply-To: wbn@equinix.com
To: nanog@merit.edu
Subject: Peering BOF IX at NANOG in Seattle - The Great Public vs. Private
    Peering Debate


Hi all -

Just wanted to invite you all to the upcoming Peering
Birds-Of-a-Feather session at the upcoming NANOG, and give you a
flavor of a couple of the topics to be discussed...

Peering Introductions
-------------------------------
For Peering Coordinators who would lke to introduce themselves to the
North American Peering Coordinator Community, we have some time set
aside for you to introduce yourself and share with the Community:
Company Name and Contact Information,
AS#, 
Where you are peering or expect to peer in the future, 
What you are looking for in a peer, and 
Why people should be interested in peering with you.

We have found these face-to-face interactions helps facilitate
peering, particularly for folks coming in from overseas. It helps to
bring network maps, and lots of business cards. If you email
wbn@equinix.com this information I will make a slide with your contact
information on it that will show up behind you are you speak.

The Public vs. Private Peering Debate
--------------------------------------------------------
We have recruited two Peering Coordinators to articulate the two sides
of the Great (Public vs. Private) Peering Debate. They have graciously
agreed to share with the audience the Strongest Arguments for Public
Peering (Maurice Dean), and the Strongest Arguments for Private
Peering (Peter Cohen). Each side will have a few minutes to present
their case, then a few minutes to attack the claims made by the other
side and/or reinforce their own side of the argument as needed. We
will perhaps add a few minutes in the middle for a couple of limited
scope audience questions; those to help the speakers clarify a point
(no speeches or attacks here). Both sides will then summarize their
argument and the audience will be asked to vote for which side made
the more compelling case.

Audience Discussion
-------------------------------
As we did last Peering BOF, we will open the floor to discussion,
focusing on points that one or both sides failed to make, or failed to
make strongly enough, that would have perhaps made a difference in the
audience vote.

Background
------------------
I researched this issue with a subset of the Peering Coordinator
Community and shared the early results at the RIPE EIX WG meeting. If
the discussions there are any indicator, I think we are in for an
interesting and educational community discussion here.

Below is an excerpt of the public vs. private peering arguments I
heard from the Peering Coordinator Community and shared at the RIPE
EIX WG meeting.  I agree with the Peering Coordinators who believe the
answer for most ISPs is a hybrid of public and private peering.  I
also agree that perhaps there sometimes emerges a transition based
upon scale and strategic intent, but we will see what the community
comes up with at the BOF.

Bill

PS - I cut and pasted the text below from the "The Great (Public vs.
Private Peering) Debate: Peering at 10G" white paper that I am using
to document these debates as they relate to 10 Gigabit-per-second
Ethernet Peering.  I am still looking for reviewers to provide
feedback here BTW...If you are interested in this stuff and can spend
a little time to provide feedback, send email to wbn@equinix.com. 
When I feel more comfortable that I have it right, I will make the
paper freely available to anyone who would like a copy.
-----------------------------------------------------------------------------------
     :

     :
The Top 4 Reasons Public Peering is better than Private Peering

1. Aggregation Benefits

a.       A network can easily aggregate a large number of relatively
small peering sessions across a single fixed-cost peering port, with
zero incremental cost per peer. (Private peering requires additional
cross connects and potentially an additional interface card, so there
are real costs associated with each incremental peering session.)
Small peering sessions often exhibit a high degree of variability in
their traffic levels, making them perfect for aggregation. Since not
all peers peak at the same time, multiple peers can be multiplexed
onto the shared peering fabric, with one peers peak traffic filling in
the valleys of another peer's traffic. This helps make peering very
cost effective: "I can't afford to dedicate a whole gigE card to
private peering with this guy, but public peering is a no-brainer."

b.       Public peering ports usually have very large gradations of
bandwidth: 100Mbps Ethernet upgrades to 1Gbps Ethernet, which upgrades
to 10Gbps Ethernet. With such large gradations, it is easier for
smaller peers to maintain several times more capacity via public
peering than they are currently using, which reduces the likelihood of
congestion due to shifting traffic patterns, bursty traffic, or
uncontrolled Denial of Service attacks. "Some peers aren't as
responsive to upgrading their peering infrastructure, nor are they of
similar mind with respect for the desire for peering bandwidth
headroom[1]." The large gradations of public peering bandwidth help
reconcile these two issues.

2. Ease of administration

a.           Public peering is the easiest and fastest way to both
turn up and turn down a peering sessions, since no physical work is
required. Peering is soft configured by the two parties on the router
and the peering session is up.

b.          It is common for a network to set up a trial peering
session to determine the amount of traffic that would be exchanged
should a session be turned up. If there is public peering capacity
available, there is no incremental cost or extra administrative work
required to turn up a trial peer, and the information gathered may
prevent choosing an incorrect private peering port size if the traffic
is moved to a private peer later.

c.           Many Peering Coordinators must work within a budget, and
do not have decision making authority for purchases within their
company. Once the public peering switch port is ordered, there is no
additional cost and therefore no additional hurdle to peering for the
Peering Coordinator.

d.          Public Peering provides financial predictability. The
hardware requirements and monthly recurring costs of peering are the
same every month[2]. This makes planning and budgeting much easier.

e.           10 Public Peering scales large peering sessions (those
greater than 1Gbps) seamlessly, while private peering beyond gigE
capacities requires private peering at 10G (very expensive), or
connecting multiple gigEs together, which can be tricky[3].
3. Public Peering is used as Selection Criteria by Customers

a.     Corporate and Enterprise customers continue to ask to see the
list of the ISP's public peering points[4].
4. Public Peering May Be the only Cost Effective way to Peer across
multiple Colos

a.     Across Europe, where public peering across multiple collocation
centers is the norm, private peering is often a much more expensive
solution. Purchasing private peering circuits within a metro is
potentially very expensive, while the same traffic can traverse a
shared peering fabric for much less.

The Top 5 Reasons Private Peering is better than Public Peering
---------------------------------------------------------------------------------------------
Here are the strongest argument private peering advocates shared with
the author.

1. Private Peering Sessions are Easier to Monitor 

a.        SNMP Counters can be easily collected on each peering port
to monitor the utilization of the Peering Session resources. No time
intensive Netflow or expensive network analysis software[5] is
required to sort through shared peering fabric data to determine
per-peering-session traffic volume.

b.       Greater Visibility: No Blind Oversubscription Problem. With
public peering, the remote peer could be congesting his port with the
other peering sessions and you have no visibility into their public
peering port utilization. Packets could be dropped due to port
oversubscription resulting in poor peering performance. Since Private
Peering involves only the two parties, when the port reaches an agreed
upon utilization (say 60% utilization for example), both parties can
see that it is time to upgrade the peering session.

2. Private Peering is Very Cost Effective 

a.     If an expected peering port and cross connect costs were $400
per month and the parties expected to send 40Mbps to each other, the
EPPR would be $400/40Mbps=$10/Mbps, a very attractive price in today's
transit market.

b.     For those who exchange traffic with a few large peers, the
80%/20% rule applies; the majority of peering benefits can be derived
by peering with the 20% of potential peers that deliver 80% of your
traffic. This suggests fewer larger peers is preferable over picking
up lots of small peers across a public peering fabric.
3. Private Peering is more reliable and easier to debug. 

a.     Private Peering involves fewer network components that could
break.[6]  It should be noted that this argument weakens when the
"private" peering are provisioned across VLANs, though optical
interconnects, telco provisioned SONET services, or other active
electronics.

b.     An architecture of private peering removes the variability of
support processes across IXes[7]. Across Europe, each IX is different,
and a NOC Operator may need to understand the processes, the levels of
support and debugging capabilities of the switch support staff on call
at the IX, and may even need to craft NOC scripts to navigate through
the IX operations tasks. A private peering architecture provides
consistency that helps the NOC debug and fix things more rapidly.

c.    The greater fear is that layer 2 fabrics could be connected
through other layer two fabrics perhaps without the knowledge or
consent of the peer, resulting in a very difficult debugging and
diagnostics situation if a peering failure occurs.

4. Private Peering Sessions are More Secure

a.           A private peering network that is directly connected only
with those with whom there is an explicit peering arrangement is more
secure than a network that connects to a public peering fabric that
includes participants with whom there is no relationship with the
company. There is some history here; early exchange points were places
where "traffic stealing" was accomplished by pointing default at an
unsuspecting and poorly secured public peer. Other problems included
peers tunneling traffic across the ocean across a peer's network.
These things are explicitly disallowed in most peering and IX terms
and conditions and can be further secured through filtering, but are
still seen as potential hazards minimized by privately peering.

b.          An architecture that solely privately peers is less likely
to be compromised. Since fiber has no active components that can be
administered, there is nothing that can be broken into.  With a switch
or other active electronics in between peers, there is the possibility
that traffic can be captured at the peering point without their
detection. It is relatively easy to mirror a public peering port as
compared with tapping into private peering fiber cross connects
without the detection of the peers involved. A few ISPs pointed to
technology that can passively tap into fiber interconnects, which if
true, would decrease the strength of this argument.

5. Private Peering Inclination Signals a More Attractive Peer. 

a.           The "Big Players" privately peer with each other and some
even loath Public Peering Fabrics for historical reasons. Adopting
this attitude puts one in the company of the largest Tier 1 ISPs in
the world. "For certain very large networks, public peering makes no
sense at all. For certain very small networks, public peering may make
perfect sense[8]."  Or put more harshly, "if you think that public
peering is a good idea, you're just not large enough yet[9]."

________________________________

[1] James Rice (LoNap), formerly engineering for BBC Internet.

[2] Modulo the incremental costs for hardware at the cost steps
described earlier and the pricing increases at the end of contract
terms with the IX Operator. It was pointed out during conversations at
the RIPE 50 meeting that the LINX charges a metered rate beyond the
flat peering fee, and that some IXes have a fractional gigabit
Ethernet rate that causes pricing steps.

[3] Patrick Gilmore (Akamai) in conversation 4/7/2005 regarding load
balancing across multiple cross connects.

[4] Frank Orlowski (T-Systems) in conversation 4/8/2005.

[5] Some of these tools cost $50,000 to license!

[6] Remco Donker (MCI) and Nina Hjorth Bargiser (Tele Danmark) point
out that some people view large capacity public peering as too risky;
losing a single large public peering port would cause massive
disruption to the infrastructure, and would result in much larger
convergence times than if a single private peering port went out.

[7] Falk Bornstaedt and Frank Orlowski (T-Systems).

[8] Richard Steenbergen conversation  3/23/2005

[9] Anonymous

From bill.norton@gmail.com Fri May 20 09:36:37 2005
Date: Thu, 19 May 2005 11:53:04 -0700
From: William B. Norton 
Reply-To: wbn@equinix.com
To: nanog@merit.edu
Cc: bill.norton@gmail.com
Subject: Peering BOF IX Meeting Minutes

Peering BOF IX Meeting Minutes              Seattle, WA  
May 16, 2005   7:30-9:00PM

...Transcribing my notes here for those who couldn't make the Peering
BOF... comments/corrections to wbn@equinix.com or wbn@umich.edu

A 5 Minute Plea for Multicast Peering
-------------------------------------------------
Celeste Anderson (ISI/Los Nettos) spoke about the need for greater
peering of native multicast routing. The audience commented that the
volume of multicast traffic was still relatively small (a few T1's
worth) in comparison to unicast traffic, and that there is still not
significant customer demand to do so. We discussed the chicken and egg
problems for a bit.

Transit Survey
------------------------
A few people in the community suggested it would be good to get a
sense of transit prices, to see if they were continuing to fall,
remaining constant, or were rising back up.  In previous white papers
I had polled the Peering Coordinator Community to get a sense of
transit prices for comparison against peering costs, so with this
previous data and an audience survey we to take a stab.  I asserted
that with transit you often get what you pay for, and we were not
interested in identifying the "lowest price", the "bottom feeder"
price points that have no customer service, and that there is in fact
a qualitative difference between transit services.  I was looking for
approximations of "Tier 1 ISP" pricing – as expected, there was some
debate here.  To avoid the rathole of "Tier 1" etc. and to focus on
"quality" of transit services for a quick price measure I pulled
together some questions to disqualify some upstream providers from the
survey:

Q1: When you complain about packet loss to your Upstream, you are told
A)	I have Bob working on it, he will call you back in 15 minutes
B)	Call this other number (which rings busy)
C)	…Let me put you on hold for an hour(Barry Manilow sings the songs
that make the whole world sing)
D)	You should have sent your packets earlier

I told folks if they answered "C" or "D" to please not answer the poll
– we were not interested in bottom feeder pricing for this poll.

Q2: My Upstream Provider's Reliability is
A)	6 9's
B)	Close to 6 9's
C)	Closer to 9 6's
D)	Don't know – I'll let you know when they stay up long enough to measure.

I told folks if they answered "C" or "D" to please not answer the
poll– we were not interested in bottom feeder pricing for this poll.


Q3: My Upstream Provider's Customer Service Makes Me Feel…
A)	Like a VIP
B)	Like a Nuisance
C)	Very, very unclean.
D)	Like I was just traded to another inmate for 2 packs of menthol cigarettes.

I told folks if they answered "C" or "D" to please not answer the
poll– we were not interested in bottom feeder pricing for this poll.

Source: A couple of these questions/answers came from a Wired magazine
article I read on the flight up, which referenced
http://www.5ives.com.

While not a particularly scientific survey, we did get 28 viable data
points with the averages being:

10Mbps commit: 	$41.75/Mbps
100Mbps commit: 	$33.83/Mbps
1000Mbps commit: 	 $20.73/Mbps
10000Mbps commit: 	 $13.80/Mbps

About 6 months or so ago I did a similar poll with the Peering
Coordinator Community and the #s were around $85, $60, $45, $30,
indicating that perhaps the transit prices have indeed fallen. If any
of you are interested in the raw data (thanks to Eric Troyer for
putting the data points into a spreadsheet) let me know and I'll email
you the spreadsheet.

Peering at 10G – introduction to The Great (Public vs. Private) Peering Debate
----------------------------------------------------------------------------------------------------------------
The motivation for the Great Debate was a white paper I am working on
called "The Great (Public vs. Private) Peering Debate: Peering at 10G"
in which I documented the Peering Coordinator Community assertion and
the math that the next best alternative to 10G Public Peering was many
1G Private Peering sessions.  I walked through the models for both
cases (10G Public and 1G Private) peering, specifying sample equipment
(thanks ras!), cost estimates, etc. so we could compare the two
alternatives side-by-side. There were many valid points raised during
this discussion including:
1)	In the modeling we were not including backbone costs, only IX and
Peering equipment fees at one location,
2)	There are other valid configurations of equipment, including
removing the peering router and using switches by themselves (debate
here as well),
3)	There were a wide variety of redundancy configurations that
could/should be considered as well in both cases.
The nice thing about modeling this stuff is the ability to identify
and discuss assumptions, to change #'s, to have something on the table
to discuss, and it was a lively discussion before and after the BOF.

The punch line for the current model I shared was that public peering
at 10G and private peering at 1G both are very cost effective
solutions, yielding less than $10/Mbps with a few gigabits of peering.
Also, the more traffic peered, the more the cost of these two models
of peering converged – so Public vs. Private becomes an architectural
/ religious issue.

This leads to the Great Debate – when do people prefer Public Peering
and when do they prefer Private Peering, and why?

The Great (Public vs. Private) Peering Debate
-------------------------------------------------------------------
I recruited Maurice Dean (who was the Peering Coordinator at Global
Crossing and now at Google) to present and defend "The Strongest
Arguments for Public Peering", and Peter Cohen (Peering Coordinator
for Telia) to share "The Strongest Arguments for Private Peering". We
used a modified Oxford Style Debate, providing each side three minutes
for opening remarks, three minutes each to attack the other side's
points and defend their own, a few minutes for the audience to ask
clarification questions, and finally two minutes for the debaters to
present summary remarks. The audience would then vote for "Who made
the more compelling case."

Maurice started out by pointing out that IXes today are not the IXEs
of 1995, but rather are large and reliable infrastructure that provide
access to a rich population of potential peers. Public peering
provides instantaneous access to peers, provides aggregation benefits,
ease of administration and is easy to scale peering capacity. He
pointed also to the marketing benefits of public peering at the IXes.

Peter opened by stating that Private Peering provided superior control
over peering infrastructure. He went on that Private Peering scaled
very well, that the same fiber pairs could be reused as both sides
simply upgraded cards. Private Peering is simpler according to Peter
since no NetFlow was needed, that there were not 20 peers all peering
across the same 10G interface. For some he claimed ISPs, the only way
to measure public peering traffic volume to a peer was to turn down a
peering session and see the difference in traffic load.

Maurice countered reinforcing that modern IXes have overcome the head
of line blocking issues of the past and indeed scale very well, and in
fact scale well even across a metro area. He defended the NetFlow
attack by mentioning that some IXes have sFlow, providing public
peering visibility on a per-peer traffic volume basis.

Peter countered that Maurice was a "Stooly for the IXes", and
depending on a single port for peering is asking for trouble. Further,
participation in some IXes requires going to member meetings, and who
has time for that. Finally that the added complexity of adding a
public peering switch was problematic – he used the analogy of air
travel. "How many of you prefer a direct flight over one with
connections?"

The audience asked a variety of questions next but I did not jot them
all down. A few points did come up though – that circuit upgrades for
private peering can take 6 months or more depending on which side is
in a hurry to upgrade. Another point was that public peering was seen
by some as easier to justify to their company since it would be used
across many peers.

The closing remarks were a rehash of the points above, but Maurice
introduced a couple additional points – that 90 day trials could be
effectively done over a public peering fabric without additional costs
to those attached, and that public peering fabrics enabled more
opportunities (for peering? For add'l services? For transit
sales?...not sure what he meant here). Peter reasserted that placing
all the eggs in one public peering port was just asking for trouble.

The audience next voted "Who presented the more compelling case?"

We had to count twice because it was so close. The final results:
Public Peering presented the more compelling case: 37
Private Peering presented the more compelling case: 33

Maurice Dean won the Great Peering Debate.

White Paper Available
--------------------------------
I captured these and other arguments for and against private peering
in the white paper mentioned above – I would love to have a few more
reviewers give me some feedback/edits/etc. If you are interested in 10
Gigabit Ethernet Peering or the debate discussion, send me an email
(wbn@equinix.com) with the Subject: "The Great (Public vs. Private)
Peering Debate – Peering at 10G" and I'll send you a copy. If you
provide comments on the draft I will add you to the acknowledgements
section of the next version of the document.

We were running out of time so we moved quickly to Peering Personals,
giving Peering Coordinators a chance to introduce themselves to the
group.

Peering Personals – (jotted down everything on the sheets)
--------------------------------------------------------------------------------------
Korea Telecom – 4766 – Bong-Hwa Song – 40 Gbps Capacity, peering in
Seattle PAIX, Palo Alto PAIX, EQLA, LINX, AMSIX, songbh@kt.co.kr

AARNET – 7575 – Mark Prior – Research Net, Seattle SIX, PAIX, LAIIX,
LAAP, FixW, P|Wave, mark.prior@aarnet.edu.au

Netflix – Vish Y – vish@netflix.com – Selective peer

Steve Gibbard – PCH – 42, 3856 – Data Collection – Open Peering 0 25
locations all over the world (SIX, EQLA, LAIIX, NYIIX, PAIXNY,
EQASH,NOTA,…)

Steve Gibbard – Hurricane Electric – 6939 – scg@stevegibbard.com,
peering@he.net – 2700 routes, public but migrating to private at
EQSJO, EQLA, EQCHI, EQDAL, EQASH, PAIXPA,NYIIX, LINX, AMSIX

Brokaw Price – Yahoo! – 10310 – OPEN – peering@yahoo-inc.com

Todd Underwood – 64597 – todd@renesys.com – NOTA, LINX, Goofy peering,
route collection only, store updates,

Google – Maurice Dean – 15169 – selective peering –
peering@google.com, NYIIX, EQ-CHI, EQCHI, EQASH, NOTA,
PAIXPA,PAIXVA,1118th, TelX (60 Hudson)., LINX< AMSIX, LAIIX

Akamai – 12222 – Patrick Gilmore – 20940 outside U.S., OPEN and at
every IX, content heavy, peering@akamai.com

ESNET – 293 – Yvonne – ipv4, ipv6, multicast, EQASH, EQSJO, MAEEast
MAEWest, PAIX, AADS, FixW, Pacific NW Gigapop, peering@es.net,
selective peering policy with AUP

Celeste Anderson – AS4, AS27, AS2152/2153, ISI/Los Nettos, research
net, lots of eyeballs, LAIIX, LAAP, Pacific Wave, PAIX LA, Mostly
OPEN~selective, eyeballs, peering@pacificwave.net, celestea@usc.edu,

Matt Peterson – SixApart , AS# pending, SIX, EQSJO, LiveJournal.com,
matt@sixapart.com
--------------------------------------------------------------------------------------------------------------
Hope this help!

-- 
//------------------------------------------------
// William B. Norton  
// Co-Founder and Chief Technical Liaison, Equinix, Inc.
// GSM Mobile: 650-315-8635
// Skype, Y!IM: williambnorton

Bill's minutes from the BOF (MS Word)
Bill's transit survey (Excel)
Bill's slides from the BOF (PDF)