Interesting Activities at the Peering Points
Presentation by Naiming Shen, Cisco
During the summer of 1997, when I was working on Internet peering issues
at iMCI, I had a chance to help track down a couple of unusual peering
activities. These involved rewriting eBGP next hops to some NAP routers;
passing third party next hops; pointing default; and registering incorrect
DNS names for NAP routers. Some of these activities were due to a
misconfiguration, such as running IGP protocols over NAP FDDIs or turning
on native IP multicasting on the NAPs.
Case #1: Rewriting eBGP Next hops
At this time iMCI had two routers at MAE-East, called cpe2 and cpe3. I
was informed by the NOC that the cpe2 FDDI inbound was very congested. I
turned on the Netflow feature on iMCI's routers and found that 15% of the
incoming traffic from cpe2 was unaccounted for. This meant that the
traffic was coming from someone iMCI did not peer with at MAE-East.
Further analysis showed that almost all the traffic was coming from a
single subnet.
The BGP routing table showed that that subnet was 3 AS hops away from
iMCI. iMCI had established private peering with ISP-1, ISP-1 peered with
ISP-2, and ISP-2 peered with ISP-3, which owned the subnet. iMCI did not
peer with any of these ISPs at the NAPs, so the only way ISP-3 could point
next hop to iMCI was to rewrite the next hop by matching iMCI's AS number
in its eBGP routes.
I placed an ACL packet filter on cpe2 to block traffic from ISP-3.
Luckily. ISP-3 only had a single block of addresses, which made it
possible to do packet-level filtering. ISP-3 then changed its routing to
ISP-2 and ISP-1. After I removed the packet filtering, however, ISP-3
pointed its traffic back to iMCI again.
I then decided to create something even more interesting. I designed a
filter to let ISP-3 pass ICMP packets, traceroute packets, and DNS
packets, but block all other IP packets, just to add some complexity to
their troubleshooting. The filter was there for four days; apparently they
could not pinpoint the problem, and finally switched traffic back to
normal.
Case #2: Passing Third-Party Next hop
This case was not flagged by the Netflow analysis, because the reverse
path lookup did not fail. iMCI peered with ISP-4 at MAE-East, but not
with ISP-5. ISP-4 not only passed our next hop to ISP-5, but also passed
ISP-5's next hop directly to us. In this way, we exchanged traffic with
ISP-5 directly. We could, of course, manually overwrite all the next hops
to ISP-4's address, but we still could not stop ISP-4 from passing our
next hop to ISP-5. After exchanging some email with ISP-4, they agreed to
fix this. Some might believe that it was more efficient to pass traffic
between third parties over multi-access media. However, I believed that
peering was not just an engineering issue, but also a business issue.
Case #3: Pointing Default
Some routers at the exchange points simply pointed default to others. One
strange example was a router at MAE-East that pointed default to UUnet;
the router name had a reverse DNS lookup of xxx.internetmci.net. UUnet
therefore asked me if this was one of our routers. I knew that we
registered mci.net and internetmci.com, but was not sure about
internetmci.net. A couple of days later, the IXP router pointed default to
us instead of to UUnet. Since they claimed to be MCI, we did an SNMP
query to the router, and it returned:
ip.ipRouteTable.ipRouteEntry.ipRouteNext
hop.0.0.0.0=IpAddress:192.41.177.180 +
The ip address was our cpe3's FDDI address at MAE-East. I also obtained
the AS number of the router, and was able to find out who the router
belonged to. After several email exchanges, the owner changed their
default route so it did not point to us any more, but to someone else ;-)
Somehow, they believed a router had to have a default route.
Other Activities
Other issues were debatable. I found some ISPs running IGPs over the FDDI
at the NAPs. When I sent the ISPs e-mail, they told me that this was due
to a misconfiguration. One could argue this was a way to create
redundancy at the NAPs. But usually an ISP's routers were at the same
location, or on the same rack at the NAPs, so redundancy could be achieved
by using a private LAN instead of bothering other routers.
One or two routers/workstations on MAE-East were using native IP
multicast. I calculated that about 2Mbps of this traffic was going over
the NAP FDDI, but didn't know if any router on the LAN was receiving the
traffic. I talked to the senders, who told me that they planned to shut
down multicast on their box.
I spent a lot of time dealing with route consistency over different
peering points. On the Internet today, we use shortest exit routing. If
the routes we learned from our peers were not consistent across all the
peering points, then we might carry some traffic unnecessarily across our
backbone. Sometime we had to avoid some peering points because of severe
congestion; this was usually done through mutual agreement.
To deal with those unusual activities, we need to detect them, communicate
with the right people, and, sometimes, find a way to stop them. To detect
problems, I was able to use:
Methods you can use to stop unwanted traffic include:
Preventive practices include:
Back to Meeting Topics