I was recently reminded of an interesting routing snafu that I experienced a few years ago which led me to believe that prepending is stupid and should generally be avoided – especially when dealing with routing on the Internet.
AS path is a mandatory BGP attribute making it present in all route advertisements. When a router sends a BGP update to a neighbor in a different autonomous system, its ASN is added to the front of the path. This process continues on and on to the ends of the Internet. It’s not uncommon for large organizations with their own IP address space to have BGP peerings with multiple providers. Let’s illustrate the scenario with the following diagram. I was in ASN 1. The rest of the networks represent the collective known as the Internet. You may have heard of it.
I worked in ASN 1 and for redundancy’s sake, the data centers had BGP peerings to two different ISPs. If the peering to ASN 2 went down, any best path to our network that was learned through ASN 2 was simply updated to the path learned through ASN 3. This is what I refer to as “letting the Internet decide.” We weren’t modifying any BGP attributes in our announcement so, in theory, the shortest AS path should always be the winning path. Simple enough.
We were presented with a situation that required us to remove all incoming traffic from the connection to ASN 2. It was suggested that we prepend our ASN in the announcement to ASN 2. Prepending the AS path in a BGP announcement simply makes that path longer, therefore making it less preferred for any given number of ASNs on the Internet. In this example, we’ve prepended twice to influence a shorter best path from different ASNs.
In theory, prepending twice to ASN 2 would make the path from ASN 11 through ASN 3 more preferred since 2 3 1 is shorter than 2 1 1 1. So, in order to completely remove incoming traffic through ASN 2, I just needed to apply a bunch of prepends on that announcement, right? Should be simple, but no matter how many prepends I applied on the announcement to ASN 2, some traffic kept coming into our network through that connection. It became apparent that certain ISPs were using their own policies to influence local preference in order to direct traffic where they wanted it to go.
Since local preference is higher on the BGP food chain than AS path, traffic was still coming in through ASN 2 even though I had applied up to 10 prepends on the announcement. In the end, ASNs 2 and 4 were to blame (and possibly others, but these are the two I observed with this behavior). ASN 2’s default local preference for customers was 100 but their non-customer default was 86. Basically what they were saying is that they didn’t care how we modified our AS path on announcements to them, they were going to send traffic destined to our prefixes through our direct peering to them because that’s better than going anywhere else. Period. So any traffic destined to our network that originated in either ASN 2 or any other network that peered exclusively with ASN 2 would use our direct peering with ASN 2 because that’s what ASN 2 had decided. It didn’t matter that the path was longer because the local preference on routes learned over our direct peering was 100 while it was 86 on paths to the rest of the Internet.
ASN 4 had, for whatever reason, decided to only accept 80 or so routes from ASN 3 – none of which were our routes so that path wasn’t an option anyway. For other unknown reasons, they had also decided to set local preference to 70 on routes learned from ASN 5. Perhaps ASN 4 viewed ASNs 3 and 5 as inferior peering partners or didn’t want to send traffic to them unless no other option existed. I’m not sure what ASN 4’s policies looked like to their other peerings, but it was clear that they wanted to send traffic to our prefixes through ASN 2 regardless of what I did with prepending.
The only way I was going to be able to completely remove traffic from our connection to ASN 2 (without shutting it off) was to simply have them make the local preference on routes learned from us lower than their non-customer default of 86. In this case, ASN 2 had a community which told them to set local preference to 70 on any routes learned from us. By doing this, they would start preferring one of their non-customer paths for our routes. In other words, any traffic destined to us that originated in their network was sent to a ASN 3. The key point about manipulating local preference is that it doesn’t provide multiple paths. By telling ASN 2 that our path was inferior to other paths from which it was learning our route, they stopped advertising it to the rest of their peers because BGP won’t advertise an inferior route. If they advertised our route to anyone else, it was from a path learned through a different ISP and not from their direct peering with us.
Simply put, you simply can’t trust to completely control your traffic with prepending because of what other network operators might be doing to manipulate the path within their own networks. If you truly want to remove traffic from a link, use communities to set local preference.