The central symptom of route instability is the disappearance of a route that previously existed in the routing table. This route might disappear and reappear intermittently, a condition sometimes referred to as flapping. What is happening at the routing protocol level is that BGP sends a routing update and then immediately withdraws it. A router that receives UPDATE or WITHDRAWN messages would have to propagate those messages to its peers. If this behavior continues to cascade, routing performance suffers.

Factors that affect route instabilities on the Internet include the following influences:

•  Instabilities of IGPs

•  Hardware Failures

•  Software Problems

•  Insufficient Horsepower

•  Insufficient Memory

•  Network Upgrades

•  Human Error

•  Backup Link Overloads

Instabilities of IGPs

Dynamically injecting IGPs into BGP can cause unnecessary route flapping. Problems that occur inside a domain can translate into problems outside the domain. As already discussed in Chapter 5, "Tuning BGP Capabilities," static injection of routing into BGP can solve this problem.

Route aggregation at the border routers can also reduce the potential unpleasant side effects associated with IGP injection into BGP. With aggregation, multiple route entries get injected into BGP as a summary aggregate. A route instability in any one element of the aggregate does not affect the stability of the aggregate itself.

Still, some network designers are forced to rely on dynamic routing for valid reasons:

•  BGP implementations can only handle a fixed number of network entries to be advertised statically. The number of static routes permitted varies from vendor to vendor. Whatever that limit is, networks that want to go beyond this limit require that administrators inject the IGP into BGP.

•  Some administrators are not too comfortable with the fact that the networks they are statically advertising might become unreachable by the router advertising them. This is understandable, especially in cases where routes are advertised from different points of the AS. Advertising a route that is not reachable can create black holes.

Hardware Failures

Faulty interfaces, faulty systems, or faulty lines can all affect route stability. An interface that is intermittently available might cause routing information to transition. Hardware failures are, to a certain degree, beyond the control of service users. System and link redundancy are important tools for reducing connectivity loss due to failures, but when a physical failure occurs, routing will be interrupted, and any interruption will initiate some kind of cascade effect down the routing path.

Software Problems

Software problems or "bugs" can cause system failures and network instabilities. Development teams try their best to catch these problems before the software is released to customers. Nevertheless, it is almost impossible to forsee every single situation that might occur in live networks. Administrators should experiment with new software or new features in test labs and low impact portions of their network in order to get some level of confidence before the software is fully deployed.

Insufficient Horsepower

The more routing updates and peering sessions the router handles, the more CPU power is required. Think of the router as your basic 4x4 truck, and think of the routing and traffic overhead as the load you carry. Would you be surprised if the truck has trouble moving with a 20-ton load? Picking the correct system with the correct CPU power is very important to satisfy your particular routing needs.

At the initial stages of building BGP tables after the BGP sessions are established, a system's processor can spend more than 90 percent of its time processing updates. When links become unstable and overloaded, the router might end up in a race condition: the CPU is too busy handling updates, which causes BGP sessions to drop, which in turn causes more instabilities.

Insufficient Memory

In addition to the memory needed by a router to run its own operating system, a router must store routing tables, cache tables, databases, and the other bits of software to permit operation. A router that reaches its memory limit might stop functioning, which causes all routes it knows of or advertises to be lost.

In BGP terms, a routing entry consists of the entry in the IP forwarding table and whatever corresponding information is available in the BGP routing table. Today, the Internet has reached more than 42,000 routes. Systems that are taking full routes from the Internet from a couple of providers are barely keeping up with 32 MB of memory. Most providers have upgraded their systems to 64 MB and even 128 MB.