Someone Broke The Internet – Major Network Issue

September 2, 2015

On Thursday August 27th, 2015, Volico started receiving calls from clients regarding a network issue around 2:55PM EST. Volico Engineers were immediately notified and investigation of the network problem was immediately escalated to top priority.

Volico Engineers noticed that multiple eBGP sessions where flapping from multiple providers. Volico immediately began opening upstream support tickets to figure out if the issue was localized to South Florida. While speaking to an upstream provider, we were alerted that a major Tier 1 provided is currently causing routing issues across North America and that multiple Tier 1 providers are currently showing eBGP Prefix List fluctuations confirming the eBGP flapping that Volico was seeing.

Upon further investigation using www.downdetector.com, it immediately became apparent that this internet issue was not isolated to South Florida but was now considered a Nation Wide Event. Volico began further investigation and measures to limit routing to carriers that did not appear to be affected. During the mitigation, Volico began witnessing that around 3:25PM EST; eBGP sessions began showing stability across the board and that networks across the United states began to stabilize.

In conclusion, the event appears to have three major ROOT causes.

1) Level3 conducted a network configuration change in the Chicago Area causing a nationwide instability in the Level3 Network.

2) Time Warner Cable was also performing a maintenance window that also caused a network outage and fluctuation of traffic across their network further saturating other Tier 1 providers.

3) Century Link conducted an emergency shutdown of one of their Core facilities in the New York area due to a Catastrophic Cooling system shutdown causing a major Century Link Network hub to go offline and further reroute traffic to other Tier 1 providers.

By this time, The network instability across the entire United States was now affected.

We have provided images below from www.downdetector.com showing the network outage Heat Maps affecting every major population zone across America. Please note that these are only a few of many carriers that were affected.

Everyone here at Volico would like to thank it’s customers for their help and dedication during the event.

http://www.neowin.net/news/level3-suffering-major-outages-effects-are-felt-worldwide

https://www.reddit.com/r/networking/comments/3imwrc/level_3_internet_outage/

http://www.neowin.net/news/level3-suffering-major-outages-effects-are-felt-worldwide

https://downdetector.com/status/time-warner-cable/news/82051-problems-at-time-warner-cable-2

https://downdetector.com/status/comcast-xfinity/news/82025-problems-at-comcast-2

Time Warner: http://money.cnn.com/2014/08/27/media/time-warner-cable-outage/

Century Link: http://www.awakenedwire.com/hfts-are-overheating-centurylink-reports-catastrophic-hvac-failure-at-nj2-data-center-starting-safe-shut-down/140585?utm_source=dlvr.it&utm_medium=twitter

Ready to See How Volico Can Help You?