The importance of secondary routes

Network World is reporting that a fire caused when a homeless man threw a lit cigarette onto a mattress under a bridge has taken down Internet2 access between Boston and New York.

Yes. A burning mattress took down an important link for a major high speed network. No, this probably wasn't specifically covered in their design and operations risk assessment.

While high speed networks are expensive, this does demonstrate the vulnerability that purpose built dedicated links suffer from. If you only have one link, either because of cost or because of specialization, you need to have plans in place for when it goes down. Copper and fiber aren't invulnerable, and even when you think you're safe you can still get hit. Just when you think it is safe to cross your physical paths, someone will go dig there with a backhoe and cut your fiber.

Two stories come to my mind in which relatively unlikely events threatened or took down Internet access.

In the first, a semi hauling a backhoe went underneath a bridge that was lower than the backhoe's retracted and stored arm. The high speed impact with the bridge severed the fiber running underneath it, cutting off Internet access to a large chunk of Michigan.

In the second, a crew working on a a sewer line hit a gas line. In the process of attempting to fix the gas line, they dug and hit a fiber conduit. Fortunately, the slack in the fiber allowed it to pull just enough to remain operational, but a series of unfortunate events might have resulted in loss of Internet access for a major institution - in addition to a pretty nightmarish repair scenario with fiber and gas lines both broken.

Lessons learned? Always ask about single points of failure, identify alternate routes if possible and financially reasonable, and make sure you have an outage handling and recovery plan.