Fastly claims innocent customer action triggered a ‘software’ bug that caused the internet to crash
A client of the cloud services company Fastly inadvertently caused the fall of thousands of pages and services on the Internet this Tuesday at 12 noon, Spanish time. The failure occurred when the user changed their service settings completely correctly. However, his action triggered a computer error that had been hidden in the software of the company and that sank 85% of its network. Fastly’s Vice President of Engineering and Infrastructure, Nick Rockwell, has given the details of the fall in a message published this Wednesday by the company.
Fastly deployed software new May 12. Hidden there was a computer bug that could only “be triggered by specific client settings and under certain circumstances,” Rockwell writes. That set of coincidences occurred on June 8. It is as if there is an imperceptible hole in the structure of a building and shortly after finishing it a resident hangs a painting right there and causes everything to fall apart.
Fastly found the bug at 11:48, one minute after the disaster started, according to Rockwell. After 10 minutes they had already published the first statement. At 12:27, always Spanish time, they found where the client’s configuration problem was and another 10 minutes later their services began to recover. It was the longest half hour in Fastly’s history, the day that his company, vital to many media and internet services, became famous.
Fastly is a company called a content delivery network (CDN). Its function is to distribute services on the internet to thousands of local points around the world to bring the pages closer to the end users.
“Although it was specific conditions that caused the crash, we should have anticipated it,” says Rockwell. “We provide critical services and treat any action that causes service problems with the highest sensitivity and priority,” he adds. Fastly is along with Cloudflare and Akamai one of the few truly global companies that sustains the internet. As it was seen this Tuesday, the global structure of the network depends on several levels of few providers.
This decentralization already occurs and is even promoted among companies, as Andy Champagne, vice president of Akamai, Fastly’s key competitor, says in commenting on Tuesday’s decline: “The use of multiple CDNs is something that has become more and more common, “he says in a statement shared by his company. “In theory, multiple CDNs provide companies with a backup in case one of them has a problem, but it is not a bulletproof strategy. The same rigor should be applied when hiring vendors even when using multiple CDNs, as the division does not guarantee 100% uptime for customers. If a CDN suffers an interruption, manual intervention is usually necessary to direct the traffic to another CDN, it is not something automatic ”.
Apart from the fact that the service would continue to be distributed among the dominators of the market, the problem Champagne refers to is that building a structure of software For CDN customers to redirect traffic in a drop of less than an hour, requires considerable effort. And, until there are worse falls, it comes out little to account.