4 Big Examples of Backup Failures (and What You Can Learn from Them)
The Average Joes of the world aren’t the only people who experience data loss and backup failures. You would assume the majority of scenarios involving data loss do some from the typical tech users you see in coffee shops or riding the subway to get to work or school, and you’d be correct -- because the common man is the biggest demographic buying technology now, of course their backup failures make up a majority of the cases and examples involving data loss.
However, sometimes the most harrowing stories about backup failures don’t come from Facebook posts or Twitter rants. They come from press releases put out by big name companies or startups that actually have a vested interest in keep data safe but, for some reason, couldn’t follow through.
Ultimately, think about it this way: if it can happen to the big guys, it can happen to you. Here are four big instances of backup failures from legitimate companies and what you can learn from their major mistakes.
Perhaps the most recent example as of the time this blog was written, GitLab went through something of a backup failure disaster. The website went offline “due to issues with [their] production database” in late January 2017 after their production data was accidentally deleted. As reported by The Register, a folder of data nearing 300 GB was accidentally deleted. By the time the command prompt was canceled, less than 5 GB remained. Despite promising users that they were working on getting a backup up and running, the process was slow going. It took the site about 48 hours to become fully operational after relying on a database copy.
Outside of the obvious lesson of “make sure you have a backup plan,” always make sure you’re sure of what you’re deleting before you delete it!
Never heard of them? The reason why is that the SaaS provider lasted less than six months after being hacked; a fate not uncommon in the SMB world. Code Spaces went under thanks to a breach using its Amazon Elastic Compute Cloud control panel, where hackers deleted their data and their backups, as well as their offsite backups. Despite reacting timely and changing passwords, the hackers had already created backup logins and made sure the company was incapable of continuing to operate.
The lesson here is also simple. While it’s convenient to have all of your backups in one place, always make sure you have multiple backup locations that aren’t all interconnected. Keep your security up and always backup to multiple sites or locations.
Prior to 2007, DreamHost was the most trusted web hosting provider out there. This all quickly changed when users informed the platform that 700 websites and 3,500 FTP accounts had been compromised by an unknown entity. While the ordeal was long and involved, the crux of their dilemma was discovering that there was significant data loss in their nameserver database. Even after using scripts to try and fix the problem, DNS records were still being deleted, causing many to lose their websites and experience massive amounts of downtime. After this, the culprit was apprehended and the hosting site apologized to its base.
Take a note out of DreamHost’s book -- if at first you don’t succeed in figuring out your data loss problem, try, try again.
Last but not least, the most major company in tech that you would never expect to be on this list is, unfortunately, on this list. Most companies suffer data breaches or losses because of human error or lazy security efforts. Google, however, went through a data disaster because of natural causes.
After lightning hit a local utility grid four different times in Belgium near one of Google’s major data centers, data was eventually lost -- though less than 0.000001% permanent disk space. The good news is that customers didn’t lose data because of data replication, making Google a pretty smart cookie.
What you can learn from this is that no matter how careful you are to not delete anything on accident or get your wires crossed, forces of nature are outside of your control. There’s good news: if you backup your data, it shouldn’t ever be a problem.