It’s a news story that no-one in the industry could possible miss – the huge script error run by 123-reg which accidentally deleted a number of customer sites and cloud servers, with a majority not able to recover their data. It’s an absolutely monumental mistake that just came from a few erroneous lines of code. So what’s the impact actually been, and what can the industry learn from the situation? 

What Actually Happened?

In simple terms, a slip-up by 123-reg means a number of their servers storing customer sites stored on Virtual Private Servers (VPS) have been completely wiped. Gone. The customers’ sites are no more, which leaves companies with no way of dealing with customers or managing internal systems such as employee payments or customer records – some are safe and can be restored, others won’t be so lucky.

By the end of the month, things still weren’t looking for 123-reg and its affected customers; the company have since not released any further updates. Based on information released by the hosting company, they’ve only been able to recover 40% of affected servers – which leaves almost two thirds of customer data currently gone. 123-reg have also admitted that a chunk of this data could be gone for good.

123-reg have tried to appease the vast majority of its other customers by stating that of the 115,000 servers they have in Europe, only 67 were affected. But that ratio doesn’t matter for the customers that are affected – downtime of sites can have big issues for a company, whether it’s an e-commerce site that manages orders or a job agency that uses online systems to manage employee payments. This is more than a website irritating customers with a 404, it’s costing businesses cold hard cash.

How Did It Happen?

123-reg have called out an error in code for a clean-up script as the cause of the issue, although they have said any specific or technical about exactly what was run or what the error was. We suspect, however, that it’s unlikely to be something simple as an undefined variable “rm -rf {foo}/{bar}” Bash script, which has happened with smaller server hosts in the past.

Regardless, the erroneous clean-up script affected a large number of websites for each VPS that was a victim of the incident. One VPS obviously means more than one site, so even a single affected VPS can have huge impacts; 123-reg have said only 67 were affected but some of their customers are not all convinced that this is the full extent of the issues.

What Does It Mean For Affected Businesses?

If you had no backups whatsoever and weren’t among the lucky 40% who had their data restored then that’s the end of it, unfortunately. Get ready to build sites, systems and databases up from scratch – it’s going to be an expensive and time consuming venture, regardless of whether you’re doing it yourself or using a specialist company. Some customers who had backups with 123-reg were a little more lucky, although it wasn’t a given that data could be restored; the offending script did also wipe backups from 123-reg as well.

Regardless of whether your data has been restored or you’re starting from square one, 123-reg have tried to alleviate the situation by offering six months’ worth of free hosting as a gesture of good will. While this is a cost saving gesture for companies still with 123-reg (and some have voted with their digital feed and gone elsewhere), it’s not gone down particularly well with those affected.

Would Separate Backup Servers Have Helped?

In essence, yes. 123-reg backups were themselves affected, and it’s a lesson learned about putting all of your eggs in one VPS basked. Separate backups on other servers hosted by a third party company would mean a guarantee of being able to restore your sites quickly, regardless of the state of your hosting company’s backups. So would a hosting company’s backups not protect you? Not in all cases – for some 123-reg customers the offending script also managed to eradicate any backups as well.

What Can Businesses Learn From The Disaster?

The key message is always keep backups – we can’t stress this enough. It sounds simple, but not all businesses affected by the accidental elimination had backups to protect them. Even then, some of those backups were also wiped out in the clean-up incident; effectively, putting all your eggs in one basket (ie. those who let 123-reg deal with backups as well as basic hosting) won’t always protect you.

External, third party backups are a good way of adding another layer of protection to your sites, databases and files. That way, regardless of the extent of any problems affecting your hosting company, there’s always an option which won’t have suffered the same fate that you’re able to restore everything and be up and running again quicker.

Otherwise, you’re left with the time-consuming and costly process of building large amounts of your sites and databases etc. back up from scratch. And every day that passes costs you money in lost business or issues with payment management. One thing we can all take away – hosting companies aren’t invincible! Even large companies such as 123-reg can be affected, and they won’t help you with everything, so some management is needed on your part to manage additional backups elsewhere.