There’s been an article on Reddit recently that I’ve seen via a couple of sources now. The post is a bit of a horror story; a junior developer on his first day of his first job was given an instruction manual on how to set up his machine, and ended up deleting production databases. It then turned out that there was no way to restore the data, resulting in massive loss to the company. The guy owned up to what had happened, and was sacked on the spot, and threatened with legal action against him. There are so many things wrong here, and the whole thing is ridiculous – and yet it’s not the first time I’ve heard of such a thing, and probably won’t be the last.
A CPU performs billions of calculations per second. A 1TB disc holds a Trillion bytes. Murphy’s Law says that “things will go wrong in any given situation, if you give them a chance”; in the context of computing it just so happens that there are a lot of things, and they can go wrong very fast! In this case, this developer followed instructions and made a mistake, with a catastrophic outcome, but in reality the problems started a long way away from this point! Should the individual developer have paid more attention? Maybe, but as a junior on his first day he shouldn’t have been put in a position where something like this could happen – and certainly shouldn’t be blamed by the very people who are really accountable for the systemic problems behind this incident. He made a convenient scapegoat, perhaps to cover for the sheer scale of the incompetence that had been exposed.
Can this kind of thing be prevented? I’d say yes; with a little forethought and diligence. Can it be recovered? Again, yes with forethought and diligence. There are many lessons that can be learned from this, and maybe some of them can help your team avoid this kind of mess.
In the replies to the original post there are a bunch of engineers from big name companies who lend their support and experiences. Many of these companies are known worldwide for the robustness of their software development practices. Needless to say that when similar events take place in these environments, the outcome is rather different!