UnderdogMindset

Accidentally deleted Gitlab's production database

Images

Gitlab
Posted: Jun 22, 2023
Computer Mistake

One night, a software developer working in the Netherlands accidentally deleted Gitlab's production database while trying to debug some issues. He did this by entering the delete all command in the wrong terminal. In trying to fix his delete mistake, more problems were uncovered; the database backups were not working as expected. Luckily, this developer took a snapshot of the database six hours before. They would use the snapshot to recover as much data as possible. It would take 18+ hours to recover fully, and Gitlab would lose six hours of merge requests and other metadata.

The company handled this incident well, sharing out the recovery process. They would not fire the developer. The developer had around 6.5 years of programming experience at that point, but he only joined Gitlab a little more than a year before the mistake. He would stay at the company for nearly five more years before leaving of his own accord. GitLab has IPOed since then and has over 30 million users today.

Social Media

Comment
by u/yorickpeterse from discussion Accidentally destroyed production database on first day of a job, and was told to leave, on top of this i was told by the CTO that they need to get legal involved, how screwed am i?
in cscareerquestions

Quotes

GitLab handled this very well,...Nobody got fired or yelled at, everybody realised this was a problem with the organisation as a whole.

Yorick Peterse

Hi, guy here who accidentally nuked GitLab.com's database earlier this year. Fortunately we did have a backup, though it was 6 hours old at that point.

Yorick Peterse
References