I'm not going to name any names: in the late 00s/early 10s I worked in one of the highest-profile, …
… I'm not going to name any names: in the late 00s/early 10s I worked in one of the highest-profile, high-growth tech startups of its era, and I've personally made a blunder that corrupted literally millions of user records in the database. This incident was known internally as one of the most disastrous technical things that happened in the company's history, among a few others. The nature of the product was one of very quickly updating data, and updates were critically important (e.g. is affected by user spends) and hence restoring from DB backups of even the night before was unfeasible. There was irreparable damage where a whole team of us had to spend the next few weeks painstakingly hand-fixing data for users, and coming up with algorithms/code to fix these things as users use the product as they go. As you expect in this anecdote, I did not get fired, I was part of the team that worked tirelessly following this incident to fix user data, and I continued to have a good, growing career in my remaining time in this company (the next few years).
Original discussionThe first time I was almost fired from Apple149 source comments · at import