Developer Deletes Entire Production Database First Day on the Job

Megalith

Megalith

24-bit/48kHz
Staff member
Joined
Aug 20, 2006
Messages
13,000
Who was (truly) in the wrong here? One junior software developer made a copy-and-paste error and accidentally erased all data from a company’s production database. Naturally, s/he was immediately fired, but the CTO also alluded that a lawsuit could be on the way. While it was a very stupid error, many seem to think that it’s the company that made the bigger mistake: not having any backups.

…any system in which humans are involved will at some point be disrupted by human error. Organizations distinguish themselves not by stamping out the possibility of error, but by handling the inevitable mistake well. As subRedditors saw it, cscareerthrowaway567 made one mistake. The company made several. It didn’t back up the database. It had poor security procedures and a sloppily-organized system that encouraged the very error cscareerthrowaway567 made. Then, rather than taking accountability for those problems, the CTO fired the rookie who revealed them. Of all the errors this company made, that last might be the most destructive to their future success.
 
Ocellaris said:
Anonymous Redditor at an anonymous company with few technical details on what happened? Story must be real.
Click to expand...
I saw the original Reddit post a week ago and it contained plenty of details without giving away his identity.

He was following a manual that gave step-by-step instructions for setting up a VM dev environment and the instructions included a script containing the production DB and credentials that he was supposed to change. He c/p'ed it instead and overwrote the database.

But nah, he didn't post it on Facebook with his real name so it must be Fake News™!
 
Won't go into if the story is real or not. I've seen stuff like this happen. Hell, I did it myself a few years ago. Thought I was on the dev server. PHP MyAdmin was warning me not to do it - but since I was sure I was on the dev server, I was in the clear. About 3 minutes later, I realized I was on the production system as root and deleted the entire database for a startup companies app. No problem, our IT department had been backing up the server for a few months. I contact them, they inform me the backups are corrupt (all of them). I guess they never tested them and the software didn't give any error messages.
I got lucky as I had recently copied the production DB to my test system so was able to restore most of it without a lot of data loss. I didn't get home that night until very late. However, it was working when I left and I have a nice scar reminding me to be careful when I'm doing a dangerous operation. I also met with IT the next day so we could figure out the backup problem - turns out this system wasn't the only one not getting backed up.
Other fun things I've done - "su rm -ef $SOMEVARIABLE/" - only to find out this terminal didn't have $SOMEVARIABLE set. Looking at the wrong terminal window and delete shit. Most of these happened on my own systems and just cost me a lot of time (and data loss). I do try to be more careful...but accidents are still going to happen.
 
I have no room to talk. I lost an entire hard drive of old archived stuff because I wasn't paying attention to what drive I was doing a wipe and reformat on. Lost almost a decade of nostalgia, including music that was HELL to find in the first place. Didn't realize it until everything was up and running, then went to go look for said stuff. Lost my shit, over losing all that shit. I even went through all my old CD's to see if I had any backups of any of it on any of them, and nothing. Even went through all 100 something blanks I have to see if they got mixed in somewhere. No luck.

I have since learned to unplug all drives except the current drive being actively worked on. That being said, a backup of that drive would have saved my ass.
 
TBH I believe there are numerous small and medium businesses out there where IT is completely neglected. I think that because I work at one. Until disaster hits the mid management looks at IT as a bottomless money waster. We literally have to beg for hardware for day to day operations. And usually they only comply when I tell them that production will stop completely if they don't get me that storage in a few days time.
And I write letters multiple times explaining the shortcomings and weaknesses in the IT infrastructure and how big the risk of data loss actually is. It probably ends up in someone's spam box. If it even gets forwarded by the guy above me.



edit: But Sure it's easier and cheaper to sue the unlucky low level operator who made the mistake that was waiting to happen, than actually have contingencies and disaster recovery plans. Or even backups!
 
Last edited:
People neglect backup all the time. That this happened is not surprising at all. I used to work for a software company whose product relied on a database on the back end. You'd have been shocked at how often we found out that our customers didn't back up their critical databases. While what happened was the fault of the employee not fully reading the document (am assuming it said to change the server/credentials in a comment someplace.. ), the result of the mistake rests on the CTO's shoulders. Every company should have recent and confirmed good (TESTED) backups of anything important, particularly a database that contains information needed for running the business.
 
Megalith said:
While it was a very stupid error, many seem to think that it’s the company that made the bigger mistake: not having any backups.
Click to expand...
Someone once deleted a production database (that is used to process all of our hardware and software orders (a few hundred a day), among other things) by mistake at my work 14 months ago. We had backups but we were still down for 3 days. It took about a month to clean up that mess. I have no idea if that person was fired, but they should have been if they weren't.
 
CTO was obviously selling inventory out the back door. He knew he was going to be busted, so he hired a newb to be the patsy, the fall guy. Hacked his login creds, wiped his trail, and is now eyeballing a new yacht.

Yeah, that's how it went down.

Next?
;)
 
I'm not sure why a junior employee on their first day of the job had this much power. They wouldn't know any of the systems yet and someone should have really done that part for them since there was a chance for massive data loss.
 
exlink said:
Not having a back-up system is an even bigger error. CTO should've been fired as well just based on that.
Click to expand...
I can understand having an infrequent back up like once a month or something but 0 backup is just idiotic. So many things can cause loss of data no back ups is eventual suicide.
 
Being a consultant now for six years, I believe it. All the ransomware outbreaks have taught me nobody has backups, and nobody tests their backups. One company I was at, actually did proper disaster recovery, they shutdown their servers acted like a tornado struck their data center what would happen. They found out, all that planning hadn't been done. Backups didn't work, their other datacenter didn't come on line, the network wasn't fault tolerant. I was blown away how thorough the test was, and how much was broken. They thought they were ok, everybody said we got all this stuff, it'll be a breeze! And talking to them after, all the stuff they took for granted, none of it worked. I have talked to lots of other companies, and not a single one of them tried what the other company did...

This post is made "AS IS" without warranties, and confers no rights.
 
Assuming the story is true or mostly true: Not the new guy's fault. It is the job of the trainer to make sure the trainee can't do anything dangerous. Trainer failed here. But crap happens. Everyone learns. Move on.

Now the other half the story: No backups. No recovery plan. MAJOR FAIL. This lands fully on the CTO. No excuses. That is the job of the CTO, to make sure the disaster plan/business continuity plan is in place and regularly tested. Appears that CTO was collecting salary and not doing job.
 
SamuraiInBlack said:
I have no room to talk. I lost an entire hard drive of old archived stuff because I wasn't paying attention to what drive I was doing a wipe and reformat on. Lost almost a decade of nostalgia, including music that was HELL to find in the first place. Didn't realize it until everything was up and running, then went to go look for said stuff. Lost my shit, over losing all that shit. I even went through all my old CD's to see if I had any backups of any of it on any of them, and nothing. Even went through all 100 something blanks I have to see if they got mixed in somewhere. No luck.

I have since learned to unplug all drives except the current drive being actively worked on. That being said, a backup of that drive would have saved my ass.
Click to expand...

-1 for not getting a forensic recovery software to regain all that data. A format loses no information. Only the stuff you overwrote was truly lost.
 
If this is true and there were no backups for the production database. Then the CTO and all senior level admin should not only be fired but publicly shamed. That is just gross incompetence at every level and I have seen some dumb shit in my 20+ years in the IT industry but that level stupidity is unimaginable as it is unacceptable.

I feel for the junior Dev on this issue because more than likely while they were the one that overwrote the database. It was probable one of the senior staff that told them to work on the database without supervision and now his career is all but done.
 
Last edited:
  • Like
Reactions: Elios
like this
When I started my current job years ago, I found out how bad some IT people are.
The previous IT guy was supposed to come in a couple days to go over the network, but didn't bother showing up.
He was supposed to leave a working system on my desk, yet it didn't even have the network drivers loaded.

After getting the laptop working on the network and poking around to document the network on my own, I discovered the backups hadn't been working for the past 3 months due to a dead tape drive. Of course he swore it was working fine for him.
I also heard numerous complaints about the old IT person, the network being down, lost data, etc.
It was so bad, the owner kept boxes of files in his office because he didn't trust storing them on the network.

Luckily the company was growing (and I was likely the 1st read IT person they could afford to hire), so they where not afraid to spend money when it was needed.
One of the 1st thing I did was get the tape drive fixed so I could start running backups.
My current backup solution makes incremental backups throughout the day. (using D2D2T)
All production databases are backup up each hour, other data is backup up anywhere from every 2 hours to once a day, depending on it's importance.
Everything is then copied to tape over the weekend and taken off-site. The tape copy does take a while since it takes (8) LTO-6 tapes. (that's a little over 40TB) :eek:

I can restore files from disk, for any day in the past 2 weeks, or from tape (if something happens to the backup server).
Getting ready to start adding laptops to the backup schedule since some people can't remember to occasionally copy their documents folder to their server drive. :(

Don't understand all these companies paying ransom because they don't have backups.
I'm just a 1 man IT department for a company with 100 employees.
I not only handle backups, but everything else from desktops, laptops, phones, exchange, firewalls, etc.

My main worry about someone getting one of these ransomware viruses, is how much of my time it will waste cleaning the affected system and restoring the damaged files.
 
not only is not having back ups stupid. But why have a script that has the admin username and password, along with production servers in it as the base for people to use to create scripts for what why need.
 
M76 said:
TBH I believe there are numerous small and medium businesses out there where IT is completely neglected. I think that because I work at one. Until disaster hits the mid management looks at IT as a bottomless money waster. We literally have to beg for hardware for day to day operations. And usually they only comply when I tell them that production will stop completely if they don't get me that storage in a few days time.
And I write letters multiple times explaining the shortcomings and weaknesses in the IT infrastructure and how big the risk of data loss actually is. It probably ends up in someone's spam box. If it even gets forwarded by the guy above me.



edit: But Sure it's easier and cheaper to sue the unlucky low level operator who made the mistake that was waiting to happen, than actually have contingencies and disaster recovery plans. Or even backups!
Click to expand...
I feel your pain! It amazes me how IT is viewed as a money pit UNTIL SHTF. Then, IT is the culprit and takes the blame.
Our IT department has a very limited budget. I can't ask them for resources. As a result, we scrounge around and take care of things ourselves (which is bad!). We have stuff in DropBox, Google Drive, etc that should be under the control of IT approved resources. Without us scrounging, we couldn't do our job.
 
I see this all the time. Pitch a backup solution, even something simple like external drives for a Windows Server backup and people will balk at the price of an 1TB HDD.
 
IT worker for the last 25 years and have spent a LOT of time with backups and DR.

1) Newb developer guy should never have write access to production environment.
2) production environment should be backed up, and backups should be tested from time to time
3) production environment should have a duplicate DR site plan ready to roll in case of disaster

So why did the newb do anything without consulting a seasoned engineer? You're fired
So who gave the newb access? You're fired
Why are there no backups? Or backups that don't work? Pick a person that might have something resembling data protection in their job description, they're fired
Why is there not a DR plan? Everyone is fired

So yeah, I believe it. Now, a lawsuit for damages? Nah.

But honestly, I've witnessed idiots doing far worse and getting a slap on the wrist, or some other type of punishment in lieu of termination. Night shift, clean toilets, other unhappy work. But still employed. Everyone makes mistakes. Now, if said person screwed up big TWICE? GONE BABY, GONE.
 
At my previous Job, we used to fly out of state for disaster recovery, had our actual production tape backups shipped to us, and then we had 48 hours to recover everything. When I first showed up it was a god awful mess and only about 40% was recovered within 48 hours and nobody got more than 2 hours of sleep. By the time I left that job I could walk into the DR DC, drop a tape in a tape drive, run 1 command and restore an autoyast server in about an hour. After that, I could PXE boot every other host, select the appropriate build, and have the whole damn linux environment online within ~3 more hours, and just waiting for the rest of the data to roll off of tape. The windows servers (domain controllers and exchange server) were the only real annoying things left, and even those got done quickly enough we could get a good nights sleep, check the data in the morning verify everything was grand and then head out for a night in NYC. I can only imagine how easy this would all be with vms and containers now instead of actually having to build servers....
 
vmxEx
 

Attachments

  • steve-urkel-did-i-do-that.jpg
    steve-urkel-did-i-do-that.jpg
    21.8 KB · Views: 68
steakman1971 said:
I feel your pain! It amazes me how IT is viewed as a money pit UNTIL SHTF. Then, IT is the culprit and takes the blame.
Our IT department has a very limited budget. I can't ask them for resources. As a result, we scrounge around and take care of things ourselves (which is bad!). We have stuff in DropBox, Google Drive, etc that should be under the control of IT approved resources. Without us scrounging, we couldn't do our job.
Click to expand...
With your previous remark that I.T. should test your backups and now you saying you have things stored in Dropbox and Google, it might be time you consider you are a cowboy and a problem for I.T. and not an ally.

Those backups they run? It's not protecting data THEY generate.
Test YOUR own backups.

That data needing stored? I.T. did not generate it.

"Scrounging" only occurs when cowboys are out there trying to make stuff happen when management won't spend the money. It's not on you to go rogue and stick undocumented solutions out of house. Push back and get it funded.

Help I.T. justify those costs to test YOUR data restores and store YOUR/CUSTOMER data in house.
 
Not only should the CTO be fired for the backup situation, but the fact that there is a simple to follow instuction set posted that can be followed by someone with limited admin privilages that would end this way is worthy of an ass beating.
 
All but one of you didn't read the original thread properly.

The guy was following compiled instructions on how to set up a test environment.

The instructions had valid username/password. (Why?)

The instructions as written, pointed to production. (Why?)

The username/password, had both read and write privileges. (Why?)

The short story: The instructions as stated, were designed to ruin production. It was only a matter of time before someone did it.
 
OracleofEpirus said:
All but one of you didn't read the original thread properly.

The guy was following compiled instructions on how to set up a test environment.

The instructions had valid username/password. (Why?)

The instructions as written, pointed to production. (Why?)

The username/password, had both read and write privileges. (Why?)

The short story: The instructions as stated, were designed to ruin production. It was only a matter of time before someone did it.
Click to expand...

Naw, I still don't even believe the entire thing happened. "Someone on Reddit" isn't a valid source when that user is on a throwaway account with no confirmation details. People on Reddit are so easily trolled into turning into white knights at every available opportunity, I've known someone who posts on throwaway accounts just for the lols in the replies, and it worked every freaking time.
 
You must log in or register to reply here.
Back
Top