Developer Deletes Entire Production Database First Day on the Job

Backups. This is basic. As posted above, until you TEST your backup, you just don't know.

Stupid home computer was being backed up every week. Hey, I knew the backups were good: the software gave me a green check mark each week. When the (only) hard-drive failed, I sprang into action to prove my manhood. I put a new drive in, shoved the rescue boot disk in...and promptly found out that there was no backup. I learned. The next software I used, I made a backup and then removed the hard drive, simulating a total hardware failure/replacement.

If I were a CTO, that would be the ONLY strategy which would assure the system's redundancy. Take it offline (gracefully) and see what happens. Oh, make the vendors sign guarantees.

Maybe that's why I'm not a CTO? ;)
 
c3k said:
... Oh, make the vendors sign guarantees.

Maybe that's why I'm not a CTO? ;)
Click to expand...
They'll sign a guarantee but at their price, you definitely won't sign the sales agreement. :D
 
  • Like
Reactions: c3k
like this
better question how did this new person have the permission to nuke the whole thing? seems like IT fail to me not the employee
 
Our beat company does a role swap every 3 months and that system stays live for 3 months. Backups are also tested with the target at the time.
 
OracleofEpirus said:
All but one of you didn't read the original thread properly.

The guy was following compiled instructions on how to set up a test environment.

The instructions had valid username/password. (Why?)

The instructions as written, pointed to production. (Why?)

The username/password, had both read and write privileges. (Why?)

The short story: The instructions as stated, were designed to ruin production. It was only a matter of time before someone did it.
Click to expand...
Get trolled much?
 
First place i worked out of college, our stupid exchange server ran out of space and exchange shut down. This other dude we had working for us had a great solution, "i can run this script and it will clean up alot of temp files and logs"... ok...


Deleted the exchange database logs. He's probably that CTO.
 
Who the fook gives any dev creds to the production database on their first day of the job?
Who the fook doesn't keep a backup of the production database?
And then to be threatened with legal action when someone makes an honest mistake copying and pasting instructions?

Almost seems like a win to me - the dev probably doesn't want to work there anyway.
 
Last edited:
Oh man. Whether true or not, still a scary thought.
When I was junior admin years ago, my boss had everything in check. Main systems backed up to a raid, which was backed up to a redundant raid, which was backed up to another redundant raid, which was backed up to an offsite redundant raid, which was backed up to tape AND dvd-ram.

So redundant it's not even funny. He was very OCD. :)

Loved the guy, minimal downtime, ever. Best plant in the company. They gave him free reign with both budget and implementation. And boy, did he go crazy with it.
 
I'm starting my first real job soon, and its a similar position.

I'm terrified of something similar to this happening. (or not being able to do everything they want)
 
As mentioned many times, if a day one employee has that level of access, there has been numerous failures at numerous levels. If anything they probably did him a favor by terminating him. He will likely have good legal grounds against them. That said, more than a few people there should be terminated over this, the CTO among them.
 
Well I suspect the truthfulness of this story, I would like to point out that having a script with a production system pointer takes the cake.

As for dumbass IT stories, I was contracted to fix a small bridal shops IT situation, I was configuring backups and forgot I replaced the network drive's path with another in clipboard. Set it to run on schedule and forgot about it. It wasn't until a fire that I realized all of their backups were writing to the same drive as production, which was a molten slab by time it was recovered.
 
steakman1971 said:
I feel your pain! It amazes me how IT is viewed as a money pit UNTIL SHTF. Then, IT is the culprit and takes the blame.
Our IT department has a very limited budget. I can't ask them for resources. As a result, we scrounge around and take care of things ourselves (which is bad!). We have stuff in DropBox, Google Drive, etc that should be under the control of IT approved resources. Without us scrounging, we couldn't do our job.
Click to expand...
That's what we do as well. Part out old hardware that lost it's original functionality and re-purpose it. But that can only get you so far.

We actually had data loss before about 8 years ago, but the entire management has been replaced since then. And as the bottom line went down, the first department they got rid of completely was IT. So now it falls on me to keep things up and running and take care of backups as well as I can, because noone else would.
 
B00nie said:
-1 for not getting a forensic recovery software to regain all that data. A format loses no information. Only the stuff you overwrote was truly lost.
Click to expand...

If I had that kind of software on hand, there's a good chance I would have remembered to back up everything like I'm archiving the secrets of the Illuminati, instead of lamenting my loss.
 
SamuraiInBlack said:
If I had that kind of software on hand, there's a good chance I would have remembered to back up everything like I'm archiving the secrets of the Illuminati, instead of lamenting my loss.
Click to expand...

R-studio recovery costs like 14 bucks. Too much?
 
Ocellaris said:
Naw, I still don't even believe the entire thing happened. "Someone on Reddit" isn't a valid source when that user is on a throwaway account with no confirmation details. People on Reddit are so easily trolled into turning into white knights at every available opportunity, I've known someone who posts on throwaway accounts just for the lols in the replies, and it worked every freaking time.
Click to expand...

Fair response. As long as you don't include other people's beliefs in yours. The entire point of having different beliefs is so no one person can fool everyone at once.

Jim Kim said:
Get trolled much?
Click to expand...

Here's one for you, then. I've worked at a global multi-billion dollar company where some computers were still running Windows 3.1 for Workgroups as recently as 2010. Not even the latest version, which is 3.11.
 
SamuraiInBlack said:
I have no room to talk. I lost an entire hard drive of old archived stuff because I wasn't paying attention to what drive I was doing a wipe and reformat on. Lost almost a decade of nostalgia, including music that was HELL to find in the first place. Didn't realize it until everything was up and running, then went to go look for said stuff. Lost my shit, over losing all that shit. I even went through all my old CD's to see if I had any backups of any of it on any of them, and nothing. Even went through all 100 something blanks I have to see if they got mixed in somewhere. No luck.

I have since learned to unplug all drives except the current drive being actively worked on. That being said, a backup of that drive would have saved my ass.
Click to expand...
This post gave me the chills. I lost a bunch of writing I had been working on for years exactly the same way. Now I backup to NAS, external drive, Dropbox and my office, heh.

The worst is that frantic search through all your CDs and floppies that always ends up being ultimately futile.
 
deton8 said:
This post gave me the chills. I lost a bunch of writing I had been working on for years exactly the same way. Now I backup to NAS, external drive, Dropbox and my office, heh.

The worst is that frantic search through all your CDs and floppies that always ends up being ultimately futile.
Click to expand...
Learned my lesson the exact same way.
SamuraiInBlack said:
I have no room to talk. I lost an entire hard drive of old archived stuff because I wasn't paying attention to what drive I was doing a wipe and reformat on. Lost almost a decade of nostalgia, including music that was HELL to find in the first place. Didn't realize it until everything was up and running, then went to go look for said stuff. Lost my shit, over losing all that shit. I even went through all my old CD's to see if I had any backups of any of it on any of them, and nothing. Even went through all 100 something blanks I have to see if they got mixed in somewhere. No luck.

I have since learned to unplug all drives except the current drive being actively worked on. That being said, a backup of that drive would have saved my ass.
Click to expand...

I learned my lesson the exact same way. Had 2 3TB drives. Picked the wrong one in diskpart. Never again. Now anything I care about goes to a NAS which auto replicates to an external drive and does an encrypted 1 way sync to a one drive account.
 
Ocellaris said:
Naw, I still don't even believe the entire thing happened. "Someone on Reddit" isn't a valid source when that user is on a throwaway account with no confirmation details. People on Reddit are so easily trolled into turning into white knights at every available opportunity, I've known someone who posts on throwaway accounts just for the lols in the replies, and it worked every freaking time.
Click to expand...

the issue with that is if true (which it might not be) the person would need that to be a throwaway account and not be something that can be traced back to him or the company. If sued he needs to have it look as good as possible for him. Bashing the company online would not help there. You also have what if this is a public company, if it was know before any company announcement that they had some type of set back in all projects that would greatly hurt their stock and be something else they could go after him for.

so that by itself could be a person doing what they should be doing or could be a person trolling. Both fit the same profile from the viewpoint if everyone else
 
How is this news? Assuming the story is real which it may not be, new people screw up all the time and companies screw up all the time giving out access to new people they shouldn't.
 
exactly what it sounds like, developers dont push changes directly to production. goes through change control process and personnel.
 
You must log in or register to reply here.
Back
Top