A bad day in Server land

L

lone wolf

Gawd
Joined
Feb 4, 2003
Messages
705
just a rant - thanks for listening

The server gods are not happy today.
This morning I get called to one of my largest clients with just a simple DNS issue, nothing much just had to remove a DNS entry (btw SBS 2003) now they can get to the MX record. I heard several complaints that the systems were very sluggish access network drives, I was thinking that a reboot may be in order after hrs. Normal Stuff Right?
A couple of hrs later I get a paniced call saying that the server has crashed and getting an error loading operating system. I rush down, and sure enough, server had crashed. All data gone, I have been nursing the partition along trying to recover data for the last 5 hrs, but no luck, now in the process of reimaging the server :(
btw: if anyone cares to know i did have a successfull backup last night, so only one day lost
 
Atleast you only lost a day's worth of data but it still sucks when that happens. Hope all goes well.

BTW, was it hardware or software?
 
hardware, SATA HDD lost partition info, I have several tools for recovery but alas none worked.
 
bigshooter said:
Not running raid?
Click to expand...
Amen to that brother. I run RAID5 on everything mission critical now. That and with regular backups (knocks on wood) things should go smoothly in the event that something bad happens.

Use this as a case for some sort of disk redundancy system. Cite that the cost to production could have been much worse and could be avoided by spending a few bucks on a RAID subsystem.
 
RAID-5 + D2D or DDT backups = bliss :D

Glad you had a handy full backup!!
 
SVT4ME said:
RAID-5 + D2D or DDT backups = bliss :D

Glad you had a handy full backup!!
Click to expand...

On some applications RAID-5 is overkill.

Give me LTO/LTO-2/LTO-3 backups over that any day :D.
 
[H]exx said:
Raid 5 is NEVER overkill on a business/system/mission critical system. YOU HEAR ME, NEVARRRRR!!!! :p

It's always good to have redundancy ;)
It's always good to have redundancy ;)
It's always good to have redundancy ;)
Click to expand...

Then why not go all out and put it on RAID51 with a D2D to D2T and finally T2I (Tape-to-IronMountain)
 
I hear ya on the angry server gods. For some reason, our datacenter had half its internal network go bugshit so a 8th of our servers went down putting maybe 10,000 clients on OMFGWTFBBQSAUCE angry mode.

Luckily its back up now. It was quite scarey for a good hour though.
 
LittleMe said:
Then why not go all out and put it on RAID51 with a D2D to D2T and finally T2I (Tape-to-IronMountain)
Click to expand...
Its a matter of what you can afford and if its applicable to your situation.
 
You must log in or register to reply here.
Back
Top