Anyone ever have one of those days that makes you go "WTH!?!"

P

peanuthead

Supreme [H]ardness
Joined
Feb 1, 2006
Messages
4,699
/Rant On

In short, one of our primary backup devices looks like it's beginning to go belly up. We've had little issues with it here and there but nothing too painful, just a nuisance. The most painful one (I think) is a slow copy rate of about 40MB/sec over 1GbE with large files. We all know that it's not too bad but when you are moving GB and GB of files it's slow. Heck, wouldn't we all like more speed if stability stays the same? :)

So I pitched a new storage device today that has a better UI, faster in copy rates, longer warranty period (the other unit is out of warranty) and the ability to grow out the unit with more bays if needed all for just south of $1200. Sounds good right? We were then told to see what we can Frankenstein together with FreeNAS using a decommissioned server. I just want to smack myself in the head and go "What the Heavens!?!" Am I just being pissy or is my logic sound? I know extraneous funds for non-essential items is not there but that is ridiculous in my mind.

/Rant off
 
Last edited:
I actually had about 8 months of that at one job I worked at.

This is how bad it was... They were trying to run Autocad on P4s with onboard graphics.

They would not order any new machines.

Needless to say, that place is not in business anymore.. because of stuff like that and because it was union run.

It really is not worth it to frankenstein something together using an old server unless you have one on hand that can take larger drives.

The older servers are going to use a lot more power as well as have a higher cooling requirement which is going to up the power bill for the A/C.

Just not worth it.
 
The $1200 solution sounds like a Frankeinstein project as well. ;)

The universal problem is that payroll budget has already been allocated, so your time is sunk cost, whereas buying new hardware hasn't been part of the budget, so "finding" $1200 may be a lot harder than just paying you $1200 in payroll to get something working. It's crazy, I know.

Depending on how much you like the company it may just be time to look for a different place of employment.
 
Thuleman said:
The $1200 solution sounds like a Frankeinstein project as well. ;)

The universal problem is that payroll budget has already been allocated, so your time is sunk cost, whereas buying new hardware hasn't been part of the budget, so "finding" $1200 may be a lot harder than just paying you $1200 in payroll to get something working. It's crazy, I know.

Depending on how much you like the company it may just be time to look for a different place of employment.
Click to expand...

It was a Synology solution so no Frankenstein project here. :) I love their UI and speed of the units. Tech support as warranty is good too which to me is a must to me in a production environment. The idea was to start inexpensively add a second Synology RS or RX sometime down the line if we needed more space or if we wanted to have one NAS handle odd days and the other even days. I figured this would allow us to not have all of the eggs in one basket while allowing a prompt response time if we need to restore back further than the retention period we keep locally. We have backups sent offsite for a second copy/archiving purpose. Knowing that something isn't done right the first time just so we need to visit it again is frustrating. Not saying that FreeNAS is not an option, but to me in a production environment you need vendor knowledge and backing on certain things. To me this is one of them. This is not an event that will cause me to leave my present employment, however this event has been well noted by me. I need to cool down, collect my thoughts, separate fact from emotion and talk with my wife. Any decisions we make we make together.
 
Last edited:
cyclone3d said:
I actually had about 8 months of that at one job I worked at.

This is how bad it was... They were trying to run Autocad on P4s with onboard graphics.

They would not order any new machines.

Needless to say, that place is not in business anymore.. because of stuff like that and because it was union run.

It really is not worth it to frankenstein something together using an old server unless you have one on hand that can take larger drives.

The older servers are going to use a lot more power as well as have a higher cooling requirement which is going to up the power bill for the A/C.

Just not worth it.
Click to expand...

That is jacked up there with regard to running AutoCAD on P4's. The decomm. server equipment is still pretty good actually. I am looking at this from the angle of providing a solution rather than just fixing a problem. I want to provide solutions and not just fix problems. I expect the same from someone that does any kind of work for me (i.e. - car, house, etc.) so I feel people/my employer should expect the same.
 
Sounds like our very last experience - EVER - with Drobo products. We also had a come to Jesus moment with Datto not too long ago when their BDR failed to do what a BDR is supposed to do - recover.

We are pretty fortunate in that generally we are stripping Frankenstein's out and putting in proper equipment.
 
Since this is for backup, work with your powers-that-be to pin management down on expected RPO/RTO values. The road may be long but ultimately they either need to fund their backup expectations or lower their backup expectations to match funding. At least your butt won't be on the line if it ends up being the latter.

I'm going through something similar to this now where my org acquired shiny new storage and vSphere hosts but is being reluctant to acquire a proper backup solution for it (we were all physical until recently).
 
SGalbincea said:
Sounds like our very last experience - EVER - with Drobo products. We also had a come to Jesus moment with Datto not too long ago when their BDR failed to do what a BDR is supposed to do - recover.

We are pretty fortunate in that generally we are stripping Frankenstein's out and putting in proper equipment.
Click to expand...

Now this is humorous reading this. The unit causing the headache is a Drobo. I was on the phone in the car and supposedly we have between ~5 servers backing up to it (not sure if all at once) and they now suspect that maybe it's a network bandwidth issue in that we are pushing too much to it. I told them that I have bowed out of this task and they have it under control.
 
That kind of situation (cobble something together with old hardware vs buy a Synology, and not the AutoCAD on P4s situation) is pretty par for the course where I work. Of course, my employer (hosting company) has the staffing and hardware on hand to usually make the cobbled solutions work as long as it's technically sound.

Not sure if you're over reacting in your situation. Sounds like warranty coverage should be a bigger deal for your company. At the same time, hey, there's this decommissioned server just sitting around and perhaps down the line that could simply be a spare should something warrantied like a synology fail.
 
peanuthead said:
Now this is humorous reading this. The unit causing the headache is a Drobo. I was on the phone in the car and supposedly we have between ~5 servers backing up to it (not sure if all at once) and they now suspect that maybe it's a network bandwidth issue in that we are pushing too much to it. I told them that I have bowed out of this task and they have it under control.
Click to expand...

DROBO... lmao. They DO NOT work as advertised, not even sort of. We have 4 B800i's and even when used as test beds they suck, so, so badly.

Our storage matrix is now a mix of Dell Equallogic's (primary storage group, replication), Dell Powervault MD122x's (secondary storage for stand alone services), HP lefthands (VMware Shared pool) and a PV124T for offsite zombie apocalypse disaster recovery. We are looking at new backup solutions from a few companies to replace Backup Exec (barf).

Spend the money and do it properly.
 
fantabulous said:
That kind of situation (cobble something together with old hardware vs buy a Synology, and not the AutoCAD on P4s situation) is pretty par for the course where I work. Of course, my employer (hosting company) has the staffing and hardware on hand to usually make the cobbled solutions work as long as it's technically sound.

Not sure if you're over reacting in your situation. Sounds like warranty coverage should be a bigger deal for your company. At the same time, hey, there's this decommissioned server just sitting around and perhaps down the line that could simply be a spare should something warrantied like a synology fail.
Click to expand...

IMO if we planned better we would not be reacting but be proactive. When you seem to put out fires more than prevent them it starts to wear you down. We seem to does this more than not. Note I say seems, so this is just my opinion. It is not my issue anymore as I have washed my hands of it.
 
These stories just crack me up. What was my company using for our backups? Buffalo Terastations. It would be faster for me to carve the data into stone then it would be to copy something to those. At our height we had......10 or so in circulation.
 
Remember folks: there is never time/money to do it right, but there is ALWAYS time/money to do it over!
 
peanuthead said:
It is not my issue anymore as I have washed my hands of it.
Click to expand...

I think that the above statement is problematic.
For one, you obviously care enough to post about it here, so odds are you will continue to care about it even though you *may* be in the clear when it comes to performance/reliability/RPO/RTO.

Are you actually in the clear? Where does this issue fall within your position description, within your actual responsibilities regardless of PD? How much money will be lost when this whole thing goes south and how exposed will you personally be when that happens?

At the minimum you need a paper (electronic) trail that proves that you found the deficiency, you notified your superiors of the deficiency, of the risk associated with the deficiency, and that you provided a viable suggestion on how to correct the deficiency. At least you'll have something in writing that will show that you pointed this out before it happened.

All this is assuming you are a proper employee and not "employed" as independent contractor.
 
Thuleman said:
I think that the above statement is problematic.
For one, you obviously care enough to post about it here, so odds are you will continue to care about it even though you *may* be in the clear when it comes to performance/reliability/RPO/RTO.
Click to expand...

I will continue to care because that's one of the reasons I am in IT. I don't deny that.

In short, I work as a PM but have some of the background, etc. that Engineering needs so I am asked for advice and input from time to time. Recently I was asked to become more involved in that side of the house, however I keep getting met with frustration after frustration as the proper solutions keep getting shot down. IMO you are not able to keep band-aiding an issue before you have a situation. I think that is what we are heading towards if not out and when something goes boom we will be up the creek. Unfortunately, this is all in via voice but the other Engineers in that department know where I stand and don't totally disagree. They are as frustrated as I am if not more.
 
For a primary backup, you want it to be super good. I cant believe they would say to cobble crap together.
Its always funny when you see what computers some multi-million dollar company use. Like stated above, trying to run CAD on a P4 with on-board graphics, small 15 inch screens, etc.
I wouldnt even use that P4 to surf the web! :D
 
I'll chip in on the OMG-I-HATE-Drobo bandwagon. Bought one of their units several years back and it never hit anywhere near its rated speeds, and their "engineers" were clueless idiots.
 
For replacing the Drobo units, we have had very good success with the QNAP TS-Series Enterprise level products. Early on there were some firmware hiccups, but over the past couple of years they have been rock solid and quite versatile. We use them for NAS, iSCSI, NFS, TimeMachine, FTP, and iTunes media servers at quite a few clients. Definitely worth a look if you have been burned in the past by Drobo.
 
You must log in or register to reply here.
Back
Top