production vm storage?

hutchingsp

Limp Gawd
Joined
Dec 24, 2006
Messages
150
We have a couple of baby HDS arrays - cheap enough, pretty dumb, HDS reputation for being deathly reliable.

We're an average SME but I take a simple view which is if the storage is having a bad day everything else is going to have a bad day so I don't take chances - frustrating sometimes as there's a shitload of really cool products out there that I'd love to try out :)
 

Thuleman

Supreme [H]ardness
Joined
Apr 13, 2004
Messages
5,833
We're an average SME but I take a simple view which is if the storage is having a bad day everything else is going to have a bad day so I don't take chances - frustrating sometimes as there's a shitload of really cool products out there that I'd love to try out :)

So much this. We are a larger shop, and I could probably cut the annual storage budget in half if we were to abandon the VMAX, Isilon, VPLEX setup, but the chance that departing EMC will be a resume-generating event for me is just not worth taking. So we continue to spend millions on storage and I am keeping my job.

Having an "Oops, I selected a storage solution that resulted in a huge downtime event" may be OK when one is in their 20's, but when one is in their 40's it is a lot harder to professionally recover from such an event.

Getting the C-suite enthused about saving money is simple, but when something goes south they aren't the ones who's livelihood will be on the chopping block.
 

feffrey

Gawd
Joined
Oct 26, 2010
Messages
585
Anyone using vsan for small deployments? We have a number of remote sites that need 2-4 hosts and shared storage. Going to be testing vsan in our lab soon.
 

hutchingsp

Limp Gawd
Joined
Dec 24, 2006
Messages
150
Not used it but keep in mind it only works with 3 hosts upwards (and 3 won't give you much resilience).
 

hutchingsp

Limp Gawd
Joined
Dec 24, 2006
Messages
150
So much this. We are a larger shop, and I could probably cut the annual storage budget in half if we were to abandon the VMAX, Isilon, VPLEX setup, but the chance that departing EMC will be a resume-generating event for me is just not worth taking. So we continue to spend millions on storage and I am keeping my job.

I don't know if I wish I was on that scale or if I'm glad I'm not :)

I suspect there's a fair bit of CYA in there - terminal failures with any vendor seem fairly rare but when it happens if I was in the spotlight I'm sure HDS or EMC buys a few points vs. explaining why I chose $startup_of_the_week.
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,711
<Having an "Oops, I selected a storage solution that resulted in a huge downtime event" may be OK when one is in their 20's, but when one is in their 40's it is a lot harder to professionally recover from such an event.>

I hear ya. It's like a variation of the old 'no one ever got fired for recommending ibm/dec/vmware/xxx'
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,608
I don't know if I wish I was on that scale or if I'm glad I'm not :)

I suspect there's a fair bit of CYA in there - terminal failures with any vendor seem fairly rare but when it happens if I was in the spotlight I'm sure HDS or EMC buys a few points vs. explaining why I chose $startup_of_the_week.

<Having an "Oops, I selected a storage solution that resulted in a huge downtime event" may be OK when one is in their 20's, but when one is in their 40's it is a lot harder to professionally recover from such an event.>

I hear ya. It's like a variation of the old 'no one ever got fired for recommending ibm/dec/vmware/xxx'


While absolutely true, there is a second part to that axiom - "No one ever got fired for going IBM, till they ~do~". Companies often have a turning point where quality finally slips enough that things go horrifically wrong in such a spectacular fashion that everyone ~does~ get fired. Watch for that turning point. (See: IBM + Queensland affair, IBM + City of Austin, etc).
 

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,711
True enough, but as a general "Robotic Law #1 of HW procurement", it tends to be a safe default...
 

niomosy

Limp Gawd
Joined
Nov 21, 2005
Messages
247
Crazy. I run netapp and FCP on it has been the most stable during any of my NDU's or failovers. Usually the blip in NFS will cause a VM or two to bomb out, while my Fiber attached systems truck right along.

I wish we could say that. Regardless of OS (Linux / RHEV, VMWare, Windows, AIX, Solaris). They all have the potential for (and have experienced) problems during many a filer failover. The worst part is that it's inconsistent and NetApp's not been able to really hone down a solution to the problem. We've patched, upgraded, updated firmware and drivers, tuned the OS for higher SCSI timeout values per their recommendation. Nothing.

You're doing it wrong :p

Although NetApp's FCP implementation, to put it simply, blows goats.

Yeah, not entirely impressed with how they handle FCP at all. Don't get my started on tools like Snapdrive on Linux and AIX. Ugh. Run command. Wait 5 minutes for response. Sigh.
 

Thuleman

Supreme [H]ardness
Joined
Apr 13, 2004
Messages
5,833
I don't know if I wish I was on that scale or if I'm glad I'm not :)

It's definitely very cool when you first walk in, after a while it's just normal. I could say that I am in charge of millions of dollars of combined CAPEX and OPEX, but in reality the overwhelming majority of that isn't discretionary funding, it's all money that goes to specific vendors in predetermined ways without that I have the option to swing that around toward different technology. So I am not really in charge of it even if it is my budget. At my old job we had a lot of agility but no money. At this job we have a enough money but no agility. Based on my experience it's definitely better to be properly funded even if that means that you don't get to deviate from an established corporate standard.

While absolutely true, there is a second part to that axiom - "No one ever got fired for going IBM, till they ~do~". [...] IBM + City of Austin

In all fairness, the City of Austin thing was like Healthcare.gov, not a hardware problem but a codemonkey and PMO induced issue.
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,608
True, but would you buy anything from them outside of a mainframe or aix box these days?
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,608
It must be a health care thing I have friends that work in healthcare and they use all IBM gear too.

Lots of stuff on that side has been "certified" for IBM only (especially EPIC) so it's easier to buy from one place. It's starting to shift though, as IBM gets out of that line of business
 

Thuleman

Supreme [H]ardness
Joined
Apr 13, 2004
Messages
5,833
Lots of stuff on that side has been "certified" for IBM only (especially EPIC) so it's easier to buy from one place. It's starting to shift though, as IBM gets out of that line of business

Exactly, we are an Epic shop too, so it's all on AIX. It's really an Intersystems thing, Epic runs on a DB named Cache, which in turn needs AIX. However, since recently Cache also runs on RHEL, but Intersystems wants millions of dollars to upgrade the licenses to be able to be used on RHEL. Doesn't make sense to pay millions just to avoid the Power server refresh every 4 years or so. Now we are completely off topic, lol.
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,608
Exactly, we are an Epic shop too, so it's all on AIX. It's really an Intersystems thing, Epic runs on a DB named Cache, which in turn needs AIX. However, since recently Cache also runs on RHEL, but Intersystems wants millions of dollars to upgrade the licenses to be able to be used on RHEL. Doesn't make sense to pay millions just to avoid the Power server refresh every 4 years or so. Now we are completely off topic, lol.

Been there, done that. :) I worked in that field for a year :p
 

QHalo

2[H]4U
Joined
Sep 30, 2002
Messages
3,432
I have customers in both boats. But we can monitor all of it with the Epic CSA and vR Ops. :) Want a demo? :D
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
The number of customers that need a true active/active setup is very small - most would be fine with 10-15 seconds of RPO, and that can be done in software. :)

define small.

I work in the financial sector which is a huge sector.. would suck if you lost access to your money while trying to pay for dinner on a hot date.. all cause an active/passive system with a 10-15 second RPO is in place.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
company does outsourcing of non-critical IT to consulting company and these guys offered to put into production for branch offices and depts some free solution for storage and it would be consulting company who'll support it instead of a storage vendor

i'm a bit scared of this business model and i don't like the idea of putting something into production without support and ability to reach vendor

do you see this a show stopper?

Yes, because at the end of the day.. you are the guy supporting some half baked solution, and its your head when it falls over.

Absolute show stopper.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
We have a couple of baby HDS arrays - cheap enough, pretty dumb, HDS reputation for being deathly reliable.

We're an average SME but I take a simple view which is if the storage is having a bad day everything else is going to have a bad day so I don't take chances - frustrating sometimes as there's a shitload of really cool products out there that I'd love to try out :)

I kicked HDS out of my data center almost 4 years ago. They did an amazing job at deleting a 14TB pool with out asking me first, and tanking a few of my DBs.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
So much this. We are a larger shop, and I could probably cut the annual storage budget in half if we were to abandon the VMAX, Isilon, VPLEX setup, but the chance that departing EMC will be a resume-generating event for me is just not worth taking. So we continue to spend millions on storage and I am keeping my job.

Having an "Oops, I selected a storage solution that resulted in a huge downtime event" may be OK when one is in their 20's, but when one is in their 40's it is a lot harder to professionally recover from such an event.

Getting the C-suite enthused about saving money is simple, but when something goes south they aren't the ones who's livelihood will be on the chopping block.

There is a reason why EMC is so expensive.. Its the number 1 storage provider out there. To kill a vmax takes some serious effort and you need to have engineering involved to do it. If it tanked on its own, there was most likely a EPAC you did not apply or ignored your warnings from your CEs. On top of that.. having EMC on your resume is a much better look then some chumpy ZFS software based solution.
 

Hurin

2[H]4U
Joined
Oct 8, 2003
Messages
2,411
My goodness. Don't I feel like small potatoes?

Running about 15 servers (windows and linux) as virtual machines in not-for-profit environment. Not life and death nor monetary losses involved if we're down for a few hours.

So I roll with a 20TB RAID 6 array of direct attached storage on one hypervisor (Windows Server 2012 R2). And then I have a near-identical hypervisor on the other side of the building running a few VMs. Either one can act as the "lifeboat" for the other in case of hardware failure. Thankfully, *knock on wood* it has never come to that in the six years I've been running things this way.

Oh, we also backup nightly to a Microsoft DPM server and weekly to tape. Tapes are taken offsite.
 

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
You said "tapes"....:p

Please let's not talk about tapes. We had a customer using them up until about 1 month ago. A wonderful HP autoloader with 8 tapes. Funny thing is just 2-3 months ago they needed a restore from 2013 and still had the tapes. The restore was actually successful.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
I completely crapped on tape, and got rid of it in my gig. However.. I'm forced to bring it back. I am in a legal hold, and due to the legal hold I am unable to delete any data. The legal hold can go on for another 6 months or 6 years. Don't know. Trust me.. I did my research.. There is no cheaper long term solution out there other then tape. Right now I'm backing up to 3 completely maxed out data domain 860s, (124Ts, raw, I'm getting 6:1 compression and dedupe), and then moving anything that's older then 60 days to my 600TB isilon install. Yes, I'm using dedupe and compression via my backup software over to Isilon.

And its still not enough. So here comes a new tape lib with LTO6 drives. *kicks dirt*
 

Thuleman

Supreme [H]ardness
Joined
Apr 13, 2004
Messages
5,833
Although I do have a couple running Epic on Tintri.

Somewhat curious on that. Given the size of a Tintri, even the smallest one, those Epic customers essentially mix Epic storage workloads with some/all of their other VM workloads?
 

niomosy

Limp Gawd
Joined
Nov 21, 2005
Messages
247
Please let's not talk about tapes. We had a customer using them up until about 1 month ago. A wonderful HP autoloader with 8 tapes. Funny thing is just 2-3 months ago they needed a restore from 2013 and still had the tapes. The restore was actually successful.

Tapes. Ugh. Somewhere in the data center is a Sun Ultra 60 still on Solaris 8 that runs ACSLS to handle tape robotic arm duties for whatever tape unit we've got left for legacy stuff the business said we just had to keep.
 

Hurin

2[H]4U
Joined
Oct 8, 2003
Messages
2,411
We're on LTO-4, but eyeing LTO-6 for our next library. I'm required to keep quarterly archives because you just never know when someone in our organization will say: "I need something that I deleted off the server sometime in 2012." Between the 30 days of D2D backups, and the offsite tapes, I worry a lot less that a gap in our backups is going to bite someone or that a plane hitting the building will erase our very IT existence. =)
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,608
Somewhat curious on that. Given the size of a Tintri, even the smallest one, those Epic customers essentially mix Epic storage workloads with some/all of their other VM workloads?

Yes.
Oh, did you move away from VMware?

Yep, 6 months ago :)

define small.

I work in the financial sector which is a huge sector.. would suck if you lost access to your money while trying to pay for dinner on a hot date.. all cause an active/passive system with a 10-15 second RPO is in place.

Many of the financial places I worked with did various forms of clustering or active/active datacenters without a stretched cluster using other methods of data movement or redundancy - it all depends on the size and need. The big CC processors are also very heavy on the UNIX side still, which is its own world in terms of storage and architecture.
 

kdh

Gawd
Joined
Mar 16, 2005
Messages
806
my worked is roughly 75% Linux, 20% ESXI, and 5% Windows. You bet your bottom dollar that our key financial systems run on Linux and not Windows.
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,608
my worked is roughly 75% Linux, 20% ESXI, and 5% Windows. You bet your bottom dollar that our key financial systems run on Linux and not Windows.

I didn't even mean windows - I meant UNIX. Last few I dealt with were still running AIX/HP-UX or even old mainframe setups for it :p
 

niomosy

Limp Gawd
Joined
Nov 21, 2005
Messages
247
Hell, we had a nice quantity of OSes running for a while. AIX, Linux, Solaris, and HP-UX along with the other stuff. Got rid of HP-UX since that was, fortunately, only a single app that migrated off an HP 3000. It's now on Linux.

These days the vast majority of my time is spent dealing with Linux, occasionally wishing to be back on the old proprietary Sun/IBM platforms where drivers had a better chance of working. Looking at you, HP Gen9 hardware and RHEL 6 with you driver/firmware updates in the last few weeks! Having 5 Splunk servers simultaneously shit themselves and reboot every few minutes after that, doing kernel panic's and resetting each time, isn't tremendous amounts of fun.

Not that I'm angry and bitter about this happening Friday or anything... ;)
 

lopoetve

Fully [H]
Joined
Oct 11, 2001
Messages
32,608
Oh lord. Try slipstreaming drivers into a PE-boot image for those boxes for windows... It'll make you cry - it's all a 100% binary blob that WANTS to run the exec installer instead of providing files and inf bits for slipstreaming. We finally gave up.
 

Eickst

[H]ard|Gawd
Joined
Aug 24, 2005
Messages
1,884
We're on 3par and love it. About to buy another one for second datacenter.

Looked at EMC, NetApp, and Compellent. In the end it was between 3par and compellent and one of the deciding factors was 3par's veeam integration since we already used veeam.
 
Top