Netapp

Foulsmell

Weaksauce
Joined
Jun 29, 2000
Messages
88
Anyone use a netapp 3000 series san? we are looking to get a enterprise grade san for virtualization and I am personally leaning towards a netapp 3040 with 10gbe connectivity using it in nfs mode. Starting out with around 5 hosts with 50ish vm's and soon adding 4 to 5 more hosts running VDI. Total starting capacity around 8 to 10TB usable. Currently talking to several vendors all with different solutions. Just wanted to get a feeling for anyone doing this kind of thing. I know once this really catches on its going to explode... ever since I implemented ESX for production shit went wild real quick with server growth. lol.:D
 
I'm there man. I've got 2 NetApp SANs - both supporting 17 fat ESX hosts and 300ish VMs. I'm the storage guy and the server guy and the VMWare guy. I'm running a 3070C cluster with 16 shelves of 15K FC, and a 3020C cluster running 8 shelves of 15K and 10K FC mixes. I'm all 1gbe iSCSI right now, but am desperately trying to convince my uppers to move to 10gbe NFS. Honestly though, performance has been pretty stellar on iSCSI. I want to skip fiber channel altogether and get into 10gbe - that's where it's at.

You're thinking is right on the money with NetApp and NFS - good stuff, fast and rock solid. Customer support is amazing. And since NetApp only sells 15K drives now, it's gonna be a screamer. I plan on upgrading my 3020's to the 3040's or whatever's down the pipe a year from now. My 3070 cluster is a beast. I've got those 2 filers running our entire Exchange environment, a ton of heavy I/O SQL servers and a crapload of VMs and more.
I assume you're buying fiber channel drives, right?

Remember, spindle count is king with NetApp. Buy as many drives as you can afford and throw as many as you can afford into one fat aggregate. Try to make your ESX aggregate as big as possible to fit as many of the the recommended 16-disk RAID groups as possible. Keep the root vol on it's own tiny aggregate for flexibility and stability.

Any questions, let me know man - I live n breathe VMWare (VC, VDI, ESX), NetApp and Server 03/08 all day, every day.

Oh, and yeah, once you open the door with ESX, it's like a big ass bomb blows up and 6 months later you've got hundreds of VMs. Every T, D & H wants their own server for every environment. Virtualization is supposed to SAVE money, I've yet to see the savings, ha.
 
thank you for the info. its coming down to a netapp versus hp lefthand networks san. lefthand doesnt do nfs but the management seems easy, even though from what it looks like you gotta build more luns with it.
 
I guess you like bending over for marketing lies?

I run a lot more VMs - more than double what you're planning - on a 4 node IBM SVC fronting a badly configured DS4800. With a lot, lot more disk.

It's overpriced overkill. You could do that load with an IBM DS3400 at less than 1/4 the cost - about $25K out the door, including the SAN switches.
 
I guess you like bending over for marketing lies?

I run a lot more VMs - more than double what you're planning - on a 4 node IBM SVC fronting a badly configured DS4800. With a lot, lot more disk.

It's overpriced overkill. You could do that load with an IBM DS3400 at less than 1/4 the cost - about $25K out the door, including the SAN switches.

Who are you referring to?

NetApp is solid stuff. Would I use it if I'm ONLY doing VMWare ESX? I'm not sure, but NetApp+NFS+ESX is defintely a winning combo right now. Other cheaper options (EMC Clariion, for example, will NOT support an enterprise ESX environment)... I've seen first hand those cheaper SANs get dropped to their knees... out of IOPS... and need to be upgraded to real storage.

$25K for an enterprise SAN? I LOL'd. Some of us have differing opinions on SLA's, criticality, availablity and performance.
 
Nope you are wrong. We run EMC Clarion behind all of our VM's just fine. In fact, I would have to say the performance is pretty stellar. Clarions support both Fiber and ISCSI now (up to 10gb).
 
Who are you referring to?

NetApp is solid stuff. Would I use it if I'm ONLY doing VMWare ESX? I'm not sure, but NetApp+NFS+ESX is defintely a winning combo right now. Other cheaper options (EMC Clariion, for example, will NOT support an enterprise ESX environment)... I've seen first hand those cheaper SANs get dropped to their knees... out of IOPS... and need to be upgraded to real storage.

$25K for an enterprise SAN? I LOL'd. Some of us have differing opinions on SLA's, criticality, availablity and performance.

Did I say Netapp wasn't? Nope. Did you demonstrate you know nothing about IBM's offerings or the proposed configuration? Absolutely without a doubt. Did you prove that you're not adult enough to ask intelligent, reasonable questions before opening your mouth to slam people or hardware? Beyond any hope of redemption. Please exit the thread now; we have no use in the industry for more money-wasting zealots who have no interest in having a clue about storage. That, and we already have enough of them.

Beware anyone who slams anyone without any facts. Funny things. I know what Netapp costs. I know exactly what a 3040 costs, and I also know for a fact that IBM can sell you the 3040 for less than Netapp. I also know for a fact that the DS3400 is more reliable than the 3040, by sheer virtue of less moving parts (and having several of it's little brother DS3300's in active production.)

For the adults, option B would be to go DS3400 and a pair of Qlogic SB1404's - these aren't as solid (no redundant power supply) but with two of them it's not as big a deal. They're 10x4Gbit and about $1500 each before SFPs (~$100/ea), and you can get two into 1U.
The DS3400 scales to 5.4TB with 450GB SAS (recommended) or 12.0TB of SATA (absolutely do not use SATA for VMWare period. I don't care who's array. DON'T.) - this is before EXP3000 shelves which will add 5.4TB each to a maximum of 3 additional shelves (21.6TB total.)
Dual power supply and dual controller? DO IT. It's a minor price difference, and outweighs the cost in reliability. I have not had any of these fail ever, but it's not worth the chance. Again, it's a drop in the bucket - especially compared to a Netapp or EMC.
For switches, there's the IBM SAN24-B (which is a rebranded Brocade) which I do not recommend, due to licensing antics. Or the Cisco MDS9124 Express, which again, I do not recommend due to licensing antics. If you must though, the MDS9124 has dual hot swap power supplies. But your guess is as good as mine as to when Cisco will chuck it to the curb, with everything going "OMG NEXUS."
You need two switches - FC requires two distinct fabrics for reliability and scalability. This is just how it works. If you're in a BladeCenter, simply drop in two 20-port QLogic blade switches, and you're done. Seriously. They're awesome and give you two fabrics. If you can't, QLogic is the hot ticket for cost-benefit. I very highly recommend the 5600 series of QLogic switches. They are very affordable, very reasonable licensing, and very high performance. Make sure to get 5602s, which have dual hot swap power supplies. They're licensed 8 ports at a time, so for your configuration, you would only need the initial 8 port license - the 4x 10Gbit ports are always on and already licensed permanently.

IBM support is IBM support. Be aware that if you buy an IBM SAN24 or a Cisco MDS through IBM, you will be forced to call IBM first, who will then call Brocade or Cisco as appropriate when needed. This can delay support calls significantly in some situations. This does NOT affect hardware repair and replacement; IBM keeps parts local. (If you think they might not, ask for your area service manager.) QLogic has excellent support, including for hardware failures, and you do talk to them directly so there's no delay. You also have the option of another seller and getting a SmartNET contract on a Cisco MDS, or the same for Brocade. I would not recommend anything larger than the QLogic 5602 - even that is arguably overkill, but this is the smallest stuff available short of the QLogic 1400.

Here's some relevant and useful links for you:
http://www-03.ibm.com/systems/storage/disk/ds3000/ds3400/index.html
http://www-03.ibm.com/systems/storage/san/b-type/san24b-4/express/browse.html
http://www.cisco.com/en/US/products/ps7079/index.html
http://www.qlogic.com/Products/SAN_products_FCS_san1400.aspx
http://www.qlogic.com/Products/SAN_products_FCS_san5600Q.aspx

Understand as well that FC is most likely going to be easier than iSCSI (which is a crapshoot due to initiator hell - that's just a fact. People who say otherwise are zealots or have no real FC experience.) It also buys you NPIV, depending on switch, going forward. Which buys you a lot in terms of scalability and reliability with VMotion on 3.5 - disks follow virutalized WWPNs attached to the VM rather than having to zone disks to every host. Understand, that's not fully implemented everywhere yet. You'll need to talk to VMware and whoever makes your switch. The DS3x00 family are NPIV okay. Send me a PM if you have NPIV questions, as they're not totally appropriate here. (I've had to become an expert with our p595 and AIX 6.1 deployment here.)

Hope this was actually helpful, as opposed to what others have wasted our time with.
 
Did I say Netapp wasn't? Nope. Did you demonstrate you know nothing about IBM's offerings or the proposed configuration? Absolutely without a doubt. Did you prove that you're not adult enough to ask intelligent, reasonable questions before opening your mouth to slam people or hardware? Beyond any hope of redemption. Please exit the thread now; we have no use in the industry for more money-wasting zealots who have no interest in having a clue about storage. That, and we already have enough of them.

Beware anyone who slams anyone without any facts. Funny things. I know what Netapp costs. I know exactly what a 3040 costs, and I also know for a fact that IBM can sell you the 3040 for less than Netapp. I also know for a fact that the DS3400 is more reliable than the 3040, by sheer virtue of less moving parts (and having several of it's little brother DS3300's in active production.)

For the adults, option B would be to go DS3400 and a pair of Qlogic SB1404's - these aren't as solid (no redundant power supply) but with two of them it's not as big a deal. They're 10x4Gbit and about $1500 each before SFPs (~$100/ea), and you can get two into 1U.
The DS3400 scales to 5.4TB with 450GB SAS (recommended) or 12.0TB of SATA (absolutely do not use SATA for VMWare period. I don't care who's array. DON'T.) - this is before EXP3000 shelves which will add 5.4TB each to a maximum of 3 additional shelves (21.6TB total.)
Dual power supply and dual controller? DO IT. It's a minor price difference, and outweighs the cost in reliability. I have not had any of these fail ever, but it's not worth the chance. Again, it's a drop in the bucket - especially compared to a Netapp or EMC.
For switches, there's the IBM SAN24-B (which is a rebranded Brocade) which I do not recommend, due to licensing antics. Or the Cisco MDS9124 Express, which again, I do not recommend due to licensing antics. If you must though, the MDS9124 has dual hot swap power supplies. But your guess is as good as mine as to when Cisco will chuck it to the curb, with everything going "OMG NEXUS."
You need two switches - FC requires two distinct fabrics for reliability and scalability. This is just how it works. If you're in a BladeCenter, simply drop in two 20-port QLogic blade switches, and you're done. Seriously. They're awesome and give you two fabrics. If you can't, QLogic is the hot ticket for cost-benefit. I very highly recommend the 5600 series of QLogic switches. They are very affordable, very reasonable licensing, and very high performance. Make sure to get 5602s, which have dual hot swap power supplies. They're licensed 8 ports at a time, so for your configuration, you would only need the initial 8 port license - the 4x 10Gbit ports are always on and already licensed permanently.

IBM support is IBM support. Be aware that if you buy an IBM SAN24 or a Cisco MDS through IBM, you will be forced to call IBM first, who will then call Brocade or Cisco as appropriate when needed. This can delay support calls significantly in some situations. This does NOT affect hardware repair and replacement; IBM keeps parts local. (If you think they might not, ask for your area service manager.) QLogic has excellent support, including for hardware failures, and you do talk to them directly so there's no delay. You also have the option of another seller and getting a SmartNET contract on a Cisco MDS, or the same for Brocade. I would not recommend anything larger than the QLogic 5602 - even that is arguably overkill, but this is the smallest stuff available short of the QLogic 1400.

Here's some relevant and useful links for you:
http://www-03.ibm.com/systems/storage/disk/ds3000/ds3400/index.html
http://www-03.ibm.com/systems/storage/san/b-type/san24b-4/express/browse.html
http://www.cisco.com/en/US/products/ps7079/index.html
http://www.qlogic.com/Products/SAN_products_FCS_san1400.aspx
http://www.qlogic.com/Products/SAN_products_FCS_san5600Q.aspx

Understand as well that FC is most likely going to be easier than iSCSI (which is a crapshoot due to initiator hell - that's just a fact. People who say otherwise are zealots or have no real FC experience.) It also buys you NPIV, depending on switch, going forward. Which buys you a lot in terms of scalability and reliability with VMotion on 3.5 - disks follow virutalized WWPNs attached to the VM rather than having to zone disks to every host. Understand, that's not fully implemented everywhere yet. You'll need to talk to VMware and whoever makes your switch. The DS3x00 family are NPIV okay. Send me a PM if you have NPIV questions, as they're not totally appropriate here. (I've had to become an expert with our p595 and AIX 6.1 deployment here.)

Hope this was actually helpful, as opposed to what others have wasted our time with.

Holy cow dude. Are you an IBM salesman? Will your previous post net you some kind of commission? Arguing storage is a losing battle for everyone involved with your kind of gustapo attitude, because eveyone thinks they've got the only solution worth buying -because it happens to be working for what they're using it for. You don't even know me, what I do, how many years of enterprise storage/systems experience I have and how many productions SANs I've worked with. You don't need to get shitty with me or anybody else because it only makes you look like a defensive, biased prick. I've got fellow storage guys replacing their EMC Clariions with NetApps and others because they simply couldn't handle the their virtual environment. That's a fact. Not poor implimentation, simply not enough IOPs to serve their needs. Period. We have no idea if the OP is going to use the SAN strictly for ESX or not. Does he already have or want the flexiblity to serve conventional SAN LUNs? Just how much redundancy / criticality is required of his inital purchase? How scalable does he need it to be? Those are questions that should have been asked. If I knew this was going to be a thread on architecting / designing a storage solution for the OP, I would have spouted off a page of FC switch / fabric design and redundant power supply lessons as well... where the industry is going etc.

Regardless, a $25K SAN will be obsolete as soon as it's racked in my opinion. There are no magic bullets. There are smart choices, but enterprise level storage is expensive, no doubt. Can you overbuild and overspend? Absolutely. Can you underspec and waste money? Absolutely. When I am asked to spend money, I'm doing it with a boatload of requirements placed upon me that simply remove the possibilty of potentially cheaper storage options. Also, at the time of purchase of my last two SANs, SAS wasn't even an option, in any brand, only FC and SATA. And we all hopefully know about SATA. Hence, WHY I said if I was going to buy a SAN ONLY for ESX, I don't know if I would make the same choice with the technology out now.
Also, I never once attacked IBM like you apparently think I did. I work with IBM, Cisco, QLogic, HP and tons of other stuff all day everyday.
Chill out man. No need to get all worked up. I really hope [H] isn't the only place the OP is getting storage purchasing advice from.
 
You guys are great. Tons of great information. On my part i can provide the fact that this san will be for a health care environment thus be critical in uptime since it is 24x7x365. It will mostly be used for ESX production environment with some testing and a few RDM's thrown in for a couple exceptionally large (well at least for us lol) 1.5 terabyte SQL databases. Many of the servers run SQL databases (vendors dont wanna share instances), with a couple sybase and i believe one oracle db in there somewhere in the mix. Down the road we do plan on doing desktop virtualization and migrating to exchange 2007 on a VM (the last one is still undecided). Like many shops im the sole person doing everything vm related (network,storage, esx, and backup), thus all your input is greatly appreciated.

As for what SAN we choose, well it is still up in the air. So far have seen a lefthand demo, looked pretty easy to manage but question performance of their whole network raid special sauce. lol
 
Who are you referring to?
Other cheaper options (EMC Clariion, for example, will NOT support an enterprise ESX environment).

I have to say that i've found the opposite as have quite a few other people....which is why the Gartner server virtualisation study showed that some 50% of VMware storage was sat on EMC...while only 4% was sat on Netapp.

Also, cost doesn't really come into it. If you think your Netapp SAN was expensive and EMC was cheap then Netapp ripped you off. Remember - there are no set prices. You have list price and then whatever the vendor wants to give you off.
 
as a netapp shareholder this thread makes me happy :)

We also have 12+ shelves in production and a few in qa as well; we're testing out that netapp feature that snapshots and moves blocks of data around in minutes.
 
I pretty much do the same work as the OP in a 450 bed hospital. We are currently running an IBM n5300 (Rebranded Netapp FAS3040) for Exchange and VMware with 10 drawers and are considering adding another 12 drawers (two drawers from max config). We've been running the config for almost two years now and it works great. We also run 2 x 2node IBM SVC cluster front ending two DS4500 and a DS8100 over MDS 9509s. We are a pure IBM shop on SAN and I have to disagree with IBM zealout above. IBM essentially hasn't done a thing to refresh their storage in 5 years other than resell Netapp SAN and buy XIV (which is cool but not really ready for 24x7x365 from what we saw).

The ds4500s and the DS8100 are due for a refresh in the next year and we've done some major shopping around. Netapp (direct) was our initial first choice but after talking to HP, Hitachi, Netapp, IBM, EMC, and Compellent I would definately recommend you at least talk to Compellent. They are clearly the next Netapp with 10 years in the business and many large installs (The FBI uses them for instance). They do real block level automatic tiering, snapshots that are mountable read/write, and from we are seeing will save us a great deal on bulk storage. They do some innovative things with writes (all writes are done as RAID10 and then migrated to RAID5 using background processes). I don't want to sound like a salesmen but there are definitely hospitals running their hardware 24x7x365 without an issue (They use the same drawers as Netapp even). I would at least sit through their WebEx before you drop money on a Netapp (which I also like a lot).

Oh and on the FAS3040 don't get it, get the FAS3140 the performance is supposedly much much higher and it is replacing the FAS3040 in the channel now.

V@nill@: We've quoted EMC against Netapp before and EMC basically said "We can't beat that price for that amount of storage or spindles." Followed by their usual "EMC is superior to everyone else and you would have to be stupid to suggest that another company could do storage as well as us regardless of what the price is." Speech.
 
I
V@nill@: We've quoted EMC against Netapp before and EMC basically said "We can't beat that price for that amount of storage or spindles." Followed by their usual "EMC is superior to everyone else and you would have to be stupid to suggest that another company could do storage as well as us regardless of what the price is." Speech.

Fair enough, in different situations different vendors propose different solutions that may have cost differences. Gartner (again) have a good doc where they benchmark the vendors' (EMC, Netapp, IBM, HDS, 3Par) costs based on a defined set of requirements.

If any of you guys can get a Gartner account through your work it's well worth it to get general input on what the industry is doing and how much things should cost.
 
What kills me about Gartner is that they place Hitachi and Dell in the far left side of their magic quadrant yet HP and EMC are on the right-hand side. HP's arrays are made almost exclusively by Hitachi (as are Sun's) and Dells arrays are mostly made by EMC these days. It also makes me chuckle when they still put IBM in the upper right-hand side in 2008... are they kidding? When was the last time IBM updated their product lines (other then rebranding Netapp products?) the DS8100 is a dinosaur now and the DS4xxx series just keeps getting a facelift to support new drives with little changes in its capabilities. They had to buy XIV just to have a product that competes in the next gen SAN market... but a digress. No doubt an IBM zealot will chime in to disagree but where has IBM innovated lately with storage in the last 3 years?
 
What kills me about Gartner is that they place Hitachi and Dell in the far left side of their magic quadrant yet HP and EMC are on the right-hand side. HP's arrays are made almost exclusively by Hitachi (as are Sun's) and Dells arrays are mostly made by EMC these days. It also makes me chuckle when they still put IBM in the upper right-hand side in 2008... are they kidding? When was the last time IBM updated their product lines (other then rebranding Netapp products?) the DS8100 is a dinosaur now and the DS4xxx series just keeps getting a facelift to support new drives with little changes in its capabilities. They had to buy XIV just to have a product that competes in the next gen SAN market... but a digress. No doubt an IBM zealot will chime in to disagree but where has IBM innovated lately with storage in the last 3 years?

There are many Gartner magic quadrants and you have to be aware that there are different ones for midrange, high end, NAS, Management, etc.

HP and HDS will be split out because HP have their own arrays (the EVA, pos!) and HDS have their mid range array. In the high end HP use the HDS arrays (which are really very good from a hardware perspective).

EMC and Dell will be split out because Dell support their own rebradged EMC arrays and most recently they have their own arrays in the form of Equalogic.

In the 2008 mid range quadrant EMC and Netapp lead, followed by Dell, IBM, HDS and HP a little way away.

I agree that IBM shouldn't be in there (and also disagree that XIV is a competitor to the main vendors yet) and think only EMC and Netapp deserve to be up there. I think 3PAR deserves a little more recognition. IBM is so high because the quadrants cover other factors such as overall business service, overall business health, execution, etc - things IBM score high on because of their name and the fact they do manage a whole lot of environments end-to-end.
 
Honestly I haven't looked at them much as they appear to specialize in smaller arrays (half rack and less) from what I can see. A Dell rep called me a couple weeks ago to try and sell us on EqualLogic but frankly he seemed to know nothing about his own product so I gave up pretty quickly.

V@nill@: On the XIV, IBM owns them now and is pushing it as their new Tier1/2 option quite hard. We've met with IBM on two occasions where they basically told us it is replacing their DS8000 series in the near future for everyone except the top 1% of shops for IO requirements. It struck me as kind of desperate as IBM typically hasn't been one to advocate early adoption (they are so far the other way sometimes that it leaves them in the dark ages). That and although I am willing to believe that you can mask the slower performance of SATA drives with enough frontend cache and horsepower it would be a near impossible sell to any IT manager as Tier1/2 as soon as you mentioned it used SATA disk only.
 
No doubt an IBM zealot will chime in to disagree but where has IBM innovated lately with storage in the last 3 years?

Don't go knocking India Business Machines. How about help desk innovation? If your company has laid off talented American employees and replace them with cheap H1B Visa slave workforce, IBM offers a "native" speaking help desk for your IT staff.

IBM sold their Storage division years ago, so now they're no more than VAR for other storage companies.
 
V@nill@: On the XIV, IBM owns them now and is pushing it as their new Tier1/2 option quite hard. We've met with IBM on two occasions where they basically told us it is replacing their DS8000 series in the near future for everyone except the top 1% of shops for IO requirements. It struck me as kind of desperate as IBM typically hasn't been one to advocate early adoption (they are so far the other way sometimes that it leaves them in the dark ages). That and although I am willing to believe that you can mask the slower performance of SATA drives with enough frontend cache and horsepower it would be a near impossible sell to any IT manager as Tier1/2 as soon as you mentioned it used SATA disk only.

Sure, I know they're around and pushing hard. They're currently struggling to get any sales. They've given away many boxes to the big banks to play with, and most are seeing it as a good tier 4/5 product but are not buying.

There are just too many things about it that leave people shying away.

* All SATA, as you said, is one reason.
* Only one RAID choice, RAID X.
* Only two configs, 80TB half rack or 180TB full rack. No other configs to date, so even if you only want 10TB day 1 you need at least 80TB (which only offer 27TB usable or 72TB usable per array!?!? That's pretty poor efficiency from 80 or 180 drives).
* You cannot control data placement, applications can interfere with eachother and you can't tweak the box.
* The RAID config means that if a drive fails...and you lose a second drive in that window before the rebuild completes you lose all data on the array. All of it. The whole thing because the way RAID X works.

Tier 4/5? I'd be happy to take those risks. Tier 1/2? You'd be mad and out of a job no matter how much you thought you'd never get fired for buying IBM.
 
EqualLogic doesn't have anyone anywhere that knows the product. You can only make one array per controller - period. The controller is extremely unreliable. Performance is best represented by a trip to the bathroom. We had a hard time not laughing through a presentation where they claimed 2x1Gbit ethernet was faster than 4x2Gbit FC.

XIV - stay the hell away. Seriously. XIV has several very bad design choices. You will never, ever be able to expand an XIV by adding disks. Ever. You are locked into forklift upgrade with software limiting, period. It's impossible to properly load level the 1MB blocks if you add physical drives, leading to very bad hotspots. The whole core design of XIV can be summarized as thus:
If X goes at Y IOPS and Z MB/sec, then let's just bodge a TON of X into a single package.
In other words, it's quite literally identical to saying that you need so many CPU cycles, and rather than buying one or two efficient systems, you buy two or three cabinets of significantly less efficient systems and just use a sledgehammer till it's fast enough. And you can't buy less than a cabinet, and you can't ever change anything without replacing it all.
V@nill@ might be wrong about data failure though, from what I've been told by IBM. That's been addressed by prioritizing block rebuild, and backed by the block randomization supposedly. However, I don't buy into it 100% (obviously) and haven't been able to get my hands on a demo or lab. This is based on some low level discussions with IBM reps - I have not yet spoken to any high level people about it. Our local rep is working on getting me some face time with PE's, so yes, I will be asking about that.
The statement that XIV is replacing DS8k is decidedly false. IBM may be stupid, but they're not that stupid; DS8k customers will not touch XIV. It is not mature enough, not stable enough, not enough fault tolerance, and not expandable.

rabident, afraid you're flat out wrong. IBM sold the hard drive division, yes. But they've never not been majority rebranded on storage with at least some in house. Period. The DS4500 was... okay, let's skip that particular horror. But the DS4100, DS4200, DS4700, DS4800, DS5100 and DS5300 are LSI/Engenio designs. The DS3k-family is also LSI/Engenio designed and built. The DS300 ISCSI trash, Adaptec.
All SSA storage has always been 100% purely IBM (and still offers more than respectable performance.) The DS6k family (which haven't had their guts replaced by DS4700 controllers yet) and the DS8k family too, are 100% pure IBM in house.

The DS4k/DS5k? After the numerous firmware problems, over and over again, since October basically have moved them to the "not recommended" branch. Thankfully, the DS3k's have been relatively unaffected. I have several humming along happily. (If you're running 7.36, get the hell off it if you can, if not? Pray, and put 800-IBM-SERV on speed dial with your MTM memorized.)
Unfortunately, there is not exactly any competition at those price-performance point. A DS5300 controller can be had for about $84K out the door including 3 year 24x7x4. Expanding on that, a DS5300-53A with 8GB of cache, 5x EXP5000 shelves, and 5x16x 450GB 15k FC disks will set you back around $315-320K list. Bump it $16k for the 8-16GB cache upgrade. (Obviously, you will never pay list.) In RAID5, that gives you 5.4TB/shelf including hotspare, or ~27TB @ $320K. It also gets you somewhere over 500MB/sec simultaneous read and write. (You can easily verify this at ibm.com, and I do invite you to. I also invite you to challenge any other vendor to achieve that price point.)

Expanding on this some more, the DS3400 is capable of over 8000 IOPS and will set you back well under $100K including a full SAN setup with redundant switches and HBAs. And IBM sets records with it. See: ftp://ftp.software.ibm.com/eserver/benchmarks/news/newsblurb_x3950M2-8P_DB2_tpcc_081908.pdf
That was set August 19th of 2008. So, yes, the DS3400 is one hot as hell box for the money and then some. How hot? >800MB/sec, >8000 IOPS. I've put one through the paces, and it blew the doors off the much more expensive DS4200 - which has been dropped with the DS3400 as the marketing replacement. Remember, the DS3400 is not the DS3200 or DS3300 - it's a multi-channel SAS platform with FC host connection, versus multi-port SATA for the smaller models.

Anyways, I'm again sticking with the DS3400 recommendation for a 10 host environment. If you're trashing a DS3400, then you need more hosts, or you need to look at what you're doing. Those little boxes are happily capable of saturating 3 LTO4 drives with TSM - no small feat. Chances are that you would be using <50% of the DS3400's capability, even in a RAID5 configuration without the cache upgrade. Our average IO load for about 100 VMs on ESX across 12 blades - including all the network shares - is less than 70MB/s and less than 5000 IO/s.

Now, if you want to get something seriously enterprise? Break out the checkbook. Throw down multiple DS3400's, and get an IBM SVC. It's back end agnostic, so hell, go EMC. Go Netapp. Go HDS. (I like HDS' AMS stuff, for the record.) Go 3PAR. Go BlueArc! It doesn't care.
It wants FC LUNs. It turns FC LUNs into MDisks, spreading your data across not just multiple spindles, but multiple arrays and multiple controllers. The SVC coupled with the DS8300, owns every SPC benchmark lock stock and barrel. (>10GB/s random.) You can get insanely advanced with the SVC, tweaking things at multiple points. You also get the excellent SDD driver, which is just faboo. Oh, and ESX 3.5 and later <3's the SVC. I can go into more detail if folks would like, as I'm running SVC.

It's stupid late here, so I skipped over lots. I want to touch on HDS / HP XP-series (which are Hitachi, period.) I also wanna talk about EVA and BlueArc. I'll try and get back to it at a later point if I have time. I'm kinda in the middle of a forklift of everything in the datacenter (and I do mean everything) so I've been researching all this stuff and talking with vendors since last October. If you have a question, I have probably already asked it, and gotten answers. So ask away. If I don't have the answer, I'll see about getting it.

P.S. Please save me from more marketing drivel. ARRGH.

(Edit Note; I typo'd and said <50MB/s for our environment, it should have read 70MB/s - average is ~62MB/s excluding the internal SAS disks with the ESX paging. They are not SAN boot for ESX.)
 
V@nill@: AreEss is right on the rebuild times, we actually forced a rebuild on a demo unit by yanking a drive and the XIV apparently can tell how full a drive is and only need to rebuild that portion of it. That being said because of the way they do raid (They actually never referred to it as RAID X but rather virtualized RAID10 during any discussion we had with IBM) they showed us a full disk (1TB) rebuild time of less than an hour. It was probably one of the most impressive things about it.

AreEss: IBM is that stupid, note: "The IBM XIV Storage System™ is an enterprise-class, Tier-1 disk system designed to deliver high performance...". If you click through to the brochure they actually show the market placement of the XIV as right next to the DS8100. Also note how it is listed with the "Enterprise Disk Arrays". IBM is definitely pushing this as Tier 1 and I would bet $1000 that they will add FC/SD disk support to the XIV in its next revision and phase out the DS8100 for it. I'd also bet that they'll quietly rename it to shed any prior bad press the platform got when it was younger >>> http://www-03.ibm.com/systems/storage/disk/enterprise/index.html

IBM clearly has nothing else in the pipe for a Tier 1 replacement of the DS8000 series, while every other storage vendor has been pushing forward with next gen systems.

I still wouldn't recommend a DS3400, its just an LSI array with an IBM brand on it (actually it appears it may be a Left Hand OEM, it looks identical to the HP 2000FC which is Left Hand). Yeah IBM has good support, but its still just an expanded disk array with fiber attach. For a small shop iSCSI probably makes more sense anyways with fiber HBAs running $2k for a pair plus the cost of a fiber switch.
 
V@nill@: AreEss is right on the rebuild times, we actually forced a rebuild on a demo unit by yanking a drive and the XIV apparently can tell how full a drive is and only need to rebuild that portion of it. That being said because of the way they do raid (They actually never referred to it as RAID X but rather virtualized RAID10 during any discussion we had with IBM) they showed us a full disk (1TB) rebuild time of less than an hour. It was probably one of the most impressive things about it.

AreEss: IBM is that stupid, note: "The IBM XIV Storage System™ is an enterprise-class, Tier-1 disk system designed to deliver high performance...". If you click through to the brochure they actually show the market placement of the XIV as right next to the DS8100. Also note how it is listed with the "Enterprise Disk Arrays". IBM is definitely pushing this as Tier 1 and I would bet $1000 that they will add FC/SD disk support to the XIV in its next revision and phase out the DS8100 for it. I'd also bet that they'll quietly rename it to shed any prior bad press the platform got when it was younger >>> http://www-03.ibm.com/systems/storage/disk/enterprise/index.html

IBM clearly has nothing else in the pipe for a Tier 1 replacement of the DS8000 series, while every other storage vendor has been pushing forward with next gen systems.

I still wouldn't recommend a DS3400, its just an LSI array with an IBM brand on it (actually it appears it may be a Left Hand OEM, it looks identical to the HP 2000FC which is Left Hand). Yeah IBM has good support, but its still just an expanded disk array with fiber attach. For a small shop iSCSI probably makes more sense anyways with fiber HBAs running $2k for a pair plus the cost of a fiber switch.

I'd recommend most mid tier boxes over an XIV. I think it will be ok one day, but today it's not good - it's got a long long way to go. If you're an IBM shop stick with the DS8000. If you can't stand the DS8000 start looking to EMC / HDS for tier 1.

Although they may have improved the rebuild times that's still one hour in which if you lose a second drive you lose everything. That's data loss and the next week recovering everything from tape (IF you have somewhere to recover to!).

I was in a DC last year where the air conditioning failed. One large array lost 6 spindles but had plenty of spares to compensate. If that has been an XIV I can't help but think there would have been data loss right there.

I would never call RAID X RAID 10! RAID X is AKA "Shotgun RAID" and is only taken on by a few vendors. What XIV does is instead of having hot spares they portion off a percentage of each drive for rebuilds. You lose a drive and it starts to replicate the surviving images to the space on other disks - so if a second drive fails before protection is restored the unprotected data is lost and the BIG problem is you wont know what is lost as a piece of each LUN may have been lost but you've no way to find out where or how much! Get those tapes out!
 
V@nill@: AreEss is right on the rebuild times, we actually forced a rebuild on a demo unit by yanking a drive and the XIV apparently can tell how full a drive is and only need to rebuild that portion of it. That being said because of the way they do raid (They actually never referred to it as RAID X but rather virtualized RAID10 during any discussion we had with IBM) they showed us a full disk (1TB) rebuild time of less than an hour. It was probably one of the most impressive things about it.

Yes, but that does NOT answer the multiple disk failure scenario. Nor does it handle any other number of scenarios.

AreEss: IBM is that stupid, note: "The IBM XIV Storage System™ is an enterprise-class, Tier-1 disk system designed to deliver high performance...". If you click through to the brochure they actually show the market placement of the XIV as right next to the DS8100. Also note how it is listed with the "Enterprise Disk Arrays". IBM is definitely pushing this as Tier 1 and I would bet $1000 that they will add FC/SD disk support to the XIV in its next revision and phase out the DS8100 for it. I'd also bet that they'll quietly rename it to shed any prior bad press the platform got when it was younger >>> http://www-03.ibm.com/systems/storage/disk/enterprise/index.html

It's put in Tier 1 because marketing is that stupid. Only an idiot believes that they expect it to replace the DS8100/DS8300. I've talked with our rep about it, at length. They have been thrown out of customers any time they were stupid enough to try it - they are not even remotely equivalent products. They are not even in the same class. They also have told me that under absolutely no circumstances whatsoever will they be adding FC disk support to the XIV. Unequivocal and absolutely authoritative denial - the controllers can't handle it, and it's too expensive. The possibility of even SAS is extremely low at best and would be at soonest 2-3 years out. SATA SSD disks possibly, but based on the thermal characteristics alone, forget it. There's also zero advantage to SSD SATA over mechanical SATA due to the XIV's inherent design defects.

IBM clearly has nothing else in the pipe for a Tier 1 replacement of the DS8000 series, while every other storage vendor has been pushing forward with next gen systems.

No, they've been pushing forward forklift upgrades. The DS8100/DS8300 will more than match anyone else's offerings, and has the same - or better - disk options. This is just cold hard fact, whether or not people like it. The reason they haven't done much with the DS8k's is because they got them right the first time around. Customers who want all the super-duper virtualization goodies are advised to put SVCs in front of DS8k's, which teaches them why the DS8k is still king of the hill.

I still wouldn't recommend a DS3400, its just an LSI array with an IBM brand on it (actually it appears it may be a Left Hand OEM, it looks identical to the HP 2000FC which is Left Hand). Yeah IBM has good support, but its still just an expanded disk array with fiber attach. For a small shop iSCSI probably makes more sense anyways with fiber HBAs running $2k for a pair plus the cost of a fiber switch.

You can not recommend all you like. However, you have major factual errors to put it mildly. Like, on the level of you shouldn't have opened your mouth.
Firstly, the DS3k are LSI. Period. I don't care if you question it; I have LSI PE's on speed dial. I know exactly what every LSI offering is. Secondly, it is not just "an expanded disk array" any less than an iSCSI pile of crap is. That is, again, cold hard fact - and often iSCSI isn't even a proper hardware array. EqualLogic is most decidedly not. iSCSI requires HBAs, period, if you want to approach "enterprise" - except iSCSI has absolutely no support for failover or path redundancy. That's a fact I got from every iSCSI vendor that wasn't lying - Adaptec, Qlogic and IBM all admitted that you cannot have iSCSI resiliency of any sort host-side. The IOE cards are also 20-30% more expensive per port than an 8Gbit Fiber HBA. And you still require a separate standalone non-resilient non-redundant server as an initiator. And no, you can't make it at all redundant.

The fact is that the only thing iSCSI buys you is a buzzword that locks you into a storage "platform" that they've been trying to push as the future for over 5 years without success. Oh, and massive, repeated headaches with a very low likelihood of ever getting things stable. What, you think it's been repeatedly slammed, thrown out, and laughed out of shops because it's faster than FC, like all these vendors lie? Get real.
 
I'd recommend most mid tier boxes over an XIV. I think it will be ok one day, but today it's not good - it's got a long long way to go. If you're an IBM shop stick with the DS8000. If you can't stand the DS8000 start looking to EMC / HDS for tier 1.

Don't waste your time or money on EMC. They've jumped the shark hard. Don't even get me started on their "virtualization" - it's anything but. It's less than worthless. And only fools pay licensing on their mandatory multipathing driver.

Although they may have improved the rebuild times that's still one hour in which if you lose a second drive you lose everything. That's data loss and the next week recovering everything from tape (IF you have somewhere to recover to!).
I was in a DC last year where the air conditioning failed. One large array lost 6 spindles but had plenty of spares to compensate. If that has been an XIV I can't help but think there would have been data loss right there.

Actually, it's far worse than that by what I've seen. Not just data loss of a LUN; because there's configuration blocks all over the place, there's apparently risk of dropping your entire XIV. The whole thing.
Also, you have absolutely no hotspare control on XIV. None at all. So, if you lose more than the fixed amount of hotspares - which is a guaranteed in that situation, given it's the hottest (thermally) box on the market - you are GONE. Not damaged. GONE. Irrecoverable. You are starting as though it were a new install.

I would never call RAID X RAID 10! RAID X is AKA "Shotgun RAID" and is only taken on by a few vendors. What XIV does is instead of having hot spares they portion off a percentage of each drive for rebuilds. You lose a drive and it starts to replicate the surviving images to the space on other disks - so if a second drive fails before protection is restored the unprotected data is lost and the BIG problem is you wont know what is lost as a piece of each LUN may have been lost but you've no way to find out where or how much! Get those tapes out!

There are physical hotspares, supposedly - a very, very small number. But that's because I'm also given conflicting data as to how the physical layout is. There's no question that it's 1MB blocks mirrored, but they have yet to answer whether it's a 1:1 mirroring, or this supposedly random garbage. They also haven't answered how many actual physical hotspares there are, other than "a very small number because it will never fail that bad," or even truly confirmed that.
In fact, every XIV person I have had occasion to talk with, has about as much technical data on the product as EqualLogic has on their trash. Maybe even less. The one thing they did agree on is that if you lose both copies of one block? Your LUN is gone for good. There is no recovering it. Get the tapes.
They're not happy dealing with me. I ask those hard questions that they can't answer or don't have an appropriate answer for.
 
Don't waste your time or money on EMC. They've jumped the shark hard. Don't even get me started on their "virtualization" - it's anything but. It's less than worthless. And only fools pay licensing on their mandatory multipathing driver.

I've never been a fan of Invista, their equivalent of SVC. HDS are smart because they sell it with every array so quite a few poeple end up using it. With EMC it's a separate solution and a hard sell, so there's a lot less of it out there although I do agree with its architecture (out of band and a collaboration with Cisco and Brocade). Incipient do exactly the same.

The big killer is that replication is replaced with something else....and if it's the one thing an EMC fan likes it's SRDF. They will not replace it. They will not entertain solutions that replace SRDF.

Also - do you mean powerpath multipathing? I don't think it's mandatory, they support MPIO, etc too.

Actually, it's far worse than that by what I've seen. Not just data loss of a LUN; because there's configuration blocks all over the place, there's apparently risk of dropping your entire XIV. The whole thing.
Also, you have absolutely no hotspare control on XIV. None at all. So, if you lose more than the fixed amount of hotspares - which is a guaranteed in that situation, given it's the hottest (thermally) box on the market - you are GONE. Not damaged. GONE. Irrecoverable. You are starting as though it were a new install.

There are physical hotspares, supposedly - a very, very small number. But that's because I'm also given conflicting data as to how the physical layout is. There's no question that it's 1MB blocks mirrored, but they have yet to answer whether it's a 1:1 mirroring, or this supposedly random garbage. They also haven't answered how many actual physical hotspares there are, other than "a very small number because it will never fail that bad," or even truly confirmed that.
In fact, every XIV person I have had occasion to talk with, has about as much technical data on the product as EqualLogic has on their trash. Maybe even less. The one thing they did agree on is that if you lose both copies of one block? Your LUN is gone for good. There is no recovering it. Get the tapes.
They're not happy dealing with me. I ask those hard questions that they can't answer or don't have an appropriate answer for.

Amen, this is a tier 4/5 box. I'd take anything over it. The only thing good about it is it is cheap.

I read a blog recently where a customer was at a demo where XIV asked the customer to pull two drives to simulate data loss. The trick was when you pull the first the rebuild starts but this takes seconds because the array is 99% empty.

Long story short when they put the drives back in they swapped them around so they each went in the wrong slot. Box spat out many errors and then, amazingly, copied the contents of the drive onto the data that was already on the system. They did this a few times to be sure.

I just hope it's an untrue and really bad rumour! but if true i'm not that shocked.
 
I've never been a fan of Invista, their equivalent of SVC. HDS are smart because they sell it with every array so quite a few poeple end up using it. With EMC it's a separate solution and a hard sell, so there's a lot less of it out there although I do agree with its architecture (out of band and a collaboration with Cisco and Brocade). Incipient do exactly the same.

OOB and non-standard inline are really, really bad ideas. Then you start screwing yourself over on the compatability side of the house. That's never a good thing. Invista, though, is about 10% of what SVC can do at it's best. And it locks you into Powerpath; EMC won't let anyone else work it right. (Like that's a surprise.) We're in the process of throwing out Cisco and Brocade due to licensing games.
Cisco yanked 90% of the advanced features on the MDS9500 family with a SAN-OS update and didn't let anyone know, then made it a mandatory update. Then they gave people on SUP1's and SUP2's the shaft with NEX-OS, which will not support SUP2's in future releases. Brocade and their per-port per-feature games is just completely ridiculous, I'm not interested. Especially when I can get everything I want and need in Qlogics at less than 25% of the price. (Oh, and the Qlogic is all full bandwidth ports, unlike Cisco.)

The big killer is that replication is replaced with something else....and if it's the one thing an EMC fan likes it's SRDF. They will not replace it. They will not entertain solutions that replace SRDF.

Yeah, SRDF and BCP spawns zealots. I should brick them in the face with SVC MetroMirror.

Also - do you mean powerpath multipathing? I don't think it's mandatory, they support MPIO, etc too.

They do and don't; without PowerPath you have a very limited feature set, and the official line I have always gotten is that if you have a host connection problem? Unless it's Emulex with EMC firmware, and PowerPath, you're on your own. Good luck! *click*

Amen, this is a tier 4/5 box. I'd take anything over it. The only thing good about it is it is cheap.

Actually, this piece of trash is anything but cheap. I have quotes here. I can stomp the crap out of it with DS5300's. Seriously. This RAID1 garbage is ridiculous; 12 drawers of 1TB SATA on a DS5300 gives you 180 drives (including hotspare on every shelf,) which works out to 36 RAID5 arrays (4+1) at 3725.29GB each, or 134100GB. Or roughly double what XIV gives you in the same space, with higher power draw and significantly higher heat output on the XIV. You can easily split it out to multiple DS4700's behind an SVC and quite literally trash the XIV in every test you like.
I wouldn't even take this as a Tier5 or scratch.

I read a blog recently where a customer was at a demo where XIV asked the customer to pull two drives to simulate data loss. The trick was when you pull the first the rebuild starts but this takes seconds because the array is 99% empty.

Somehow, I'm not surprised. I'm not at all surprised. That's a common trick. One of my favorite legitimate SVC tricks, was to demo it by putting it in front of a LOT of controllers. It's a perfectly valid and recommended configuration, except from a price standpoint. But the SVC in that configuration can make even 500GB Seagates beyond stupidly fast.

Long story short when they put the drives back in they swapped them around so they each went in the wrong slot. Box spat out many errors and then, amazingly, copied the contents of the drive onto the data that was already on the system. They did this a few times to be sure.

I just hope it's an untrue and really bad rumour! but if true i'm not that shocked.

Now THAT I haven't heard. I have XIV engineers coming here in an hour or so, and our IBM rep, so I have many many questions for them now. I will be absolutely thrilled to share their answers where I'm able (got reminded of various NDAs.) It certainly would NOT surprise me if it were true, it's a simple, stupid and easy design mistake to make. Don't verify data, if disk is marked "okay" just presume they put it back right and roll.
Incredibly stupid on their part, to put it mildly. But they also have an attitude that you'll never ever see two disks fail at once.
 
So, I was reminded before, during, and after what I saw that most of it was under NDA.

Yeah, I'll say it's under NDA. And with good reason. Suffice to say, IBM marketing was hard at work. They swiftly and quickly eliminated themselves from consideration as Tier 1 storage, and did it with a vengeance. I have never before in my life been so terrified at the thought of owning a storage platform as I was when I saw the details.

Since it's already been mentioned by multiple people, I believe I can confirm one thing. They did concede that yes, should you have two disks fail - the second failing during a rebuild - you WILL immediately have a data loss event. I can't go into further details on this though; again, NDAs.
 
Back
Top