• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Need SAN advice

d3mon187

n00b
Joined
Dec 6, 2011
Messages
7
I've been struggling to find a place for an unbiased opinion, so I figured I'd post on the good old Hard Forum. If you can put up with my novel, I'd really appreciate some feedback from my fellow IT brothers. For some reason I find it really difficult to trust vendors to give me an unbiased opinion...crazy huh?

I'm fairly new to the SMB/Enterprise sector, and I'm struggling with what to do with our storage. We are currently running an EMC AX4-5i, loaded down with 33 450gb 15k SAS drives. I have it broken down into a Raid 5 disk pool for backing up to, and a R10 disk pool that has all vmware datastores, VMs, file shares, etc loaded on to it. My problem started when I noticed almost no improvement in the growing of the R10 disk pool. When I first started, everything was split out onto multiple R5 disk pools. I read that best practice on the AX4 was to combine everything into one pool, so that every request was hitting the full performance of the array. Sounded like a great idea, so I started with a small R10 pool and began to grow it. I ran HD Tune Pro and PerformanceTest against the pool as I added each drive, but after only 6 drives it became clear that some of the tests had flat lined. I figured there had to be a bottleneck somewhere. Being an iSCSI array, I first turned to the network and saw high utilization on the ports. So I setup round robin in VMware, and tried the tests again. This then gave me 50-60% utilization max on each port, and no improvement in tests. The R10 pool is now at 18 drives, and still performs poorly. I have tried and read everything, talked with EMC for months, and I'm still not sure if the bottleneck lies in the SAN storage processors, the gigabit ports, or somewhere else in my network. I will be giving the SAN its own dedicated cisco 3560 switch soon, so I will see if that helps. EMC has been 0 help, and I'm really so tired of their non-service, which has been a big driving factor to get a new SAN. EMC says I'm getting good performance on IOmeter tests, but doesn't want to address the fact that I get worse numbers in HD Tune pro than the guy in this thread - http://hardforum.com/showthread.php?t=1582718 who is using 7200rpm sata disks. Crappy performance, horrible service, no future support, no new features, and it's not hard to see why I'm looking at getting a new SAN.

My network itself really isn't all that big though. 8 physical servers, 3 of which host about 4 or 5 VMs. We only have about 100 users total, but all are all extremely heavy on the network. Its a construction company, so there are constant huge reports run against several different SQL instances. We use Exchange, SharePoint, 2 terminal servers, and quite a few applications. The owners are generous with our budget because time is very important to the people in the office who are constantly in and out, and are responsible for commanding another 400 people in the field. Waiting 20 minutes or more for some of the reports to run can be a huge slow down for a lot of people, so I really work hard to make sure everything is running as fast as possible. Storage wise we only use about 3.5TB, which gives us about 1 TB to grow on our current system. We have another 3TB of fast drives used for backup. Would have liked to have just gone with 1TB 7200rpm drives for backup, but with EMC discontinuing everything on our 2 year "old" system, they were hard to find and crazy expensive.

So we've been looking for a new SAN. Compellent has been pushed hard, and I love all the features that come with it. Reviews seem to be pretty good online, and the price only makes me cringe a little bit (until I look at their drive costs). Still, I know there are a lot of options out there. I also cant really see me just throwing away our old AX4, or selling it for a crazy low price. Offsite DR has been a big goal of ours, and not even EMC supports using the AX4 for offsite replication to one of our remote offices. This lead me to the thought of building my own SAN for a little while, but the two software vendors I found with the most features (Starwind and Datacore) are pretty expensive themselves. To add to the confusion, I found someone willing to sell me their Compellent Series 20 with more drives than we will need for quite some time, for super cheap. It's all quite confusing. I certainly don't want to rush into this, and risk getting another crap SAN, or even waste money on something we didn't actually need.

Any advice or recommendations are greatly appreciated!
 
How are the various workloads separated accross the LUNs(SQL, Fileserver, APP)?

What types of servers are you using and how much RAM are they using?

How are the NICs configured in ESX?

What is the storage nic utilization on the box with SQL, RAM?
 
I have the disk pool split into 4 luns. One is a fileshare with the company's documents on it, one has the exchange databases, and the other two each have all of the VMs with seperate VHDs for the SQL and application drives.

Two servers are DL360G7s running dual Quad core e5640 Xeons, with 32GB ram (48GB after this week). Third VMware server is the same, except with dual hexcore e5645s. Most VMs are allowed 4-6GB ram, while the SQL and Exchange server currently have 12GB. All have quad port nics with 2 for the network, and two for toe iscsi with paths to all 4 storage ports on the SAN.

SQL server is in VM on a shared host, so I don't think I can really see the storage NIC utilization for that server alone?
 
yes get rid of raid5

yes rebuild array

datacore has tiering no idea how good it works however no dedupe at all

starwind has good dedupe usable for primary storage but no tiering at least for now
 
I have the disk pool split into 4 luns. One is a fileshare with the company's documents on it, one has the exchange databases, and the other two each have all of the VMs with seperate VHDs for the SQL and application drives.

Two servers are DL360G7s running dual Quad core e5640 Xeons, with 32GB ram (48GB after this week). Third VMware server is the same, except with dual hexcore e5645s. Most VMs are allowed 4-6GB ram, while the SQL and Exchange server currently have 12GB. All have quad port nics with 2 for the network, and two for toe iscsi with paths to all 4 storage ports on the SAN.

SQL server is in VM on a shared host, so I don't think I can really see the storage NIC utilization for that server alone?

So you have all disks lumped into the same raid group, with 4 lun's carved out?

Is the SQL server virtualized on VMWare? Are they managed through vCenter, you can monitor the storage NIC usage through the host.
 
So you have all disks lumped into the same raid group, with 4 lun's carved out?

Is the SQL server virtualized on VMWare? Are they managed through vCenter, you can monitor the storage NIC usage through the host.

Yes I do.

I see some spikes of 50mbps and 20mbps on the Disk (KBps) chart. Looking at realtime, I can see where someone was putting some strain on it for 40 minutes with a sustained 25MBps. At that time, disk requests were at 4,000, and network packets were between 175,000-200,000.
 
yes get rid of raid5

yes rebuild array

datacore has tiering no idea how good it works however no dedupe at all

starwind has good dedupe usable for primary storage but no tiering at least for now

We are using R10 for everything that matters. R5 is only used for the backup.

From what I've been hearing, I certainly need to rebuild the raid array. I can't believe these things aren't smart enough to re-stripe.
 
Well according to the tech at EMC, I don't need to rebuild the raid since it does re-stripe.

"On the AX4, when you add more disks to the Virtual Pool (called a raid group on the CLARiiON CX series), you're performing what's called a raid group expansion. This operation does re-stripe all the data over all the disks. On the CX4 and VNX they have a new concept called "Pool" - before we only had raid groups. With these new Pools, when you add more disks to the Pool, it does not restripe the data."
 
Yes, if you add spindles to a RAID Group it does restripe the data. Currently on the VNX it does not restripe when you add disks to a pool, but pools are very different from RAID Groups. Restriping for pools is coming soon.
 
Also, if you want you can PM me and I'll see about pulling analyzer data off that AX4 and looking at it to see if I can find the bottleneck.

And, the AX4 isn't really two years old unless you bought in right before they discontinued that model. Let me know what pricing you're seeing on drives and I can make sure it's right.

As for a new array... Compellent is "good" but not great. It's all or nothing. The answer to performance issues is "add more disks" for more striping. The AX4 line is now the VNXe line with pools and support for SSD, SAS, and NL-SATA. The SPs are also much faster and they have the best VMware integration in the industry.

I'm also happy to look at your VMware environment...if you have everything hitting the same drives you may want to look at splitting that out. 95% of our VMware datastore deployments are against RAID5, not RAID1/0. I'll do that on special cases but the vast majority is 4+1 and 8+1 RAID5 RGs.
 
One last thing. Who said EMC doesn't support the AX4 for offsite replication? It absolutely supports MirrorView for replication which is still a supported and deployed product/technology.
 
Sorry about the late reply. I guess it would be a good idea to subscribe to my own thread so I get notifications, lol.

Sadly, my boss did purchase the AX4 right towards the end of the models availability. He always likes to wait quite a while for any device or software to be out, and with the short life of the AX4, it put him at the end of the product cycle.

As for MirrorView, I'm not sure where I heard it, but I know someone had told me that the VNXe doesn't support it. Maybe they meant it just doesn't support synchronous, but does asynchronous?

We have a really bad taste in our mouth when it comes to EMC right now, so it just doesn't seem likely that we would go with another one. It may just be me, but the customer support we've gotten with EMC has been terrible. I'm sure the EMC tech hates me, and just thinks I'm complaining about nothing. Their only real solution to my concerns about performance was to add more disks, when the entire reason I became concerned about performance was that adding more disks wasn't increasing performance. EMC said Analyzer data showed it performing normally. Maybe its just my ignorance/stupidity of the differences between IOPS vs throughput, but I would think throughput would increase as well as IOPS when adding disks. I ran Passmark Disk test and HD Tune Pro tests, and EMC just dismissed them as bogus and would only use IOmeter. Not that I would dismiss IOmeter, but I just don't think Passmark and HD tune are in anyway bad tests either. We had run split Raid 5 before, but since we had the disks and didn't need the space, we decided to say the hell with it and just give it the beans with one big Raid 10.

One thing I have noticed is that something is going on with caching in VMware or windows. Running HD Tune Pro, on one of the 2008r2 VMs, I get some strange results. On the VM I have 5 drives/luns. First is a 4 disk r5 vhd on its own datastore, second one is a 6 disk r5 presented RAW to the VM, the other 3 are all on the same 16 disk r10 pool but are vhds on different luns. Only one of the vhds on the r10 pool seems to show caching in HD Tune pro though. Running a test on that drive I get around 70-80mbps followed by a huge jump to 4000mbps. On all 4 of the other drives they just stay around 70-80mbps for the entire test. I get the same results every single time. In windows all drives are set to better performance, and I can't seem to find any settings in VMware that would explain it. Symantec was actually the ones who had pointed out the problem, since I was getting crazy slow performance in Backup Exec when allocating the 4gb file before backing up. They pointed out that the problem only happens when the SAN is not caching. EMC says SAN is fine, but clearly something isn't working. I'd be hugely thankful with any help with this. Hell, if you can figure out what's going on I'll be happy to fedex you a 6 pack for the trouble :).
 
Last edited:
As a DBA, I've yet to find a SAN vendor that wasn't 85% useless when it comes to support. We've had EMC, Sun/Oracle, and now Netapp. Netapp may be the best so far... but it's like winning at the Special Olympics with this group.

We once had a controller battery get marked as expired on a controller. This disabled the SAN cache since each controller mirrors the other, and one controller considered it's battery failed. It wasn't bad, just past it's expiration date. We have databases running on the SAN, so we have it set to 100% write caching. Performance went through the floor and we couldn't tell why. Had to dig into obscure stats to find the damn battery had expired and disabled the cache, but even worse, everywhere else said the cache was enabled. It took our vendor a week to figure this out.

As to your issues, we've not had good luck with iscsi. We've stayed with Fibre Channel. I have seen issues with vmware beating the heck out of a SAN when it's not using the VMware Paravirtual SCSI Adapter. One VM wasn't configured correctly, and it was apparently killing the I/O to the other VM's sharing the same disks. Once it was fixed, everything was nice and fast for all VM's.
 
Wow, well that's awesome that you found the expired battery problem. I'm terrified of my problem being so obscure. What SAN was that on?

I haven't tried the PVSCSI adapter yet, so maybe I'll have to give that a shot. Just did some reading on it, and it looks like its reached the point where it's good for most all cases now and not just ones with crazy high IOPS.

Doing IOmeter tests from my 2008r2 VM against the 16 disk r10 15k SAS disk pool, I get about 4-5k IOPS in a 4kb/25% read/Sequential test and 2.5-3k IOPS on random. I'm a complete n00b when it comes to IOPS. Is this good, bad, average?
 
Last edited:
Don't have time right now to dig in all details but two things.

1. VNXe won't do MirrorView as the VNXe doesn't do block storage..it's a NAS that does CIFS and NFS. AX4 is a block array...so there is your problem. You'd need to replicate against a VNX5100 or larger that does block.

2. The reason they use IOMeter is that it's standard and well understand. That's what everyone uses and knows.
 
Not intending to hijack the thread, but I'm just looking for clarification of some terms used, since I know next to nothing about enterprise storage.

1) "RAID Group" - is this basically a single RAID array?
2) "Pool" - is this basically like a tank in ZFS or a PV under LVM ?
3) LUN - I know what a Logical Unit Number is, in SCSI terms. How is it applied here? Would it be like creating an LV under LVM and then exporting it as an iSCSI target?
 
ive done a lot with emc.. everything from ax4 all the way up to vmax..

what version of Flare are you running on your AX4?

You said pools.. so you must be running flare 30 or higher.. Correct?

If you are running flare 30, make sure its above patch 517. Also, if you are doing pools.. Always build your pools the largest you can build them in one shot. Do not start the pool out with 4 disks and then add 4 disks at a time. The bigger the pool is up front the better your data is spanned accross all the under lying raid groups.

Do you have the Navi Anaylzer running on your system? That will tell you if its your array or not.

From your work load posted above.. it does not sound like a a disk problem.

Do you have SPA0,A1 and SPB0,1 zone to all of your hosts? Check to make sure you dont have all your luns tresspassed over to 1 sp and not the other.

Have you run host grabs on every host connected to your san? I shit you not.. i've personally seen firmware issues that will make a box perform like crap if its not been tested and approved by emc. I can also tell you, I've personally seen Qlogic cards running in ESX under a known good firmware revision cause massive tresspass storms which will crush the array. IE, lun flop to 1 sp, then flop to the other hundreds of times a minute.

What kind of HBAs, Emulex or Qlogic.. Check your san switches and reset all your port counters. Possible one gbic is flopping up and down causing performance issues.

When you build out your pools/raidgroups, are they spanning your busses? or do you have all the mirrored stuff on bus 0 and all the raid5 stuff on bus 1? If you did that, you might be just pounding the crap out of your bus.

Finally.. I know this sounds retarted.. make sure write caching is on. If you have a Failed SP battery backup.. The array will auto disable write caching to save itself incase this loses power.

the ax4 is a pretty decent array. If you don't want to do a vnx.. then look at a NS120, emc is still pushing them.. I have one at a remote office running Flare 30/512 with FAST, 3 trays of Sata 2tb drives and 1 tray of FC running the double the work load you posted and it runs just fine.

hope that helps.

edit.. for grins.. do an SP Collect and upload the data to EMC.. they might see something you don't. hell just look at the sp logs.. the'll flat out tell you if you got a hardware issue, or tresspass storm.
 
One last thing. Who said EMC doesn't support the AX4 for offsite replication? It absolutely supports MirrorView for replication which is still a supported and deployed product/technology.

agreed.

could get all out of control with vplex metro/geo.. but thats way beyond the scope of this conversation.
 
Back
Top