• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Storage Question - What do you recommend?

R

RiDDLeRThC

2[H]4U
2FA
Joined
Jun 13, 2002
Messages
3,963
Ok so this question comes up it seems on every forum. What is the best method of carving up your array. I am giving myself the chance to fix what I feel is a mistake I made a few years ago when I first put our MD3000i into production.

I currently have a MD3000i. I have been given the budget to add 3 additional SAS drives to it.

Currently this is how things sit-

5x 300gb SAS 10k drives (3 additional drives coming for this group)

7x 500gb SATA 7.2k drives (one hot spare)

I have enough space to move all my current VM's (while powered off) to the SATA drives for temp storage while I re-config the SAS storage.

Now our workload varies between VM, we have one SQL server, an Exchange server, a File Server, ETC. Its a small office so they don't get hit too hard.

Total storage space on the SAS drives is somewhat of a concern, we currently have 1.089TB usable and need to keep AT LEAST that much.

Now at least one of the 3 new drives is going to be a hot spare, the question is should I remain with just one large disk group for the SAS drives and if I do this, should I convert this disk group to a RAID 6. Does it matter how I carve up the Virtual Disks?

None of my VM's have HD's that ard RDM to this storage group.

What is everyones thoughts and experiences? I know everyones work load varies but mine is pretty generic and all over the place and I just want to maximize the speed/space.
 
looks like my 3 drives i ordered at 5pm today will actually be in tomorrow.

I'm thinking a raid 5 with a hot spare and breaking up that raid 5 into two equally sized luns.

I don't know if raid 6 is worth it or not.

thoughts?
 
I think the general consensus is that the RAID6 penalty isn't worth it. I'll have to wait for loepotve to speak up though...
 
yeah thats what i am reading too. i'm waiting for him to grace us with his expertise :)

Now RAID 10 is another story only problem is I would have to use all 8 of my drives and wouldn't have a hot spare.
 
Raid6 isn't worth it unless you REALLY have to guarantee the array won't die. I'll take 5 + backups any day (and hot spares!).

RAID5, 2 luns, and then mix and match the SATA to whatever you need.

I never, ever, ever recommend not having a hot spare. Too many times a batch of drives has more than one failure at a time, and RMA's sometimes take WEEKS.
 
Honestly, 7 SAS drives at that speed will probably be fine for any high-load apps. Put exchange, SQL, oracle on there and test to verify.

Put file servers/web servers on SATA as needed. :)
 
Sounds like a plan.

Yeah the SATA drives are used for crap stuff. my BES VM that has my single blackberry on it, term server that only myself and one other person uses very lightly is on there. just random stuff that actually isn't in "production".

I will be going with the RAID 5 w/ hot spare then with two LUNs.

Hopefully the migration to the SATA storage goes well. Think that storage could actually handle running a few VM's?

Seems pretty strait forward seeing as the ESX Servers have access already and I no longer run my vCenter server as a VM (well I do just not on my ESX cluster :p)
 
We run Microsoft EBS so whats bad is our File Server also runs SQL and the System Center Essentials DB along with a bunch of other random crap. I don't know how the SATA storage would like it.
 
SATA can definitely handle a few vms.

Just remember - 10k SAS is going to be over twice as fast as the SATA disks in use... so they'll run VMs, just not as many :) And it depends on the load.

I run 10 VMs on 4 sata disks. None of them are remotely high load (VC server with 6 people using it, DR server, one VERY light use DB server for VC/testing, VDR, and a few other test machines) - none of them complain for disk speed, but they don't NEED a lot.

If you've got that much on a VM - check the disk usage over a few days, and find out the IOPS load. That'll tell you if you can use it on SATA or not :)
 
Also seeing as I am getting the chance of "starting over" any other considerations I need to take?

I know when I make a new LUN I get to select my Segment size of either 128kb or 256kb. Should I make any changes or just leave it on the default 128kb?
 
RiDDLeRThC said:
I know when I make a new LUN I get to select my Segment size of either 128kb or 256kb. Should I make any changes or just leave it on the default 128kb?
Click to expand...

keep it at the default. there was quite a big debate on this recently and it basically came down to stick with the vendors defaults unless there is a specific need.

and i'd like to echo the earlier question of "why 2 luns" - i can understand presenting a lun to each VM but splitting it in two doesn't make sense to me.
 
I believe it has to do with ESX limitations with how many concurrent connections to a single lun.

I'm totally speaking off no actually knowledge of this.

lopoetve will have a real answer for us :)
 
RiDDLeRThC said:
I believe it has to do with ESX limitations with how many concurrent connections to a single lun.

I'm totally speaking off no actually knowledge of this.

lopoetve will have a real answer for us :)
Click to expand...

i can give you a real answer. there is no limitation on connections per lun.

The only concern you might have is the maximum limit of 2TB with the VMFS file system, however your storage does not exceed 2TB therefor it does not need to be split up.
 
Kinda jumping back to the block size

from another thread-

Originally Posted by oDii View Post
I'd be interested to see what lopoetve has to say about your stripe size question. My thinking would be that you'd be wanting the stripe size of the controller and the block size of the filesystem to match, so bar any other requirements (maximum file size being the main consideration in choosing the VMFS block size) matching 1MB to 1MB or (if a PERC5 is capable of this, I know the PERC6 is) 2MB to 2MB would be preferable?

Where sub blocks come in (1/8th of the size of the VMFS blocksize) and how operations relating to them affect performance are a question in my mind. I know they'd be a factor in thin provisioning where used capacity in a volume is under tighter scruitiny, but would they affect performance unduly if the wrong stripe size was chosen?
Well, there are really two parts there... VMFS block size is a bit different. Most places run a 64k stripe, IIRC. some 32k. NetApp uses 4k by default

I'd stick with default, or test against your workload. Hard to say from this high a level.
Click to expand...

Not sure exactly what this means...
 
outspoken said:
i can give you a real answer. there is no limitation on connections per lun.

The only concern you might have is the maximum limit of 2TB with the VMFS file system, however your storage does not exceed 2TB therefor it does not need to be split up.
Click to expand...

Ahh this makes sense. Actually I have noticed when using iSCSI and port channel groups (link agregation), you really only get 1GBit per connection (when using gig networking), even if you have multiple links as part of your port channel. I can understand breaking up your luns to allow seperate TCP sessions, thus allowing you to utilize a greater overall bandwidth on your iSCSI SAN.
 
Well if the 2tb limit is the only reason to do more than one LUN then i will just stick with one LUN. I remember reading more on this, just can't find it right now.
 
RiDDLeRThC said:
Kinda jumping back to the block size

from another thread-



Not sure exactly what this means...
Click to expand...

whoever posted that sounds quite confused. I believe that they are trying to talk about the VMFS blocksize, which should generally be set to 2048 so you can obtain the 2TB limit.

in regards to the stripe size it is different per vendor and device, to say netapp uses 4k is incorrect.

EDIT: just to re-iterate, leave your stripe size at 128k which is your vendor's default for that card.
 
Last edited:
tormentum said:
out of curiosity, why 2 LUNs?
Click to expand...

1. Limit exposure to metadata failures
2. Limit exposure to reservations (somewhat minor, but still a consideration)
3. More pathing flexibility (more paths you can take, since pathing is per lun).
4. More queue depth (each lun can get a new queue, although swiscsi is a bit more limited in this regard).

outspoken said:
i can give you a real answer. there is no limitation on connections per lun.

The only concern you might have is the maximum limit of 2TB with the VMFS file system, however your storage does not exceed 2TB therefor it does not need to be split up.
Click to expand...

Well, there is, but you're no where near it. I'm not a fan of massive single luns for many reasons :)

tormentum said:
Ahh this makes sense. Actually I have noticed when using iSCSI and port channel groups (link agregation), you really only get 1GBit per connection (when using gig networking), even if you have multiple links as part of your port channel. I can understand breaking up your luns to allow seperate TCP sessions, thus allowing you to utilize a greater overall bandwidth on your iSCSI SAN.
Click to expand...

Bingo.

outspoken said:
whoever posted that sounds quite confused. I believe that they are trying to talk about the VMFS blocksize, which should generally be set to 2048 so you can obtain the 2TB limit.

in regards to the stripe size it is different per vendor and device, to say netapp uses 4k is incorrect.

EDIT: just to re-iterate, leave your stripe size at 128k which is your vendor's default for that card.
Click to expand...

No, he was on, but far more in depth than we need to get. Performance concerns at the sub-block level for VMFS is not something 99% of customers need to worry about. Minimal impact in vSphere to boot.

WAFL (NetApp's default filesystem) uses a 4k segment by default (good and bad sides to this that need to be considered - don't forget to align your disks!). Makes for awesome fast snapshots though.

Oh, and 2TB is 8MB block, not 2MB.
 
AceXsmurF said:
If anyone has an answer on this, I would appreciate it too :)
Click to expand...

Quick and dirty ways-
1. Fire up esxtop. Hit 'v' for VM. 'e' will let you expand out the world. Look at what cmd/s is over time, and get an average for read/write. It's a dirty one, but it's closeish. Then fire up the RAID calculator and see what the back-end IOPS usage equates to (remember, writes on RAID have a penalty).
2. Most apps have a performance monitor that can tell you, or some 3rd party ones can examine the entire system.
3. VMware PSO can measure it as well for you.
4. Google average use based on users/size/etc ;) Gives you a ballpark to start with.

Lots of choices out there. Depends on the app, to be honest, for the best answer.
 
lopoetve said:
WAFL (NetApp's default filesystem) uses a 4k segment by default (good and bad sides to this that need to be considered - don't forget to align your disks!). Makes for awesome fast snapshots though.

Oh, and 2TB is 8MB block, not 2MB.
Click to expand...

How do you align the disks?
 
Doesn't matter nearly as much for the 3k (if at all). NetApp it does. :) They have a big nice best-practice guide to follow for their arrays.
 
just trying to squeeze every little bit out of this thing lol figured you can never optimize too much and now is the best time seeing as I have the spare room to blow alway the LUN on my SAS RAID.
 
lopoetve said:
Well, there is, but you're no where near it. I'm not a fan of massive single luns for many reasons :)
Click to expand...

i wouldnt consider a 1.xTB lun to be "massive". is there really a limit as far as vmware is concened. I understand that the queue limit would be reached well before that, whatever it is.

lopoetve said:
No, he was on, but far more in depth than we need to get. Performance concerns at the sub-block level for VMFS is not something 99% of customers need to worry about. Minimal impact in vSphere to boot.

WAFL (NetApp's default filesystem) uses a 4k segment by default (good and bad sides to this that need to be considered - don't forget to align your disks!). Makes for awesome fast snapshots though.
Click to expand...

ok, i didn't pick up on that he was talking about the defaul filesystem on netapp. if you're running a lot of small files it would be fine i suppose.

Oh, and 2TB is 8MB block, not 2MB.[/QUOTE]

yes, my mistake. ;)
 
Larger the lun, the more VMs, that's all - so more of an impact if you ~do~ lose something. :) Given what I work on and what I have to repair, I'm always thinking of limit impact :)

Riddler, you can probably get about 2-3% by aligning the filesystem. Without a tool like MBRscan, it's a destructive operation. And it's only done on data disks most of the time.
 
lopoetve said:
Riddler, you can probably get about 2-3% by aligning the filesystem. Without a tool like MBRscan, it's a destructive operation. And it's only done on data disks most of the time.
Click to expand...

You talking about the VMFS file system or the VM's file system?

I'm redoing my LUN's so i'm not too worried about being destructive :).
 
VM's filesystem - the actual VMs themselves. :)

VMFS is automatically aligned unless you do it from the command line and do it wrong :p
 
lopoetve said:
VM's filesystem - the actual VMs themselves. :)

VMFS is automatically aligned unless you do it from the command line and do it wrong :p
Click to expand...

Yah i was just reading that a few minutes ago. I used the VI client years ago back on ESX 3 so i'm guessing it would be aligned.

I will be re-creating them anyways to split into two or three LUNs so vCenter will do it for me.
 
You must log in or register to reply here.
Back
Top