• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

SAN Solution Sata or Sas?

C

Chandler

Limp Gawd
Joined
Jul 7, 2013
Messages
136
I need some sort of SAN solution for potentially 20 VMs, most application servers, two SQL Servers, and a few DCs.

Here is what I am unsure of:
500GB LUN for each data store? (App servers will be spread across multiple LUNs but should only need 80-160Gb virtual drives at most (user profiles)

I am sure I want a RAID 10 configuration (Stripe and mirror)
But I am not sure if I want hardware raid or if i want ZFS

If I go ZFS route, I will need a lot of ECC Ram - 64Gb+?
Which means I can't use a $350.00 Super Micro board and a cheaper E3 series processor.
(Assuming I build the SAN)

I like the idea of Hardware Raid and the Equalogic system from Dell

If I use Dell's Equalogic approach should I looks at SAS or use sata?
How many arrays should I use?

The application will be a hosted solution for a dental program using MS SQL to access it's data.
 
Are you designing this from the 80's? Your unlikely to have a lot of luns.

If you go zfs, you will need ECC yes, how much, no idea.

You need to know how many gigs your working set size is to know how much ram you want to have.

What really is needed, is to know how the load looks like that hits the disks, all random i/o? how many iops do you need?
 
Zfs is pretty superior. The good part of zfs is that it will scale perfectly as you throw more disks at it. So that has my vote.

Just make sure you use SAS disks, and not SATA to avoid any potential conflicts.
 
Take it this way, this is my home storage server that serves 3x ESXI nodes running 8-10 VM's each.

2x opteron 270's
8GB ECC RAM
Qlogic 4x 2gbps FC HBA
14x 146GB 2gb FC disks in an enclosure with split links (7 disks per link 4gbs total bandwidth)
Running Nas4Free, ZFS is 4x RAIDz1 with 2 hot spares.

Benchmarks at 330MB/sec on the nas4free box.

The storage is presented to the ESXi boxes as NFS over 1Gb/Sec links. I average 100MB/sec sequential read and write and 100MB/sec read 89MB/sec write random qd32.

Even with a fairly low buck setup with old hardware i am still getting very, very good numbers.

Think of what you would be able to do with a few SATA6 SSD's for cache and some decent SAS disks running mirrors and stripes...
 
Last edited:
I am using Nas4Free in my home as well. I do like it - I am just afraid of using something like that and building my own solution for production.

As far as having multiple LUNs, I read somewhere that over 500GB LUNs was not ideal for VMWare/ they did not recommend it.

I will be storage large amounts of data (X-Rays) as well - for those I would use SATA, not SAS.

I want my first try to be future proof for growth, and I mean small growth. Going from needing 200 IOPS per database application to 400, then being able to host at least 5 different clients on the one SAN solution - that's 2K IOPS (Average IO Size is anywhere from 20-64Kb) - This is without speaking of the application servers.

Application server are less IOPS intensive, only around 10-40 IOPS depending on what is going one, this is with about 20 users connected. Average IO Size is 31Kb.

Now ideally I will have an application server for each physical office - each client will be composed of 3-5 offices
__

Client example
SQL Server Database (Server 2008R2 ENT full install with a 50GB Database)
400 IOPS
Average IO size 64Kb
5 x Application Server
Total 150 IOPS
Average IO Size 31Kb

So right now my needs are not great, but I still want a safe solution.

The databases will probably be accessed right on the vDisk or else connected via ISCSI through Server 2008 to my SAN. I like the ISCSI idea, having Different folders for clients, within there having the databases - Then having only one or two database servers running all the databases.

I would like to plan for double the need IOPS for these smaller clients so I am looking at about 5K IOPS excluding my Imaging machines, and excluding VMWare overhead for the vDisks.

The imaging servers will run on LUNs attached to the SAS drives in the SAN, however I am thinking data drives attached via ISCSI for the actual images using SATA drives in the SAN.

They need decent through put though, so maybe I should consider SAS? the issue is that each client will need potentially 5TBs of storage, each file size being 10-25Mb - so SAS gets expensive

How many SAS spindles should I have for vDisk? should I have a separate array of physical disks for database storage - or should I just throw 16 15K 300GB SAS (Cheaper seagate ST3300657SS - $210 a drive) drives in a "Raid 10" in ZFS and just create multiple mounting points and call it a day? - That would still only put me in the 2K random IOPs range I believe. - according to some online calculators I have been playing with

Edit:

Any Switch suggestions?
 
Call Dell and let them figure it out for you. You should not be building your own SAN/server/etc for something like this.
 
I am not sure where this 500GB per vmfs comes from for vmware.

If you are using old vmware, maybe, v3?

If you are using v4, I would likely stick to 2tb luns, as that is the max you can do on it.
Using v5, I am currently using 10tb luns, for simplicity.

The issue with vmware and luns is locking, it locks the whole lun when it does some things, this is mainly a poweron/off vm issue. vmware does take care to not keep the lock for very long, so unless your growing your vm's very often (used thin disks) or powercycling them a lot, it's not an issue.

Even if you do, I hadn't ever noticed the issue, using a few hundred thin vm's

If your storage supports VAAI, this is really even less of an issue, as VAAI offloads the locking.

Normally the recommendations aren't about lun size, but about how many vm's you should put on a single lun, If you avg vm is 10gigs, yes 500gigs would be considered large. But I think 100vm's per lun is where people begin to notice this issue, could be off, since I have VAAI support for awhile now, and haven't been concerned with these issues.
 
I highly recommend SAS over SATA. SATA has serious contention issues with multiple 5+ VM's. 15K speeds are ideal for SQL database, but the prices on them escalate fairly quickly. Running RAID10 with SAS 10K is probably fine though. You're thinking correctly about the tiered nature of your storage, SATA is excellent for large images that you are basically just storing.

Equalogic's are a nice product, from my experience 10GigE isn't terribly expensive, and totally worth it if you go the iSCSI route.

I've also used the SAS DAS MD3220 solution from Dell, and it's pretty fast as well, cheap. I prefer EMC's VNX series over them all, but pricing is pretty high on them.
 
Monday I have my new Dell account manager coming to meet with me to go over my I/O requirements.

I agree that I should not be building my own solution for this - which is why I wanted suggestions at what to look for.

I did start to consider building my own ZFS box - this is the hardware I chose:

Code: 
24x Western Digital XE WD3001BKHG 300GB
4x Intel DC S3700 100GB MLC Drives (There are few SLC options now on New egg and MLC for ZIL is not a huge issue anymore)
3x LSI 9207 8i controllers (24 ports total no expander)
8x Kingston 16GB 240-Pin DDR3 SDRAM DDR3 1600
2x Intel Xeon E5-2609
SUPERMICRO MBD-X9DR3-F-O
2x intel I340-T4  (Quad port nic)

As of right now I am not set on my SSD choices, also I may have an issue mounting them.
The SAS drives (10k 2.5") choices appeared to be slimmer then I expected, and I am weary of the drive being less then $200.00 - the whole idea seems unreal to me when I think of having paid $600+ for the same drive specs in the past

I will create the exact same box filled with 2TB SATA drives as well - or should I just get a bigger super micro chassis?
 
That would be many times overkill for what kind of specs you stated.
Your looking at around a good 3000iops write on that.
What is going kill overall performance is all the gigabit connections.
 
A 10gbe connection is just so expensive. If I am using dell c6100s for hosts, I would have to purchase 600 dollar cards just to upgrade them so they have the same connection. Will 10 1gbs ports not be enough? I rather buy two quad port cards for 600 then a cheap 10gbe controller and an expensive switch... What is over kill about my set up? The ram or the CPU? I wanted to be able to have more then 33 gb of ram wich means I have to get an e5 board and CPU. I couldn't find a decent board with enough pcie slots with getting a dual CPU board. To use all the channels on the board I have to have two CPUs. I want to have 48 drives every 4us in my rack since it will be colo and I don't want to run out of rack space so quickly. For the data drives I could use external sad cards and link them to the main San chassis but I also do not want a single point of failure. Getting as much ram as I specified isn't to expensive compared to half the ram. I figured why not for better preformance in the long run...
 
That is some very attractive throughput. I thought several 1GB Nic ports can come close to the 10gbe while also giving me some redundancy if a port fails.

Again though, if I purchase c6100s for under 1k for 96Gb of RAM with four nodes, only 2x1gb on each node - then I feel like $600.00 per node isn't justified for a 10Gbe card on the host. Of course we are talking about the SAN it self - but If I only have a total of 8-12 nodes, I don't think I will saturate the SAN traffic with the 8Gb or even 12 Ports. I think each node will be out first. If I could have 6 ports per node I would, that way I could dedicate a vSwitch to ISCSI, but with this cheap solution - I can't. Four nodes for under a single 1,000 seems like a good starting point though...
 
I just realized that the C6100 has a single riser card for PCIe x16 slot - for some reason I was thinking I had to have the Dell 10Gbe card module.

Maybe I can make this more economical on the host side as well. But if I have 2x 1GB and 2x 10Gbe - How should I approach traffic on each node? 1x 10Gbe for ISCSi? a 1Gb port for management? The 10Gbe and 1Gb port for VMs - I feel like there still isn't much redundancy - at all - but I suppose that's the beauty of having multiple nodes.

Thanks!
 
You must log in or register to reply here.
Back
Top