Datastore locations/ methods - tired question

[Chandler]

I know this is a tired question, but I can never seem to find the appropriate answer.

I want some sort of all in one solution (local storage? NAS/ nas4free with HBAs pass through for hard drives) But I also want to be able to add a host in the future if I needed to, so local storage may not be a good idea, maybe use the NAS set up the same way I intend to have local storage, hardware raid - just use a 9260-8i in RAID 10 and pass it through to the NAS as well.

Now my question is whether or not SAS is a valid investment for me.

If I use SAS I will use 8x 146GB SAS Hitachi UltraStar C10K300
Or maybe 4-8 300gb disks if I can get them close to the same price/ performance.
Can I use more then 8 disks on the same array with the 9260-8i card? Do I just have to buy a sas expander? If it is pcie8, would I be nearing my bandwidth limit if I had and array of 12 drives?

I want to be able to put 20 vms on this array. They will just be smaller servers.. ie, domain controller. Probably two sql servers, maybe play with citrix. Anything that I have, I will map a data drive to it to another data store from a Freenas box using sata drives.

Will I still keep my sas performance with hardware raid if I port it through to Freenas to have the ability to share it out to another host? - I am probably thinking to big here for myself.

How could I build out a NAS with Hardware raid? I want NAS storage for ESXi with the reliability and preference of a RAID10 array if possible. I am almost contemplating just buying a NAS box, but I do not think I will get the I/Os I want (Even though I do not need necessarily them)

As an alternative I might just mirror some cheap SAS drives to install the freenas os on then put everything in a Raidz2 with Sata drives and have a separate mount point/ disks for the esxi data stores. I have this now on another all in one solution but I would say the performance lacks, but I am also not using any sort of server that requires heavy I/Os so I would assume it is just awful.

Any suggestions/ direction to push me in would be great

 

[NetJunkie]

If you want to add hosts later why are you still looking at local storage? Just do a NAS. Just go buy a Synology (what I use) or a QNAP. You can build one if you want but the prebuilt boxes are VERY good and will do anything you want.

 

[MrGuvernment]

Synology and QNAP are limited to single 1G connections are they not? Easily saturated with that many VM's.

 

[Chandler]

Synology and QNAP are limited to single 1G connections are they not? Easily saturated with that many VM's.

^^ This

--

a 1Gb connection maxs at what, 110mb/s? so if I had three hosts and each one had something that was IO intensive...

I am trying to find some sort of happy medium, But I don't think there is one.

Can I build some sort of DAS or SAN and connect to multiple hosts on a budget, or is that pretty much out of the question without spending less then 1,200 minus the drives

 

[Wrench00]

Synology and QNAP are limited to single 1G connections are they not? Easily saturated with that many VM's.

Thats why you have round robin

Trust me if you have 2 iscsi ports and a device capable of pushing hard drives you can easily run 10-15 vms. As long as they are not heavy load machine you will be ok..

Oh and I have a Qnap 870u-rp with a 10gb interface I get around 900mb/s.

Don't overthink your solution.
You can also build a seperate box for a nas or a do a nested using, openfiler, nexenta, etc..

Many many solutions.

For cheap stuff I just do Freenas or Openfiler NFS connector this is nested. You will use your internal storage via vm network it will do what you want. The drives and possibly your controller will be the expensive items.

 

[QHalo]

I want to be able to put 20 vms on this array. They will just be smaller servers.. ie, domain controller. Probably two sql servers, maybe play with citrix. Anything that I have, I will map a data drive to it to another data store from a Freenas box using sata drives.

I think people grossly underestimate just how much 110MB/s really is in terms of bandwidth especially given what you listed above as what you're going to be running on that device. Is this a lab or a production configuration or what? I wouldn't build anything for a production environment. I'd be buying something with a warranty and support. If this is a lab, that's cool too. Ask yourself if you're really going to be pushing more than that amount of bandwidth all the time. If you are, then maybe building something is for you. In that case, you may find a better answer in the SSD and Storage forum just up a bit in the main list. They'll be able to help you put together a ZFS monster or whatever you're really wanting. I think at the end of the day, IOPS are really what you're going to be needing, not throughput.

FWIW, QNAP and Synology offer 10GbE devices but they are exorbitantly expensive.

GL

 

[Wrench00]

I think people grossly underestimate just how much 110MB/s really is in terms of bandwidth especially given what you listed above as what you're going to be running on that device. Is this a lab or a production configuration or what? I wouldn't build anything for a production environment. I'd be buying something with a warranty and support. If this is a lab, that's cool too. Ask yourself if you're really going to be pushing more than that amount of bandwidth all the time. If you are, then maybe building something is for you. In that case, you may find a better answer in the SSD and Storage forum just up a bit in the main list. They'll be able to help you put together a ZFS monster or whatever you're really wanting. I think at the end of the day, IOPS are really what you're going to be needing, not throughput.

FWIW, QNAP and Synology offer 10GbE devices but they are exorbitantly expensive.

GL

+1

People often don't understand through put and IOPS.

The only time you need massive throughput is when vm boot. This is why you stagger them. But once a server is running its really not using a hell of a lot of disk access unless your running a giant DB or some disk intensive application and thats IOP intensive any way.

Buying expensive disks is not always the best answer either. If you have a good raid controller you can get slower disks just way more of them. 10 7200rpm sas drives will out perform 4 15k rpm sas drives.

My qnap wasn't terrible. 3.3k for 879u + 8 disks 1tb SATA $110/each+ $600 for 10 gb interface and 1.7k for a switch. Not a terrible amount of cash imho.

 

[Chandler]

I think people grossly underestimate just how much 110MB/s really is in terms of bandwidth especially given what you listed above as what you're going to be running on that device. Is this a lab or a production configuration or what? I wouldn't build anything for a production environment. I'd be buying something with a warranty and support. If this is a lab, that's cool too. Ask yourself if you're really going to be pushing more than that amount of bandwidth all the time. If you are, then maybe building something is for you. In that case, you may find a better answer in the SSD and Storage forum just up a bit in the main list. They'll be able to help you put together a ZFS monster or whatever you're really wanting. I think at the end of the day, IOPS are really what you're going to be needing, not throughput.

FWIW, QNAP and Synology offer 10GbE devices but they are exorbitantly expensive.

GL

a ZFS monster sounds good to me. I want to have a ZFS server no matter what as I will be housing a backup server and want to map drives to a ZFS pool for backups. I was thinking I would nest the ZFS server and port over a few IBM m1015s with sata drives... but I don't think that will provide the I/O performance I need for the all data stores/ It will definitely be the bottle neck in my system. Even a separate box for the ZFS server would make since, but I want to make sure I am using ECC ram so building out another box adds 1200 in hardware costs instead of throwing everything into a norco 4220 as I would like.

I'll search around for a "ZFS monster" solution in the storage forums and hopefully I will find more useful information on performance. thanks!

Thats why you have round robin

Trust me if you have 2 iscsi ports and a device capable of pushing hard drives you can easily run 10-15 vms. As long as they are not heavy load machine you will be ok..

Oh and I have a Qnap 870u-rp with a 10gb interface I get around 900mb/s.

Don't overthink your solution.
You can also build a seperate box for a nas or a do a nested using, openfiler, nexenta, etc..

Many many solutions.

For cheap stuff I just do Freenas or Openfiler NFS connector this is nested. You will use your internal storage via vm network it will do what you want. The drives and possibly your controller will be the expensive items.

That Qnap for 3k without disks would be more on the exspensive side i believe. But in the end I suppose I can always move my datastores over to a NAS if I choose to get another host anyway right? So maybe right now a Local RAID10 of SSDs or 10k SAS would be more then suitable for now.

 

[NetJunkie]

Yep. 1Gb isn't usually a problem at all unless you boot a bunch of VMs at once. It's all about the IOPS, not the throughput.

 

[Netwerkz101]

Synology and QNAP are limited to single 1G connections are they not? Easily saturated with that many VM's.

Many of the QNAP boxes have dual gigabit ethernet ports with a few methods
to link them (trunking as they call it).

 

[Netwerkz101]

I<snipped>
Now my question is whether or not SAS is a valid investment for me.

<snipped>
How could I build out a NAS with Hardware raid? I want NAS storage for ESXi with the reliability and preference of a RAID10 array if possible. I am almost contemplating just buying a NAS box, but I do not think I will get the I/Os I want (Even though I do not need necessarily them)

As an alternative I might just mirror some cheap SAS drives to install the freenas os on then put everything in a Raidz2 with Sata drives and have a separate mount point/ disks for the esxi data stores. I have this now on another all in one solution but I would say the performance lacks, but I am also not using any sort of server that requires heavy I/Os so I would assume it is just awful.

Any suggestions/ direction to push me in would be great

You build a hardware NAS by placing a hardware controller in a "computer" and running an OS and sharing out the drive space.

My "hardware" box is a PowerEdge T110 with 5 Gb NIC ports + a PERC H700 controller (similar if not rebadged 9260-8i):
8x2.5" 7200rpm WD consumer drives in RAID10 + 4x1TB Seagate running on the motherboard built-in SATA controller.
... I've tried Windows iSCSI, Starwind, OpenFiler, and FreeNAS on it.
In FreeNAS the hardware RAID10 is presented as one volume and then I form a software RAID5 with the 4 1TB drives ...
so a hybrid is possible.

Have you considered 128GB SSD instead of the 146GB SAS drives?
RAID10 is doable with a 9260-8i as is RAID6 (RAIDZ2 similar)
If you go the FreeNAS route, know that it can boot from USB so you can use all your drives for storage.

I also have a QNAP TS-659 Pro II .... it is my full time storage box because
it is quiet and uses less power than the PowerEdge box.

Quite frankly ...with only two XenServer host and no more than 20 active VMs, I can not tell a difference in performance between the two.

 

[DigitalDaz]

If you're looking to shift data fairly quickly without having to go 10 gig, have a look at the qle2462 4Gbit Fiber Channel cards. There are plenty to be had on ebay and perform great with an esxi/zfs san combo.

 

[shade91]

Synology and QNAP are limited to single 1G connections are they not? Easily saturated with that many VM's.

Depends on the model. Some Synology's have 4 NICs (with the ability to add on more), some have 2 built-in with the added ability to aggregate them, however, if he wants to run iSCSI and realize the benefits of them he won't want to bond the links since iSCSI is path aware. The really low-end units have 1 NIC and are dogs in anything but the lightest application.

 

[Phog]

Two 1GigE ports for iSCSI is plenty if you're concerned about IOPS, which is what it sounds like you are. As per a previous suggestion, if this is for home lab use/etc, get a Synology NAS and put some SSD's in it. SSD's absolutely blow SAS out of the water. It's not even close when it comes to IOPS (running multiple VM's at once requires IOPS, not throughput). Plus, the Synology is insanely flexible, and has so much more to offer on top of iSCSI. Most 120GB SSD's these days rate in at 50,000 - 80,000 IOPS, a 146GB SAS drive is what, maybe 250 IOPS?