VMware homelab storage

[Stensby]

Hey guys, I am looking for some advice on storage for my VMware lab. I am a student, so through my school I get a huge variety of Windows and VMware licenses and decided to through together a lab to play with all the cool stuff VMware has to offer.


My current setup:

Gaming Desktop Whitebox:
i7-2600K
ASUS Gene-Z
8GB DDR3
GTX 570
Intel 510 120GB SSD
Caviar Black 1TB


ESXi Hardware:
Dell R610: 2xX5570, 48GB RAM, 4x1GbE. Running ESXi 5.5
Sunfire X4170: 2xE5520, 48GB RAM, 4x1GbE. Running ESXi 5.1 (Box doesn't support ESXi 5.5 )

Current Storage:
Media/ISOs: Synology DS1513+ 2x4TB & 2x3TB in SHR.
VMs: Assorted 2.5" SATA drives in both servers (for basic testing/exploring, while I figure out my real storage)

What I would like to do is build a dedicated FreeNAS whitebox, and use that for my growing media collection, starting with 5x4TB drives in RAIDZ1, with the option to more vdevs as needed. I was thinking something like this:

Norco 4220 with 5x mini-SAS
SUPERMICRO MBD-X10SL7-F-O
Intel Xeon 1230 V3
16GB ECC
500W PSU

I would then use the Synology as an NFS/iSCSI box for all my VM's/ISOs etc. I am having trouble deciding what to fill the Synology with - I was thinking of using the 2x 3TB drives I already own in RAID 1, as a bulk storage for my ESXi hosts, and 3x240GB Crucial M550's in RAID 0, which will run all my VM's with daily mirroring to the 3TB volume (none of my VM data is super critical, losing a few hours of work if a SSD fails doesn't matter to me).

Would this provide enough space/IOPS for running a ton of VM's?( 20+) Most of them will be Windows variants, with some linux. Googling around it seems like if I aim for 50 IOPS per VM for 20 VMs, that should give me what I need with some room to grow.

Thanks!

 

[The_Moves]

Assuming your not doing a lot of IO at once, then you should be fine. Though you may want to invest in some more drives to increase the drive density in one of the arrays to spread IO across the platters.

Most of the times I assume your VMs will be idle and I can only think of converting video that would be IO intensive

 

[_Gea]

I would then use the Synology as an NFS/iSCSI box for all my VM's/ISOs etc. I am having trouble deciding what to fill the Synology with - I was thinking of using the 2x 3TB drives I already own in RAID 1, as a bulk storage for my ESXi hosts, and 3x240GB Crucial M550's in RAID 0, which will run all my VM's with daily mirroring to the 3TB volume (none of my VM data is super critical, losing a few hours of work if a SSD fails doesn't matter to me).

Would this provide enough space/IOPS for running a ton of VM's?( 20+) Most of them will be Windows variants, with some linux. Googling around it seems like if I aim for 50 IOPS per VM for 20 VMs, that should give me what I need with some room to grow.

Thanks!

One disk has about 100 IOPS. If you want 50 IOPS per VM on 20 VM you need about 1000 IOPS on 10 disks. A single SSD can deliver the same IOPS unless you do not use the whole space. If you use the SSDs to hold your VMs you are ok.

How do you intend to mirror the SSD to the disks as you have open files?
You either need a ESXi backup software or a snapshot capable filesystem like ZFS.

Next problem with 20 VMs is a quite low network performance and the synology is not too fast as its CPU is quite slow and you do not have a lot of RAM to cache files.

Have you thought of a config where you hold all VMs locally on the ESXi server and use a separate NAS only for backups (or a pair of ESXis for cross backup). You can use some sort of vSAN (virtual local SAN). This gives you a low latency high performance storage with up to several GB/s between ESXi and SAN. I do this for years with my napp-in-one concept with high quality local ZFS storage. I offer a ready to use ZFS Storage VM that you can download and run from http://napp-it.org/downloads/index.html

miniHowTo: http://www.napp-it.org/doc/downloads/napp-in-one.pdf

 

[Stensby]

Okay so it seems like I have overestimated my IOPS quite a bit, I wasn't really considering that the VM's would be idle a lot.

How do you intend to mirror the SSD to the disks as you have open files?
You either need a ESXi backup software or a snapshot capable filesystem like ZFS.

That's a great point, I hadn't really considered that, I would probably be better off running the SSDs in a RAID array to eliminate the need for regular backups. (I know RAID is not a backup but for my homelab purposes occasional snapshots of the VMs and RAID is more than sufficent.)

Next problem with 20 VMs is a quite low network performance and the synology is not too fast as its CPU is quite slow and you do not have a lot of RAM to cache files.

Have you thought of a config where you hold all VMs locally on the ESXi server and use a separate NAS only for backups (or a pair of ESXis for cross backup). You can use some sort of vSAN (virtual local SAN). This gives you a low latency high performance storage with up to several GB/s between ESXi and SAN. I do this for years with my napp-in-one concept with high quality local ZFS storage. I offer a ready to use ZFS Storage VM that you can download and run from http://napp-it.org/downloads/index.html

miniHowTo: http://www.napp-it.org/doc/downloads/napp-in-one.pdf

From what I understood the Synology box was plenty fast for storage, as is has an Atom processor and can support up to 3GB of RAM. I originally got the idea of a flash array inside a Synology from a blog by Chris Wahl http://wahlnetwork.com/. The reason I am looking at keeping the storage outside the hosts themselves is for exploring vMotion/HA/cluster etc. but does the napp-it still allow this?

Thanks for the detailed reply

 

[_Gea]

From what I understood the Synology box was plenty fast for storage, as is has an Atom processor and can support up to 3GB of RAM. I originally got the idea of a flash array inside a Synology from a blog by Chris Wahl http://wahlnetwork.com/. The reason I am looking at keeping the storage outside the hosts themselves is for exploring vMotion/HA/cluster etc. but does the napp-it still allow this?

Thanks for the detailed reply

Fast storage is not Intel Atom with 3 GB RAM.
Fast storage is a MultiCore Xeon with 64GB+ RAM.
Highend storage is something like NetApp or similar like the quite comparable ZFS based ones - not solutions based on ext4 or other older filesystems.

Beside that a NAS/SAN offers shared NFS/iSCSI storage - does not matter if you provide the storage on hardware or virtualized. Advantage of a virtualized solution a s superiour performance (low latency and several gigabyte/s) without a high cost FC network and no single point of failure like with a single dedicated highend SAN.

So yes, vmotion/storagemotion or HA features are possible with several napp-in-one solutions. ESXi is not aware that the storage server is provided by ESXi itself as a virtual machine.

 

[Stensby]

Fast storage is not Intel Atom with 3 GB RAM.
Fast storage is a MultiCore Xeon with 64GB+ RAM.
Highend storage is something like NetApp or similar like the quite comparable ZFS based ones - not solutions based on ext4 or other older filesystems.

Beside that a NAS/SAN offers shared NFS/iSCSI storage - does not matter if you provide the storage on hardware or virtualized. Advantage of a virtualized solution a s superiour performance (low latency and several gigabyte/s) without a high cost FC network and no single point of failure like with a single dedicated highend SAN.

So yes, vmotion/storagemotion or HA features are possible with several napp-in-one solutions. ESXi is not aware that the storage server is provided by ESXi itself as a virtual machine.

Great thanks for the advice. Definitely given me some stuff to look into.