Combining ESXi server and ZFS server into one

I

idea

Gawd
Joined
Jan 24, 2005
Messages
615
This seems to be a pretty resourceful forum for both ESXi and storage related questions so here we go....

In my sig I have my two servers that make up my "home lab." They were really fun to build but now the heat, noise, and electricity usage is annoying me. My next phase of the project is now combining them into one. It's going to cost some money that I hope to recoup by selling the leftover parts.

My ESXi server hosts around 10 servers of various OS's, all with very low, near zero load.

My questions:
  1. In order for this all to work, I'm banking on the idea that I can configure "passthrough" in ESXi for a Solaris VM to make use of a set of disks to make a ZFS pool with. Is this smart?
  2. Can anyone help me figure out how to make the primary datastore cheaper? 2.5" disks is all I have space for in my enclosure. $600 worth of them. Should I go with regular SATA laptop disks? SSD maybe? RAID1 instead of RAID10?

  • $0.00 ESXi 4.1
  • $0.00 Supermicro X8SIL-F
  • $240.00 Intel Xeon X3440 or maybe $300.00 Xeon L3426
  • $0.00 Dell PERC 5/i (for the primary datastore)
  • $0.00 Dell SAS 6/iR (for disks to use with ZFS)
  • $140.00 8GB ECC memory (2x 4GB), will upgrade to 16GB if needed
  • $600.00 292GB RAID10 for primary datastore (4x 146GB 10K 2.5" SAS)
  • $300.00 6TB RAIDZ ZFS (4x cheap slow 2TB drives with passthrough to Solaris VM)

Copy of my sig in case it changes in the future:

Hypervisor: ESXi 4.1 - Intel Xeon 5160 | Supermicro X7DVL | 12GB ECC FBDIMM | 4x300GB SAS (RAID10) | Dell PERC 5/i
SAN: Solaris 11 Express - Intel Core i3 530 | Supermicro X8SIL-F | 4GB ECC UDIMM | 6x750GB SATA (RAID-Z) | Dell SAS 6/iR
Click to expand...
 
Last edited:
The primary datastore will only hold the solaris VM, correct? If so, I wouldn't do anything special for that, just a regular disk. If you have one or more laptop drives, sure. I think a raid10 is overkill for that though. Do you have the solaris physical host serving up storage to esxi via nfs or iscsi for other VMs? If not, you should in this case. If so, are you using iscsi or nfs?
 
danswartz said:
The primary datastore will only hold the solaris VM, correct? If so, I wouldn't do anything special for that, just a regular disk. If you have one or more laptop drives, sure. I think a raid10 is overkill for that though. Do you have the solaris physical host serving up storage to esxi via nfs or iscsi for other VMs? If not, you should in this case. If so, are you using iscsi or nfs?
Click to expand...

To answer your questions: I should have stated that the ESXi server hosts around 10 VMs as it is. Yes, the purpose of the existing physical Solaris host is to serve files via NFS/CIFS. Maybe I will do RAID10 with some cheap laptop drives. I have 4 slots so why not? They are cheap enough.
 
Well, power, complication, etc :) If you have a HW raid that will do raid10 and it works with esxi and what-all, more power to you :)
 
Ok so your solaris machine is just file serving to the vm's now but is not the datastore for the virtual disks of those VM's.

But if your if you are going to rebuild and combine the solaris host into an all in one why not go all the way and combine all disks into a ZFS pool and have it host the datastore for all VM's AND file data that it is serving to those VM's. That is all VM's except for the Solaris VM which would need to start up first on it's own datastore/disk...which danswartz pointed out can be any old disk.

I think it would be quite interesting to see the the 4x2TB RAIDz combined with the 2x146GB SAS RAID1 added as ZIL, and 2x146GB RAID0 as L2ARC. You can then add a folder and share as NFS for the datastore (or do iSCSI) with the rest be whatever nfs/cifs/smb share setup you have currently. The only question is how much disk I/O do you really need for the the 10 VM's? I think the above would be an interesting test. Perhaps not the same raw performance as your hardware RAID10 datastore...but could well be enough.

Of course this is a big rework and you would have to offload all the virtual disks somewhere to set it up. You could off course just leave your hardware RAID 10 as is and just virtualize the Solaris setup and passthrough the controller for the big disks. That would be fairly simple and you could keep all data in place. Just export your pool, setup VM, set pasthrough of card, reinstall solaris as a VM, reimport pool. You could then try out your filesharing via 10Gb (vmxnet3) internal network if you like.

Edit: It looks like you don't have the SAS drives now but were considering them. Sorry. Same thing goes though. What is enough disk I/O for your VM's? Perhaps just a RAID 1 mirror of SATA drives is enough?? Or maybe just ZFS it all and add two SSD's (one ZIL one L2ARC) to the big drive and run everything off that, datastore and all.
 
Last edited:
Silenus said:
Ok so your solaris machine is just file serving to the vm's now but is not the datastore for the virtual disks of those VM's.

But if your if you are going to rebuild and combine the solaris host into an all in one why not go all the way and combine all disks into a ZFS pool and have it host the datastore for all VM's AND file data that it is serving to those VM's. That is all VM's except for the Solaris VM which would need to start up first on it's own datastore/disk...which danswartz pointed out can be any old disk.

I think it would be quite interesting to see the the 4x2TB RAIDz combined with the 2x146GB SAS RAID1 added as ZIL, and 2x146GB RAID0 as L2ARC. You can then add a folder and share as NFS for the datastore (or do iSCSI) with the rest be whatever nfs/cifs/smb share setup you have currently. The only question is how much disk I/O do you really need for the the 10 VM's? I think the above would be an interesting test. Perhaps not the same raw performance as your hardware RAID10 datastore...but could well be enough.

Of course this is a big rework and you would have to offload all the virtual disks somewhere to set it up. You could off course just leave your hardware RAID 10 as is and just virtualize the Solaris setup and passthrough the controller for the big disks. That would be fairly simple and you could keep all data in place. Just export your pool, setup VM, set pasthrough of card, reinstall solaris as a VM, reimport pool. You could then try out your filesharing via 10Gb (vmxnet3) internal network if you like.

Edit: It looks like you don't have the SAS drives now but were considering them. Sorry. Same thing goes though. What is enough disk I/O for your VM's? Perhaps just a RAID 1 mirror of SATA drives is enough?? Or maybe just ZFS it all and add two SSD's (one ZIL one L2ARC) to the big drive and run everything off that, datastore and all.
Click to expand...

To be honest, the VM's would be fine with regular SATA storage. Not much IO going on. It is just that I happen to have them currently running on 4x 15K SAS 3.5" disks that I had lying around and I didn't want to lose the performance. But if I am going to combine these two machines, I won't have enough space for 4x 3.5" disks, so I need to buy 2.5" ones.

Also, I would much rather put all VMs on local storage. I do not want to complicate things by using the ZFS datastore to host the VMs over iSCSI back to ESXi.
 
idea said:
Priceless. How did I miss that?
Click to expand...

Ha! I assumed that you were referring to that in your original post.

Anyway you can definitely combine them. The only question is whether you want to keep your primary datastore as a separate hardware RAID or put the datastore on ZFS. Edit: just read your post above. Ok it is simpler to keep your datastore local. THough going NFS share back to ESX is not too complicated but will definitely lose performance over local for sure.
 
yeah, I can confirm no vt-d with an i3 on a 3400/3420 chipset - but you do get ECC. x3430 is the cheapest chip that gives you vt-d I think (intel anyway), and x3440 is normally a worthwile upgrade as it's not much more.
 
ChrisBenn said:
yeah, I can confirm no vt-d with an i3 on a 3400/3420 chipset - but you do get ECC. x3430 is the cheapest chip that gives you vt-d I think (intel anyway), and x3440 is normally a worthwile upgrade as it's not much more.
Click to expand...

OK thanks very much I have edited my first post to indicate a new CPU needs to be purchased
 
danswartz said:
According to this:

http://www.phoronix.com/scan.php?page=article&item=intel_corei3_530&num=1

I then looked around more and found other articles that contradicted that. Given that it did not work for you, I know where my money is. Sorry for the bad info.
Click to expand...

No worries. I was hoping I was wrong.

I could not find L3426 anywhere, even when they showed stock, it was the same supplier behind it all and they have none.

Then I tried to order an X3440 and had 0 luck as well. So I'm running an X3430 and will learn to "live with compromise" or perhaps hunt a bargain L3426.

Or perhaps I'll just wait until the X9 + E3 12xx setups become viable and add another "lab in 1 box".
 
barista said:
No worries. I was hoping I was wrong.

I could not find L3426 anywhere, even when they showed stock, it was the same supplier behind it all and they have none.

Then I tried to order an X3440 and had 0 luck as well. So I'm running an X3430 and will learn to "live with compromise" or perhaps hunt a bargain L3426.

Or perhaps I'll just wait until the X9 + E3 12xx setups become viable and add another "lab in 1 box".
Click to expand...

Can you please explain your hard drive config? You have 2x SSD for "boot", I assume RAID1? It can't be software RAID so how did you do it? Also, I assume you keep your Solaris VM on this two disk array, but where do you put the rest of the VMs?
 
Here is some specs I am toying with. Storing the VMs on the SAS disks (and shared from ZFS) removes the PERC5/i from the setup. I'm concerned about performance with a virtualized network though. I still wish I could replace these big hot SAS disks. I need at least 256GB for VM storage and I can't afford that in SSD...

Code: 
    $0.00 ESXi 4.1
    $0.00 Supermicro X8SIL-F 
    $0.00 Dell SAS 6/iR (PCI passthrough to the Solaris VM)
    $0.00 4x 300GB 15K 3.5" SAS (2x mirror vdev, 600GB usable)
$???.?? 30GB SSD for ESXi + one Solaris VM (with onboard Intel 3420 controller)
$300.00 4x 2TB 5400 SATA (1x RAIDZ vdev)
$240.00 Intel Xeon X3440 or maybe $300.00 Xeon L3426
$140.00 8GB ECC memory (2x 4GB), will upgrade to 16GB if needed
 $50.00 4 x 2.5" HDD in 1 x 5.25" bay SAS / SATA 2.5" Hot Swap Backplane RAID cage

What type of performance can I expect from this setup: The Solaris VM will live on the single SSD drive. All other VMs will live on an iSCSI disk shared from the Solaris VM (composed of the 600GB 2x mirror virtual device). Assuming the Solaris VM is configured with VMXNET3, the network should not be the bottleneck, right? The SAS disks should get full speeds?
 
I would think that the primary bottleneck will be CPU consumption due to I/O protocol overhead. I worry that neither Solaris's vmxnet3 driver and possibly others will do much effective offload to NICs. So, you are taking a CPU hit exporting over iSCSI, another CPU hit consuming it in vmware, and then you have your vmfs, guest OS FS and other layers of overhead on top of that. I think you'll be more compute limited than before. Then again if you generally are not generating much load anyway, this might be OK.

I'm trying to setup a similar test myself.
 
The other thing that bothers me about this setup (and take this with a grain of salt for I am no expert) is that Solaris and ZFS pretty much fall to pieces under ESXi if you get into a memory overcommitment situation in ESXi. You need to make sure Solaris gets its reservation, or one of your non-Solaris VMs could push the whole box off a cliff if it starves Solaris.
 
sphinx99 said:
The other thing that bothers me about this setup (and take this with a grain of salt for I am no expert) is that Solaris and ZFS pretty much fall to pieces under ESXi if you get into a memory overcommitment situation in ESXi. You need to make sure Solaris gets its reservation, or one of your non-Solaris VMs could push the whole box off a cliff if it starves Solaris.
Click to expand...

Ugh, things like this make this setup so unrealistic...
 
Well, I think the notion of a single box is perfectly realistic. However, looping iSCSI back to ESXi through the vmware NIC adapter is something I'd consider a "try it and see what happens" architecture rather than a "it's proven to work, so go ahead and depend on it" option.
 
sphinx99 said:
Well, I think the notion of a single box is perfectly realistic. However, looping iSCSI back to ESXi through the vmware NIC adapter is something I'd consider a "try it and see what happens" architecture rather than a "it's proven to work, so go ahead and depend on it" option.
Click to expand...

But the entire notion of a single box depends upon virtualizing Solaris and sharing ZFS data via iSCSI/NFS through the vmware NIC adapter. So I guess the whole notion of a single box is a "try it and see what happens" architecture anyway
 
idea said:
But the entire notion of a single box depends upon virtualizing Solaris and sharing ZFS data via iSCSI/NFS through the vmware NIC adapter. So I guess the whole notion of a single box is a "try it and see what happens" architecture anyway
Click to expand...

From ESXii-view, you have a SAN-Server without difference between self-hosted or external.
I have such configs running for more than a year. It is stable and ultrafast with highspeed internal links.

You have to care about power on/off (SAN must be the first/ last guest) and you have to care about
update/ crash scenarious especially of the SAN part (SAN Hot failover is slow, complicated and expensive).
For the last, i have always a second identical configured machine, where i can hot-plugin the ZFS-pool.
(enough free disk slots)

I need about 20 min to move and reimport the pool, reshare it via NFS and restart 10 VM's with the original data.
I have not needed my backups since. ZFS snaps are the way, if a guest needs a state of yesterday or last friday
or end of last year.

Gea
 
idea said:
Here is some specs I am toying with. Storing the VMs on the SAS disks (and shared from ZFS) removes the PERC5/i from the setup. I'm concerned about performance with a virtualized network though. I still wish I could replace these big hot SAS disks. I need at least 256GB for VM storage and I can't afford that in SSD...

Code: 
    $0.00 ESXi 4.1
    $0.00 Supermicro X8SIL-F 
    $0.00 Dell SAS 6/iR (PCI passthrough to the Solaris VM)
    $0.00 4x 300GB 15K 3.5" SAS (2x mirror vdev, 600GB usable)
$???.?? 30GB SSD for ESXi + one Solaris VM (with onboard Intel 3420 controller)
$300.00 4x 2TB 5400 SATA (1x RAIDZ vdev)
$240.00 Intel Xeon X3440 or maybe $300.00 Xeon L3426
$140.00 8GB ECC memory (2x 4GB), will upgrade to 16GB if needed
 $50.00 4 x 2.5" HDD in 1 x 5.25" bay SAS / SATA 2.5" Hot Swap Backplane RAID cage

What type of performance can I expect from this setup: The Solaris VM will live on the single SSD drive. All other VMs will live on an iSCSI disk shared from the Solaris VM (composed of the 600GB 2x mirror virtual device). Assuming the Solaris VM is configured with VMXNET3, the network should not be the bottleneck, right? The SAS disks should get full speeds?
Click to expand...

This is similar to my setup, but I am sharing my setup to ESXi with NFS instead of iSCSI. The network will definitely not be your bottleneck. Solaris will run fine on the SSD. I would expect even better performance to all of your VM guests with iSCSI than NFS, so you should be good there.

I feel pretty confident that you will be happy with the performance of the above setup. The disk access on my VMs isn't screaming fast, but it is plenty fast enough for what I do (an active web server, and a bunch of "services" servers for my house).

The only things I'm concerned about is the amount of RAM you have, and the single CPU. How much of that are you dedicating to Solaris? How much CPU do your other VMs use? My Solaris box doesn't use a lot of CPU, but it does use some. I'm pretty sure NFS (my setup) would use less CPU than iSCSI as well. But everyone always tells me that your VM environment is ideal if it sits between 60-85% CPU utilization most of the time and mine sits at 5-10%, so maybe mine is overkill.

I've never run into a memory overcommitment situation with my setup, but I have significantly more RAM (48GB). RAM is pretty cheap, so I would probably pick up a little more just to make sure that you can fulfill your Solaris reservation without starving your other VMs. It really depends on what the rest of the VMs are doing.

_Gea's last comments here are extremely helpful, too. Listen to him. I did, and it helped me get mine built the way it is now. It's been running since January or February without a single issue.

Good luck! Let me know if I can help at all.
 
idea said:
Can you please explain your hard drive config? You have 2x SSD for "boot", I assume RAID1? It can't be software RAID so how did you do it? Also, I assume you keep your Solaris VM on this two disk array, but where do you put the rest of the VMs?
Click to expand...

The signature is where the information resides:

Patriot Convoy XL is a 2x2.5 SATA device using a Silicon Image 57xx chipset to deliver RAID0/1 or some odd combinations thereof. I run it RAID1.
 
sphinx99 said:
The other thing that bothers me about this setup (and take this with a grain of salt for I am no expert) is that Solaris and ZFS pretty much fall to pieces under ESXi if you get into a memory overcommitment situation in ESXi. You need to make sure Solaris gets its reservation, or one of your non-Solaris VMs could push the whole box off a cliff if it starves Solaris.
Click to expand...

ESXi forcibly reserves 100% off the memory assigned to VMs with VMDirectPath I/O pass-through devices (e.g. PCI/PCIe devices). I have 16 GB, and I've given 6 to Solaris, with the rest going to my Windows VMs. Works decently enough.

FWIW, I've benchmarked about 5 Gb/sec between my VMs through their virtual NICS using iperf, with Windows getting 1-2 Gb for inter-vm file transfers. I've not put any effort into thorough benchmarking or optimizations.
 
Last edited:
Maybe I should just take these SAS disks out of the setup. I don't know why I want them so bad. They just take up 4 slots and heat up like a furnace. I just like saying that my server has huge 15K SAS disks, you know? If I sell them I can get some decently sized SSDs for the L2ARC/ZIL and it would probably be just as fast (that's a guesstimate)
 
You must log in or register to reply here.
Back
Top