• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ESXi host and NAS on single computer

Y

yankees9920

Weaksauce
Joined
Apr 23, 2010
Messages
68
I am looking to build a cheap ESXi whitebox while not increasing the number of computers that I run 24/7 so I am looking to virtualize a NAS on top of this.

I am wondering if it is a good idea/possible to have it set up so that ESXi boots off a flash drive, a FreeNAS VM is installed and runs from another flash drive (or other partition). The FreeNAS machine has RDMs to the physical hard drives and the FreeNAS VM provides the ESXi vm datastore via iSCSI or NFS so that all my other VMs are running on a raidz array.

The other VMs will be primarily testing and fooling around and I'd probably also like to run a torrent/media indexing vm and a DNS server so I can do network booting for a HTPC.

This is the way I envision the system running and I am looking for any thoughts or comments. I am also looking for a relatively cheap motherboard that may be suitable for this job that works with ESXi.
 
I plan to be testing this out myself in the next week or so.

Same concept. Install FreeNAS as a VM and try to give it direct access to a few physical disks in a ZFS pool.

I'm not sure if I really want to do this as a "permanent" solution.. more or less just to see how it works. Assuming I can write the file system directly to the drives instead of being in a .vmdk container, it should work out quite well though.

If it works out well enough, I may also try out the NFS route also. I've already had decent success having ESXi 3.5 connect to an NFS share on a separate FreeNAS system.

I was going to start using the HTPC I've recently thrown together for some of my backup uses, but if I were able to efficiently and reliably virtualize FreeNAS how we're talking here, I could stuff a couple drives in the ESXi machine and leave the HTPC off most of the time. The board I'm using has 5 internal SATA ports, so I might as well try to use them if I can.
 
I think we are on the same page. I'd want FreeNAS to have direct access to the disks so you could theoretically "unvirtualize" (if that is even such a thing) freenas and plug in the existing hard drives and maintain your array. (I don't know if that would technically work).
Then .vmdk's would be stored in the ZFS pool for the other VMs.

Please let me know how that works out for you and I'm also interested to hear what hardware you use for it.
 
Are you planning on running VMs you care about on this box or just some to play with?

If you're just playing, you could install Windows or Linux on the box, create an NFS share, then install VMware Workstation and install as many ESXi instances as you want. Those ESXi installs would use the NFS share as their datastore.
 
yankees9920 said:
I think we are on the same page. I'd want FreeNAS to have direct access to the disks so you could theoretically "unvirtualize" (if that is even such a thing) freenas and plug in the existing hard drives and maintain your array. (I don't know if that would technically work).
Then .vmdk's would be stored in the ZFS pool for the other VMs.

Please let me know how that works out for you and I'm also interested to hear what hardware you use for it.
Click to expand...

Yeah, it would be nice to just pull the drives out and drop them in another non-ESXi system if the need came up.

We'll see how it works, now I just need to wait for a pair of Intel gigabit cards to show up since I don't apparently have any that are compatible with ESXi4.1. I knew the onboard wasn't going to work without cobbling some drivers together, but I was hoping my old 3Com 10/100 would've worked.

Child of Wonder said:
Are you planning on running VMs you care about on this box or just some to play with?

If you're just playing, you could install Windows or Linux on the box, create an NFS share, then install VMware Workstation and install as many ESXi instances as you want. Those ESXi installs would use the NFS share as their datastore.
Click to expand...

True, this would also be an option, but that's just adding another abstraction layer, and I'm trying to focus more on ESXi right now. I haven't purchased Workstation yet, but I do have VMware Player running on my laptop.
 
DuronClocker said:
True, this would also be an option, but that's just adding another abstraction layer, and I'm trying to focus more on ESXi right now. I haven't purchased Workstation yet, but I do have VMware Player running on my laptop.
Click to expand...

But this way you would be able to use VMotion, DRS, HA, etc. which are all features that require two or more ESXi hosts with shared storage.
 
Child of Wonder said:
But this way you would be able to use VMotion, DRS, HA, etc. which are all features that require two or more ESXi hosts with shared storage.
Click to expand...

My plan would be to have a single ESXi/FreeNAS machine and then in order to test with the VMotion, HA, DRS stuff, I could just install ESXi in vmware Workstation on my desktop. I imagine running windows, and vmware workstation with 2 esxi hosts each running vm's would require enormous amounts of ram on a single pc which I don't have.

I have to look more into the StoreMagic stuff, but it seems like something I may experiment with. Thanks!
 
It's possible... still not sure how reliable it is though :)

Currently I have my ESXi box set up with a Nexentastor VM for testing NAS features. The NAS features have direct access to the onboard 8-port SAS controller (1068e-based) which has 4 WD20EADS drives in a RAIDZ configuration, and 4x160 GB 7200RPM laptop drives in a RAID 10 configuration (2 in mirror mode, then added another 2 in mirror to the pool). The RAID10 is shared to ESXi via ISCSI in one whole partition, while the RAIDZ is shared in 2x2TB LUNs and a 1.5TB LUN.

I then installed a WHS VM, Windows 7 VM, and a Windows 2003 VM (for VirtualCenter) on the RAID10 drives and mapped the RAIDZ luns to the WHS VM for testing.

I also configured the startup options for ESXi to automatically boot the NAS VM first with a 180 second delay or Continue immediatly if VMWare Tools starts, followed by the VirtualCenter VM, and then WHS (since the other two depend on the RAID10 share on the NAS).

The NAS VM runs off a local 300GB SATA drive which also holds my ISO images.
 
farscapesg1 said:
It's possible... still not sure how reliable it is though :)

Currently I have my ESXi box set up with a Nexentastor VM for testing NAS features. The NAS features have direct access to the onboard 8-port SAS controller (1068e-based) which has 4 WD20EADS drives in a RAIDZ configuration, and 4x160 GB 7200RPM laptop drives in a RAID 10 configuration (2 in mirror mode, then added another 2 in mirror to the pool). The RAID10 is shared to ESXi via ISCSI in one whole partition, while the RAIDZ is shared in 2x2TB LUNs and a 1.5TB LUN.

I then installed a WHS VM, Windows 7 VM, and a Windows 2003 VM (for VirtualCenter) on the RAID10 drives and mapped the RAIDZ luns to the WHS VM for testing.

I also configured the startup options for ESXi to automatically boot the NAS VM first with a 180 second delay or Continue immediatly if VMWare Tools starts, followed by the VirtualCenter VM, and then WHS (since the other two depend on the RAID10 share on the NAS).

The NAS VM runs off a local 300GB SATA drive which also holds my ISO images.
Click to expand...

Very interesting. Have you seen any issues with this setup? How is your read/write performance vs having a physical nas?
 
Well, it is still very preliminary. I tried an Openfiler VM and the performance was horrible (about 10 MB/s transfers). That was using a software RAID5 for the WD20EADS drives.

I currently have a Dell 760 with a single 250 GB hard drive set up as a FreeNas iSCSI connected to the host also, so I could create a virtual disk and compare the transfers with the RAIDZ configuration. I will say that the transfer rate for the RAIDZ was between 50-60 MB/s, compared to the Openfiler VM, which is more in-line with what I need for multimedia streaming via the WHS VM.

I've only had the NexentaStor VM running for about 5 days, so I really can't comment on reliability, but so far there hasn't been any issues. The FreeNas setup is still attached specifically for the reason of testng out different NAS solutions (transfer my VMs off the virtualized NAS, rebuild, transfer back and test... repeat).
 
Good info farscapesg1.

I look forward to playing with this once my NICs show up (and memory!! picked up 2x 4GB DDR3 sticks for $72 shipped w/ Newegg's 15% off coupon code!)
 
Hey Duron I just wanted to check in and see if you got your NICs and have tested this setup at all.
 
i have several esxi-servers running with a virtualized embedded NAS/SAN. The major problem about is the virtualized SAN if you need performance and stability similar to the performance on real hardware. Usually you should avoid to use ESXI formatted disks for your NAS/SAN or to use unsupported things like RAW mapping.

I have found only one way to have it running fast and stable.

1. your virtualized NAS needs full access to its own storage controller and disks
this is done via pci-pass-through. this feature is mostly only availybe on mainboards
based on server-chipsets. they are from 150 Euro up.
If you have enough RAM, such a combo can perform just similar to solutions with a VMServer and a separate SAN connected via a good hardware-switch.

2. you should have a small stable server-class os that can deliver NAS services like SMB or AFP and SAN services like NFS or iSCSI for your VM-Server.

i have created such a all-in-one concept based on free components like ESXi 4.1 and a ZFS-Server OS like Nexenta + my Web-gui + hardware recommendations to build.

I have a thread at http://hardforum.com/showthread.php?t=1573272 for more details about.

Gea
 
I am also running a setup similar in principle. I have an ESX 4 box with an Areca 1880i in it. I boot ESX off of a 2.5" drive. I created 1 Windows Storage Server VM and used PCI passthrough to assign the Areca raid controller to the Storage Server VM.

The Storage Server VM runs microsoft's iSCSI target software, and serves that to the ESX running on the host.

Works great so far.

Someone mentioned performance issues in running the iSCSI in a virtualized environment. I'm sure there are some, but one huge benefit is that all connections between the iSCSI target and the other VMs are 10gbe, which is not really affordable for home usage yet.
 
randomlychosen said:
I am also running a setup similar in principle. I have an ESX 4 box with an Areca 1880i in it. I boot ESX off of a 2.5" drive. I created 1 Windows Storage Server VM and used PCI passthrough to assign the Areca raid controller to the Storage Server VM.

The Storage Server VM runs microsoft's iSCSI target software, and serves that to the ESX running on the host.

Works great so far.

Someone mentioned performance issues in running the iSCSI in a virtualized environment. I'm sure there are some, but one huge benefit is that all connections between the iSCSI target and the other VMs are 10gbe, which is not really affordable for home usage yet.
Click to expand...

about performance of a integrated SAN
if you use the ESXi vmxnet network driver on your Storage Server and if you have enough CPU-power and RAM, performance of internal datatransfers should be better than with an external 10GBe IP-SAN-Network

about Windows
two years ago, most of our virtual and storage server were based on Windows. From Hyper-V, we moved away due to the always needed security updates + reboot and the problem, that you can't just copy a VM without correct export. ESXI is much more easy to handle.

All our storage servers are now based on ZFS. We moved away not only because of the
monthly patches but mainly because of the more advanced features of ZFS like scrubbing (online filecheck/ refresh, something like a online chkdsk).With Windows you have to unmout a disk prior a filecheck - a huge problem because of silent data-errors. Also Data-Security is much better due to end-to-end-checksums and copy on write. With ZFS, files on disk are always correct or not written. Other features like controller independant pools, better snapshots, bootable snapshots, hybrid storage and os/kernel-integrated support for SMB, NFS and iSCSI convnced me.

If you have a test-machine, try ZFS. I'm quite sure, i will hear a WOW.

Gea
 
Well, here's the good news of running FreeNAS virtualized with raw device mapping of my 4x 250GB WD RE (first gen) drives on the AMD SB710 southbridge's SATA ports in AHCI.

Below are speeds I got from a simple dd command of a 6.1GB file with 6GB physical memory assigned using ZFS.

Single drive
read 54MB/s
write 43MB/s

raid1 (4 drives mirrored)
read 189MB/s
write 41MB/s

raid0 (4 drives striped)
read 219MB/s
write 152MB/s

raid10 (striped pair of mirrors)
read 190MB/s
write 85MB/s

raidz1 (3 drives)
read 107MB/s
write 51MB/s

raidz1 (4 drives)
read 121MB/s
write 61MB/s

I thought these were pretty respectable numbers, and it is very neat to see the scaling of ZFS. I ran each test multiple times (between 5 and 10) and took the average. Of course this is only sequential reads and does not take into account any random reads or access time/IOP numbers, but it means I'll be sticking with the RAID10 setup for this particular array.

Now the bad news... I've been very disappointed with the NFS performance of FreeNAS though. Can't seem to get much more than a few MB/s out of it. I've installed OpenIndiana and Gea's front-end and will be testing the NFS performance of that soon. Based on others' experience with OI's ZFS/NFS, I think I am going to be very happy with it. My primary concern is that I cannot run it from an SD card as I was doing with FreeNAS, but it isn't a problem to run off an IDE drive (just was hoping for less moving parts, especially old ones).
 
Thanks for posting that.
Why not try iSCSI if you are not the getting the performance out of NFS?

I'd also be interested to see what SMB performance is and if its hindered by the virtual nics/switches.
 
Last edited:
Best performance I've gotten out of any NAS/iSCSI solution in a virtualized NAS/SAN machine was... Windows 2008 R2 with Microsoft iSCSI Target 3.3.

I used a ram disk to remove any disk based limits during my testing, all VM's were given 4GB Ram, 2vCPUs. Tested outside the internal vSwitch, on the "real network" from my laptop connected to procurve 1800-24G switch at 1GbE link.

It was a tad surprising, Microsoft being the ultimate winner I mean, I get full wire speeds. There are a few downsides to using the MS solution though, no HA support without using full "MS Clustering", which is a little retarded IMO. Tested by copying a 10GB file over, speed posted is average. I decided to try Windows on a whim, and low and behold it has given me the best performance., I haven't tried FreeNAS yet, but performance from highest to lowest in all my tests was this:

Read - 112MB/s, Write - 114MB/s - Windows 2008 R2 with Microsoft iSCSI Target 3.3
Read - 111MB/s, Write - 84MB/s - Windows 2003 R2 with Starwind iSCSI SAN Target 5.6
Read - 111MB/s, Write - 67MB/s - Windows 2003 R2 with Starwind iSCSI SAN Target 5.6 - Using HA
Read - 94MB/s, Write - 43MB/s - NexentaStor Community 3.0.4 iSCSI Target
Read - 74MB/s, Write - 48MB/s - Openfiler (latest updated build) iSCSI Target

I haven't tested yet, but plan to test: FreeNAS, Datacore SANsymphony-V R8, and Anything else interesting I come across.

I plan on re-doing all these tests again soon before I decide on a final solution for my home lab. On my home lab I plan on setting up a HA SAN, virtualized with an HBA card passed through VT-d to the virtual instance of the SAN VM. Unfortunately Datacore and Starwind are both paid products with no community, limited free, or TechNet style way of gaining access to more than a 30 day trial, even if for use only on a home testing lab. So... I'll use them for testing and playing around with but that’s about it.

If any ones interested, when I do my next round of testing I'll actually do more proper recording of the results and whatnot.
 
Duron, I was wondering if you have any numbers regarding the read/write speed for a VM that has a virtual disk located on the ESXi datastore located on your virtual FreeNAS w/ RDMs (or whatever you went with instead).
I want to make sure the datastore performance for guest VMs is fine.
 
Sounds strange you did not get wire speed for write with StarWind. StarWind used very effective write-back cache and MS uses no caching at all so I cannot get how you've managed to have lower speed with SW if configured properly. Also the same with Nexenta. Did you enable ARC on it? Was ZFS configured to run de-duplication? If yes what block size used?

From hypervisor storage point of view Microsoft has zero sense to use because it has no de-duplication. Virtual machines images dedupe very good (1:10 ratio is not uncommon) and you also has less disk access so both peformance and capacity raise. Win-win situation. And MS is still in mid-2005 when they got StringBean: no high availability, no caching, no dedupe... Pretty ghetto :)

Olga-SAN

Chilly said:
Best performance I've gotten out of any NAS/iSCSI solution in a virtualized NAS/SAN machine was... Windows 2008 R2 with Microsoft iSCSI Target 3.3.

I used a ram disk to remove any disk based limits during my testing, all VM's were given 4GB Ram, 2vCPUs. Tested outside the internal vSwitch, on the "real network" from my laptop connected to procurve 1800-24G switch at 1GbE link.

It was a tad surprising, Microsoft being the ultimate winner I mean, I get full wire speeds. There are a few downsides to using the MS solution though, no HA support without using full "MS Clustering", which is a little retarded IMO. Tested by copying a 10GB file over, speed posted is average. I decided to try Windows on a whim, and low and behold it has given me the best performance., I haven't tried FreeNAS yet, but performance from highest to lowest in all my tests was this:

Read - 112MB/s, Write - 114MB/s - Windows 2008 R2 with Microsoft iSCSI Target 3.3
Read - 111MB/s, Write - 84MB/s - Windows 2003 R2 with Starwind iSCSI SAN Target 5.6
Read - 111MB/s, Write - 67MB/s - Windows 2003 R2 with Starwind iSCSI SAN Target 5.6 - Using HA
Read - 94MB/s, Write - 43MB/s - NexentaStor Community 3.0.4 iSCSI Target
Read - 74MB/s, Write - 48MB/s - Openfiler (latest updated build) iSCSI Target

I haven't tested yet, but plan to test: FreeNAS, Datacore SANsymphony-V R8, and Anything else interesting I come across.

I plan on re-doing all these tests again soon before I decide on a final solution for my home lab. On my home lab I plan on setting up a HA SAN, virtualized with an HBA card passed through VT-d to the virtual instance of the SAN VM. Unfortunately Datacore and Starwind are both paid products with no community, limited free, or TechNet style way of gaining access to more than a 30 day trial, even if for use only on a home testing lab. So... I'll use them for testing and playing around with but that’s about it.

If any ones interested, when I do my next round of testing I'll actually do more proper recording of the results and whatnot.
Click to expand...
 
Oh... And you messed everything with a licenses a bit :)

1) Microsoft. TechNet only. And no production (if you're crazy to put it into production). Or go buy Dell or HP hardware with Storage Server. Sick...

2) StarWind. 2TB limited free version and no HA for free. You can get full version if you're VCP, CCP, MCP and so on on NFR basis however.

3) Nexenta. Paid version if you want to pay. You can go free if you want. The only problem is lack of true HA DataCore & StarWind have. But ZFS rocks! StarWind has some de-duplication and DataCore claims we don't need any :)

4) Openfiler. Paid version is very expensive but not obvious what you pay for. You can go free one if you want. The only problem is - HA is shared IP Linux cluster so 1/2 HA actually :)

5) DataCore. Was performance leader. Price leader for sure ;) No dedupe and no free version (or did I miss anything?) Cheap version discontinued :(

Other ones?

Olga
 
yankees9920 said:
Duron, I was wondering if you have any numbers regarding the read/write speed for a VM that has a virtual disk located on the ESXi datastore located on your virtual FreeNAS w/ RDMs (or whatever you went with instead).
I want to make sure the datastore performance for guest VMs is fine.
Click to expand...

Have not. Those numbers I posted seemed promising, but it turns out it either wasn't reliable at all, or I did something wrong. Those tests were using a simple dd transfer of a file larger than RAM.

Then I started to try running Bonnie++ and it would never complete. I then discovered that every time I ran Bonnie++ on a ZFS volume, it would go into degraded mode. This makes sense considering I did have a few files from the dd tests that I wasn't able to remove after creating them.

Everything seemed correct with the RDM mapping, drives showed up correctly and all looked normal, but I couldn't write anything to them reliably. I tested all scenarios from a 1-drive pool to a 4-drive pool with no luck.

Ended up deciding to scrap the setup and am now in the process of creating a separate NAS again. Will be using OpenIndiana (was kind of hoping for the 2011.02 to be out by now since I'm almost ready to get it up and running) with _Gea's Napp-It.

At least being able to see the scaling of ZFS in my earlier numbers was cool. I have NOT tested the drives since this, but I would find it really hard to believe that all 4 drives decided to fail on me at the exact same time. I do plan on using them in the new setup:

Asus P5B Deluxe
Celeron E3400
4GB DDR2
4x 250GB RE drives in a RAID10-style pool

Going to add a 32GB SSD drive I recently picked up as L2ARC, and I've got a 250GB WD Black 2.5" drive that I plan on using as a hotspare.. just need to check the sector count to make sure that will work out. That is basically just for short-term protection. Once a RE fails, and the hot-spare is needed, I'll start replacing the 250's with 300GB VelociRaptors.

I just bought one of those IBM M1015 cards for my normal media/PC datastore. Since I'm adding the hot-spare and SSD to the above setup, I won't have enough onboard SATA ports for my current 2x500GB drives (likely upgrading to 4-6 2TB drives in a RAIDZ at the end of the year). I'm hoping that is going to work in the x16 PCI-e slot on this board as I'd like to also add an Infiniband setup between the two machines in the not-so-distant future. I guess I'll find out soon if this board will let me do that, but it realistically isn't a huge deal at this point in time. If it doesn't work, I'll just put the SAS card in the second PCI-e slot (x4 is still more than I'll saturate). I'll just know I'll need to upgrade the motherboard before I can go to Infiniband in the future.
 
Last edited:
_Gea said:
new miniHowto uploaded

ESXi barebone Hypervisor + included OpenIndiana ZFS NAS/ SAN Storage Server
+ other guests like Windows, Linux and others in one box

see draft copy
part 2 of my ESXi all-in-one mini HowTo
http://www.napp-it.org/doc/downloads/all-in-one.pdf

please report errors

Gea
Click to expand...

Gea,

I read your document as I have been wanting to do exactly as the others have posted in this thread (ESXi hosting a VM NAS/SAN with direct hardware access). I noticed that one of the notes in your document is that you need to use VT-D for the passthrough. The box that I currently have is using a celeron that supports VT-x, however it does not support VT-D. Is this going to be a problem?
 
Hm, back to trying one system for everything. Ended up seeing an 890FX (for IOMMU) board for sale in FS/T, and assuming all goes well my parents' ancient P4 2.4 system will get upgraded to the P5B/E3400.

This will basically be a _Gea-style setup but AMD instead of Intel.
 
DuronClocker said:
Hm, back to trying one system for everything. Ended up seeing an 890FX (for IOMMU) board for sale in FS/T, and assuming all goes well my parents' ancient P4 2.4 system will get upgraded to the P5B/E3400.

This will basically be a _Gea-style setup but AMD instead of Intel.
Click to expand...

Isn't the only reason that you would need IOMMU to pass through a RAID controller? I though your disks were directly connected to the motherboard.

Could your issues be virtual network confutation related. Look how one user has it setup here - http://hardforum.com/showthread.php?t=1494591
 
My speed performance concerns were only with the poor NFS speed using FreeNAS on a separate system. Honestly, I didn't spend much time troubleshooting it. When I researched online and saw a ton of people struggling with it, I said there has to be other options.. which is when I came across the OpenIndiana project, and then here the Napp-it front-end.

My drives were connected straight to the motherboard (using RDM) for the performance numbers I posted earlier in this thread, and I was definitely happy with those for these drives. However, I just couldn't write to the drives without errors/degrading pools (even a single-disk pool). Instead of troubleshooting that (there seems to be very little support for that method of RDM, and I don't see what I could have done wrong if the vmdk's created without errors), I decided to just go with a separate system (7 SATA ports on the mobo) with an easier method of adding drives (drop in card and go). I'm pretty sure that method wasn't giving me S.M.A.R.T. information either.

I was all for that route then until I came across this 890FX board for $100. I'll get more than that back out of the parts I picked up for the second system in addition to the current AM3 board I'll no longer need. Now I no longer have to deal with the unsupported RDM method.. which would be quite a pain as far as adding/replacing disks. Downside to this particular board is that it only has 5 internal SATA ports, but the plus side is I can easily add extra SAS/SATA cards and pass them through to the NAS VM directly.. so I don't need to limit myself to hardware with only ESXi support.

Another added benefit is that I should also be able to pass an old Intel 10/100 dual-port NIC through (ESXi dropped support for anything less than gigabit) through to an IPCOP VM, which should simplify things for me a bit.
 
DuronClocker said:
I was all for that route then until I came across this 890FX board for $100. I'll get more than that back out of the parts I picked up for the second system in addition to the current AM3 board I'll no longer need. Now I no longer have to deal with the unsupported RDM method.. which would be quite a pain as far as adding/replacing disks. Downside to this particular board is that it only has 5 internal SATA ports, but the plus side is I can easily add extra SAS/SATA cards and pass them through to the NAS VM directly.. so I don't need to limit myself to hardware with only ESXi support.
Click to expand...

If I am correct I don't believe all 890FX boards support IOMMU, I think it varies by the model as the option must be available in the BIOS. What board did you go with and how did you possible get it for only $100?
 
Biostar TA890FXE, someone was selling it in the FS/T forum here. I've looked around the web a bit and have seen mixed reviews of IOMMU on this board (pretty much the same with all 890FX boards). There is a BIOS available from Biostar that says it added IOMMU support.. we'll see.

I'm not sure if I'll get around to playing with it much for a few days as I'm waiting on a case to show up.
 
With the brief testing I did tonight, IOMMU can be turned on in the BIOS, and ESXi recognizes it and allows me to select devices to pass-through (SAS controller). I haven't actually gotten the system together yet as far as installing any VM's yet though. I likely won't be able to now for a while as I need to get a few other items figured out... but so far it is promising.
 
I just put a proof of concept together on the cheap (already had the drives, memory, and intel vt-x processor) and I can confirm that DQ35MP works with the IBM M1015 (flashed with LSI 20.10.1-0020). The DQ45CB does NOT WORK with the M1015. I also tried the SASUC8I and it appears to work with either board, however I didn't actually flash it with IT FIRMWARE and setup drives. The SASUC8I has drivers in OpenIndiana, but you will need to add them for the M1015. Keep in mind that these boards only support 8GB of memory and you must reserve all memory for a VM that has passthrough enabled. Here's the best deals I found (I purchased the first two and had no problems).

$90 INTEL DQ35MP (ONLY 2 LEFT)
$35 IBM M1015
$115 INTEL DQ45CB
$150 SASUC8I
 
As an eBay Associate, HardForum may earn from qualifying purchases.
As an Amazon Associate, HardForum may earn from qualifying purchases.
DuronClocker said:
With the brief testing I did tonight, IOMMU can be turned on in the BIOS, and ESXi recognizes it and allows me to select devices to pass-through (SAS controller). I haven't actually gotten the system together yet as far as installing any VM's yet though. I likely won't be able to now for a while as I need to get a few other items figured out... but so far it is promising.
Click to expand...

Great news! Are you able to pass through the onboard SATA controller and NIC or does it require adding a seperate hardware controller/NIC?
 
cooter said:
Gea,

I read your document as I have been wanting to do exactly as the others have posted in this thread (ESXi hosting a VM NAS/SAN with direct hardware access). I noticed that one of the notes in your document is that you need to use VT-D for the passthrough. The box that I currently have is using a celeron that supports VT-x, however it does not support VT-D. Is this going to be a problem?
Click to expand...

With vt-x, you can virtualize basic hardware parts and use them with emulated drivers.
you cannot pass-through hardware directly to guests.

This feature was added with hardware I/O virtualizuation and is called vt-d (Intel)
or IOMMU (AMD) and allows to assign a entire pci-e device to guests.

Without vt-d you can only try raw disk mapping (RDM)
= direct access to single drives.

!! BUT !!
RDM is unsupported with ESXi, it may work,
but i have heard mostly about problems with this config.

I have not tried by myself. Bit i would say its more a sort of proof of concept,
not suggested or problem-free - avoid it -


Gea
 
Olga-SAN said:
Other ones?

Olga
Click to expand...

There is Open-E http://www.open-e.com which offers their Open-E Data Storage Software V6 Lite for free though it has a 2TB limit. I haven't tried it myself so I can't comment.

I do want to comment on NexentaStor though. I see a number of issues with it. For one you can't separate your management traffic from your storage traffic, at least not easily. You could hack something together in iptables or whatnot but out of the box all shares are exposed to all network interfaces without an option to limit access to the storage by interface or network.

I spun up a NexentaStor Community Edition VM and the first thing I noticed is that the user interface is bugged. Changes I made to the default config during installation did not show up in the web ui even though the changes were committed and functioning (i.e. network interfaces).

On top of that the NexentaStor user community is pretty dead, not a whole lot of dev news either although there's a post somewhere saying that they are working on a new version.
 
The whole idea of having storage node virtualized is just plain brain dead :( Hypervisor machine needs some service applied. What are going to do? Move virtual machine to another hypervisor. With all the storage it controls? This means you're back with DAS concept as you need to have ALL hypervisor nodes having their DAS storage synchronized. It works more or less for two node setup. But for more and heavy load the whole thing starts breaking to parts. So yes to home setup. Yes to very basic commercial installation. But no even for a bit serious setup.

Olga

NetJunkie said:
EMC Celerra VSA:

http://nickapedia.com/2010/09/12/ubertastic-celerra-uber-vsa-v3-unisphere/
Click to expand...
 
Olga-SAN said:
The whole idea of having storage node virtualized is just plain brain dead :( Hypervisor machine needs some service applied. What are going to do? Move virtual machine to another hypervisor. With all the storage it controls? This means you're back with DAS concept as you need to have ALL hypervisor nodes having their DAS storage synchronized. It works more or less for two node setup. But for more and heavy load the whole thing starts breaking to parts. So yes to home setup. Yes to very basic commercial installation. But no even for a bit serious setup.

Olga
Click to expand...

In most cases I won't argue that..but there are often good use cases for virtualized storage. Remote sites being one. I can drop a Celerra VSA on a remote site's vSphere host and then have it replicate back to a Celerra physical SAN in HQ.

The usual use for these is just lab environments, though...which is the point here.
 
Used to deal with open-e couple of years ago when hired by one of local TV channels. I'm still shaking :) I don't like virtually everything about this company. It distribution model through local resellers with OEM hardware. It's support model when guys who had sold the hardware can only accept complain for burnt CPU fan or HDD being dead on arrival and we need days and weeks to decide do we have hardware or software issue and who's responsible it it. Don't like their broken HA model with having third unprotected pinging node to wake up virtual ip cluster as faiover happens too slow and your media streaming gets corrupted. And finally they sell something they don't control. You need to have either a lot of money to have own working linux distro like a lot of big companies do or you need to go open source so open community would take care of our problems. But as open-e is not a big company and they are closed source if qlogic releases buggy driver for FC target mode and open-e includes it as an update you have non-working config! And it's your personal problem. No one can help. Been there seen this. So open-e never again. Thank you very much!

Good idea and commonly accepted practice to have SAN ethernet traffic physically separated from anything else trevelling your company copper or fiber tunnels :) If you *can* ran iSCSI over the same wire you use to tweat facebook pages it does not really mean you *should* do it :)

I can only partially agree about nexenta but still agree. They are like open-e but they don't go closed source. So they have community. And you don't need to pay to nexenta for ZFS as you can say go :http://openindiana.org/ What I've really tried to say you always pay :) With money or your own time.

Thuleman said:
There is Open-E http://www.open-e.com which offers their Open-E Data Storage Software V6 Lite for free though it has a 2TB limit. I haven't tried it myself so I can't comment.

I do want to comment on NexentaStor though. I see a number of issues with it. For one you can't separate your management traffic from your storage traffic, at least not easily. You could hack something together in iptables or whatnot but out of the box all shares are exposed to all network interfaces without an option to limit access to the storage by interface or network.

I spun up a NexentaStor Community Edition VM and the first thing I noticed is that the user interface is bugged. Changes I made to the default config during installation did not show up in the web ui even though the changes were committed and functioning (i.e. network interfaces).

On top of that the NexentaStor user community is pretty dead, not a whole lot of dev news either although there's a post somewhere saying that they are working on a new version.
Click to expand...
 
You're 2000% correct on this! Thank you for bringing this one up! :))

Again you're right here. I though we;ve already went generic :))

Olga

NetJunkie said:
In most cases I won't argue that..but there are often good use cases for virtualized storage. Remote sites being one. I can drop a Celerra VSA on a remote site's vSphere host and then have it replicate back to a Celerra physical SAN in HQ.

The usual use for these is just lab environments, though...which is the point here.
Click to expand...
 
You must log in or register to reply here.
Back
Top