Encrypted ZFS iSCSI target in VMWare Server?

[iroc409]

I skimmed a how-to article recently on building a GELI-encrypted ZFS volume on FreeBSD as an iSCSI target for a Windows Server. The person using it wanted the benefits of ZFS but still keep WS2008--it was a bit of a "best of both worlds" situation. I don't remember if the article stated, but I think the iSCSI box was a separate machine.

Would it be feasible to run a FreeBSD machine in VMWare Server on a Windows Server machine, or would the performance be too poor to use well?

In my instance, I wouldn't be using dedup.

 

[danswartz]

In general, running some sort of storage VM will have crappy performance, unless you are running ESXi or XEN and passing a PCI disk controller to the VM, which requires mobo/cpu that support vt-d or the amd equivalent.

 

[lopoetve]

We finally figured that one out, we think. You know anything much about the IO schedulers in BSD? I'm trying to learn...

 

[danswartz]

No idea, sorry My experience with BSD is mostly end-user...

 

[iroc409]

In general, running some sort of storage VM will have crappy performance, unless you are running ESXi or XEN and passing a PCI disk controller to the VM, which requires mobo/cpu that support vt-d or the amd equivalent.

The system in question has AMD-V currently. If I upgrade it (maybe out of necessity), it will have VT-d and probably VT-x.

Xen is similar to VMWare Server in that it runs on top of a host OS (as I understand it). How is that a major improvement over VMWare Server, or are you talking about XenServer? Can you not pass through a physical device in VMWare Server? I guess I've never done it, but I am pretty sure you can hand it hard drives.

Maybe I have to bite the bullet and go over the learning curve to make FreeBSD my base OS. I have experience with it, and have used it off and on since 2.2.7, but haven't had an install in a couple years. There are a number of things I've not done with it, though. I guess backups are my biggest concern.

Then what are my virtualization options on FreeBSD? Jails and VirtualBox, right?

 

[lopoetve]

heh. On linux, if you switch from CFQ to Noop, you get identical performance as raw hardware, at least storage wise (or within a few percentage points)

Trick is figuring out what the same is for BSD...

edit: all this is @ dan.

 

[danswartz]

I don't have much xen experience, just repeating what i was told . lopoetve, any ideas about solaris? this is kinda moot for me now, since i have my virtualized OI server working with pass thru mode and no real interest in messing with it

 

[danswartz]

Then what are my virtualization options on FreeBSD? Jails and VirtualBox, right?

Virtualbox doesn't have to do anything with jails, AFAIK. Not sure why you would want to use freebsd for your hypervisor? It seems to lag solaris/linux/windows as far as support and feature set (although has caught up some...)

 

[iroc409]

Virtualbox doesn't have to do anything with jails, AFAIK. Not sure why you would want to use freebsd for your hypervisor? It seems to lag solaris/linux/windows as far as support and feature set (although has caught up some...)

I wouldn't, but I have no experience with Solaris. I have used SuSE and Ubuntu. I like FreeBSD, mostly. I'm looking for a hypervisor on top of the primary file server to run WHS (for backups only), Untangle and probably pfsense.

I didn't mean to tie jails to Virtualbox--I meant them as the two options that I know of. OpenBSD apparently plays very well with Xen, but I've not used either of those.

The reason for Windows Server, is that my file server is already on that (2008 Standard). I could just plug in VMWare and go (currently using Hyper-V). I would also like to run a SATA online backup, where it backs up daily and I can just swap the backup drive out once a month. I've never done that with any *nix distro, but easy and painless with Windows. I've done a bunch of reading, and I'm not sure I could do it in ESXi without a lot of pain and suffering.

 

[lopoetve]

I don't have much xen experience, just repeating what i was told . lopoetve, any ideas about solaris? this is kinda moot for me now, since i have my virtualized OI server working with pass thru mode and no real interest in messing with it

Solaris... join the 20th century?

No idea yet. I'm still researching.

 

[danswartz]

LOL. Well, if I could get state of the art ZFS any other way, I would

 

[lopoetve]

LOL. Well, if I could get state of the art ZFS any other way, I would

Have to admit I've never understood the love for ZFS... but oh well.

 

[danswartz]

Have to admit I've never understood the love for ZFS... but oh well.

if you look at the things you can do with it, it is really something else.

 

[Olga-SAN]

...and even in such a case the same storage virtualization software but running on a bare metal should run circles around VM insider.

Olga

In general, running some sort of storage VM will have crappy performance, unless you are running ESXi or XEN and passing a PCI disk controller to the VM, which requires mobo/cpu that support vt-d or the amd equivalent.

 

[Olga-SAN]

What task are you trying to get solved? Even with encrypted target you going to end with non-encrypted iSCSI traffic hitting the wire between target and initiator. Big security hole (if you're considering encryption). Start with keeping iSCSI content unencrypted and using IPsec or any other tunneling to protect the wire. Resulting performance should suck however )

Using FreeBSD with VMware should add too much overhead... And mixed with Windows it just don't have any sense ) In your case I'd use native Windows solution running on top of Windows server and use encrypted disk (Windows built-in NTFS encryption, BitLocker or just exporting TrueCrypt managed strong encrypted volumes) as a container for iSCSI volumes. We've been using all listed approaches and they run just fine. There are free Windows targets like Microsoft own iSCSI target 3.3 and StarWind iSCSI SAN so you're not gonna break the bank )

Hope this helped.

Olga

I skimmed a how-to article recently on building a GELI-encrypted ZFS volume on FreeBSD as an iSCSI target for a Windows Server. The person using it wanted the benefits of ZFS but still keep WS2008--it was a bit of a "best of both worlds" situation. I don't remember if the article stated, but I think the iSCSI box was a separate machine.

Would it be feasible to run a FreeBSD machine in VMWare Server on a Windows Server machine, or would the performance be too poor to use well?

In my instance, I wouldn't be using dedup.

 

[Olga-SAN]

Did you ever give a try to the most recent ZFS-based products? )

Olga

Have to admit I've never understood the love for ZFS... but oh well.

 

[danswartz]

...and even in such a case the same storage virtualization software but running on a bare metal should run circles around VM insider.
Olga

Do you mean bare metal will run circles around a passthrough setup? If so, I'm curious what you base this on, as it completely contradicts my experience (and that of everyone I've talked to who has tried it.)

 

[iroc409]

What task are you trying to get solved? Even with encrypted target you going to end with non-encrypted iSCSI traffic hitting the wire between target and initiator. Big security hole (if you're considering encryption). Start with keeping iSCSI content unencrypted and using IPsec or any other tunneling to protect the wire. Resulting performance should suck however )

Using FreeBSD with VMware should add too much overhead... And mixed with Windows it just don't have any sense ) In your case I'd use native Windows solution running on top of Windows server and use encrypted disk (Windows built-in NTFS encryption, BitLocker or just exporting TrueCrypt managed strong encrypted volumes) as a container for iSCSI volumes. We've been using all listed approaches and they run just fine. There are free Windows targets like Microsoft own iSCSI target 3.3 and StarWind iSCSI SAN so you're not gonna break the bank )

Hope this helped.

Olga

Basically, I want to add the file system protection of ZFS to my Windows Server. The only reason I'm looking to encrypt everything is in case someone does a smash & grab in my house and takes the server, without starting it up the data is useless. I'm not really worried about data traveling over iSCSI un-encrypted. I've been using TrueCrypt on my laptop, and I basically can't tell it's running performance-wise. I suppose in that case I can encrypt the system from Windows, instead of encrypting the ZFS filesystem, right?

If I'm not using ZFS, there's no need for iSCSI, I just need a couple other virtual machines.

Although, I'm not sure how much data protection would be offered by a ZFS server. My current storage model at home is most of our personal documents/pictures/etc are stored locally, and the server is backup. The server does have some archived data and some common items--my TechNet file storage, MP3s, etc. The local machines do not have ECC or ZFS, so they are still subject to error. The server does not currently have ECC, but I'm planning on changing that.

 

[danswartz]

Idea: have a ZFS dataset on the server, and export it to windows as an CIFS share. You can use snapshots on the ZFS dataset, and windows clients can access previous versions of files.

 

[iroc409]

Looks like I have some reading to do. I've never used CIFS, or even looked into it.

 

[danswartz]

Looks like I have some reading to do. I've never used CIFS, or even looked into it.

cifs is another name for smb, i assume you have experience with that?

 

[iroc409]

cifs is another name for smb, i assume you have experience with that?

Oh yeah, duh. I have experience with that, but I thought the whole goal with iSCSI was that it's faster and more transparent to the system accessing the target? How would performance improve over iSCSI?

The goal is for an all-in-one box. I already have multiple boxes, I want to consolidate into a single system. My home office runs too hot.

 

[danswartz]

I understand. My motivation too Point is: it's not mutually exclusive. You can serve up content via CIFS as a network share, or by windows formatting an iSCSI share (which would look like a local disk to the client.) Not sure why it needs to be 100% transparent? Everyone in the window world is used to the concept of network shares/folders, no? In point of fact, I do both - my Win7/pro box has a CIFS share for backups, whereas my two kids use "local disks" which are actually iSCSI targets (reason being that Win7/home does not allow the backup tool to save to a network target.)

 

[iroc409]

Hmm... I don't think that would work would it? The clients (all Windows) would be connecting to the files through Server 2008. The FreeBSD box would not be sharing files directly. If I created an iSCSI target, formatted it in Server and put files on it, FreeBSD wouldn't be able to share those files directly would it? I can see Windows machines, all using NTFS, might be able to do simultaneous CIFS/iSCSI, but since Server 2008 and FreeBSD don't share file systems, it seems it wouldn't work. It could be my ignorance showing. Does this work because FreeBSD understands NTFS (assuming it does--only interacted with NTFS on Linux)?

A lot of this has to do with server management, learning more stuff when I have a lot of new stuff to learn right now (delving into AutoCAD and have been going back to school). I can set up a FreeBSD file server easy enough, but haven't done backups. It's easier to manage backups as well (recovery) if they are all in NTFS.

 

[danswartz]

I think we have a disconnect here. I am advocating using OI for the ZFS. If so, why not have it serve the files up instead of through the 2008 server box?

 

[lopoetve]

if you look at the things you can do with it, it is really something else.

I can do all of that with a decent SAN too, and even more But I'll admit, given what I work with on a daily basis, it's a bit of a different market demographic than ZFS was aimed for.

 

[lopoetve]

...and even in such a case the same storage virtualization software but running on a bare metal should run circles around VM insider.

Olga

Baloney.

/me looks at the CX4 he's been doing performance benchmarking on, as well as the LHVSA he's been comparing against.

What task are you trying to get solved? Even with encrypted target you going to end with non-encrypted iSCSI traffic hitting the wire between target and initiator. Big security hole (if you're considering encryption). Start with keeping iSCSI content unencrypted and using IPsec or any other tunneling to protect the wire. Resulting performance should suck however )

Using FreeBSD with VMware should add too much overhead... And mixed with Windows it just don't have any sense ) In your case I'd use native Windows solution running on top of Windows server and use encrypted disk (Windows built-in NTFS encryption, BitLocker or just exporting TrueCrypt managed strong encrypted volumes) as a container for iSCSI volumes. We've been using all listed approaches and they run just fine. There are free Windows targets like Microsoft own iSCSI target 3.3 and StarWind iSCSI SAN so you're not gonna break the bank )

Hope this helped.

Olga

Windows as an iSCSI target? Completely unsupported because it ~doesn't work~ right. Hope ya' got good backups! NFS works, ish, using SFU, but not super well. (Note: He's talking VMware VMFS datastores - I don't pay attention to the other portion of the world at this point).

Hmm... I don't think that would work would it? The clients (all Windows) would be connecting to the files through Server 2008. The FreeBSD box would not be sharing files directly. If I created an iSCSI target, formatted it in Server and put files on it, FreeBSD wouldn't be able to share those files directly would it? I can see Windows machines, all using NTFS, might be able to do simultaneous CIFS/iSCSI, but since Server 2008 and FreeBSD don't share file systems, it seems it wouldn't work. It could be my ignorance showing. Does this work because FreeBSD understands NTFS (assuming it does--only interacted with NTFS on Linux)?

A lot of this has to do with server management, learning more stuff when I have a lot of new stuff to learn right now (delving into AutoCAD and have been going back to school). I can set up a FreeBSD file server easy enough, but haven't done backups. It's easier to manage backups as well (recovery) if they are all in NTFS.

This would be correct. FreeBSD can read NTFS (probably, if not, meh) but it's not a super intelligent way.

I think we have a disconnect here. I am advocating using OI for the ZFS. If so, why not have it serve the files up instead of through the 2008 server box?

Agreed.

 

[danswartz]

I can do all of that with a decent SAN too, and even more But I'll admit, given what I work with on a daily basis, it's a bit of a different market demographic than ZFS was aimed for.

ZFS can be deployed lower down the price-performance curve than any SAN that provides the features ZFS does...

 

[lopoetve]

ZFS can be deployed lower down the price-performance curve than any SAN that provides the features ZFS does...

Exactly my point - I'm used to dealing with a very different market.

It also doesn't offer the features that the SAN in question would either Nor does it offer certain enterprise level features that may be required that other filesystems do. It's for a specific market.

 

[danswartz]

We are in violent agreement then

 

[Olga-SAN]

Hyper-V host running Windows based iSCSI target as native app Vs. Hyper-V host running the same target but virtualized with exported iSCSI disk configured as pass-thru instead of fixed VHD (default).

I don't have the numbers sorry for this but AFAIK bandwidth in a long run was pretty much the same but IOps were 1.5x-2x lower and average response time changed from 6-8 ms to 10-12 ms.

In general PT is not that great as it looks initially. Here's a good link to read BTW, I can only confirm the results and comment on a software SAN the difference is even less noticable:

http://clusteringformeremortals.com...-dynamic-vhd-files-in-windows-server-2008-r2/

Do you mean bare metal will run circles around a passthrough setup? If so, I'm curious what you base this on, as it completely contradicts my experience (and that of everyone I've talked to who has tried it.)

 

[Olga-SAN]

I would keep shared volumes unencrypted to boost performance just a bit and keep "secret" data on Windows encrypted volumes. I mean - solve encryption task on "per VM" basis instead of "per whole storage" basis. IMHO of course )

O.

Basically, I want to add the file system protection of ZFS to my Windows Server. The only reason I'm looking to encrypt everything is in case someone does a smash & grab in my house and takes the server, without starting it up the data is useless. I'm not really worried about data traveling over iSCSI un-encrypted. I've been using TrueCrypt on my laptop, and I basically can't tell it's running performance-wise. I suppose in that case I can encrypt the system from Windows, instead of encrypting the ZFS filesystem, right?

If I'm not using ZFS, there's no need for iSCSI, I just need a couple other virtual machines.

Although, I'm not sure how much data protection would be offered by a ZFS server. My current storage model at home is most of our personal documents/pictures/etc are stored locally, and the server is backup. The server does have some archived data and some common items--my TechNet file storage, MP3s, etc. The local machines do not have ECC or ZFS, so they are still subject to error. The server does not currently have ECC, but I'm planning on changing that.

 

[Olga-SAN]

Bad idea ) Unless you have VSS (Volume Snapshot Services) hardware provider running on your Windows box... ZFS server has no clue about what Windows does at this moment ZFS is willing to take a snapshot so you're definitely going to end with creating snapshots keeping Windows state between huge flushes of Windows Cache Manager's lazy writer. In other words without VSS back end any SAN snapshots is inconstant. Windows aware software iSCSI SAN vendors like DataCore, FalconStor, StarWind and Microsoft itself (surprise!) do ship this their products with VSS agents you should install at the client side. That's the way to go )

Olga

Idea: have a ZFS dataset on the server, and export it to windows as an CIFS share. You can use snapshots on the ZFS dataset, and windows clients can access previous versions of files.

 

[Olga-SAN]

+1000

For either using OpenIndiana to serve as ZFS iSCSI/SMB box *or* keeping everything done on Windows Server 2008 R2 (some free iSCSI target to feed Hyper-V and built-in SMB thing for file level sharing).

O.

I think we have a disconnect here. I am advocating using OI for the ZFS. If so, why not have it serve the files up instead of through the 2008 server box?

 

[Olga-SAN]

Clarify please ) What's wrong with Windows based iSCSI SAN?

P.S. And yes backups are good and up to date )

Windows as an iSCSI target? Completely unsupported because it ~doesn't work~ right. Hope ya' got good backups! NFS works, ish, using SFU, but not super well. (Note: He's talking VMware VMFS datastores - I don't pay attention to the other portion of the world at this point).

 

[danswartz]

Hyper-V host running Windows based iSCSI target as native app Vs. Hyper-V host running the same target but virtualized with exported iSCSI disk configured as pass-thru instead of fixed VHD (default).

I don't have the numbers sorry for this but AFAIK bandwidth in a long run was pretty much the same but IOps were 1.5x-2x lower and average response time changed from 6-8 ms to 10-12 ms.

In general PT is not that great as it looks initially. Here's a good link to read BTW, I can only confirm the results and comment on a software SAN the difference is even less noticable:

http://clusteringformeremortals.com...-dynamic-vhd-files-in-windows-server-2008-r2/

I haven't used hyper-v, my and others experiences were with esxi. I think the article has it completely backwards, at least as far as zfs is concerned. Disadvantages of vhd method: no way to monitor smart status of drives. No way to hot-plug/unplug drives from zpool.

 

[danswartz]

Hyper-V host running Windows based iSCSI target as native app Vs. Hyper-V host running the same target but virtualized with exported iSCSI disk configured as pass-thru instead of fixed VHD (default).

I don't have the numbers sorry for this but AFAIK bandwidth in a long run was pretty much the same but IOps were 1.5x-2x lower and average response time changed from 6-8 ms to 10-12 ms.

In general PT is not that great as it looks initially. Here's a good link to read BTW, I can only confirm the results and comment on a software SAN the difference is even less noticable:

http://clusteringformeremortals.com...-dynamic-vhd-files-in-windows-server-2008-r2/

I read that article more thoroughly. It sounds like we are talking apples and oranges. I am not talking about pass-thru disks, but passing thru the entire controller, and the disks go with it.

 

[lopoetve]

Clarify please ) What's wrong with Windows based iSCSI SAN?

P.S. And yes backups are good and up to date )

Not supported

We're talking about ESX and VMware bits here, where the windows targets are not functional.

Hyper-V host running Windows based iSCSI target as native app Vs. Hyper-V host running the same target but virtualized with exported iSCSI disk configured as pass-thru instead of fixed VHD (default).

I don't have the numbers sorry for this but AFAIK bandwidth in a long run was pretty much the same but IOps were 1.5x-2x lower and average response time changed from 6-8 ms to 10-12 ms.

In general PT is not that great as it looks initially. Here's a good link to read BTW, I can only confirm the results and comment on a software SAN the difference is even less noticable:

http://clusteringformeremortals.com...-dynamic-vhd-files-in-windows-server-2008-r2/

Don't know hyper-V much, but I can definitely call shens on the ESX side. I can get you within a percentage point of the same IOPS on RDM or VMDK on VMFS, all things being equal. And then re-export it if needed and get you within a point of what was under that too, with the right scheduler.

 

[danswartz]

Not supported

We're talking about ESX and VMware bits here, where the windows targets are not functional.



Don't know hyper-V much, but I can definitely call shens on the ESX side. I can get you within a percentage point of the same IOPS on RDM or VMDK on VMFS, all things being equal. And then re-export it if needed and get you within a point of what was under that too, with the right scheduler.

you figured out the freebsd I/O issue. on the other hand, like i said, if you do want to do a virtualized SAN, using multiple disks in some kind of software RAID (ZFS or otherwise), it's kind of a PITA to have to put one big "disk" file on a datastore on each disk, since you have no access to smart info, and no easy way to hot plug/unplug drives. With ZFS at least, if a drive fails, you can have a spare in the pool and it will automatically be put into service and start resilvering the pool. Can't do that with the non-pass through mode, I wouldn't think... I'm curious, did you ever figure out WHY the wrong disk scheduler wreaks such havoc?

 

[lopoetve]

Well, still working on BSD, but I can do linux or windows

It has to do with the schedulers conflicting, since ESX has an async IO scheduler built in, and putting another asynch IO scheduler (the default in linux) on top results in... well, amusement. It's like shifting without synchros and no clutch. If you line up JUST right, it'll work. Otherwise? Crunch.

As for ZFS... put a single VMDK on each VMFS volume, and then a spare one if needed... could work just fine - ESX can handle some of the hardware notifications, depending on what they are, and the hot-swap will work with a vmdk as well as anything.

I won't argue the utility of passthrough (I'm all for run-the-san-on-bare-metal) - but performance, that we should be able to fix