iSCSI and zvols

No1451

Weaksauce
Joined
Oct 19, 2010
Messages
75
Beginning the finals stages of my new server setup and I am aiming to use iSCSI to share my ZFS storage out to a Windows machine(WHS 2011 that will manage it and serve it to the PCs in my network), however I'm a little confused.

Can I simply use iSCSI to share an entire ZFS pool? I have read a lot of guides that all show sharing a zvol, if I DO use a zvol is it possible in the future to expand it and thereby increase the iSCSI volume that the remote computer will see?

:confused:help please!
 
iSCSI is a SAN-protocol, and as such the CLIENT computer (windows) will control the filesystem, not the server which is running ZFS.

So how does this work: ZFS reserves a specific amount of space (say 20GB) in a zvol which acts as a virtual harddrive with block-level storage. This zvol is passed to iSCSI-target daemon which exports over the network. Finally your windows iSCSI driver presents a local disk, which you can then format with NTFS and actually use.

In this example, the server is not aware of any files stored on the iSCSI volume. As such you cannot share your entire pool; you can only share zvols or files. ZVOLs obey flush commands and as such are the preferred way to handle iSCSI images where data security/integrity is important. For performance bulk data which is less important, a file-based iSCSI disk is possible. This would just be a 8GB file or something that you export.

You can of course make zvol or file very big to share your data this way, but keep in mind only ONE computer can access this data at one time. So you wouldn't be running a NAS in this case, but only a SAN.
 
One nice thing about zfs is that you can make the zvol sparse (aka 'thin provisioned') so that you don't need to guess how much space to allot and then have the client run short later...
 
Thanks for the quick reply! My reasoning for wanting to do it this way is that WHS 2011 can use iSCSI volumes as storage, allowing me to have a simple and clean way to present my data to others on my network(none of my roommates are terribly tech-savvy and a simple way of presenting the share to them is preferred).

Is it possible to create a ZVOL that is larger than my current pool or have it expand in size along-side the pool itself? My primary goal is to be able to expand as I add additional vdevs to my pool(I'm a student, incremental upgrades net me the most bang for buck).
 
You can "thin provision" the ZVOL as big as you want it. Just watch that the ZVOL doesn't use more space than you actually have. As far as keeping up with ZVOL growth, expanding the array will help there.

"growing" fixed size ZVOL's isn't really doable. That is, when I've changed the ZVOL size in Solaris (which is easy to do) the iSCSI initiator in Windows wouldn't recognize the additional size, even detaching and re-attaching to the target and if I remember right even diskpart couldn't expand it.
 
One nice thing about zfs is that you can make the zvol sparse (aka 'thin provisioned') so that you don't need to guess how much space to allot and then have the client run short later...

You can "thin provision" the ZVOL as big as you want it. Just watch that the ZVOL doesn't use more space than you actually have. As far as keeping up with ZVOL growth, expanding the array will help there.

"growing" fixed size ZVOL's isn't really doable. That is, when I've changed the ZVOL size in Solaris (which is easy to do) the iSCSI initiator in Windows wouldn't recognize the additional size, even detaching and re-attaching to the target and if I remember right even diskpart couldn't expand it.

Looks like I missed some replies while replying to Sub.Mesa, so if I were to do this I can for instance create a very large ZVOL without actually having that amount of space available and then expand my pool as I go to keep pace with my demands? If I do this, what space does the initiator see? Will it only think I have the current pool space or will it believe it has whatever amount I stated was there?
 
Looks like I missed some replies while replying to Sub.Mesa, so if I were to do this I can for instance create a very large ZVOL without actually having that amount of space available and then expand my pool as I go to keep pace with my demands? If I do this, what space does the initiator see? Will it only think I have the current pool space or will it believe it has whatever amount I stated was there?


The initiator is going to see whatever you provision the ZVOL as. if you make it 10TB it'll see it as 10TB.
 
The initiator is going to see whatever you provision the ZVOL as. if you make it 10TB it'll see it as 10TB.

Alright, not a perfect solution but a lot better than trying to get my roommates to manage connecting to the ZFS server directly. Thanks!
 
If the zvol is created by ZFS and formatted with NTFS, do all of the benefits of ZFS go out of the window?
 
Just curious as I dumped a WHS system (2003) for a ZFS system recently - What functionality are you getting out of the WHS2011 system that you couldn't get directly from the ZFS server?
 
Just curious as I dumped a WHS system (2003) for a ZFS system recently - What functionality are you getting out of the WHS2011 system that you couldn't get directly from the ZFS server?

The biggest thing for me is my roommates, it's remarkably easy to just use the connector and bam, they have access. For the most part for me this is an effort to make the solution stupid-proof. For myself, I am not terribly comfortable with FreeBSD and have yet to find a reasonable FTP solution that meets my needs. I suppose I could probably use an FTP client to download directly onto a shared(via CIFS) drive, but I am unsure how well that would pan out.
 
One nice thing about zfs is that you can make the zvol sparse (aka 'thin provisioned') so that you don't need to guess how much space to allot and then have the client run short later...
Yes, but the downside of this approach is that even if you keep only 50% of the zvol filled, you would eventually have written to all blocks on the zvol; thus claiming the maximum space even though 50% is used. ZFS does not know of stale (unused) data that was deleted; such is the design of Storage Area Networks (SAN). To ZFS that zvol is just one big file of zeroes and ones.

One other downside of sparse files is that initially, writing even random writes would be fast, but after a while it would mean your disks have to seek more, causing lower performance. Performing a defragment on the client side would make the problem even worse, as the physical layout and virtual layout of ZVOL as not the same anymore. These problems are nonexistent when using normal reservation, where if the client starts writing at the middle, ZFS would also write at the middle of the zvol, keeping virtual and physical layout the same.

If the zvol is created by ZFS and formatted with NTFS, do all of the benefits of ZFS go out of the window?
SAN is what it is: the client controls the end-consumer (filesystem). You cannot use all the nifty functions of ZFS, and if NTFS metadata screws up then you're in a world of hurt since NTFS has little protection of its metadata; only light journaling. But ZFS should make sure the NTFS filesystem stays consistent, due to it obeying flush commands and processing in transactions.

So you get part of the goodness of ZFS. You also have a reliable 'RAID layer' while not all RAID layers are so problem-free as ZFS is, especially considering ZFS pretty much fixes all damage on the spot, and due to its redundant metadata things like large directories disappearing won't happen easily to ZFS.

Finally, you can enable compression and/or dedupe on ZVOLs exported as SCSI disk. So you would export a 20GB disk, but due to compression it consumes less space. This is not really recommended though on harddrives due to added seeks and fragments, but on SSDs could increase throughput since SSDs won't mind a few seeks here and there and are pretty much transfer capped so compression could increase speeds even, and allow more data to be cached in memory.

I don't like the idea of exporting all ZFS space by a single iSCSI ZVOL though. I like the idea of a NAS alot more: centralized storage where ZFS acts as primary end-user filesystem, where multiple computers can share data on. iSCSI, being a SAN protocol, is a completely different beast. You can do both at the same time though.

iSCSI can be used to circumvent protections on Windows that prevent the use of normal networked disks, such as in Photoshop and Outlook i believe, and probably more commercial packages as well. But iSCSI itself is not a NAS protocol; all it does is present block-level storage over the network. Block-level storage is the kind of storage a harddrive offers: it knows about sectors and LBA, but it does not know about files or any other meaning of the data.
 
Yes, but the downside of this approach is that even if you keep only 50% of the zvol filled, you would eventually have written to all blocks on the zvol; thus claiming the maximum space even though 50% is used. ZFS does not know of stale (unused) data that was deleted; such is the design of Storage Area Networks (SAN). To ZFS that zvol is just one big file of zeroes and ones.

iSCSI unmap that's been added to Illumos will fix this problem. :)

Though, using Nexenta here at home and work with iSCSI LUNS. Yes eventually the iSCSI LUN would balloon up to it's provisioned size, however the capacity will change upon write/deletions within the mounted LUN. So, I'm not so sure this is problematic from a Client standpoint. From a Host standpoint yes that much of the pool is 'reserved' (and a refreserve adds even more to this) for the ZVOL. Won't be a huge problem once iSCSI unmap gets rolled out though.


Regarding "other" ZFS stuff you lose. Yeah.. it kinda stinks. Snapshots still work, though you have to clone the snapshot to a new LUN and mount it to pull a particular file out (that you might want to roll back), or you can roll back the entire ZVol to a prior snapshot.
 
Last edited:
That's interesting, mindflux! If that would work then sparse provisioning would be more sexy, it would be like a TRIM command like SSDs use. No idea about the support for this in FreeBSD, probably istgt-target daemon has to support it? FreeBSD kernel infrastructure does allow for BIO_DELETE commands to travel to SSDs, though.

Likewise, patches that integrate TRIM support into ZFS has also surfaced, but i'll wait for the FreeBSD devs to import that one day. :)
 
Oh that's lame, I'm just going to do the setup myself for each of their machines and run ZFSGuru as my NAS. Thanks for all the advice guys!:)
 
Regarding "other" ZFS stuff you lose. Yeah.. it kinda stinks. Snapshots still work, though you have to clone the snapshot to a new LUN and mount it to pull a particular file out (that you might want to roll back), or you can roll back the entire ZVol to a prior snapshot.

You have no way to make sure the snapshot contains a consistent file system on creation. Except shutting down the iSCSI initiator.
 
You have no way to make sure the snapshot contains a consistent file system on creation. Except shutting down the iSCSI initiator.

If you clone a snapshot just make another target out of it and check the FS for consistency or pull the file you want. I'm not advocating blowing away the original target for a recovery. As far as shutting down the initiator, I'd either bring the other LUN up in the same initiator or use another machine to check for consistency, not detach from the original LUN.
 
I'm not understanding why your roommates can't just access the ZFS server over CIFS. It's built in and super easy?
 
I'm not understanding why your roommates can't just access the ZFS server over CIFS. It's built in and super easy?

They are dumb. Simple as that. Unless it's done for them they will screw it up, hence why I will have to do it for them :)
 
They are dumb. Simple as that. Unless it's done for them they will screw it up, hence why I will have to do it for them :)

Umm, mapping H: to \\servername\username is easy.

Setting up iSCSI targets, portal groups, target groups, views, mapping to initiators, and then setting up client initiators, formatting and assigning a drive letter isn't.

Secondly, you do realize that any data on the iSCSI volume isn't automatically shared with anyone else? And only one person can connect to an iSCSI volume at a time with NTFS?

Edit- I guess I'm a little lost. You are going to have 2 servers, one Windows, one ZFS. The ultimate goal is to share folders over CIFS. To do this you are going to have the storage on ZFS, shared over iSCSI, formatted as NTFS, and shared via Windows?
 
Last edited:
Umm, mapping H: to \\servername\username is easy.

Setting up iSCSI targets, portal groups, target groups, views, mapping to initiators, and then setting up client initiators, formatting and assigning a drive letter isn't.

Secondly, you do realize that any data on the iSCSI volume isn't automatically shared with anyone else? And only one person can connect to an iSCSI volume at a time with NTFS?

Edit- I guess I'm a little lost. You are going to have 2 servers, one Windows, one ZFS. The ultimate goal is to share folders over CIFS. To do this you are going to have the storage on ZFS, shared over iSCSI, formatted as NTFS, and shared via Windows?

That was the goal, because then they can just use the Launchpad and go "View shared folders" and it would take them into the share. I scratched it though after seeing the issues with doing so, I'll just need to do the setup for them. I know it's hard for anyone here to understand, but these people DO NOT UNDERSTAND computers. Nor do they want to. They want to click and see their files. Anything beyond that is entirely too much for them to understand or care about.
 
Uhh...I think you don't understand how CIFS works. Why connect a windows box to an ISCI on a ZFS box so you can share it out form the windows box when the ZFS server can just handle the shares itself?

They can just click on a share and see their files. That's what you wanted, it's in the server already. No need for 23 servers and 86 iSCSI targets with 146 windows shares on 34 different disks on 18 different internet connections. You're just making it hard when there is no reason to.
 
Uhh...I think you don't understand how CIFS works. Why connect a windows box to an ISCI on a ZFS box so you can share it out form the windows box when the ZFS server can just handle the shares itself?

They can just click on a share and see their files. That's what you wanted, it's in the server already. No need for 23 servers and 86 iSCSI targets with 146 windows shares on 34 different disks on 18 different internet connections. You're just making it hard when there is no reason to.

Thanks for calling me stupid. No, I understand full well how it works, and if you had bothered to read rather than just a (stupid) conclusion you would also see that I wanted WHS for backups as well. Have you EVER dealt with someone who doesn't want to learn and is not comfortable with a computer? You can show them how to do it 20 times and they will still be confused and unable to duplicate it.

Regardless, also if you had bothered to read you would see that after seeing the negatives I abandoned this idea, several days ago. Next time read the thread. Thanks. Bye.
 
I didn't mean to imply that sparse zvols were perfect. They are certainly better than any kind of non-sparse arrangement though :)
 
I didn't mean to imply that sparse zvols were perfect. They are certainly better than any kind of non-sparse arrangement though :)

Thanks for the information, I'm probably going to be using something like this to provide additional storage to ESXi and my WHS guest for use by backups. Many thanks!
 
Re backups - that was one of my reasons for hanging onto WHS for so long, but assuming they are on windows 7 the built in backup is really a viable alternative - I was surprised by how decent it is now.
 
Backups for my computers on the network. Not all of us are running Windows 7(some of them still insist on XP) so the built in WHS backup is clean and easy. This isn't about the backend storage, this is about backing up machines on the network.
 
How are WHS backups any better than ZFS snapshots?

The WHS backup client is pretty slick - so the advantage is in getting the data to the server, not in how it is stored when there ( and the ease of accessing the backups, bar metal restore, etc.). With 7 this is mitigated, with xp still an issue.

Still, one or two acronis licenses would probably be simpler, and still cheaper than whs2011 ( for the xp boxes)
 
Back
Top