Home built iSCSI for VMware

A

AMD_Gamer

Fully [H]
Joined
Jan 20, 2002
Messages
18,287
What have people been using software wise as their own home built iSCSI targets for ESXi? A lot of people in the Virtualization forum i notice just get a Synology or something like that but A) more fun building your own B) cheaper.
 
AMD_Gamer said:
What have people been using software wise as their own home built iSCSI targets for ESXi? A lot of people in the Virtualization forum i notice just get a Synology or something like that but A) more fun building your own B) cheaper.
Click to expand...

If you are looking for a free High-End SAN solution, look at ZFS (ex OpenIndiana)
But I would prefer NFS over iSCSI (You can access Snapshots and Move/ Copy/ Backup
from Windows via NFS or SMB) while its similar in speed.

Gea
 
I setup an OpenIndiana/Napp-it box and haven't looked back. Even though it has iSCSI as an option, I just use NFS and SMB as Gea suggested. On top of that..my raidz pools are performaning exceptionally well especially with the large amount of ram and SSD cash I have.

Believe me when I tell you, i've used a lot of storage target software such as Starwind iSCSI, Windows iSCSI targets, HPP4000 VSA, the EMC Uber VSA..etc, and nothing has come close in performance, from a build your own standpoint. On top of that, it's simple to setup, just make sure that you have supported hardware.
 
_Gea said:
If you are looking for a free High-End SAN solution, look at ZFS (ex OpenIndiana)
But I would prefer NFS over iSCSI (You can access Snapshots and Move/ Copy/ Backup
from Windows via NFS or SMB) while its similar in speed.
Gea
Click to expand...

Is it safe to use snapshots/copy/etc even though Windows VSS is unaware of it? I thought that wasn't recommended.
 
thefreeaccount said:
Is it safe to use snapshots/copy/etc even though Windows VSS is unaware of it? I thought that wasn't recommended.
Click to expand...

Pure ZFS snapshots do not care about ESXi and guest-OS
-> good if you just need the file state, you can hold thousands of such snaps

If you need hot snaps for ESXi (with memory state)
do a ESXi snap, do a ZFS snap and delete ESXi snap (ESXi snaps are very limited)
now you can go back to this ZFS snap and restore a running ESXi VM

if you need hot snaps within a special VM where a Hot-Snap if ESXi is not enough,
you have to pause/ shutdown vor VM or you need VM internal snaps.

A ZFS only snapshot is like the state after a power failure.
Most systems can handle such a failure in most cases especially
because the filesystem itself is always consistent.

Gea
 
This means you've failed to configure your SAN properly :)

OK, MS iSCSI target is definitely on a slow side, HP & EMC were virtualized so everything is clear and you should have wire speed with StarWind or any other GOOD iSCSI SAN. ~950MB/sec and 5 ms latency is what you should expect with 10 GbE. No NFS would stand even close to this. So... Show us your benchmark results or it never happened :)

Olga

Vader said:
I setup an OpenIndiana/Napp-it box and haven't looked back. Even though it has iSCSI as an option, I just use NFS and SMB as Gea suggested. On top of that..my raidz pools are performaning exceptionally well especially with the large amount of ram and SSD cash I have.

Believe me when I tell you, i've used a lot of storage target software such as Starwind iSCSI, Windows iSCSI targets, HPP4000 VSA, the EMC Uber VSA..etc, and nothing has come close in performance, from a build your own standpoint. On top of that, it's simple to setup, just make sure that you have supported hardware.
Click to expand...
 
It's not safe it's just begging for troubles. And they should come pretty soon :) Shot data would be inconsistent as snapshot taking action could and would happen between two flushes of lazy writer.

Olga

thefreeaccount said:
Is it safe to use snapshots/copy/etc even though Windows VSS is unaware of it? I thought that wasn't recommended.
Click to expand...
 
Shutting down VM to take a snapshot does not sound like a good solution for any production environment I know. In a nutshell: even solid ZFS is basically useless without proper VSS agent running inside hypervisor.

Olga

PS Don't be goofed with Microsoft target and their hardware VSS. You cannot use it to backup CSV so it's again close to being DOA - all backup / snapshot jobs would eat all CPU cycles and proceed serialized.

_Gea said:
Pure ZFS snapshots do not care about ESXi and guest-OS
-> good if you just need the file state, you can hold thousands of such snaps

If you need hot snaps for ESXi (with memory state)
do a ESXi snap, do a ZFS snap and delete ESXi snap (ESXi snaps are very limited)
now you can go back to this ZFS snap and restore a running ESXi VM

if you need hot snaps within a special VM where a Hot-Snap if ESXi is not enough,
you have to pause/ shutdown vor VM or you need VM internal snaps.

A ZFS only snapshot is like the state after a power failure.
Most systems can handle such a failure in most cases especially
because the filesystem itself is always consistent.

Gea
Click to expand...
 
Olga-SAN said:
This means you've failed to configure your SAN properly :)

OK, MS iSCSI target is definitely on a slow side, HP & EMC were virtualized so everything is clear and you should have wire speed with StarWind or any other GOOD iSCSI SAN. ~950MB/sec and 5 ms latency is what you should expect with 10 GbE. No NFS would stand even close to this. So... Show us your benchmark results or it never happened :)

Olga
Click to expand...

Here's a benchmark on 1 Gbit and 10 Gbit Ethernet (FCoE[10Gbit only], iSCSI, NFS) and 4/8 Gbit FC. The conclusion? The differences weren't that dramatic.

http://media.netapp.com/documents/tr-3916.pdf
 
Thank you very much for that link tonyb - that was a very interesting article!
 
This means you've failed to configure your SAN properly :)

OK, MS iSCSI target is definitely on a slow side, HP & EMC were virtualized so everything is clear and you should have wire speed with StarWind or any other GOOD iSCSI SAN. ~950MB/sec and 5 ms latency is what you should expect with 10 GbE. No NFS would stand even close to this. So... Show us your benchmark results or it never happened :)

Olga
Click to expand...

It doesn't take a genius to setup Starwind iSCSI, and let's look at reality, barely anyone is using 10Gbe for home use. That's what I was talking about. I didn't take any benchmarks and i'm not about to set up a new environment, but I can tell you this, using NFS w/OpenIndiana and my ZFS pools w/SSD cache for my VMware lab, there is a visible difference. Now, this could be the fact that I was using SATA only with Starwind and there wasn't a similiar setup for Cache like I have with my ZFS pools.

And I agree..thank you tonyb for the article.
 
OMG

Did anybody actually bother to read this one? Or at least take a closer look at the pictures they had published?

They used 4KB and 8KB block sizes with random I/O pattern. Good for NetApp's implementation of NFS cached entirely at hardware level (no actual data is written but transaction log is held for a while before writing WAFL consistency point), bad for 10 GbE iSCSI (hard disk seeks are comparable with the network latency they're trying to measure) and absolutely killer for FC (obvious, payload size is pretty close to the frame size).

Good luck..........................

tonyb said:
Here's a benchmark on 1 Gbit and 10 Gbit Ethernet (FCoE[10Gbit only], iSCSI, NFS) and 4/8 Gbit FC. The conclusion? The differences weren't that dramatic.

http://media.netapp.com/documents/tr-3916.pdf
Click to expand...
 
I don't think so. Running StarWind setup is trivial indeed but it takes days to have everything up and running giving away appropriate performance numbers. We gave up and ended with hardware solution (no names here, sorry for this) for production and OpenIndiana powered test bed configs. I would not consider this a being entirely software issue but 50/50 split is pretty close to being truth.

10 GbE adoption is growing on exponential basis. So it's not for homes yet. Right. In year of 2011. But in 2013 half of the SOHO COTS hardware is expected to be equipped with on-board 10 GbE ports.

Wow! SSD cache on ZFS works! Unbelievable! Who could predict this? Did you at least configure StarWind to use dedupe? Or you've been running flat disk access with NTFS on StarWind with dedupe heavily minimized I/O on ZFS and OpenIndiana? I can bet I know your answer.

Good luck!

Vader said:
It doesn't take a genius to setup Starwind iSCSI, and let's look at reality, barely anyone is using 10Gbe for home use. That's what I was talking about. I didn't take any benchmarks and i'm not about to set up a new environment, but I can tell you this, using NFS w/OpenIndiana and my ZFS pools w/SSD cache for my VMware lab, there is a visible difference. Now, this could be the fact that I was using SATA only with Starwind and there wasn't a similiar setup for Cache like I have with my ZFS pools.

And I agree..thank you tonyb for the article.
Click to expand...
 
Olga-SAN said:
OMG

Did anybody actually bother to read this one? Or at least take a closer look at the pictures they had published?

They used 4KB and 8KB block sizes with random I/O pattern. Good for NetApp's implementation of NFS cached entirely at hardware level (no actual data is written but transaction log is held for a while before writing WAFL consistency point), bad for 10 GbE iSCSI (hard disk seeks are comparable with the network latency they're trying to measure) and absolutely killer for FC (obvious, payload size is pretty close to the frame size).

Good luck..........................
Click to expand...

The data sets were done with large enough data sets to ensure it couldn't all be cached. Each VM had a 10 GByte virtual disk for the I/O test, with up to 128 VMs run concurrently,

4KB is larger than a fibre channel frame (FC payload is 2048 bytes max, so 2, probably 3 FC frames were required). 8K is smaller than a single iSCSI TCP segment if using jumbo frames around 9K (which they did).

I think the conclusion of the article is that the transport doesn't matter as much as it used to, despite the perception. It's all about the IOPS.
 
If you want to play with iSCSI for a homelab then just download MS iSCSI....its free.

That said I will throw my hat in NFS >iSCSI due to flexibility it provides and expandability via Trunking opposed to upgrading to 10gbe
 
One thing that really bothers me with Netapp's whitepapers and tests and things they don't tell you is that Netapp's implementation of iSCSI really isn't native, and that it's implemented ontop of their propietary file system:

http://thesantechnologist.com/?p=52

I would like to see Netapp's NFS comparied to other vendors ISCSI numbers and not Netapp's 'version' of it. I'm not debating the ease of use that NFS presents, just the skewed performance numbers.
 
MassiveOverkill said:
One thing that really bothers me with Netapp's whitepapers and tests and things they don't tell you is that Netapp's implementation of iSCSI really isn't native, and that it's implemented ontop of their propietary file system:

http://thesantechnologist.com/?p=52

I would like to see Netapp's NFS comparied to other vendors ISCSI numbers and not Netapp's 'version' of it. I'm not debating the ease of use that NFS presents, just the skewed performance numbers.
Click to expand...

That's common on many NAS boxes doing iSCSI. Even EMC has that as an option on some arrays. If you do iSCSI via the datamovers it's a blob on a filesystem. Do it through the back-end array and it's native block iSCSI. Just depends.

Bottom line. For VMware there is very little reason to do iSCSI these days. I used to do a good bit of it when Site Recovery Manager did not support NFS...but now it does and most vendors have far better integration of vSphere and vCenter with NFS than iSCSI. You can do things (speaking in EMC terms here as that's what I do) like right-click a VM in vCenter and say "compress that one...but not these other 3 on the same datastore". Can't do that right now with a block level protocol. Also, you can offload snaps to the backend array instead of the vSphere host. Some things you can do with block level if your array does VAAI....but some you still can't.

That's why I haven't done an iSCSI VMware design in a year and a half...or more. The only reason some people still push for it is that you can multipath iSCSI...can't do that with NFS, yet. But in almost all cases the limiting factor is IOPS, not throughput and a single Gb NFS link does that fine. If you need more we can split datastores across NFS exports and balance that way...or just suck it up and go 10Gb.

For my 3-host vSphere lab I use a Synology DS1010 with NFS.
 
I would also add then in an NFS setup, you can expand your storage by replacing disk with higher capacity or add a new shelf of disks.
Open up VCenter and click refresh and your new storage is instantly realized. Pretty convenient.

@NetJunkie
I realize you cant do multipathing but could you not just use LACP/Trunking & VLANs to obtain the same results w/ NFS?
 
Netjunkie, +1. I was using iSCSI on my all in one originally, but now the datastore is NFS only, and I've never regretted it. The only thing I am using iSCSI for is so my kids win7 laptops can back up to the "local disk" the win7 iSCSI initiator presents to them (with win7 home, the backup app will not work with network shares...)
 
nitrobass24 said:
@NetJunkie
I realize you cant do multipathing but could you not just use LACP/Trunking & VLANs to obtain the same results w/ NFS?
Click to expand...

No...because it's a single connection from a vSphere host to the NFS export. One IP/MAC to one IP/MAC so you're only going to hash across one physical connection if you channel.
 
So let me pose two questions for you:

You're going to run a SQL server VM with the data files on your SAN on a RAID 10 datastore (mdf within the VM, not RDM). Let's say without stating specific IOP requirements that its generally intensive and you want to maximize performance. Would you put it on the SAN carving via iSCSI or via NFS?

2nd question.....same SAN, but now you have a LUN allocated as storage for large PACS images (which you can consider as video files). These 'images' need to have good sequential throughput for transfer from the X-ray to the server as well as from the server to Reviewing station. Let's assume that the reviewing station has either 10k raptors or 15k SAS drives for their local data storage, and let's say we have multiple reviewing stations requesting various 'images'. Would you still do NFS or go with multipathed iSCSI, keeping in mind that existing gigabit infrastructure has to be used?

Thanks.
 
1. NFS or iSCSI doesn't matter. In most cases SQL is not throughput intensive..it's IOP intensive with low throughput. I'd do NFS due to other reasons.

2. How many PACS servers do you have? If you have a single VM that will pull those images then it doesn't matter what you're doing as you'll only get one connection's worth of bandwidth at a time. If it's multiple then you could see benefit from iSCSI multipathing..but honestly, when I see PACS (and I see it a lot...work a lot with hospitals) I push toward Fibre Channel or 10Gb. It's a more guaranteed solution than multipathing iSCSI.

EDIT: Also understand that all these "recommendations" are based on no workload information. I'd have pulled IOPS and throughput info on each server before nailing down a config.
 
Once again, I realize all the benefits of NFS datastores, and we'll probably use both iSCSI and NFS on the same SAN depending on the LUN needs for the VMs. Thanks for the input
 
NetJunkie said:
The only reason some people still push for it is that you can multipath iSCSI...can't do that with NFS, yet.
Click to expand...

I am curious about the yet qualifier. How could nfs be hashed across more than one physical connection when channeling? pnfs?
 
You must log in or register to reply here.
Back
Top