A popular concept for ZFS lovers is the all-in-one. A server with ESXi, and a storage appliance (usually based on some flavor of solaris). It has an HBA passed in via pci passthrough, and serves up the storage to other guests by sharing it to ESXi using NFS. Works just fine. I decided to try the same thing with iSCSI. Not so fine. ESXi can write to the datastore just fine, but as soon as a guest tries to (for example, when you are installing a guest, and the installer writes to the 'hard drive'), the storage appliance iSCSI target software starts throwing errors about task aborts (apparently from the initiator.) I finally tried creating the VM on the other host of the cluster (this is a lab setup), and it works as expected. So this works: guest (A) => hypervisor (A) => network => iSCSI target (B), but this doesn't: guest (A) => Hypervisor (A) => iSCSI target (A). If I can't do this, that's fine, but I'd really like to know why it fails - I've done quite a bit of google-fu and come up empty.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
"All in one" using iSCSI - major fail?
- Thread starter danswartz
- Start date
Yeah more details would be good. Here's what i do in a lab environment and it works fine:
ESXi1 runs locally and has local datastores.
On ESXi1 a Windows 2012 VM (Win2012) is running on the ESXi1 local datastore.
Win2012 in turn provides iSCSI targets.
The Win2012 targets can be accessed without any issues by other VMs on ESXi1 and ESXi2, some of the included VMs are nested ESXi and other guest OSes on ESXi1 and ESXi2.
I think that you are trying to do something like that, and it's working fine for me using Win2012 as iSCSI target provider.
ESXi1 runs locally and has local datastores.
On ESXi1 a Windows 2012 VM (Win2012) is running on the ESXi1 local datastore.
Win2012 in turn provides iSCSI targets.
The Win2012 targets can be accessed without any issues by other VMs on ESXi1 and ESXi2, some of the included VMs are nested ESXi and other guest OSes on ESXi1 and ESXi2.
I think that you are trying to do something like that, and it's working fine for me using Win2012 as iSCSI target provider.
sorry for the paucity. it was late at night and i was wiped from fighting this all day 
anyway, both hosts are running 5.1. the iscsi targets were ubuntu 13 and centos6 (tried both). the target software was LIO (built in to both distros) as well as scst. i can't use ietd (my previous favorite) as it doesn't support scsi3 persistent reservations or some-such. the odd thing is it isn't an innate incompatibility, since it works if the vsa is on the other host, but that kinda obviates the whole point (as well as introducing a guaranteed single point of failure...) I found one post on google that looked similar where the 'fix' was to reduce some iscsi queue depth from 256 to 128, but that was never really followed up on by the OP or the responders. It's almost like being on the same host, vsphere can push too much data too fast at the target? Dunno... I'd love to keep poking at this, if I have some clue where to start. One idea I had was to run the vsa on both hosts, and have them do block level replication with something like drbd using a dedicated 1gb link. A trick I saw similar to this was: give each host a vmkernel port/vm port group with the same IP address, but no external network connection. This way, each host will think the vsa is shared storage, so if you migrate a guest to the other host, it will start talking to the other vsa and it should all 'just work'. I don't know any way to get something like that working with windows (block level replication of some device), as well as wanting that storage to live on my existing zfs data pool.
anyway, both hosts are running 5.1. the iscsi targets were ubuntu 13 and centos6 (tried both). the target software was LIO (built in to both distros) as well as scst. i can't use ietd (my previous favorite) as it doesn't support scsi3 persistent reservations or some-such. the odd thing is it isn't an innate incompatibility, since it works if the vsa is on the other host, but that kinda obviates the whole point (as well as introducing a guaranteed single point of failure...) I found one post on google that looked similar where the 'fix' was to reduce some iscsi queue depth from 256 to 128, but that was never really followed up on by the OP or the responders. It's almost like being on the same host, vsphere can push too much data too fast at the target? Dunno... I'd love to keep poking at this, if I have some clue where to start. One idea I had was to run the vsa on both hosts, and have them do block level replication with something like drbd using a dedicated 1gb link. A trick I saw similar to this was: give each host a vmkernel port/vm port group with the same IP address, but no external network connection. This way, each host will think the vsa is shared storage, so if you migrate a guest to the other host, it will start talking to the other vsa and it should all 'just work'. I don't know any way to get something like that working with windows (block level replication of some device), as well as wanting that storage to live on my existing zfs data pool.From my previous post, given that doing it all on one host, maybe I am swamping something, but that doesn't happen when I am limited by the 1gb link? I doubt it's malformed commands, since it does work if the vsa is on the other host...
I should have mentioned that I have a 3rd host running omnios with the existing data pool, served to both vsphere hosts via iscsi and it all just works. One reason I was interested in the all in one, is it would let me repurpose the 3rd host to allow me to experiment with 3 host clusters, which I can't currently do. If I could get this to work, I could just move the existing data pool into a vsa and little disruption.
Blech. Just use NFS, problem solved, or a storage appliance software package. Spinning your own is pretty pointless these days.
You don't want to try to build your own replication (it can be done), as you'll run headfirst into quorum joy and you really don't want to design your own clustering software.
You don't want to try to build your own replication (it can be done), as you'll run headfirst into quorum joy and you really don't want to design your own clustering software.
sorry for the paucity. it was late at night and i was wiped from fighting this all day
IETD is dead and has been dead for ages. It was time to abandon it 4 years ago.
This has nothing to do with queue depth, almost certainly.
Grab the vmkernel logs and stash them someplace, but you really should build a nexenta/omnios/something AIO if yuo want to do that, and not try to hand-spin LIO or SCST.
All good points. Being anal, I'd love to know *why* it wasn't working when the guest and the vsa are on the same host, though My san is already omnios-based, so virtualizing it and providing the same storage pool is pretty easy. I did play a bit with gluster on two different VSA, and it worked okay. Since it shares out the data via NFS, that would be handy.
My san is already omnios-based, so virtualizing it and providing the same storage pool is pretty easy. I did play a bit with gluster on two different VSA, and it worked okay. Since it shares out the data via NFS, that would be handy.All good points. Being anal, I'd love to know *why* it wasn't working when the guest and the vsa are on the same host, though
Probably has to do with how you have LIO handling reservations, taking a wild stab in the dark.
Gluster will hit latency problems a lot of times for VM workloads unless you have it set to not always be in sync... in which case you have data inconsistency problems
Clustering is the hardest part of storage to solve for VM workloads, since things "move".
Hmm, interesting. I appreciate the insights. I have no idea how lio or scst were doing whatever they were doing, but whatever it was, was the default. I did out of curiosity, check esxtop and I noticed the comstar lun had a QD of 128 but the LIO one was only 32. That said, I think I've p*ssed away enough time and energy on this. I've heard mixed reviews of gluster. I do know about the trusted-sync option, but as you said, that is kind of useless for my proposed use case. What I would have wanted was a setup where I migrate a VM to the other host (or maybe it fails over to it via HA), and picks up seamlessly. If the guest was doing disk writes when the failover occurs, I can't see how it would be seamless, since you would lose some writes, no? Failover via HA doesn't matter so much since the guest is being forcibly restarted anyway, I assume...
Aren't you exposed to that anyway? If a guest is writing data out, and the host crashes before the writes all make to local/shared storage, isn't that a risk anyhow? I guess depending on the filesystem and write barriers and such for metadata, not so much? However unlikely, though, even for a lab environment, not something I really want to risk
Aren't you exposed to that anyway? If a guest is writing data out, and the host crashes before the writes all make to local/shared storage, isn't that a risk anyhow? I guess depending on the filesystem and write barriers and such for metadata, not so much? However unlikely, though, even for a lab environment, not something I really want to risk
Exceedingly small - we're talking microseconds to milliseconds here for that, vs possibly seconds on replication at times (or longer).