• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Fiber Channel SAN

B

BSDMan

Weaksauce
Joined
May 23, 2012
Messages
105
Hi All

I'm about to setup a lab environment at home for my VCP6 studies. It'll consiist of two ESXi6 servers and a FreeNAS server for storage. Most of the storage in the FreeNAS box will be SSD based (to store the VMs) and there will proabably be no disks in the ESXi6 servers (except for a bootable USB drive).

My question is about storage. I'd like the storage performance to be good and after much reading it looks like a Fiber Channel SAN setup is the way to go. Having a look on eBay suggests that the prices are quite reasonable for such great performance.

I was thinking of getting the following:

  • For the FreeNAS server: QLogic QLE2564 8Gb/s - this has 4 ports
  • For each ESXI 6 server: QLpgic QLE2562 8Gb/s - this has 2 ports
(I know 8Gb fiber is more than 4Gb fiber but I am ok with paying the extra considering most of my storage will be SSD based)

I was then going to connect both ports from each ESXi 6 server to the 4 ports on the FreeNAS server so I:

  • Don't need a fiber switch
  • Can make use of multi pathing

My questions are:

  1. Will the hardware be compatible with ESXi 6 and FreeNAS? (research suggests yes)
  2. Will connecting the ESXi hosts directly to the FreeNAS server without a fiber switch wotk?
  3. Could I boot off the SAN? Then I could ditch the bootable USB drives for ESXi
  4. What fiber cables do I need for these cards? There seem to be so many to choose from

Further info: I'll have 8 SSD drives in the FreeNAS server and will be running at least 24 VMs.

I did look into iSCSI and MPIO but I decided against it due to cost, speed, and needing too much cabling/ports on the switch. I think FC gives better performance and (can be) cheaper. Also, I am interested in learning FC.

Thanks for any help!
 
The FreeNAS FC method looks extremely hackjob and not a representation of real world FC.

I would recommend checking ebay for intel X520 10GE based cards and snagging some passive twinax cables, much less frustrating in the long run.
 
geiger said:
The FreeNAS FC method looks extremely hackjob and not a representation of real world FC.

I would recommend checking ebay for intel X520 10GE based cards and snagging some passive twinax cables, much less frustrating in the long run.
Click to expand...

This. Don't bother with FC unless you like headaches. Chelsio cards are also extremely easy to setup with Freenas.
 
If you zone everything correctly, FC can be very reliable and perform very good. It's a little expensive for a home lab though. iSCSI is easier to setup and cheaper. But for a home lab I would just recommend using servers with local or NAS storage.
 
The FreeNAS FC method looks extremely hackjob and not a representation of real world FC.

I would recommend checking ebay for intel X520 10GE based cards and snagging some passive twinax cables, much less frustrating in the long run.
Click to expand...

How would I use the Intel X520 cards without a 10GB switch? In the long run I am looking to have 3 ESXi nodes and shared storage.

This. Don't bother with FC unless you like headaches. Chelsio cards are also extremely easy to setup with Freenas.
Click to expand...

The reason I actually got interested in FC was due to the (from what I can tell) relative ease of setting it up and cost:

Fibre Channel SAN on the Cheap(ish) - Just Shinfo
https://www.justsh.info/2016/01/08/fibre-channel-san-on-the-cheap/
Switching to Fiber Channel for SAN access. Part 1 | EpiJunkie

What headaches are you referring to?

If you zone everything correctly, FC can be very reliable and perform very good. It's a little expensive for a home lab though. iSCSI is easier to setup and cheaper. But for a home lab I would just recommend using servers with local or NAS storage.
Click to expand...

I didn't think the cost of FC expemsive at all. 4gb FC is super cheap. 8gb is more but still quite affordable considering what you are getting.

I've never worked with FC and think it would be a good skill to learn since I am studying VMware currently.

From all the videos and books I have read so far on vSphere they pretty much all say FC is better performing than iSCSI and has lower latency.

I'm open to ideas but basically I want fast performing shared storage which uses SSD drives. It has to be shared storage as I want to use features in vSphere such as vMotion, HA, DRS and Fault Tolerance (all of which require shared storage). I'm also aiming to start with ESXi 2 nodes and then grow that to 3 nodes. Getting the shared storage correct from the word go is crucial in making this a success!

Thanks for the comments!
 
The headaches are with FC and Freenas together. You can almost certainly make it work, it will just likely be more of a pain than an Intel 10GbE or Chelsio 10Gb DAC setup.
 
boss6021 said:
The headaches are with FC and Freenas together. You can almost certainly make it work, it will just likely be more of a pain than an Intel 10GbE or Chelsio 10Gb DAC setup.
Click to expand...

Ok, lets tackle this from another angle ;-)

What software can I use for my SAN at home that will work with FC in my lab? I have a Supermicro X10SL7-F motherboard with 32GB of ECC RAM and a Xeon 1230v3 CPU. I'll have 8 SSD drives and one SATA HDD.

I'd really like to give FC a try as I have read that the performance is really good compared to iSCSI and it also saves me buying an expensive 10gb switch. And 10gb switches are not that practical for home use (too noisy for me).

So i was thinking of putting a dual port FC HBA in each of the two ESXi hosts and then installing a 4 port FC HBA in the SAN server (my Supermicro server) and then using multipathing between the hosts and the SAN.

If theres a better solution to setup shared storage for a vSphere cluster then I'm open to any ideas! (I have considered VSAN but its just too costly for me and you need a minimum of 3 nodes).

Thanks!
 
You may check Comstar, a framework for FC/iSCSI targets on Oracle Solaris
(free for noncommercial demo and development) or its free fork OmniOS
Configuring Storage Devices With COMSTAR - Oracle Solaris Administration: Devices and File Systems

You can also use bridging.
Your NAS can then work similar to a 10G switch for 10G Ethernet.
(This is what I would prefer together with NFS for easyness)
Bridging Overview - Oracle Solaris Administration: Network Interfaces and Network Virtualization

I use this in my setups with napp-it
 
Thanks for the links!

I've been giving this some more thought and maybe I should be considering iSCSI (again!) over 10gb NICs using MPIO. The two Supermicro servers I am considering come with Dual 10G SFP+ so would the following work and be a good setup:

1) Use Windows Server 2012 R2 as an iSCSI target with MPIO enabled AND have two dual port 10gb NICs in the storage server (I was thinking of using the Intel X520-DA2 Dual Port cards with twinax DAC cables)

2) Connect two twinax DAC cables from each Supermicro server to two ports on the storage server

I'm not too familiar with 10gb and SFPs so may have some of this wrong. Can I connect two ESXi nodes via 10gb connections to two different 10gb NICs in the same storage server? And still use MPIO?
 
We've been using 10g iSCSI in production for the last 4 years with no issues.

The twinax cable is essentially a copper cable with built-in SFP+ modules.
 
My ESX storage box is a VM with a raid controller passed through... sharing iSCSI back to ESX over the internal 10Gig-E loopback...

Sounds kludgy, but it's damn fast =)
 
Last edited:
You must log in or register to reply here.
Back
Top