Question about FC duplex performance

[sphinx99]

Simple question - is fiber channel a full-duplex protocol? How to verify?

Scenario - I've setup a storage system (NetApp 3200-series with gobs of RAM, flash), Connection is simple, single path 4Gb fiber from the storage system to a host running Server 2008 R2. I use sqlio to generate a big read load, and I get my expected ~ 370MB/sec throughput with big enough block sizes.

Then, I do the same test but with big writes, and again get the same speed. Both times, I am verifying on the storage system that the actual reads/writes are happening. (Albeit, largely from cache in the case of the reads.)

Then I kick off both loads at once from the same host, and end up with ~ 230MB/sec reads, ~ 230MB/sec writes simultaneously across the single 4Gb path. This tells me that (a) I'm clearly not getting full duplex and (b) I do seem to be getting a smidge more than I would expect the interface to be capable of if operating half duplex, so maybe I am full duplex and just hitting a HBA chipset or other limitation.

Thoughts?

Note, I have duplicated the above both direct-attached (SAN to host) and with a FC switch in the middle.

Edit: I reduced the number of threads I was generating and am now seeing a solid 300MB/sec reads and 300MB/sec writes, each against its own LUN, simultaneously, across a single 4 gigabit fiber port, so I guess they really are full duplex.

 

[Blue Fox]

Yes, FC is full duplex (it's why each SFP has a TX and RX port).

 

[sphinx99]

I trust it's a fair assumption that the processor in typical FC HBAs may not be capable of saturating the pipe in both directions? Or, might I be running into some kind of cross-talk issue? (Shouldn't be, right - two strands for two different paths?)

I am setting up a Linux host to execute a similar test, to see what the behavior is like there. The HBA I'm using is on a PCI express 4x slot which I think should have enough bandwidth.

 

[Blue Fox]

Not completely sure what the limiting factor is, but it could possibly be the HBAs (and PCIe x4 does have enough bandwidth). 300mb/s is fairly good though. Cross talk is impossible with fibre optics (unless you're running copper for all your FC stuff).

 

[sphinx99]

Nope, it's optical, so must be OS or HBAs on either end. This is kick-the-tire data collection for scoping a larger project, so I'm just trying to document bottlenecks as I come upon them. I agree that 300MB/sec each direction is solid.