• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Network speed for storage

What is your storage head network speed

  • Total voters
    49
T

titusc

n00b
Joined
Feb 26, 2014
Messages
22
Curious what network speed people are using for connecting to the storage heads, and whether 1Gbps is too slow considering a sustained read / write speed of 100MB/s (800Mbps) is easily achievable and 200MB/s (1.6Gbps) is available for some of the enterprise SAS.

If you can get the Intel® TrueScale Fabric Host Adapter QLE7340 - 40Gbps InfiniBand (1x QSFP) for $327, which is cheaper than some of the 10Gbps Ethernet NIC, is this worthwhile of getting it? What will be the decision factor for using this type of speed?
 
Have 4x1Gb in to my lab Synology DS1813+. But it really depends on the workload. I rarely need even a full 1Gb/s using spinning disks because it's all about the IOPS. With SSDs that changes a bit..bit it's usually just bursts and often is below 1Gb/s.
 
dual-quad 1GB from hosts to quad 1GB in SAN's. We've hit our limits on bandwidth and are upgrading to quad 10GB in everything.
 
HydroBudz said:
dual-quad 1GB from hosts to quad 1GB in SAN's. We've hit our limits on bandwidth and are upgrading to quad 10GB in everything.
Click to expand...
Sorry you are using Dual 1Gbps or Quad 1Gbps? What type of setup do you have and how is it used?
 
HydroBudz said:
dual 4port gig NIC's in our DL580 G7 hosts.
Click to expand...
Wow that should be fast enough to handle a SSD or the fastest enterprise SAS in RAID 0. What type of requests is your storage used for? Is it for office or web?

Have you considered using the Intel QDR Infiniband cards? They appear to be cheaper than some Dual 10Gbps cards.
 
titusc said:
Wow that should be fast enough to handle a SSD or the fastest enterprise SAS in RAID 0. What type of requests is your storage used for? Is it for office or web?

Have you considered using the Intel QDR Infiniband cards? They appear to be cheaper than some Dual 10Gbps cards.
Click to expand...

it doesn't work like that directly, LACP will only let you get 1Gig max per transfer or equal. so they in theory have 8Gig of total bandwidth which would let 8 x 1Gig connections max it out.

not do a single 8Gig transfer
 
titusc said:
Wow that should be fast enough to handle a SSD or the fastest enterprise SAS in RAID 0. What type of requests is your storage used for? Is it for office or web?

Have you considered using the Intel QDR Infiniband cards? They appear to be cheaper than some Dual 10Gbps cards.
Click to expand...

We host over 150 very active medical EHR databases on 4x HP 4500 SAN's with over 1000 connections. All 15k spinners. It's not so much bandwidth but packet saturation. Would make sense to go SSD, but cost prohibitive at this time.

MrGuvernment said:
it doesn't work like that directly, LACP will only let you get 1Gig max per transfer or equal. so they in theory have 8Gig of total bandwidth which would let 8 x 1Gig connections max it out.

not do a single 8Gig transfer
Click to expand...

This.
 
spazoid said:
No. There's the option of 4x 1Gbps connections, which is based on an entirely different protocol and has completely different performance characteristics.
Click to expand...
Okay sorry thought 4Gbps FC is more or less similar to Quad 1Gbps Eithernet. In this case which one is faster?

We host over 150 very active medical EHR databases on 4x HP 4500 SAN's with over 1000 connections. All 15k spinners. It's not so much bandwidth but packet saturation. Would make sense to go SSD, but cost prohibitive at this time.
Click to expand...
Okay so I guess it's IOPS between the databases and the iSCSI targets.

it doesn't work like that directly, LACP will only let you get 1Gig max per transfer or equal. so they in theory have 8Gig of total bandwidth which would let 8 x 1Gig connections max it out.

not do a single 8Gig transfer
Click to expand...
Sorry why can't we team up all 8 x 1Gbps for an effective 8Gbps connection via LACP?
 
titusc said:
Okay sorry thought 4Gbps FC is more or less similar to Quad 1Gbps Eithernet. In this case which one is faster?


Okay so I guess it's IOPS between the databases and the iSCSI targets.


Sorry why can't we team up all 8 x 1Gbps for an effective 8Gbps connection via LACP?
Click to expand...

IOPS and bandwidth. We've pretty much hit the proverbial wall with LACP on 8x 1GB connections. Time to step it up to 4x 10GB connections. Add another 4x 4530 SAN's and move from data around for more IOPS.
 
mtrupi said:
To be nit picky they spell it Fibre.
Click to expand...

I've seen it both ways, and if you search ebay you find a lot more under fiber channel than fibre channel :D
 
My nexenta filers are 4 x 10gbps. One LACP aggregate of 2 ports and 2 x 10gbps links not bonded for iscsi traffic.

Compute nodes are 2 x 10gbps LACP. I rarely see more than 10% saturation of the compute data links.

Also, in terms of pure IOPs its actually fairly difficult to saturate even a 1gbps link as you'll run into the latency wall before you max out the link. Even over 10gbps I have trouble sustaining over 50MB/s on an 8K block size 100% random 100% write test pattern. I can sustain a few of those streams at once but individually they'll top out around 55MB/s and right around 6000 IOPs and about a .85ms latency. This is after fully tweaking esx, nexenta, and disabling access time within the running guest OSes.

Trying to get some loaner hardware from Mellanox so that I can begin testing NFS over RDMA. Not for the bandwidth but for the significant decrease in latency. I don't think 10gig is capable of meeting the needs of the top 3% or so of workloads within a public cloud type environment. Those types of workloads are exceptionally lucrative if you can deliver them and right now I haven't seen or heard of a public IaaS cloud that can honestly deliver on the high end.
 
I used 2x1Gbps teamed when I ran ESXi at home (until last year). At work we use 4x1Gbps but for heavy loads no amount of 1Gbps connections will be acceptable. Fortunately OP doesn't seem to have extreme requirements. Teaming 1Gbps connections is only going to read at 200MB/s if you use a protocol with native multipathing like SMB 3.0 (and I think NFS 4 also) or if you can split the data up into multiple streams (which could then be load-balanced).
 
titusc said:
If you can get the Intel® TrueScale Fabric Host Adapter QLE7340 - 40Gbps InfiniBand (1x QSFP) for $327, which is cheaper than some of the 10Gbps Ethernet NIC, is this worthwhile of getting it? What will be the decision factor for using this type of speed?
Click to expand...
I'm using a single gigabit link for my storage, because it suits my needs just fine. That being said, I was considering picking up some of the 40Gbps Inifiniband adapters. They were, last I checked, hovering around $100 per adapter on eBay. Two of them and a $50 cable would increase my speeds dramatically. The issue is, the InfiniBand switches are still quite expensive. A "good deal" on a used one is about $1250. And they're noisy as hell, bad for home use. Oh, and if you want to mix Cat6 and InfiniBand, open up your pocketbook, because a switch that can do both is VERY pricey.
 
madrebel said:
My nexenta filers are 4 x 10gbps. One LACP aggregate of 2 ports and 2 x 10gbps links not bonded for iscsi traffic.

Compute nodes are 2 x 10gbps LACP. I rarely see more than 10% saturation of the compute data links.

Also, in terms of pure IOPs its actually fairly difficult to saturate even a 1gbps link as you'll run into the latency wall before you max out the link. Even over 10gbps I have trouble sustaining over 50MB/s on an 8K block size 100% random 100% write test pattern. I can sustain a few of those streams at once but individually they'll top out around 55MB/s and right around 6000 IOPs and about a .85ms latency. This is after fully tweaking esx, nexenta, and disabling access time within the running guest OSes.

Trying to get some loaner hardware from Mellanox so that I can begin testing NFS over RDMA. Not for the bandwidth but for the significant decrease in latency. I don't think 10gig is capable of meeting the needs of the top 3% or so of workloads within a public cloud type environment. Those types of workloads are exceptionally lucrative if you can deliver them and right now I haven't seen or heard of a public IaaS cloud that can honestly deliver on the high end.
Click to expand...
Okay so what you are saying is we'll reach IOPS bottleneck before link speed bottleneck. Can I ask what type of traffic are you serving for both of your iSCSI and non iSCSI setup?

dandragonrage said:
I used 2x1Gbps teamed when I ran ESXi at home (until last year). At work we use 4x1Gbps but for heavy loads no amount of 1Gbps connections will be acceptable. Fortunately OP doesn't seem to have extreme requirements. Teaming 1Gbps connections is only going to read at 200MB/s if you use a protocol with native multipathing like SMB 3.0 (and I think NFS 4 also) or if you can split the data up into multiple streams (which could then be load-balanced).
Click to expand...
And you are saying 2*1Gbps * 1024Mb/Gb * 8/10 ~ 200MB/s only so it's not fast enough. Can I also ask what type of traffic you have at work? Also do you use any active-active setup for storage?
 
titusc said:
Okay so what you are saying is we'll reach IOPS bottleneck before link speed bottleneck. Can I ask what type of traffic are you serving for both of your iSCSI and non iSCSI setup?
Click to expand...
NFS is almost exclusively VM traffic. ISCSI traffic is a mix of VM datastores and LUNs connected to VMs.
 
You must log in or register to reply here.
Back
Top