Freenas/ZFS capped out 40-50MB/s?

G

Guldan

Weaksauce
Joined
Jan 16, 2013
Messages
113
I have a Dell R510 with (Dual CPU / 6 Core) and 130GB of ram. My pool consists of three 12 disk raidz3 plus a mirrored SSD ZIL and a four disk SSD L2ARC. It's also on a fully 10 gig network (and reflects that in config). HBA is a beastly lsi 9300-16e.

Nothing I can do is making this thing break 50MB/s whereas my home budget raidz1 does 100MB/s and in comparison my windows boxes push 300MB/s (REFS raid60s). I'm just using a windows 2012 ISO file for every test.

I tried many different configurations (raidz1/raidz2/raidz3,mirror) even a single disk!.. always maxes at 50MB/s (with a flat transfer graph) Smelled like a protocol issue so I mounted NFS which was actually slower.. FTP same thing. Now if I transfer from multiple sources I'll get 40-50MB/s from each source.

I ran a local IOzone test and I was getting 250MB/s+ writes and 600MB/s+ reads (more what I expected)

Can anyone help me pinpoint the bottleneck? everything else has been flawless aside from this.
 
Could it be a network cable issue? Really it could be anything, bug within the NIC driver, faulty switch, misconfiguration, client issue and so on.
 
olavgg said:
Could it be a network cable issue? Really it could be anything, bug within the NIC driver, faulty switch, misconfiguration, client issue and so on.
Click to expand...

I'm currently setup in a failover configuration although I tried individual 10gig links, each link goes to a different switch and I tried both. I'll get someone to check the port(s) to rule it out but pretty certain that is not the cause.
 
Does the box have a standard gigE port? If you try anything but the 10G do you get better throughput?
 
mwroobel said:
Does the box have a standard gigE port? If you try anything but the 10G do you get better throughput?
Click to expand...

Yep two of them, I'll hook one up this afternoon and also check the 10g switch then report back on the results.
 
The 10 gig link is showing no errors on the switch side so I hooked up the 1 gig link.

1GBE ~ 39MB/s
10GBE ~ 47MB/s

Once again I tried hitting the 10 gig with multiple streams, they all hit that limit in parallel. So if I send four streams from four servers it ends up at 47MB/s x 4.

Not sure whats limiting the single streams of data.

Edit:

I've been playing with tunables
https://calomel.org/freebsd_network_tuning.html
None of these made any difference to the bandwidth im getting
 
Last edited:
Do you have the network adapters setup for LACP? Do you have the same frame size set everywhere adapter/switch? What are you using to gauge the speed? What does iPerf say?
 
Last edited:
Try installing a different version of the OS, or just reinstalling.

Also try solaris
 
Network adapters are setup for failover on freenas and each individual link goes to a seperate switch. I have de-coupled them and run single links and the speed remains the same.

I just ran an Iperf for you, shows 1.41 Gbytes @ 1.21Gbits/sec

For a real world test I've been copying a 4.7GB ISO file from various windows file servers I have.

edit: regarding re-installing, I have an identical box offsite and it is having the same bottleneck.
 
Guldan said:
Network adapters are setup for failover on freenas and each individual link goes to a seperate switch. I have de-coupled them and run single links and the speed remains the same.

I just ran an Iperf for you, shows 1.41 Gbytes @ 1.21Gbits/sec

For a real world test I've been copying a 4.7GB ISO file from various windows file servers I have.

edit: regarding re-installing, I have an identical box offsite and it is having the same bottleneck.
Click to expand...

I was having the same issues with FreeNas that made me leave it. Now I use regular solaris and it works great for me.
 
How does it preform if you remove the Zil and L2arc? I've seen them hurt performance.
 
staticlag said:
I was having the same issues with FreeNas that made me leave it. Now I use regular solaris and it works great for me.
Click to expand...

I know for a fact there are people running this version of Freenas without issue, getting 100+ MB/s so there is a logical root cause, I just need to find it. Pretty frustrating to me because I am a huge Freenas fan and we spent a lot of money putting together this system


smangular said:
How does it preform if you remove the Zil and L2arc? I've seen them hurt performance.
Click to expand...

Zero impact, literally every configuration I have made performs exactly the same. raidz1/raidz2/raidz3/mirror/single disk.. with and without zil, l2arc.. I've tried a dozen or more tunables.
 
Guldan said:
I just ran an Iperf for you, shows 1.41 Gbytes @ 1.21Gbits/sec
Click to expand...

1.2 Gb/s or GB/s? 1.2 Gb/s on a 10GbE adapter is quite bad, you should be aiming for 9.x Gb/s. 1.2 GB/s on the other hand is about right under ideal conditions.

For comparison my 10GbE linux boxes are capable of maxing out client SSDs while transferring large files to/from network shares. You might consider trying another OS, if only to have a point of comparison and to check for hardware problems.
 
Silhouette said:
1.2 Gb/s or GB/s? 1.2 Gb/s on a 10GbE adapter is quite bad, you should be aiming for 9.x Gb/s. 1.2 GB/s on the other hand is about right under ideal conditions.

For comparison my 10GbE linux boxes are capable of maxing out client SSDs while transferring large files to/from network shares. You might consider trying another OS, if only to have a point of comparison and to check for hardware problems.
Click to expand...

1215 Mbits/sec to be exact, Yeah I suppose I could bootup in another OS and just see what rates I get to compare good call.
 
I'm assuming that you have compression turned on.
- Default compression? (Lz4?)
- Is Hyperthreading disabled? (Especially since CIFS is single threaded)
- What do you have for the client (Does it have a SSD?)
 
CRaschNet said:
I'm assuming that you have compression turned on.
- Default compression? (Lz4?)
- Is Hyperthreading disabled? (Especially since CIFS is single threaded)
- What do you have for the client (Does it have a SSD?)
Click to expand...

- LZ4 is enabled however on the main pool however I have tried multiple pools without compression which net the same speed
- I've monitored samba during transfers and the CPU is quite low, NFS/FTP is quite slow so I doubt it's CPU related.
- I've tried multiple clients, I've tried it from three different file servers all which hit 300MB/s to each other, but to the Freenas box all do 40-50MB/s max.

safelyRemove said:
What's the client? Maybe the problem isn't server side?
Click to expand...

- I have four file servers, all are 2012 / 2012 R2 on 10 gig GBe links connected to MD1000 arrays. When they transfer to each other they do 300MB/s easy, same file to the Freenas box does 50MB/s (on the same network/same protocol).

Also to add on all of this.. I was doing an zpool iostat while a transfer was going on and I got these odd results, could the erratic writes be because of write cache taking hits or something else? last column is write bandwidth

pool alloc free read write read write

storage01 121G 130T 0 224 0 2.14M
storage01 121G 130T 0 0 0 0
storage01 121G 130T 0 1.16K 0 69.7M
storage01 121G 130T 0 155 0 779K
storage01 121G 130T 0 0 0 0
storage01 121G 130T 0 224 0 2.14M
storage01 121G 130T 0 0 0 0
storage01 121G 130T 0 146 0 767K
storage01 121G 130T 0 146 0 767K
storage01 121G 130T 0 0 0 0
storage01 121G 130T 0 146 0 767K
storage01 121G 130T 0 0 0 0
storage01 121G 130T 0 146 0 767K
storage01 121G 130T 0 224 0 2.19M
storage01 121G 130T 0 0 0 0
storage01 121G 130T 0 317 0 1.67M
storage01 121G 130T 0 0 0 0
storage01 121G 130T 0 290 0 1.56M
storage01 121G 130T 0 146 0 803K
 
Iostat numbers will be erratic even under normal circumstances, especially writes. You have a lot of RAM as well for caching so rereading the same files might not even show up at all on iostat. I have a system with similar specifications and testing the raw disk transfer rate is not so easy, especially if you have compression enabled. Async writes are cached for some seconds before they are flushed to disk.

Your iperf numbers are pretty bad. You could connect a single SSD and see if transfer rates are any different from that, but unless that shows a real difference it looks like a network config/driver/hw issue. Boot a linux flash drive or live cd and test your performance from there.
 
Fair points thanks guys, I'll give Solaris and/or Ubuntu or something a try when I have time and report back.
 
Guldan said:
Fair points thanks guys, I'll give Solaris and/or Ubuntu or something a try when I have time and report back.
Click to expand...

We have two freenas machines in production. Running a single E5-1620V2. 64GB of RAM. 4x vdevs of 6xZ2's. 8 drives per LSI 9207-8i HBA's

CIFS sucks, you need to tune it. I was hitting similar speeds, although not as bad prior to tuning over 2x10Gb on a Netgear XS712T. We were capped at around 40-100MB/s. After tuning I can saturate 10G easily as long as the file is in ARC, other wise I'm limited depending on file size.

As for iperf, try with larger window sizes. Try 9000 byte. You should be able to saturate your 10G easily. I was only able to push 1-2gb/s over default TCP window sizes.


Once I get back into the office, I'll post some settings that you can tweak. Also research on FreeNAS forums, there's great information there =)

Nothing is out of the box on high performance hardware.


Have you tried an internal disk benchmark? Throw some files on it and run a scrub. If the scrub runs slow, then I'm guessing it's just a bug with the new LSI chips. Your CPU's are also quite old man.
 
Last edited:
cookiesowns said:
We have two freenas machines in production. Running a single E5-1620V2. 64GB of RAM. 4x vdevs of 6xZ2's. 8 drives per LSI 9207-8i HBA's

CIFS sucks, you need to tune it. I was hitting similar speeds, although not as bad prior to tuning over 2x10Gb on a Netgear XS712T. We were capped at around 40-100MB/s. After tuning I can saturate 10G easily as long as the file is in ARC, other wise I'm limited depending on file size.

As for iperf, try with larger window sizes. Try 9000 byte. You should be able to saturate your 10G easily. I was only able to push 1-2gb/s over default TCP window sizes.


Once I get back into the office, I'll post some settings that you can tweak. Also research on FreeNAS forums, there's great information there =)

Nothing is out of the box on high performance hardware.


Have you tried an internal disk benchmark? Throw some files on it and run a scrub. If the scrub runs slow, then I'm guessing it's just a bug with the new LSI chips. Your CPU's are also quite old man.
Click to expand...

So I loaded up Ubuntu livecd on the box that has freenas then connected to another ubuntu machine and did 100MB/s + with a single stream, something freenas has not been able to do under any circumstance.

Moreover I've found that what is happening is freenas will max out streams from a particular IP. I'll give an example.

fileserver 1 - transfers 1 stream of data and gets 40MB/s, from same server I open up a new stream and both go to 20MB/s.. four stream and I get 10MB/s.

fileserver 2 - If I open a new transfer (while file server one is still going) I get a fresh 40 MB/s

I look at the freenas network graph and its literally 300Mbit steps like a staircase.
 
When I was using freenas in the past, I remember needing to play with the CIFS settings, like buffer sizes and shit. Basically just set them as large as they can go, also turn on jumbo frames.

After that I could keep a rock steady 112MB/s

Its not that CIFS sucks, not even than FreeNAS sucks. Its just that the default settings suck.
 
This seems to be a much bigger problem on the 12th gen dells but I'd check it on yours. Check the power management in the bios and set it to high preformance. There is a good chance yours is set to os controled which generally is fine but a lot are set to the lifestyle controller for management.

This has caused all sorts of issues in windows for us. Pretty much the dell controller slows down the cpu to seemly nothing and starts messing with everything.
 
swatbat said:
This seems to be a much bigger problem on the 12th gen dells but I'd check it on yours. Check the power management in the bios and set it to high preformance. There is a good chance yours is set to os controled which generally is fine but a lot are set to the lifestyle controller for management.

This has caused all sorts of issues in windows for us. Pretty much the dell controller slows down the cpu to seemly nothing and starts messing with everything.
Click to expand...

its on high everything, Ive tried every BIOS setting on and off.

..........still 45MB/s

At this point i've determined it's a bandwidth cap per host but I have no idea where it's coming from and no solid leads.
 
You must log in or register to reply here.
Back
Top