Serving files out over Samba/NFS

[JoeKillaMofo]

Hey all!

So I have a file server performance issue that I have been banging my head against a wall for at least a few months now.

The clients (windows 7) mount the server (centos 5.6) via Samba for the most part and stream HD file sequences. Doing tests on just the server itself, copying files to the array from a ramdisk or vice versa and it performs admirably (sequential copies r kewl). But when I go to copy those files out via the network I get an insane drop in performance, by insane I mean from hundreds of MBps down to maybe 40-60MBps.

These clients, and the server, are 10Gbit btw.

Now before you tell me to RTFM on the NIC, I can serve out said files at almost 200MBps from a Ramdisk on the server (over Samba). So the problem has been narrowed down to not being the Raid card or the FS, and not being the NIC card (or tuning parameters). It looks to be the linux kernel itself... How that is possible I have no clue, BUT this has been replicated on 5.6 and 6.2 (both CentOS).

I am thinking there is something I am doing wrong maybe on the FS side, that I should be building the FS itself in such a way as to optimize it for serving out LAN traffic, maybe the chunks of data coming off the disks are needing excessive fragmentation to be brought down to a level acceptable for the NIC. A friend of mine suggested that maybe the RAM on the raid controller may be double penalizing somehow (fragmenting data going into the ram (the files are large) but again local performance isn't terrible (500MBps+). Is there some switch I am missing? I have done all the TCP tuning/optimization for large files, set a very large read-ahead (16MB) in the kernel and changed the scheduler to "noop" to offload the data handling to the raid card itself, etc. Nothing I have found allows me to take the full performance I get from the raid and slam it through the NIC's out on to the network, nothing!

The raid card is an Areca 1880x btw. The NIC is a Myricom 10gbe. Backplane and disk chassis is Supermicro and drives are 1TB sata 7200 seagate.

Thanks for any help!

JKM

 

[diizzy]

You're leaving out quite a bit of information,
How much ram does this machine have?
How many concurrent clients?
What filesystem are you using?
Do you have any Samba tuning at all?
//Danne

 

[JoeKillaMofo]

You're leaving out quite a bit of information,
How much ram does this machine have?
How many concurrent clients?
What filesystem are you using?
Do you have any Samba tuning at all?
//Danne

Well, that's just it, most of this info doesn't seem relevant, the individual processes work perfectly, but their interconnects don't seem to... Nonetheless here are specs:

12GB RAM
1 Client
XFS
Samba tuning:
max protocol = SMB2
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
max xmit = 65535 ; No impact no effect
read raw = no

 

[Red Falcon]

I'm interested in this as well, subscribed.

 

[jwcalla]

If your Windows client supports it, you could try setting up the NFSv4 protocol. That could rule out any problems with the SMB kernel module.

 

[ChrisBenn]

40-60MB/sec over 10GbE?

I would
1) Export a share from a drive not on the Areca and see what kind of performance you get there (yeah, you did this with the ram disk - but it would be nice to run the share through the sata subsystem as well)

2) Try connecting over a 1GbE connection without any tuning, etc. to the share in (1) as well as to the areca share. This is to see if you get the same speed over 1GbE that you do over 10GbE (some sort of hard bottleneck), or a different speed (something proportional).

3) Try a different protocol - this was suggested above, but I would try NFS from an linux client - I had wierd 40MB/sec limits on the NFS client in windows - it works, but I don't think it's very performant.


Are you running any kind of jumbo frames? If you are can you verify you aren't getting fragmenting? pint -f -l 8500 ip.of.server from the windows box should return valid ping, not a do not fragment flag (unless you are not running jumb frames, then disregard)


I know technically some of this may have been covered in the earlier ramdisk test you mentioned - but as put the results don't make sense, so I would go back to basics. Find the simplest, most "standard" configuration that gives expected performance (say 1GbE/stock tunings/single drive share), and start adding the other stuff back in.

 

[JoeKillaMofo]

40-60MB/sec over 10GbE?
I would
1) Export a share from a drive not on the Areca and see what kind of performance you get there (yeah, you did this with the ram disk - but it would be nice to run the share through the sata subsystem as well)

I can bring up a share on the system drive pretty easily, will do.

2) Try connecting over a 1GbE connection without any tuning, etc. to the share in (1) as well as to the areca share. This is to see if you get the same speed over 1GbE that you do over 10GbE (some sort of hard bottleneck), or a different speed (something proportional).

Same speed to 1gbe clients as well.

3) Try a different protocol - this was suggested above, but I would try NFS from an linux client - I had wierd 40MB/sec limits on the NFS client in windows - it works, but I don't think it's very performant.

Yeah I had the same problem with the Windows NFS client, very poor performance, don't think I tried with 10gbe, worth a shot I suppose.

Are you running any kind of jumbo frames? If you are can you verify you aren't getting fragmenting? pint -f -l 8500 ip.of.server from the windows box should return valid ping, not a do not fragment flag (unless you are not running jumb frames, then disregard)

Jumbo's throughout. I can confirm that everything in-between is jumbo's, again though even if it was fragmenting that doesn't necessarily imply that it is the bottleneck since it pulls at full speed when the files are cached on the server.

I know technically some of this may have been covered in the earlier ramdisk test you mentioned - but as put the results don't make sense, so I would go back to basics. Find the simplest, most "standard" configuration that gives expected performance (say 1GbE/stock tunings/single drive share), and start adding the other stuff back in.

[/QUOTE]
I need to double check and verify it was getting those speeds from the RamDisk as opposed to the cache on the file server, I may have misspoke there, I will go back through the tests and reconfirm. At least if it was the cache and not the ramdisk it would imply a disk subsystem issue I guess?

Thanks for all your help!