Linux file server problem - Slow samba transfers on gigabit

nry

Limp Gawd
Joined
Jul 10, 2008
Messages
409
Hi

Im new here, don't really post on forums much anymore but recently I built myself a file server using old bits and a few new bits..

Also let me begin by saying im quite new to linux so could be a big error on my half.

For those who care this is what it looks like

image283rp3.jpg


Now the major problem with this is the speed of the file transfers im getting. Im storing some Hd movies to stream to my HTPC and need to be getting some reasonable speeds for smooth playback.

Spec of file server
CPU - AMD XP 2000+
RAM - 2x 1GB PC3200 DDR
Motherboard - Asus A7N266-VM
PSU - Tagan TG480-U01
System HDD - Seagate 2.5" IDE 4200rpm 80GB
Network - Gigabit Dlink DGE-510T
SATA Card - SATA PCI Sil3114 (was a reasonably cheap one off ebay)
Storage Drive - Western Digital Green Power 1TB
OS - Ubuntu 8.04
Installed Apps - Webmin, Samba, SSH, Azureus

So personally I think this system should be able to reach more than the slow speeds im getting.

Here are some of my file transfer tests between the 3 computers on my gigabit network at the moment.

Code:
Laptop to HTPC - 23MB/s
HTPC to Laptp - 25MB/s (does hit 35MB/s at times!)

Laptop to Server - 10MB/s 
Server to Laptop - 8MB/s

Some tests to the system drive of the server IDE 2.5" 4200rpm 80GB
Laptop to Server - 13MB/s
Server to Laptop - 8MB/s


HTPC to Server - 9MB/s
Server to HTPC -  - 8MB/s


Following is with HDD mounted in a SATA USB Caddy to laptop
HDD to Laptop - 25MB/s
Laptop to HDD - 36MB/s

So this just about shows that my HDD isnt the problem.

Testing the network I used iperf
This is from the terminal of the server

Code:
root@nas:~# iperf -c 192.168.1.5
------------------------------------------------------------
Client connecting to 192.168.1.5, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.6 port 39982 connected with 192.168.1.5 port 5001
[  3]  0.0-10.0 sec  84.8 MBytes  71.2 Mbits/sec
root@nas:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.6 port 5001 connected with 192.168.1.5 port 50421
[  4]  0.0-10.0 sec    156 MBytes    131 Mbits/sec
[  5] local 192.168.1.6 port 5001 connected with 192.168.1.8 port 49268
[  5]  0.0-10.0 sec    430 MBytes    360 Mbits/sec

Last few bits show, 4 - laptop speeds, 5 - htpc speeds

Not quite sure why the laptop is slower.

hdparm results of the 1tb drive.

Code:
root@nas:~# hdparm -Tt /dev/sda1

/dev/sda1:
 Timing cached reads:   472 MB in  2.00 seconds = 235.44 MB/sec
 Timing buffered disk reads:  216 MB in  3.02 seconds =  71.48 MB/sec
root@nas:~#

Can anyone help me on this? Really don't know where to go about sorting this problem now.
Really could do with the speeds between the laptop and htpc on my server.

Any help would be appreciated.
 
Try turning off checksums in hardware: ethtool -k devname.

Also, try a different protocol and see if that's faster... but I doubt it. iperf is pretty lightweight, so it should go as fast as anything can. If you have a different network card you could put in, I'd do it - I've seen several problems of this sort fixed by moving from Realtek to Intel, and I wouldn't be surprised if the D-Link is in the same boat.

That said, even Blu-Ray maxes out at about fifty megabits, so you should be able to handle a single stream as is.
 
Players:

10.10.10.5 FreeNAS server based on FreeBSD 6.3
10.10.10.213 Vista 64bit SP1
10.10.10.215 Gentoo Linux 2.6.24-r8


Code:
torrentbox ~ # iperf -f M -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 10.10.10.215 port 5001 connected with 10.10.10.5 port 51259
[  4]  0.0-30.0 sec  3058 MBytes    102 MBytes/sec
[  5] local 10.10.10.215 port 5001 connected with 10.10.10.5 port 62435
[  5]  0.0- 1.3 sec    119 MBytes  94.2 MBytes/sec
[  4] local 10.10.10.215 port 5001 connected with 10.10.10.213 port 65104
[  4]  0.0-30.0 sec  2943 MBytes  98.1 MBytes/sec


torrentbox ~ # iperf -f M -M 9000 -c 10.10.10.213 -t 30 -w 80K
WARNING: attempt to set TCP maximum segment size to 9000, but got 536
------------------------------------------------------------
Client connecting to 10.10.10.213, TCP port 5001
TCP window size: 0.16 MByte (WARNING: requested 0.08 MByte)
------------------------------------------------------------
[  3] local 10.10.10.215 port 46562 connected with 10.10.10.213 port 5001
[  3]  0.0-30.0 sec  2972 MBytes  99.1 MBytes/sec
torrentbox ~ # iperf -f M -M 9000 -c 10.10.10.5 -t 30 -w 80K
WARNING: attempt to set TCP maximum segment size to 9000, but got 536
------------------------------------------------------------
Client connecting to 10.10.10.5, TCP port 5001
TCP window size: 0.16 MByte (WARNING: requested 0.08 MByte)
------------------------------------------------------------
[  3] local 10.10.10.215 port 55310 connected with 10.10.10.5 port 5001
[  3]  0.0-30.0 sec  2553 MBytes  85.1 MBytes/sec

also
FTP transfer from Server to vista 64


How? Intel pcie network cards with latest opensource e1000e drivers, good gigabit switch and Jumbo Frames.
 
Looking at your config the first thing that jumped out at me was this:
SATA Card - SATA PCI Sil3114

Silicon Image sata cards are somewhat notorious for causing problems, both on Linux and other os'es.

Something you might want to check out, from the Silicon Image Wikipedia article
"Silicon Image has released the SII3112, SII3114 and SII3512 S-ATA controller chips, which, in order to avoid corrupt data storage, require driver workarounds to correct compatibility problems with certain disk drive models. When activated with the affected disc drives, these workarounds will often cause significantly diminished performance in a system."

It is quite possible that since you are using a recent Linux distro that the workarounds are enabled by default. So Ubuntu has to slow down transfer rates to keep the buggy card from corrupting your data. I am making the assumption here that your transfers are going through the sata card to the 1TB drive.

sata controller cards are kind of weird. You can get a modern motherboard that is fairly inexpensive and has decent sata controllers, but if you want a pci sata addon card that is reliable and reasonably fast you can end up spending $50+

I'm kinda new here and don't know if posting links is allowed, so I won't. But the sata card I have on one of my older servers is a "Promise sata300 tx4" which can be had on newegg for about $60+shipping. With that card I have four sata drives in a Linux software raid5 for a couple years now with no problems, read speed from the server is about 26MB/s with samba and about 35MB/s with less chatty protocols like ftp or nfs.
 
BadBlock
I am looking at the Promise sata300 tx4 for a file server I am going set up over the next few months and am wondering what motherboard are you using?
I will be using a K8N Neo with a NVIDIA nForce 3 chipset and am wondering if that will work with that card
 
Thanks for all the replys. going to get testing on this over the next few days.

Regarding the info about the sata pci sil3114 card -
Wouldn't this effect the hdparm result as well?
As shown above and the result for my ide drive.

Code:
/dev/sdb1:
 Timing cached reads:   434 MB in  2.01 seconds = 216.40 MB/sec
 Timing buffered disk reads:   96 MB in  3.04 seconds =  31.60 MB/sec
root@nas:~#

But over samba they both near enough get the same speeds.

Don't want to spend any more money if I dont have to :) and think I woulrather get a new motherboard as here in the UK sata cards seem quite expencive.

EDIT: dont know how to say this, but I have changed nothing and im now getting 22MB/s off the server???

must be a problem some where though

EDIT: nope its back down the the speeds before. so I know it CAN work at full speeds just not 100% of the time
 
One thing you could try is doing a transfer to the ide drive over the network and doing one to the sata drive over the network and seeing if the speeds are dramatically different. I am assuming the workarounds for that card only slow down traffic through it and not all ide/sata bus traffic.

If you don't want to go to the hassle of making separate samba shares just to test that you could just log into the machine via ssh and copy a file to one of the drives looking at how long it takes, then doing the same for the other drive. Assuming you have ssh server on the server you can be on your client, open your file browser (nautilus, konqueror, etc) and just put in sftp://username@serverip
If your client machine is windows or mac you could use some sftp client like filezilla or winscp.

Keeping in mind that sftp is pretty slow on its own since it has to encrypt everything, but this might give you an idea if the traffic to each drive is either identical or varies. If you do try this you might get more accurate results doing transfers to the server rather than from the server since I think the encrypting end of an ssh transfer has to do more work and you don't want the server cpu to be the bottleneck.
 
A few things, I can attest that the silicon image controllers cause problems. I had 3 1tb disks in a MD raid 5 under Ubuntu 7.10 and basically a few months after I built the array a power outage caused the array to go out of sync and I could never complete a rebuild. I would get a kernel pannic about 30 mins into the rebuild.

Also, if you are trying to get that kind of performance, you need a good NIC like the intel sugested above. You will also need a good switch. You can test without the switch by using a crossover cable between your devices. Also look for duplex and physical issues with cabling. A little packet loss will cause horrible problems. A lot of crapo home switches (dlink, netgear, etc...) won't do more than a few hundred megs....

You can also do some linux tuning. If you don't care about tracking the last time the files were accessed on the disk, you can specify the "noatime" mount option on your data volume. This prevents the kernel from updating the file information with the last time it was read. You see this a lot for people using cf media as boot disks. You can also check on what is bogging the server down by running "vmstat 1". This will show you interupts "in", kernel context switches "cs", and processes waiting to be serviced "r".

You might try switching to XFS as it gives better performance for windows clients accessing Samba.

You can also disable wide links

In your smb.conf file you will want the socket options to look like this:

socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192

You may need to increase the buffers if you are using Jumbo. You may also try it without this line as the newer kernels are supposed to autotune TCP better.

In Samba-3.0.12 and later, you can set "case sensitive = True", this causes the server to only look for the file in the exact case it was requested and not try to convert it to upper or lower..

I would also do a sniffer capture and watch both the TCP window size and the SMB block size. The interpacket gap and turnaround time for the server side can be a telling tale as well.

You can quite often see where the client shrinks the TCP window to 0 and the connection is stalled.... This usually indicates a resource problem somewhere, I.E. low performance writing to the disk.

Finally, and last but not least, you can try and use something much more efficient like NFS wihch is full of other problems and quirks, but is faster.
 
Right, my netgear switch can do 25MB/s or there abouts as there are no problems doing that from Laptop to HTPC both running windows vista.

SFTP speeds to server IDE - 4.1MB/s
SFTP speeds from server IDE - 1MB/s

My biggest problem now is how the speed changes! it seems to reac23MB/s no problem but won't do it 100% of the time.

I just started transfering a a 4GB file to the server IDE drive at 26MB/s
Read speeds at 8MB/s

Going to try a diffrent network card shortley but can't see this being the problem, nor the sata card at the moment. Going to try running without the sata card in and just get the IDE drive working 100% first.

ftp://ftp.dlink.eu/datasheets/DGE-510T.pdf
EDIT: looked at the network card im using at I dont think its realtek but my spare one is, so going to stay with the d-lnik for now, as I know it can reach the speeds I want on some occasion.
 
I looked into NFS but its not really suitable for my network due to it costing and extra stuff needed to be installed?
Not too sure about the TCP window etc bit over my head there. Not the most technical person but have a rough idea what im doing.

To use jumbo frames don't you need to have all the hardware on the network compatible with that? As some computers accessing the server won't support it.

Right with the sata card removed. And basically running as little hardware as possible speeds from the IDE drive (not sure I metioned it but it uses the ext3 filesystem) ...

Samba speeds
Write - 25MB/s (seems to jump around 23-28)
Read - 9MB/s

FTP
Write - 22MB/s
Read - 8MB/s

Where can I go now to sort this? as surely if my IDE drive can write at 25mb/s it can read at least 15m/s

Really stuck here.
 
I would not play around with jumbo frames until you figure out where your bottleneck is.


As far as nfs goes, it's just another network protocol, like samba, so you don't need any special hardware to use it.

advantages of nfs:
fast
supports lots of security options and unix file permissions

disadvantages of nfs:
windows doesn't understand it
configuration is more complex than samba


If this is just for an internal home network you could set up nfs and disable most of the security options to make it easier to set up, there is a guide on the ubuntu site, most of the steps in it probably would not be necessary for your setup.

Like I said before, the performance I get out of one of my old servers is around 26MB/s with samba and about 35MB/s with nfs. So it can make a difference but if you are having some hardware related bottleneck it might not help much.

Also when monitoring network transfer speed, either use a program that measures is or use a stopwatch/calculator, you often can't trust the speed indicated by the transfer window because under some setups it will dump a bunch of the file being moved into memory (showing a high speed), then begin pushing it over the network (showing a low speed).

If you do narrow it down to the network card and need to replace it the intel cards mentioned by others are pretty good performance and driver wise, you can get one such as this for around $25 before shipping.
 
biggest problem with nfs you have to install something on the clients.

Surely if im getting slow speeds in ftp as well the problem lies else where.

It can't really be the network card either because if I get the speeds one way, why not the other?

For measuring speeds I usually let the large file (2-4GB) run for a while then look at the speed as it starts of around 40MB/s then would slow down to 25ish

This is the config of my samba

Code:
[global]
        server string = %h server (Samba, Ubuntu)
	passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
	obey pam restrictions = yes
        socket options = TCP_NODELAY IPTOS_LOWDELAY 
	log level = 0
	log file = /var/log/samba/log.%m
	read raw = No
	write raw = No
	#oplocks = yes
	map to guest = bad user
	encrypt passwords = true	
	passwd program = /usr/bin/passwd %u
	passdb backend = tdbsam
	wins support = true
	dns proxy = no
	netbios name = NAS
	invalid users = root
	unix password sync = yes
	workgroup = WORKGROUP
	os level = 20
	syslog = 0
	panic action = /usr/share/samba/panic-action %d
	usershare allow guests = yes
	max log size = 1000
	pam password change = yes
	lanman auth = Yes
	ntlm auth = Yes
        client NTLM auth = no
	client NTLMv2 auth = Yes
	client lanman auth = no
	client plaintext auth = No

But the problem in my openion is in the software side.
Going to install windows onto another disk and see how well that performs, then might try freenas.
 
It's strange that the writing would be so much faster than the reading.

One idea I am going to throw out there, if there was some problem with read/writing to drives on the file server, but the network hardware was fine, and the machine was caching aggressively you could write a file to that server and it would fly over the network and into memory and then be slowly written to the hard drive but from the client end it looks like the transfer is already done and was fast.

At the same time reads from the server would appear slow since there it can't hide anything with caching and if there was some horrible bottleneck between the drives and the motherboard you would feel it on the client side and everything looks slow.

I haven't actually seen this happen, at least not to this degree, but something you could do to test it would be to copy a file to the server and see if there is a significant difference between when the client thinks the transfer is done and when the server actually stops writing to the drive. Or copy a very large file to the server that is greater than the amount of memory it has and see if the write speed doesn't drop significantly before the transfer is complete.
 
Tested a windows install on a old 80g drive, managed to get 15-20MB/s to the HTPC both ways with no real effort.
Drive was a WD model no. WD800JB

Transfered 4.17GB from laptop, overall the speed was 25MB/s all the way through and writing to the server drive stopped around 10 seconds after it finished on the client.

Now running the server just with minimum hardware, with the realtek network card as well.

2.5" IDE Speeds.
Write - 25MB/s
Read - 35MB/s
But it will also read at 8MB/s sometimes (torrent files which are complete?)

So guess that problem is sorted? But why am I only getting 8MB/s on the completed torrent files

Also tested speeds of an incomplete torrent file and got around 15MB/s

The torrents are set to be moved from my 'Unfinished' folder to the 'Finished' folder both on the same drive.
 
Really don't understand why this is causing so many problems!

Yesterday I was getting the speeds mentioned in my last post

Today I get 8MB/s max either way

Now I have not changed anything or rebooted, then tried rebooting but still the same speeds?

Any ideas?
 
Its been quite some time, but I think there was a speed issue if IPV6 was enabled.. is that on? You might want to try disabling it and rebooting to see if that helps. I could be thinking of a totally different issue, but its worth a shot
 
I have ipv6 turned off on both all computers.

To do it on ubuntu I used:

Code:
alias net-pf-10 ipv6 off
alias net-pf-10 off
alias ipv6 off
in /etc/modprobe.d/aliases
 
How do I view the current version of my sata sil3114 drivers?

And how do I go about upating them?

Tested the sata card in my htpc, latest drivers off the website and getting 60MB/s from the 1TB drive no problem.
 
Back
Top