ZFS Mirror Array - only 30MB/s

A

AGampher

Weaksauce
Joined
Aug 20, 2003
Messages
109
Hello all!

I've got a ZFS mirror of 2 WD Red 2TB drives hosted on Ubuntu Server 12.04. When transferring a larger file (1GB+) I'm only getting around 30MB/s when writing the file to an SSD over a gigabit network. Any ideas? I am using the onboard SATA controller and am wondering if I need a dedicated card.

TIA!
 
I am using the onboard SATA controller and am wondering if I need a dedicated card.
Click to expand...

Unlikely that will help. Whatever your onboard SATA you have should be fully capable of reading from 2 hard drives as fast as they can deliver.

Any ideas?
Click to expand...

I would start by measuring the read performance of the same file on the server. I mean something like
Code: 
dd if=mybigfile of=/dev/null bs=1M

dd will print the MB/s after the operation has completed.
 
Last edited:
Its automatically a zfs read issue?

Your doing it over gigabit network? are you using jumboframes? nfs? samba? ftp?

what nic are you using on the send and receive side?
 
What is the client? If it's doing sync writes, your performance will suck. Try setting sync=disabled and re-test to see if that is it.
 
drescherjm said:
Unlikely that will help. Whatever your onboard SATA you have should be fully capable of reading from 2 hard drives as fast as they can deliver.

I would start by measuring the read performance of the same file on the server. I mean something like
Code: 
dd if=mybigfile of=/dev/null bs=1M

dd will print the MB/s after the operation has completed.
Click to expand...

Thanks for the info!

Looks like it's not the drives, nor the controller:
3075+1 records in
3075+1 records out
3224686592 bytes (3.2 GB) copied, 27.3969 s, 118 MB/s

patrickdk said:
Its automatically a zfs read issue?

Your doing it over gigabit network? are you using jumboframes? nfs? samba? ftp?

what nic are you using on the send and receive side?
Click to expand...

Didn't mean to make it sound like it was definitely a ZFS read issue, was just what I was thinking at the time.

I'm using Samba (zfs set sharesmb=on tank/storage). The server (Ubuntu Server) and endpoint (Windows 7) are both using an Intel(R) PRO/1000 PT Desktop Adapter, with a Netgear ProSafe gigabit switch in-between. I didn't install any drivers on Ubuntu Server, just plug and play. I have the most recent Intel drivers installed on the endpoint, but do not have jumbo frames enabled.

Thanks for the help!
 
Looks like it's not the drives, nor the controller:
3075+1 records in
3075+1 records out
3224686592 bytes (3.2 GB) copied, 27.3969 s, 118 MB/s
Click to expand...

That is slower than I expected but zfsonlinux is not yet optimized for performance.
 
Ideally, you should be getting around 40-50MB/sec using samba.
With samba2 on windows I'm getting 70MB/sec, but samba doesn't support that yet.

Probably need to tune up samba and maybe your windows host to get it going faster. I only did limited tuning on mine to get it going 45MB/sec
 
patrickdk said:
Ideally, you should be getting around 40-50MB/sec using samba.
With samba2 on windows I'm getting 70MB/sec, but samba doesn't support that yet.

Probably need to tune up samba and maybe your windows host to get it going faster. I only did limited tuning on mine to get it going 45MB/sec
Click to expand...

SMB2 is supported by Samba 3.6 and newer.
 
Odd, wonder why my windows -> windows samba is so much faster than samba 3.6.3 -> windows.
 
patrickdk said:
Odd, wonder why my windows -> windows samba is so much faster than samba 3.6.3 -> windows.
Click to expand...

I don't know, it is difficult to know without further detail of the machines involved and config.

You should update your samba version as 3.6.3 and older suffer from a remote code exploit (CVE-2012-1182) and since samba runs as root it is a doozy (this is one of the reasons I don't run Samba without SELinux, AppArmor, or equivalent).
 
Last edited:
patrickdk said:
Ideally, you should be getting around 40-50MB/sec using samba.
With samba2 on windows I'm getting 70MB/sec, but samba doesn't support that yet.

Probably need to tune up samba and maybe your windows host to get it going faster. I only did limited tuning on mine to get it going 45MB/sec
Click to expand...

I've enabled jumbo frames on both the server and endpoint and I'm now at around 39MB/s sustained.

How can I be sure I'm utilizing samba 2 between my Windows 7 Pro endpoint and Ubuntu Server (Samba 3.6.3)?

Thanks!
 
I'm not sure if it is smb2 or smb2.1 or smb2.2 that gives the speed increase. After some reading.

Also, just cause it is 3.6.3 doesn't mean it has an exploit.
 
patrickdk said:
I'm not sure if it is smb2 or smb2.1 or smb2.2 that gives the speed increase. After some reading.

Also, just cause it is 3.6.3 doesn't mean it has an exploit.
Click to expand...

3.6.3 is listed as being affected explicitly, although your distro may offer a patched 3.6.3. In either case, I'd check, as it is a root exploit.

Also samba 4.1 supports SMB3, if you want to use the latest version of the protocol.
 
Is samba normally slow? I get a solid 100-110MB/s from my ZoL server on Debian Wheezy to my win7 desktop via samba.
 
IIRC, the difference is that windows file sharing uses multiple TCP and UDP ports to fully saturate a data link. Whereas samba is more or less single threaded TCP.

So the windows version will show as faster in most cases. I'm not sure about the latest versions of samba though
 
SirMaster said:
I get a solid 100-110MB/s from my ZoL server on Debian Wheezy to my win7 desktop via samba.
Click to expand...

Me too (except not ZFS). I've never had any trouble saturating GigE with a Windows client uploading or downloading to a linux Samba server.
 
Have you tried commenting out your socket options in smb.conf? I seem to remember that fixing my issues.
 
You must log in or register to reply here.
Back
Top