ZFS Speed Problem

[Nolhian]

Hello,

I have a raidz2 pool with 6 drives Samsung F2 ecogreen 1.5to on Nexenta Core with 8Go ECC Ram.

time sh -c "dd if=/dev/zero of=ddfile bs=8k count=2000000 && sync" on the pool reports 310mo/s.

iperf between the NAS and my pc reports 500mbits/s in a way and 630mbits/s in the other.

With NFS I got 41mo/s and with CIFS 38mo/s. While transfering in both computer the processor load doesn't go further than 30% .

So I should at least get 60mo/s, what's the problem ?

Thanks in advance,
Nolhian

 

[danswartz]

does mo/s mean megabytes/sec? if so, MB/s is more common. if not, what do you mean? if MB/sec, given protocol overhead and such, i don't think you're doing that badly...

 

[luma]

mo/s is French for MB/s

Even with overhead nfs and smb should still be a bit more no? and iperf which is cpu/ram only should be close to maxing out the network cards?

With iperf using a 128k or 258k window I get 879MB/s average, My Nexenta box is a core2duo E6600 with 4GB of memory and the other end is a laptop (I7 dual core with 8GB ram) with iperf running on a xubuntu vbox VM.

does mo/s mean megabytes/sec? if so, MB/s is more common. if not, what do you mean? if MB/sec, given protocol overhead and such, i don't think you're doing that badly...

 

[danswartz]

what network cards on the two hosts? that can be an issue.

 

[Nolhian]

Thanks for your fast answers !

On the Client :

Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

On the NAS :

D-Link DGE-528T

I just did a test the other way around, NFS from NAS to CLient : 68MB/s , CIFS still 35MB/s.

 

[luma]

What kind of CPU is your NAS running? What Speed of ECC RAM?
In your NAS what are the drives hooked up to? PCI controller? PCIE? onboard?

That D-Link is regular PCI is it not? I am not super technical when it comes to interfaces but I do know PCI has it's limits especially if shared with something but I am sure someone can correct me/provide more info on this one!

 

[iconiK]

CIFS has massive overhead. 35MB/s is good speed.

 

[luma]

No it is not. State your facts/sources if you think 35MB/s is good speed using gigabit and a fairly decent machine.

I have a Core2Duo Nexentastore machine with 4GB of ram, nothing fancy at all and using 6x disks in a RAIDz I can do 95-105MB/s depending which machine I am using to pull the data. (using cifs going to Windows machines)

CIFS has massive overhead. 35MB/s is good speed.

 

[hotcrandel]

In the case of ZFS, I think it has to write everything before it will accept more data, as well as the transaction log and possibly checksums.

Correct me if I'm wrong, but iPerf does not access the ZFS pool and so the performance is increased without having to wait for zfs to commit everything.

 

[danswartz]

That would affect writes, not reads and writes (I thought he had slowness both ways?)

 

[Nolhian]

What kind of CPU is your NAS running? What Speed of ECC RAM?
In your NAS what are the drives hooked up to? PCI controller? PCIE? onboard?

That D-Link is regular PCI is it not? I am not super technical when it comes to interfaces but I do know PCI has it's limits especially if shared with something but I am sure someone can correct me/provide more info on this one!

CPU : Athlon X4 645
RAM : I must check but either 1333Mhz or 1600Mhz
Drives are hooked up to a LSI 1068E board on PCI-E 16x port.
Yes D-Link is regular PCI.

In the case of ZFS, I think it has to write everything before it will accept more data, as well as the transaction log and possibly checksums.

Correct me if I'm wrong, but iPerf does not access the ZFS pool and so the performance is increased without having to wait for zfs to commit everything.

I think you're right but when I make a file locally on the Raidz2 I get more than 300MB/s so more than a Gigabit cable can handle.

That would affect writes, not reads and writes (I thought he had slowness both ways?)

Well if I write on the server via NFS I've got 40-45MB/s but when I pull a file from the server to the same client, still via NFS, I've got 68MB/s so I really don't understand where the problem is. Iperf shows the LAN is OK with 500+ MB/s, Local write on Raidz2 is OK because making a file shows more than 300MB/s so i should get better data transfers.

 

[Red Falcon]

CIFS has massive overhead. 35MB/s is good speed.

Only in very old versions was the overhead a problem. It has been fixed in recent revisions. 35MB/s is not very good, even for CIFS.