• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

RAIDZ troubles. Help Please, 8 Drive NAS

S

staticlag

[H]ard|Gawd
Joined
Mar 26, 2010
Messages
1,679
So I finally bought a new gaming machine with the letdown of bulldozer I went ahead and pulled the trigger on a 2600k. With this I passed down my old Phenom II 940 (3.0 ghz) with 8GB RAM and controller card to my NAS and bought a 4220 to hold it all.

Setup is as follows:

CPU: Phenom II 940 (3.0gHz)
RAM: Corsair DDR2 1066 mHz (8GB 5-5-5-15)
Controller: LSI 9211-8i (2 SFF 8087)
LAN: Intel proE (PCI model)
Power: Antec High Current Gamer 750w (4 rail, 40a on each rail)
OS:Solaris 11 express installed in a 250GB laptop drive.
Drives: Samsung 2TB F4 HD204UI x8

Setup: RAIDZ2

I created my pool and shared it over the network successfully. Now I am trying to copy all 4TB of my movies and music to my NAS and it chokes.

Scenario:

First I tried to copy about 1 TB of movies (large single 700MB-4GB files). It started out ok, a slow IMHO but tolerable 66MB/sec. Memory and CPU usage on the NAS were nowhere near maxed out, CPU usage hovering at right around 10%. Then after about an hour of transferring the box decided to choke, transfer rates dropped to 4MB/sec, CPU and memory usage close to nothing. I halted the transfer and restarted it. Transfer rates resumed at the horrid 4MB/sec. At this point there was about 600GB on the RAIDZ2. After a few restarts of the client and the host of the transfer, speeds did not improve so I gave up.

Next I tried downloading from the pool, speeds were an abysmal 40MB/sec.

I have had no problems with this hardware previously, the motherboard, LAN, and NIC were able to max out my gigabit network (HP 18008G) under ideal drive conditions (SSD to RAID0) in its last setup.

I am willing to try anything to improve performance, but frankly I am pretty disappointed by the lack of performance of RAIDZ2.
 
This has nothing to do with raidz2. There is something networking related. The writes to the pool would be faster I think if you had a ZIL. Try setting 'sync=disabled' on the pool and retesting your writes. This makes all writes asynchronous. Not a good production setting, just to prove the point. The slow reads? I've seen that from flavors of solaris with various windows smb clients - never really found a good answer. Actually, my comment about writing is for the original 66MB/sec. Why it slows to almost zero after a bit is a different issue. What settings do you have for the pool and share?
 
Strange. Can you boot a liveCD with OpenIndiana, or FreeBSD and see if the problem persists? Then we can rule out S11E.

This has nothing to do with ZFS. It is something else, problem with a driver or something.
 
Try using CIFS. I have good luck with my very basic Neo II Dual Core and 4GB of ram. I can max out a 1GBit connection about with this setup. Transfers hover around 110-100 MB/s. (large .mkv video files)
 
Yes, CIFS is a good choice. I thought you were using CIFS? There are blogs on how to setup CIFS for OpenSolaris (compatibble with S11E). Try CIFS also.
 
As others have said, the issue is not related to raidz2 and I very much doubt this has anything to do with Solaris 11 express either. You should be able to max out gigabit in both directions.

Are you seeing any errors in iostat?

What are you using to copy the files, native windows or a copy manager? Is this single threaded copy or multiple?

Do not use a copy manager like Teracopy.

NFS or straight windows share?
 
Thanks for the posts,

I got some free time last night and decided to try re-downloading and re-creating a Solaris E11 livecd.

Results actually very much improved!

Write speeds increased to 90 MB/sec!!!

I am using the native CIFS smb function of zfs for my shares.

"zfs set sharesmb=on"

I changed the user of the pool to a dedicated share account and I set 777 permissions on the pool. I have also set it to my correct workgroup.

My method of testing has been from my other server running Server 08, I simply copy paste files into the shared pool present on the network.

So after the OS reinstall and the destruction and re-creation of my raidz2 pool, I am getting 90MB/sec writes!!! Writes are sustained at 90MB/sec until my RAM cache backs up and the ZIL slows down (then it chugs at 30-40MB/sec maybe 300-400GB later). I checked memory usage and it appears to be using 6GB of RAM (of 8) for write cache. I would say these results are very satisfactory and could be easily improved by adding a dedicated ZIL drive as Dan said above.

Now for the problem. My sequential read speeds are right at 55-70MB/sec. When looking at the notification lights on the array I notice that the lights are flashing much similar to a write cycle, such as pause a few seconds, then the array all flashes, pause.

There must be something going on that is causing the array to behave like that, any thoughts?
 
davros123 said:
As others have said, the issue is not related to raidz2 and I very much doubt this has anything to do with Solaris 11 express either. You should be able to max out gigabit in both directions.

Are you seeing any errors in iostat?

What are you using to copy the files, native windows or a copy manager? Is this single threaded copy or multiple?

Do not use a copy manager like Teracopy.

NFS or straight windows share?
Click to expand...

I see no errors with iostat or with zpool status

I am using native windows.
 
Perhaps I can explain a bit more about the intended usage of this box,

For home use I plan on keeping the box as main repository for my HTPC running XBMC. The rest of the space is for my photography business storing raw camera files. One full day wedding can create about 16GB of files after editing.

But...

In two weeks I plan on going to a LAN party and taking my server there to share files. It will be around 200 people using the windows program Dc++ to locally share files.

So i had planned on running solaris, virtualizing windows, then running Dc++ to share all my files.

I use virtualbox on windows for virtualization so I was leaning towards solaris, but I would be willing to try any OS that can accomplish this.

I think it will be interesting to see how well this box can preform in a maxed out environment.

I am also looking at getting a SSD for read cache for the event.
 
I heard that WinXP clients give 50MB/sec. Linux too. Win7 clients give 90MB/sec. This was using CIFS and OpenSolaris (or Solaris 11 Express - which is the same thing).

Can anyone confirm these speeds?
 
I just copied a 2.4GB ISO from a CIFS share (virtualized OS server) to my win7 pro box. Consistently got just under 100MB/sec.
 
You must log in or register to reply here.
Back
Top