OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

A speed drop with multiple copies is not typical for Solaris.
The problem may also be at the Win7 side (Network card, copy tools etc)

Same happen with another host running Win2k8 R2..

I am starting to look more toward ESX or something.. but all this dosen't make much sense to me as it can handle 700Mbps+ on single transfer.. I might try to install OI directly on the host without ESX..

What's also interesting is it seem to go down to 100Mbps...
 
Same happen with another host running Win2k8 R2..

I am starting to look more toward ESX or something.. but all this dosen't make much sense to me as it can handle 700Mbps+ on single transfer.. I might try to install OI directly on the host without ESX..

What's also interesting is it seem to go down to 100Mbps...

I'm using ESXi 5... ur using esx? also my esxi is patched fully. (note: each patch is cumulative so just install the latest build #)
 
Sigh. So couple months ago some kids broke in my house and trashed some stuff. One thing is my Norco case was pushed onto ground... I sorta laugh thinking someone tried to pick it up. Anyways I have two raidz2 arrays, one 8 drive and one 12 drive. Falling on ground two drives died right away, one in each array. Lazy me did not replace them and on Friday two more drives failed in the same array. Lame. Larger array is gone now. I have backups and probably lost little data but just a PITA.

To make things worse, Solaris didn't exactly handle it well. I ran 'zpool status' on the dead array and command just hangs and I cannot kill it. Annoyed I try rebooting Solaris, hoping it would give up on trying to read from the dead drives, or whatever it was hanging on. While booting up Solaris now fails at boot-image or whatever and drops me to maintenance mode. Another PITA. I cannot figure out what is going wrong, I don't see any errors/warnings anywhere.

After being in generally bad mood I said f-it and created a new VM for FreeBSD v9, thinking I might as well just switch to it. Didn't realize VMNET3 isn't supported in FreeBSD, or that FreeBSD v9 is still far behind in zpool verisons it supports, or that passing through LSI2008 controllers in ESX to FreeBSD is known not to work-- the WAR is disabling MSI and MSIX.

All that annoyed me as well so with no expectations I just reused an old Linux VM and installed ZFS kernel module (not fuse). WTF it imported array just fine. Scrubed the remaining array a few times, goes along at 500MB/sec which is fast as Solaris did. Everything worked surprisingly well. Also will maybe not have to deal with Solaris + SAS Expanders + SATA HDDs issues?

Who knows. Anyways going to try our ZFS On Linux for awhile. I read through the mailing lists for a bit-- the biggest thing I care about, data protection, has never been a problem it seems. Bugs people run into now seem to pretty specific stuff, nothing that I would hit. So got my fingers crossed.
 
Well I got some update:

Installed OI 151a_U3 on the bare metal on the same machine (directly on my SSD), and it works perfectly, getting around 80 MB/s then I start another transfer it start around 30 MB/s and they both balance around 40 MB/s after a min.

So everything is the same, except that it's not running under ESXi5.

Any idea anyone?
 
Well I got some update:

Installed OI 151a_U3 on the bare metal on the same machine (directly on my SSD), and it works perfectly, getting around 80 MB/s then I start another transfer it start around 30 MB/s and they both balance around 40 MB/s after a min.

So everything is the same, except that it's not running under ESXi5.

Any idea anyone?

other discussion, same question, some opinions
http://forums.servethehome.com/showthread.php?533-SMB-reads-are-slow-on-OpenIndiana-NAPP-IT
 

Thanks Gea, question is somewhat different because I am experiencing slow write. I am also not using 3rd party tool to copy file, just the default Win7 file transfer.

I guess the solution you were refering to is:

"i found a solution by setting flow control in openindiana for the nic card to "rx" only".

I will google how to do that and give it a try. Thanks you. Will let you know if it fix my issue.

I might also try to passthrough a Real Intel Nic if it dosen't work.
 
Well, it's not the flow control:

LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 flowctrl rw no bi no,tx,rx,bi

also, I tried vmxnet3 and it does the same, passing through my onboard intel nic dosen't seem to be detected on the OI, so can't test that..

I also found that my laptop dosen't exhibit the same issue (onboard Realtek), but my win2k8 with a marvell controller, and my win7 with realtek are affected..

Next step will be to try with Solaris 11, and maybe freebsd (NAS4free, freenas)...
 
What is the status of the not working disk spin down on Solaris 11 ? Any idea how big an issue it is for disk lifetime?
 
Last edited:
What is the status of the not working disk spin down on Solaris 11 ? Any idea how big an issue it is for disk lifetime?

status: not working
I expect a higher failure rate with a lot of spin down/up
 
Well, it's not the flow control:

LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
e1000g0 flowctrl rw no bi no,tx,rx,bi

also, I tried vmxnet3 and it does the same, passing through my onboard intel nic dosen't seem to be detected on the OI, so can't test that..

I also found that my laptop dosen't exhibit the same issue (onboard Realtek), but my win2k8 with a marvell controller, and my win7 with realtek are affected..

Next step will be to try with Solaris 11, and maybe freebsd (NAS4free, freenas)...

did you install the latest drivers for the client nics??? (not the windows default ones)
 
yea mang I had weird issues my with baremetal windows 7 machine with default realtek drivers talking to open indiana. I never had any network issues before until I started using OI. so it threw me for a while until I installed the realtek drivers from my mobo manufacturer and all my probs went away
 
yea mang I had weird issues my with baremetal windows 7 machine with default realtek drivers talking to open indiana. I never had any network issues before until I started using OI. so it threw me for a while until I installed the realtek drivers from my mobo manufacturer and all my probs went away

Thanks you so much! And everyone else too.

Updating the driver for the marvell nic fixed the issue.. But like you said, i've been using that server at max speed for 3 years and it;s only now that the nic show issue only with OI.... weird..

vmxnet3 nic also seem to be working reliably, it use much less cpu.
 
I'm re-looking at the way I am setting up my vdevs in a pool, I might go towards mirror instead of raidz1 to gain in IOPS and losing a bit in capacity.

Are Mirrored vdev able to detect AND correct error or only detect them? That would be the only reason I find to stay with raidz1.

Thanks you
 
I'm re-looking at the way I am setting up my vdevs in a pool, I might go towards mirror instead of raidz1 to gain in IOPS and losing a bit in capacity.

Are Mirrored vdev able to detect AND correct error or only detect them? That would be the only reason I find to stay with raidz1.

Thanks you

The ZFS self-healing feature works with Raid-Z and mirrors.
If ZFS detects an error during read or scrub, it reads the data from the other mirror.
Due to checksums, ZFS can identify the faulty data and repair from the other disk.

No danger to overwrite good data with bad data like with other mirror solutions
 
I'm thinking of trying this over FreeNAS, just curious if there are any people who are having a trouble free setup?
All the questions posted here makes it seem like everyone is having problems?
Or maybe for most people it's working perfect and they aren't posting?
 
Of the four fileservers I personally am aware of running OI+napp-it none have had any issues whatsoever.
 
Basic question...

Just checked my OI+Napp-It status and one of my drives in my RAIDZ2 array is showing

Error: S:0 H:27 T:30

I'm assuming this is indicating a failing hard drive (of course while I'm out of town). The drive in question is an older WD20EADS drive.

Can I replace the drive with a 3TB, or do I need to replace with another 2TB since it is a live drive (non-hotspare)? And related to that.. any recommendations on drives :) The pool is mixed between Hitachi HDS72202 and WD WD20EADS drives.

Slightly more difficult question...

I already had a spare Intel X3430 and 8 GB ECC RAM.. and got my hands on a Supermicro X8SIL-F motherboard locally for a steal ($80). While I'm currently running an All-in-one, with the spare parts I'm pretty sure I've decided to break out my storage to a seperate box instead of building another ESXi host since I've already got more CPU and memory capacity than I need. Especially since that would free up 10 GB RAM on my primary host (currently reserved for the OI install).

What is the easiest way to move from a virtual OI+Napp-it install to a physical? I know I can export/import the pool info, but what about other configurations (iscsi, NFS, ZFS folder settings,etc.). I know I could recreate them pretty easily... but export/import would be quicker :)
 
The hard error can mean, your disk will die in 5s or 5 years. These errors can indicate a problem but sometimes its only a sector marked as bad with no more consequences.

You can replace a 2 TB disk with a 3 TB disk if your controller supports 3 TB disks.
Sometimes you have problems replacing older 512b disks with 4k ones

about virtual -> physical
I would just reinstall OI and import the pool. Most relevant settings (shares etc) are stored in the pool. You must re-set user, jobs and iSCSI settings
 
I have read a lot here in the forum about ZFS and hardware recommendations. I´m using Nexentastor at the moment with 4x 2TB disc in a RaidZ.

I´m planing a backup server with 4x3TB. For performance reasons this should be a 2x2TB mirror.

The AMD E-350 and E-450 seems to be used by some people here but I can´t find any real performance informations about the transfer rates. One is writing about nearly 200mb read and 130mb write at a 32GB file. Others are complaning about write speads about only 30mb/sec.

I´m able to use every MB of the transferrate because the backup server will have a 4gb fibrechannel card installed. But something around 100 - 150mb/sec would be fine...

Does anybody have some real read and write rates for a e-350/e-450?
 
I have read a lot here in the forum about ZFS and hardware recommendations. I´m using Nexentastor at the moment with 4x 2TB disc in a RaidZ.

I´m planing a backup server with 4x3TB. For performance reasons this should be a 2x2TB mirror.

The AMD E-350 and E-450 seems to be used by some people here but I can´t find any real performance informations about the transfer rates. One is writing about nearly 200mb read and 130mb write at a 32GB file. Others are complaning about write speads about only 30mb/sec.

I´m able to use every MB of the transferrate because the backup server will have a 4gb fibrechannel card installed. But something around 100 - 150mb/sec would be fine...

Does anybody have some real read and write rates for a e-350/e-450?

A backup server with mirrors (single user, mostly large sequential writes) is not CPU limited (ok forget smallest Atom) and mostly not RAM limited if RAM is not too low (say 4 GB+). In such a case I have heard of 100 MB/s rates fom HP Microserver users on Gb Ethernet.

The smaller the writes, the worser the reported values. In these cases, you need I/O power. In extreme with pure sync writes and small files it can fall down to 1/10 of sequential values and lower. If you have multiple user/ backups at once it could become really worse with slow disks. If you add possible problems with some Nics/ drivers on Server and Client it could be quite unpredictable ...

But If you need really premium values, there is nothing better than: RAM, high speed disks and CPU-Power in this order

What I want to say: It depends on hardware/ disk and use case
 
Looks like I am having some hardware issues.

OI 151a+ Latest Napp-It

Dual x5650 (Hex core)
Supermicro X8 board
192GB DDR3
Host
2x 60gb Intel 520 OS
JBOD1- Controller 1
2x 128gb Crucial M4 Zil
2x 512gb Crucial M4 L2ARC
20x 1.5TB Seagate 7200.14 Sata 3
JBOD2- Controller 2
24x 1.5TB Seagate 7200.11 Sata 2
Disks are setup in a large mirrored pool.
Bonnie refuses to run, DD bench gives nothing.
Seeing a max of 40mb/s read/write. After a fresh boot, I can easily max my gigabit connection around 110mb/s. As I transfer files around (100mg-5gb) it slows to a snails pace of 10-30mb/s.
The controllers are on the latest firmware, latest bios, clean install. Nexanta gives the same performance. Out of idle curiosity I am going to remove the 7200.11 drives and rebuild the pool just on JBOD1.
 
Last edited:
*Update- Removed JBOD2- 89-96Mb/s over gigabit. Going to setup our Infiniband setup tomorrow and see what happens with some larger files.
 
How does zfs stripe data over vdev of various size? Let s say i have 2x 2tb in mirror and 2x 500gb in mirror in a pool ?

If the 2x500gb are slower will they slow down the pool in throughput or iops?
 
How does zfs stripe data over vdev of various size? Let s say i have 2x 2tb in mirror and 2x 500gb in mirror in a pool ?

If the 2x500gb are slower will they slow down the pool in throughput or iops?

Since the pool stripes its data over the vdevs if the 500GB mirror is slower than the 2TB mirror then yes, it will slow the pool down in terms of read/write speed vs. having another 2TB vdev instead... It's generally recommended to keep the vdevs at about the same size and performance, iirc.
 
How does zfs stripe data over vdev of various size? Let s say i have 2x 2tb in mirror and 2x 500gb in mirror in a pool ?

If the 2x500gb are slower will they slow down the pool in throughput or iops?

The extreme of such a situation is a full pool where you add a new vdev.
All new writes and updated data (due to CopyOnWrite) must go to the new and free vdev.
Iif there is enough space free on the other vdev(s) all new writes/updates are striped over all vdevs, ZFS can handle such a situation with unbalanced pools the best as possible.

If all vdevs are of the same size, the situation tends to a fully balanced pool over time where all reads/writes are striped over all vdevs,
otherwise some reads/writes are done only from part of the pool (same situation like with 'golden number of data disks without counting redundancy in a vdev=128,64,32,16,8,4,2 where all disks are used for read/write or only a part)
 
Last edited:
Does that also apply to mirrored vdev? I was planning at going in increment of 2 2tb drive in mirror for iops and capacity while creating many vdev over time for additional iops and capacity?
 
Having run OpenIndiana + Napp-It the last 9 months and being very pleased with it, I decided to replace my motherboard with one that would support 32GB of RAM. Since I would need to reload the operating system, I became tempted with the lure of VAAI that Nexenta offered. So I exported my pools and installed Nexenta.

Right off the bat, my pools could not be exported because OpenIndiana partitioned the drives with GPT (as they should). No problem, I have a backup of my data and don't mind restoring it.

Next, I ran into problems with the initial wizard. It would get to a certain point and completely freeze. Only a reboot would bring it back. Turns out, an update of Nexenta is needed to work out these bugs so I run "setup appliance upgrade."

Now I can get through the wizard and set up my networking. Two of my NICs were aggregated before so I did the same with Nexenta. Oops, no jumbo frame support on aggregates. OK, so I'll change my NFS vmkernel ports on my two VMware hosts back to MTU 1500.

Now I get my pools created along with a couple iSCSI and NFS datastores, both of which connect to VMware great. Then I create my CIFS file share and prepare to restore my file share to it. Even though I chose to grant guest read/write access, there is still a password required to access the share. *Sigh* Guess I'll need to update my group policy to include the credentials.

Now I'm ready to begin the long task of restoring my VMs and file share. After migrating about 4 VMs to my 8 disk mirrored pool, I find write latency times are higher than what they were on OI, despite having sync disabled. Whereas they would average around 2ms when I had 12+ VMs running, they now average ~6ms with only 4 VMs running. Not happy.

As the restore of my file share reaches hour 18 (2.5TB of data) suddenly it errors out. The Nexenta CIFS share is no longer accessible! My iSCSI and NFS VMware datastores are still up, but CIFS is completely dead.

So, I'm now exporting my pools and reloading with OI and Napp-It. I've learned my lesson and won't stray again!! :)
 
Gea, is 1 also a member of the golden number of data drive in a vdev? :)

Such as vdev=128,64,32,16,8,4,2,1 ?

For example in the case of mirror or raidz2 with 3 drive?

I have 4 new WD Black 2TB right now, and the total case capacity is 13 drives.

Also, when doing testing, and wiping the OI machine and re-importing the POOL I got error saying that SMB import failed, is there anyway to prevent that, possibly by re-creating the same user before importing?
 
Last edited:
I've built my pool in 2 mirrored 2x2TB and performance is pretty good around 250MB/s write and 300 MB/s Read.

Does OI support TRIM let's say I was to create another pool with 2x SSD drive in mirror?

also

I've noticed that OI vm take 100% cpu (seen from the esxi performance cpu) when doing a 1Gbps filetransfer, while the system monitor inside the VM show about 10-15% CPU for each core, I have 4 vcore assigned to OI, from a Intel 3930K (6Core/12Thread) cpu.

Thanks
 
Last edited:
Golden number= 1
these numbers are only important, if you want to stripe data over all (>1) disks.

SMB use ACL. If you import a Pool, you have only UID/GUID
If there is nor mapping, you may get errors.
Reset ACL then to current users

There is currently no trim support in OI
 
My laughs are becoming more and more hysterical... just came home from work, the NAS responded, I thought well that's good. Wanted to shut down the box to test each single drive, as coolrunnings suggested. Checked the disks first and oh-oh... Hard Errors (the one with T:1 is the spare disk).Tried to boot, but no, it wouldn't let me - I looked at this for 15 minutes until I pulled the plug. Noticed lots of stuff like that on the console. It comes back up, and this time, everything looks normal to me. I also guess a restart of the system resets the error counter?

:(

Well, I shut it down now again, and will test all the drives. Will do performance tests later. Unfortunately, Just found out that the test controller I brought home from work is an HP P400 and does not support JBOD... It's just not my day... sigh.

Greets,
Cap'
Captainquark,

I was wondering if you've made any headway. I am experiencing the exact same symptoms as you are. I have the same pool setup as you and am using the same drives.

Random drives go offline and the hot spare is activated. Recently I had two in one mirrored vdev go offline - the second halfway through the resilver. Luckily I was able to force one back online and not lose any data.

As for me, when a drive would go offline I would yank it, and throw it in another box and test it. I have found no faults on any of the drives that go offline.

I'm hoping, perhaps, that together we can figure this out. I thought about purchasing a support contract but from what I read Oracle may not help with these kinds of issue as it may not be a fault of Oracle's code. There are third-party support companies, but I'm not ready to spend $150/hr for support... yet.

I will be ordering a different sas controller to see if it changes. It will be a LSI 9211-8i as it not only was installed in computers direct from Sun but is also on the HCL. Though both the supermicro and LSI use the same chipset.

Also, have you had any PCIe device failures on the controller? You can see it in "fmadm faulty". I got one for the controller and moved it to a different slot and have yet to see it come back.

Also, what kind of transfer rates are you getting? I'm only seeing about 90MB/s write and 250MB/s-300MB/s read.

My specs are as follows:

Intel S500PSL
Dual Xeon X5355
16GB RAM
Supermicro AOC-USAS2-L8i flashed to IT mode w/ phase 10*
Adaptec 39320D w/ IBM LTO3 drive

With the exception of the supermicro controller, everything is on the HCL.

*I tried phase 10, 11, 12. I seemed to see more failures with phase 11 and more with 12 for whatever reason. Also, I noticed that phase 12 seems to display IR mode everywhere and has RAID options, though the drives are still readable by the OS.

What kind of hardware are you using. Truth be told, I'm wondering if it's the drives themselves that are causing this issue.

Also, looking at your device IDs it looks like multipathing is enabled. I have it disabled on my setup though I didn't see any change.
 
Captainquark,

I was wondering if you've made any headway. I am experiencing the exact same symptoms as you are. I have the same pool setup as you and am using the same drives.

Random drives go offline and the hot spare is activated. Recently I had two in one mirrored vdev go offline - the second halfway through the resilver. Luckily I was able to force one back online and not lose any data.

As for me, when a drive would go offline I would yank it, and throw it in another box and test it. I have found no faults on any of the drives that go offline.

I'm hoping, perhaps, that together we can figure this out. I thought about purchasing a support contract but from what I read Oracle may not help with these kinds of issue as it may not be a fault of Oracle's code. There are third-party support companies, but I'm not ready to spend $150/hr for support... yet.

I will be ordering a different sas controller to see if it changes. It will be a LSI 9211-8i as it not only was installed in computers direct from Sun but is also on the HCL. Though both the supermicro and LSI use the same chipset.

Also, have you had any PCIe device failures on the controller? You can see it in "fmadm faulty". I got one for the controller and moved it to a different slot and have yet to see it come back.

Also, what kind of transfer rates are you getting? I'm only seeing about 90MB/s write and 250MB/s-300MB/s read.

My specs are as follows:

Intel S500PSL
Dual Xeon X5355
16GB RAM
Supermicro AOC-USAS2-L8i flashed to IT mode w/ phase 10*
Adaptec 39320D w/ IBM LTO3 drive

With the exception of the supermicro controller, everything is on the HCL.

*I tried phase 10, 11, 12. I seemed to see more failures with phase 11 and more with 12 for whatever reason. Also, I noticed that phase 12 seems to display IR mode everywhere and has RAID options, though the drives are still readable by the OS.

What kind of hardware are you using. Truth be told, I'm wondering if it's the drives themselves that are causing this issue.

Also, looking at your device IDs it looks like multipathing is enabled. I have it disabled on my setup though I didn't see any change.

Check your power supply.
 
Check your power supply.

Thanks for your reply. I have no other indications the power supply is defective. It's not that old, or an 'el-cheapo' either. What, may I ask, should I look for?
 
Last edited:
is there a good general openindiana forum thats active? I'm decent with linux but looking for some direction on how to install certain apps to assist with system management/monitoring.
 
Captainquark,

I was wondering if you've made any headway. I am experiencing the exact same symptoms as you are. I have the same pool setup as you and am using the same drives.

Random drives go offline and the hot spare is activated. Recently I had two in one mirrored vdev go offline - the second halfway through the resilver. Luckily I was able to force one back online and not lose any data....

Hey liam137,
Thanks for checking back on this. Unfortunately, I could not make any progress as I didn't get back my mobo yet. Apparently, they've sent it in to the supplier in the Netherlands (I'm from Europe) to have it "repaired"... I don't know what to expect... maybe a plant in the middle of nowhere with lots of people re-soldering mobo's... :) Well, I will be selling it on ebay anyway when it gets back, as I decided that I want to use SAS as well, and therefore I ordered a Supermicro X8SI6-F. That will give me 8 additional SAS ports to the 8 ports on the LSI 9211-8i card. So I have ordered the X8SI6-F, but it will not be delivered before May 11th. :(
BTW the LSI card has been exchanged in 3 days, now waiting on my desk to be flashed again with the IT firmware. Also, did you see there's a new version P13?

I took out every single drive of my setup (7 Samsung 1 TB) and had them tested in another computer with the Samsung (Seagate really) testing tool. I also took the long tests, looked at SMART data... NOTHING. Since it affected over time all my disks in a rather random sequence, I assume it has nothing to do with the disks.

I have not looked in "fmadm faulty" for PCIe errors. I have switched slots out of pure desparation, and it got a lot better (not perfect though).

How did you measure the transfer rates? I have never managed to get Bonnie running... and the dd stats were very odd, but I don't remember them by heart. NEVER that high, I would be very satisfied with this. I just remember that copying from Windows 7 onto the NAS dropped quickly down to 10 MB/s and stayed there or even below it. It took me AGES to copy my 2+TB of data onto it...

My Setup will be
Supermicro X8SI6-F mobo with LSI2008 SAS-Ctrl for SAS Drives
Xeon L3426 (low power consumption)
12 GB of RAM, maybe upgrading to 24 once everything works as expected
LSI 9211-8i for SATA-Drives
7 x 1TB Samsung SATA

I did not quite understand what you mean by "phase" 10, 11, 12? Also I did not understand exactly what you mean with the multipathing? Can you please elaborate?

As for the Support question, I would never ask Oracle for support. This entire thing is pretty specific to home NAS'es and I doubt they will support you with this. Especially if it does not run on SUN hardware. Best thing would be if _Gea would offer some (paid) support, but I doubt he has time for this... _Gea?

About the Power Supply, I think you should check if it has enough power to supply that many devices. Maybe try connecting less disk drives. If your power supply does not have enough power to "feed" all devices accordingly, it can create very odd behaviour.

Best regards,

Cap'
 
Back
Top