OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Just replying because I have the same question. Spun down, my array is about 100w less of power. When I gave it a shot, clients fail to mount shares right away. Can't have this.
 
my filers are always up, but i suppose your options are either
a smaller second pool (ex a mirror) for critical data without spin down
or a job that disables spindown on use time.
 
my filers are always up, but i suppose your options are either
a smaller second pool (ex a mirror) for critical data without spin down
or a job that disables spindown on use time.

Thanks _Gea for your response.
Unfortuneately the "important" data is the larger set (the movies and music collection).
It is not in use on a regular basis, that is why spin-down makes sense in terms of energy savings.
Which makes both aspects, energy efficiency and easy access to the data, a direct link
to the WAF...when a media player won't connect at once or even needs a reboot
to retry, this is not funny.:D

...regarding that job you mentioned...maybe I can introduce/loop-in a short "ping-mount"
whenever my dhcp server issues an IP (for a client upon start).
That should pre-trigger the spin-up of the array and reduces possible timeouts when
the media player finally is up.
 
Or maby you can chante the timeout on the client?

For my embedded media players, this is not an option I am aware of in the firmwares.
Most devices will silently give up and you can re-initate the access to the share.
One model does need a reboot in order to get to the shares.:eek:
I also use XBMC in windoze...when the underlying win does not connect to the shares upon boot, XBMC will not see them untill you manually browse to the shares with explorer.

My linux boxes don't have a problem connecting or with timeouts.
All in all this is problem with ease of use & comfort...
 
I recently built up a server using OpenIndiana and Napp-It, and I'm seeing some weird behavior with the low cap alert.

I have a 40GB SSD drive with the root pool on the first 30GB, and the rest will be sliced up for ZILs.


------------------------------
zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 31.2G 14.8G 16.4G 47% 1.00x ONLINE -
zpool0 65T 4.53T 60.5T 6% 1.00x ONLINE -

Pool capacity from zfs list
NAME USED AVAIL MOUNTPOINT %
rpool 27.5G 3.29G /rpool 11%!
zpool0 3.75T 49.2T /zpool0 93%



Why, when I have over 16GB free on my rpool do I see errors saying I'm down to <15%?
Any suggestions?

-e
 
Last edited:
I recently built up a server using OpenIndiana and Napp-It, and I'm seeing some weird behavior with the low cap alert.

I have a 40GB SSD drive with the root pool on the first 30GB, and the rest will be sliced up for ZILs.


------------------------------
zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
rpool 31.2G 14.8G 16.4G 47% 1.00x ONLINE -
zpool0 65T 4.53T 60.5T 6% 1.00x ONLINE -

Pool capacity from zfs list
NAME USED AVAIL MOUNTPOINT %
rpool 27.5G 3.29G /rpool 11%!
zpool0 3.75T 49.2T /zpool0 93%



Why, when I have over 16GB free on my rpool do I see errors saying I'm down to <15%?
Any suggestions?

-e

zpool lists reports raw pool capacity
= raw sum of all vdevs - not regarding redundancy or anything else)

use zfs list and look at the parent folder.
that gives the really available space.

In your example
Your total space = used + avail = 27,3+3,29=30,59 GB
From this around 30 GB you have aroung 3 GB free ~ 10%

Thats the way its calculated
 
If you open the file and go to line 1221 in that file, you will find nearly the same code (only difference is that in the original file, the nfs-share-name is always the same)

It should be easy to replace the old code with the new one.
Keep the original file so you can go back in case of syntax errors.

On next update, the file will be replaced, so no problem.

Thanks, _Gea, for implementing the fix in version 0.8c - works like a charm. I have to admit I had to wait for your fix, as I didn't want to harm things on my system :)

Best regards,
Cap'
 
Hi,

what is the easiest way to test the Email Alert Feature in napp-it? I have to servers running napp-it. On one it is possible to just click "run now" in the auto tab, where the alert job is registered.

On the second system this doesnt work. How is is possible to test, if it is really working?


Thanks
 
Hi all,

Sorry for the long post... but honestly, I am kind of desperate by now, and hope anyone can help me.

It's now 4 months I am fiddling and tuning on my Sol 11 based home NAS enhanced with Napp-it, to no avail. I seem to be unable to get it to a stable and reliable state, although I followed lots of recommendations and tips in this forum and on the napp-it website.

My Hardware:
Supermicro X8SIE-F mainboard
Xeon L3426 CPU
12 GB ECC RAM (Kingston)
LSI 9211-8i controller flashed with IT firmware (v.12 by now)
7 Samsung 1 TB SATA-II Harddisks 7200rpm for datapool on LSI controller
2 mirrored disks 250 GB for OS (followed Constantin Glez's guideline) onboard
3way mirror (1 spare) 750 GB onboard for VMWare pool

Currently using Napp-it 0.8c.

I have two problems: reliability and performance

Reliability
In a first attempt, I created one large Raidz1 pool for data, one of the disks being spare. From the beginning, I had frequent lockups (every few days,see earlier posts from me) which I never could really solve. It was less frequent when I rebooted the box every 2 or 3 days. Lock-up symptoms were: 1. bad response time, 2. share is visible but not accessible under windows, 3. at some point I tried to reboot the NAS, which went down only half-way and then stopped responding, 4. I had to cold boot, 5. see here, here and here. Then I had to wait until resilvering finished, clear the errors and then it presented itself as "healthy" again, but only for a very short time until it started all over again. Lately, even that was not possible anymore, so I had to destroy the pool, flashed the firmware to the newest version v.12 and recreated a pool using 3 mirrors and 1 spare. On my first attempt to copy data on it, it was really fast (like 70 MB/s or so), but after like 1 hour, dropped to 7-14 MB/s and it took me now two days to copy roughly 1 TB of data on it (however, I didn't loose the pool ever since, that's about 3 or 4 days now). I don't know if it stays reliable now, so while waiting, I want to troubleshoot performance.

One last thing to say - I always lost the pool that was connected to the LSI controller card ONLY, never the two pools that are connected onboard - not one single time.

Performance
Well, after reading some articles on Constantin's blog, I was pretty sure that 3 mirrors ought to be (much?) faster than one large Raidz pool. When trying to copy data on it from my local harddisk in my Windows box, the performance was really slow (like 6-9 MB/s over a larger period of time). HDD LED on my PC showed activity for a few moments and then paused for 18 seconds (!), then sent some more data and paused again for about 20 seconds and so on. This was the case when trying to copy from the SATAII-conventional disk, and from my SATAII-SSD, so I guess it must have something to do with the NAS not being able to accept the data fast enough and/or not acknowledging receipt. The CPU on my NAS has 99% idle time. When I looked, it said it has 3.5 GB of physical memory available. Honestly, I do not understand at all this behaviour.
I then tried to copy a 370GB-vmdk from a local disk on my VMWare ESXi 5 box to the VMware pool on the NAS (it's an exported NFS share). After 10 hours of copying, it said it did 6 % so far...

As you can see, the situation is really bad... any hints really appreciated.

Thanks and regards,
Cap'
 
Dear Captainquark,

1. Seeing that you have been struggling for the pass 4 months with various difficulties in your setup
1.1 As noted even by yourself, only the LSI-portion has problem, maybe try a different HBA?
1.2 Other performance complication,

Question : Are you "determined" to endure this troubleshooting / software compatibility issue for many months more ? or are you perhaps open to different combination? Just genuine inquiry. Nothing else.
A. Can you please identify the exact power supply in use?
B. Are you copying using Windows 7/Vista/XP Explorer ?
C. Are you able to FTP file from Windows/Mac/Linux/BSD client to Sol-11 Host with say 20Gbyte size through gigabit ethernet and report average transfer rate result? This will identify whether it is special error during file copy, OR fundamental issue with the entire Sol-11 host machine storage. FTP service is usually very reliable on Unix host so there is no complicated software issue to deal with, just normal file transfer activities.

optional : sometimes the problem is we want to hold on to...if this setup has so many headache for you, maybe time to let go some of the hardware???
 
Hi,

what is the easiest way to test the Email Alert Feature in napp-it? I have to servers running napp-it. On one it is possible to just click "run now" in the auto tab, where the alert job is registered.

On the second system this doesnt work. How is is possible to test, if it is really working?


Thanks

use menu Jobs- Email - smtp test to check send functionality

remove a disk and try to access to check if you get alerts
(alert checks against zpool status where errors are reported after unsuccessful access)
 
Hi all,

Sorry for the long post... but honestly, I am kind of desperate by now, and hope anyone can help me.

It's now 4 months I am fiddling and tuning on my Sol 11 based home NAS enhanced with Napp-it, to no avail. I seem to be unable to get it to a stable and reliable state, although I followed lots of recommendations and tips in this forum and on the napp-it website.

You use one of the configs that should be really stable. So you need to identify the part that is not
working properly. Bugfixing in this case means excluding possible reasons

I would first look at the Bios for Active State Power Management
and disable it (if it was enabled). That fixed for me a problem with another X8 board

As all of your problems seems to be tied mostly to the LSI controller, i would check/ replace cables.
If this does not help, you need another controller to verify if the problem is the controller

It could also be one of the disks. You may discover with menu system - statistics - discs
but this does not detect failures by time. You may try to build two test pools from your disks
and compare them.

You may also remove half of the RAM and check if the problem is gone,
then use the the other half.

You may also use another (Intel) nic to check if its a Nic problem.


the same with performance.
Can you check performance against a pool on the LSI and against a pool on AHCI Sata

regarding ESXi and NFS
On performance problems, you can disable sync on the NFS shared folder and
recheck performance to see if its due to ESXi sync writes.

Also connect your PC directly with a crossed cable, check against another PC
Do not use any copy tools on your PC. Just copy a large file and measure time.
(ex zip your hard disk or do something similar to create a file > 10 GB)
 
The problem with the ipmiview app is the "discovery" function never finds the host. No idea why and a straight up browser connect works fine. Refreshing my memory, not only does the 'discover' function not find the host, if I give the explicit IP, I get some error about 'the device is offline, connect anyway?'. If I say yes, I get the a progress bar for a bit, then a fail.

I figured this out last night. I happened to be upgrading some stuff in the server and noticed the IPMI firmware on the mobo was rev 1.01. Looked at the SM site and saw the latest is 1.27! I flashed it via the web browser and voila - the app now works.
 
Captainquark:

maby try installing sol11 directly to NAS without ESX and try that... That way you will have one less thing that can make problems...
 
Guys,

Thanks a lot for your replies, I appreciate you are trying to help me. When I came home from work today, the NAS didn't respond again. I tried to reboot but it would go down only half way, so I had to cold boot it. When it came up, it said all Pools were healthy... well, that was new :p

@levak, I already have installed Sol11 directly on the hardware, I am not having the All-in-one solution. My VMware machine is a different box, and I have no intention to mix it.

@lightp2,
Well, I am determined to give it another few shots, but I will not try for another 4 months. Even though I am not a Solaris or *nix Expert, I do work in IT infrastructure for almost 15 years and never had such insistent problems with any system so far. At some point, I will have to ask myself if this is really a good setup for me, but I haven't come there yet. ;)
To answer your questions:
A. Good point. My PSU is a BeQuiet! Straight Power 680W.
B. I was trying to copy with Windows 7 64bit Explorer, and with ViceVersa Pro, with exactly the same results.
C. Interesting idea. I couldn't try yet with FTP, as I cannot activate the FTP service (I click online - submit under Services, and yet it stays on "disabled" on the overview page). I will read my way through and then try in the next few days.

I don't have problems with letting go hardware, it's more a financial thing. So far, I invested a lot of money into this NAS, I can't simply throw that away and buy something else.

@_Gea,
Thanks for the hint with Active State Power Management, but that was already disabled.
As per your proposals regarding replacing cables and controllers, I need to find such hardware first. Obviously, I can buy the cables, but a controller is a different thing. As I said, I have spent a lot of money on it, and hesitate now to invest even more, maybe finding out that I can't solve the problem and then sitting on a pile of expensive electronic waste... So I will try to borrow a controller from someone else.

As you proposed, I have now disabled sync writes on the vmware nfs export, and I am about to try to copy data to it. So far, I have the impression that it's much faster than before, but I will have to wait until it's finished.

Again, thanks a lot for your ideas. I will try them out in the next few days and let you know the outcome. If anybody has even more ideas, please bring them up!

Best regards,
Cap'
 
On your Sol11 machine, try doing some local "dd" tests to measure read I/O performance of individual drives.
Make sure you test every data drive, but one at a time (you can get onto multiple drive testing later - as a first step you need to find out if there is an inherent problem with any of the individual drives/controllers first)

If you are unsure, first get a list of drives using the "format" command.


then in one window

#iostat -xn 5 (which will print disk performance stats every 5 secs)


then, in another window, do a read test

#dd if=/dev/rdsk/cXtYd0sZ of=/dev/null bs=64k (substituting the correct XYZ numbers - Z is most likely 0 for a GPT labelled disk)

After a minute or two, you can hit CTRL-C (you'll have had plenty of time by then to see the read I/O performance in the "iostat" window) and move onto the next drive.




Also, you aren't running compression and/or deduplication and/or encryption on your zpools are you?
I only ask as you say that you have 3.5GB available on a 12GB machine - so something is using 8.5GB of memory. Any idea what? What are you using to determine "available memory"?


If the drives are OK on read, the next thing is to test local zpool performance, and then the network.
It's a process of elimination, and it can be slow and frustrating, but take it one step at a time and eventually you should be able to pinpoint the issue!
 
After I changed network interface napp-it stopped working, it wont load. I have set it to the same IP as old interface and removed the old interfcae from the PC. I updated napp-it but still, wont load.
 
Approaching my wit's end and haven't found the answer here or anywhere else I've googled.

First of all, pertinent hardware:

Intel S1200kp motherboard, latest firmware
i3-2120T processor
8 GB Kingston unbuffered ECC RAM
LSI SAS9212-4i4e HBA, latest IT firmware
1 Mushkin 120GB SSD
5 Seagate ST2000DM001 SATA drives

I'm installing OpenIndiana oi_151a from a USB stick, prepared using OpenSolaris Live USB Creator (I've tried this twice so far with the following results). According to the OI Wiki, the install media should detect that I have >1GB RAM and give me x64 installation options. However, GRUB presents only x86 (32-bit) options, and my >2TB hard disks are not dealt with, as only the 32-bit OS is installed. Gparted doesn't see them either, of course.

I'm pretty unfamiliar with Solaris, so I'm not sure what I can troubleshoot or how I can correct this. I can't believe I'm the only one who has experienced this, but I can't find any clues online.

What should I try next? What additional information should I post here? Any advice?

Many thanks, OI gurus!
 
Approaching my wit's end and haven't found the answer here or anywhere else I've googled.

First of all, pertinent hardware:

Intel S1200kp motherboard, latest firmware
i3-2120T processor
8 GB Kingston unbuffered ECC RAM
LSI SAS9212-4i4e HBA, latest IT firmware
1 Mushkin 120GB SSD
5 Seagate ST2000DM001 SATA drives

I'm installing OpenIndiana oi_151a from a USB stick, prepared using OpenSolaris Live USB Creator (I've tried this twice so far with the following results). According to the OI Wiki, the install media should detect that I have >1GB RAM and give me x64 installation options. However, GRUB presents only x86 (32-bit) options, and my >2TB hard disks are not dealt with, as only the 32-bit OS is installed. Gparted doesn't see them either, of course.

I'm pretty unfamiliar with Solaris, so I'm not sure what I can troubleshoot or how I can correct this. I can't believe I'm the only one who has experienced this, but I can't find any clues online.

What should I try next? What additional information should I post here? Any advice?

Many thanks, OI gurus!

are you sure x86 is refering to 32 bit, is there a 32bit version if OI 151? I though it is 64 bit regardless and x86 refers to an intel-type cpu vs spark.

on my system (solaris 11) I have

uname -a
SunOS name... 5.11 11.0 i86pc i386 i86pc Solaris
 
are you sure x86 is refering to 32 bit, is there a 32bit version if OI 151? I though it is 64 bit regardless and x86 refers to an intel-type cpu vs spark.

on my system (solaris 11) I have

uname -a
SunOS name... 5.11 11.0 i86pc i386 i86pc Solaris

No, the installer will install either depending on how much memory it detects. But despite the 8GB I have, it's only installing x86 (32 bit).
 
To tell which kernel you are running:

# isainfo -b


Try Solaris11 or OpenSolaris - do they only see 1GB or is the issue specific to OpenIndiana?
 
Captainquark: I've read here and elsewhere of people who have a single drive cause massive slow-downs. In my experience, drives that even have 1 re-allocated sector showing up in their SMART data are immediate replacement candidates. Since you have another machine handy, I'd suggest getting a copy of the free drive monitoring software called Acronis Drive Monitor. Shut down the NAS and pull each drive on the problematic pool and hook them to your desktop one at a time and read the SMART data on them. Worth a shot and all it costs is a tiny amount of time... Also, if you need a good source on replacement cables to test, PM me.
 
To tell which kernel you are running:

# isainfo -b
32-bit

Try Solaris11 or OpenSolaris - do they only see 1GB or is the issue specific to OpenIndiana?

Great minds think alike, although I was going to try Ubuntu first. But Solaris 11 installs properly. All disks are recognized (of course, since Sol11 is 64-bit only). Same USB stick and installation method as with OI.

Problem with the Illumos kernel, do you think? Or something else?
 
No, the installer will install either depending on how much memory it detects. But despite the 8GB I have, it's only installing x86 (32 bit).

As far as i know, the boot process is currently always 32 bit so x86 do not refer to 32 or 64 bit.
This will change in future to support large boot disks

see https://www.illumos.org/issues/1999

Maybee its a problem with USB booting (have not tried ever)
Can you try the default ISO and boot from a DVD drive?
 
After I changed network interface napp-it stopped working, it wont load. I have set it to the same IP as old interface and removed the old interfcae from the PC. I updated napp-it but still, wont load.

If you use the live version, start Firefox locally and enter http://localhost:81
(will work independently from ip settings)

With a text-only server config, you can solve this problem only from CLI.
napp-it needs a working interface, you need to know the current ip

check "ifconfig -a" at console for current config
 
anyone running solaris 11.11? I had major issues joining it to a domain, it would not set computer account password, finally after x number of tries it took. Wish my pool wasn't v31 so I can switch to something based on illumos.
 
32-bit



Great minds think alike, although I was going to try Ubuntu first. But Solaris 11 installs properly. All disks are recognized (of course, since Sol11 is 64-bit only). Same USB stick and installation method as with OI.

Problem with the Illumos kernel, do you think? Or something else?


Well it certainly looks to be an OpenIndiana issue of some sort - though what I can't say TBH. It's an odd one for sure.

You could try forcing 64bit from the grub menu.

Select the first entry in the list "OpenIndiana Development oi_151x X86", hit "e" (for edit).
You'll see two lines - hit "e" again and then just replace "$ISADIR" with "amd64" then hit Return. Repeat for the second line, and when you've done both lines, hit b (to boot).
This should boot the 64bit kernel!

The first line on the screen after hitting "b" should tell you whether it's booting 64bit or 32bit!
 
Anyone using NexentaStor CE and happy with it? I'm considering changing from OI 151a to it because of the VAAI support for iSCSI.
 
As far as i know, the boot process is currently always 32 bit so x86 do not refer to 32 or 64 bit.
This will change in future to support large boot disks

see https://www.illumos.org/issues/1999

Maybee its a problem with USB booting (have not tried ever)
Can you try the default ISO and boot from a DVD drive?

No optical drive available, but I'll try this when I get home :

http://bradrobinson.blogs.exetel.co...s-x86-64-bit-installer-for-2TB-root-disk.html

Will let you know how it goes.
 
Hi guys,

I'm setting up a ESXi5 with all-in-one ZFS data storage.
I am going to use a Supermicro 8 hotswap SATA drive case with a socket 2011 CPU.
However I am lost at which ZFS implementation I should be using. I was initially intending to use freeNAS, however it seem to be slower than other ZFS implementations(might be wrong)., also their ZFS is lagging behind, but should be upgraded to v28 by the end of next month.

What are my alternative if what I want is:

1) Easy to maintain configure (web based)
2) Easy to monitor (possibly by sending email about failure)
3) Support thoses features: Windows Active directory ACL (Permissions), SSD caching, iSCSI, NFS
4) Long term support (no out of support product)

Does napp-it support windows domain integration with the free version?
 
Last edited:
1) Easy to maintain configure (web based)
2) Easy to monitor (possibly by sending email about failure)
3) Support thoses features: Windows Active directory ACL (Permissions), SSD caching, iSCSI, NFS
4) Long term support (no out of support product)

Does napp-it support windows domain integration with the free version?

Illumian and OpenIndiana are free. They "support" these features (beside support, for this you
need to pay Nexenta or Oracle). If you use the Solaris kernel-based SMB server instead of
SAMBA , you will get also domain-support with Windows compatible ACLs and access to
ZFS snaps via Windows previous version.

Napp-it is also free for end-users (you are not allowed to sell it/ distribute without asking).
It does not reduce any functionality but adds all that is needed to setup your server with
one command and basically manage your NAS/SAN via Web-UI including e-mail alerts.

I started napp-it as a hobby project in my free time but it has reached a level where further
development neads earnings. I was not payed for it from anyone to do. Special add-ons
from me (and others) can be distributed independently under their own licence.
(Think of napp-it like a platform usable for developers or manufacturers to create individual
appliances, napp-it is extendable like a CMS for a Web-UI).

Some extensions like comfort-replication between appliances or managing Domain-ACL's comfortably
from Web-UI instead of CLI or using Windows are not free. If you like the comfort you must pay
but you can use it all without.
 
Last edited:
I've read your pdf, and my idea of using a USB key for the OpenIndiana partition seem to be impossible according to it.

My ESXi will be on a 4GB USB stick, can OpenIndiana really not be on a second 32Gb stick?

I was planning at adding a local Intel SSD 80GB for ESX to use for it's swap file, I guess I might end up using it for OpenIndiana.

If I passthrough my SSD drive as well as my 8x sata controller, would it be possible to partition it for OpenIndiana base install + fast log device + l2arc ? (So I can make use of the full 80Gb )
 
If you like the All-In-One Idea, use

a cheap SATA laptop disk or SSD (40GB+)
as local datastore, and opt for ESXi to boot

add a second controller for data-disks with pass-through

or you will not be happy with the result
Forget any solution based on USB Stick or USB DOM
or the idea of slicing disks. You selected ZFS for data security.
Avoid these "add insecurity, saves minimal money" solutions

ESXi + RDM may be the alternative at home.
(i would expect reduced stability and speed)
 
Last edited:
i meant a sata DOM. they work great for OS storage. get two and mirror the root pool.

Sata DOM's may be a good option.

But I have not jet seen any that is comparable in any way to a
common Sata mainstrem SSD ex like a Intel 320-40 GB - not in price,
performance or expected reliability.
 
and you wont, DOMs aren't meant for performance. DOMs don't consume hot swap bays though and can live on the often unused onboard sata ports quite happily. DOMs also come in smaller sizes. 8GB is doable with openIA and napp-it if you enable compression before the napp-it install but it is real tight. if openIA had an option to install to a compressed root pool though it wouldn't be tight at all.
 
Back
Top