OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

davewolfs

Limp Gawd
Joined
Nov 7, 2006
Messages
333
If you have 400gb all for L2ARC you better have a lot of memory. I wouldn't use Solaris either unless you are paying for it.
 

CopyRunStart

Limp Gawd
Joined
Apr 3, 2014
Messages
155
We pay $1,000/year per system for Solaris.

What do you define as a lot of memory? We do 128GB. Raw pool size is 80TB.
 
D

Deleted member 82943

Guest
i just downloaded solaris 11_3 from oracle. Seems to be the most feature complete version of ZFS, will try it out on a VM
 

dilidolo

Limp Gawd
Joined
Jun 30, 2011
Messages
180
Thanks, good to know. :D

Actually I wasn't asking if it was enough, I was asking if it was too much lol. I've heard that every 100GB of L2ARC steals ~2GB of RAM from ARC. So I just wanted to ask if a 400GB to 128GB L2ARC:ARC ratio was good.

ARC is adaptive, meaning hot data gets added to ARC and cold data gets removed dynamically. It never hurts to get more.

I think it's 1TB data per 2GB of RAM. It's not the pool size but actual data size. Again, it depends on data as well. If it's all cold data, it's waste of money.

You can use arc_summary.pl script to see how arc is used.
 

spazoid

Limp Gawd
Joined
Jun 16, 2008
Messages
289
Thanks, good to know. :D

Actually I wasn't asking if it was enough, I was asking if it was too much lol. I've heard that every 100GB of L2ARC steals ~2GB of RAM from ARC. So I just wanted to ask if a 400GB to 128GB L2ARC:ARC ratio was good.

The memory requirement of the L2ARC depends on the block size of the cached data. IIRC each L2ARC chuck requires 180 bytes of memory, and by default a maximum of 25% of RAM can be used for the L2ARC index.

I made a basic calculator in Excel a few years back using the numbers I found in some documentation, and with 128 GB of RAM and a block size of 8 KB, it says you can have up to ~2800 GB of L2ARC before you start having issues.

I wouldn't add any L2ARC device before actually needing it though. Check your hit rate with arcstat.pl with a hot system (might be weeks of use before you reach that point).
 

DlStreamnet

Limp Gawd
Joined
Mar 10, 2005
Messages
359
Guys,

I replaced a drive at the weekend and it has 4k sectors. My other 4 disks are 512byte.

Is there a way to force the new drive to be 512?

I have tried -o ashift 9 but it has rejected the command, I assume the results I am getting on Google are out of date.

Thanks,
dL
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
Not possible, you cannot replace a 512B disk in a vdev with ashift=9 with a 4k disk and current ZFS.

You can try to buy a 512B disk or do a backup and recreate a pool with ashift 12.
Ashift is a property of a vdev that is set during creating a pool or adding a vdev.
 

DlStreamnet

Limp Gawd
Joined
Mar 10, 2005
Messages
359
Not possible, you cannot replace a 512B disk in a vdev with ashift=9 with a 4k disk and current ZFS.

You can try to buy a 512B disk or do a backup and recreate a pool with ashift 12.
Ashift is a property of a vdev that is set during creating a pool or adding a vdev.

Thanks Gea.

I have a mirror backup of the data so I will destroy the vdev and re-create it with my 4k+512b disks with ashift=12.

I'll take your advice from a previous thread and retire my Open Indiana installation as well I think!

Thanks,
dL
 

enigmah

n00b
Joined
May 13, 2010
Messages
23
Can somebody give me a hint on the basis of a simple shell script how to configure the nappit "other job" feature? Especially the error management is the my problem.

I can start my own rsync script (sh script.sh) and it will run through, but the nappit log shows errors (first printf for own log file) cause it seems to expect other return values.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
Its all about the question if or if not the jobstate displays an error in joblist after the job is run.

If you set the return value of errormanagement to
- never, means that jobstate is always ok does not matter if your job returns a value or not
- always, means that jobstate displays an error if your job returns any value
- condition, means that you get an ok on a certain return value or an error only on another return value
that you can set under conditions
 

enigmah

n00b
Joined
May 13, 2010
Messages
23
Thanks Gea.
Always is the default. That is what I use at the moment and is not differentiated enough for my usage.
Never "looks" good but is not what I want not see too.
So I've to stick with condition. But what should I write at the end of the shell script to see an OK in nappit? And what should the check in nappit command look like?
I had a look at job-other.pl but I'm not so good at perl and I don't know what nappit expects from my shell script.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
If you do not need to check a return value, set to "never"

Most console commands do not return a value on success,
Only on errors, they return something. In such a case "always" is ok.

Other commands return a value always,
Example a command either returns "x files copied" or "file not found"
In such a case, you can use "condition" with "copied" and "not found" as condition.

In a script, you can use the echo command at the end to return maybe ok or failed.
 

seijirou

n00b
Joined
Feb 28, 2012
Messages
34
It looks like omnios won't boot on my machine. I need to import a ZOL pool so I'm pretty sure it has feature flags, ruling out solaris. How bad are things on OI? I used to run napp-it on OI 2 years ago on this same box and it ran well for me, but it sounds like updates to napp-it since then aren't working on OI?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
If OI boots, OmniOS should boot as well as they share the same bootmechanism and both are Illumos distributions.

If you want to stay with OI 151a, your problem is that there have been no security updates in the meantime like the critical OpenSSL fixes. If you want to use an old OI with a newer napp-it, check https://www.napp-it.org/downloads/openindiana_en.html about howto achieve this.

Other options:
OI Hipster (no stable but newer)
OmniOS (what is the problem, have you tried USB or CD boot optionally with a reloaded iso)
 

seijirou

n00b
Joined
Feb 28, 2012
Messages
34
If OI boots, OmniOS should boot as well as they share the same bootmechanism and both are Illumos distributions.

If you want to stay with OI 151a, your problem is that there have been no security updates in the meantime like the critical OpenSSL fixes. If you want to use an old OI with a newer napp-it, check https://www.napp-it.org/downloads/openindiana_en.html about howto achieve this.

Other options:
OI Hipster (no stable but newer)
OmniOS (what is the problem, have you tried USB or CD boot optionally with a reloaded iso)

I tried the portable napp-it image and I tried the installer usb image. Both load to where it reads "Solaris" and a bunch of other stuff across the top, and just stall there with a blinking cursor.
 

seijirou

n00b
Joined
Feb 28, 2012
Messages
34
I've got OI Hipster installed with napp-it and things are mostly working. I'm having trouble with SMB shares though and I think I just don't understand what's going on with ACLs. The sharing is just done with guest access, I don't have a need for access control I just want it wide open.

When I create a file, the access mask is 000. Here's an example..

drwxrwxrwx+ 2 99 99 5 Dec 30 17:54 .
drwxrwxrwx+ 3 uucp staff 8 Dec 16 18:10 ..
----------+ 1 2147483649 2147483650 411K Dec 30 17:48 0049_151230115037_001.pdf
----------+ 1 2147483649 2147483650 0 Dec 30 17:54 New Text Document (2).txt
----------+ 1 2147483649 2147483650 0 Dec 30 17:53 New Text Document.txt

I can create the file but then I can't do anything with it. I'm not sure how relevant this is though because after I chmod 777 I still can't open,edit, or delete files.

If I look in filesystems I see there is folder ACL and share ACL.
tank/store store, guest.. on off off off n.a. zfs unset on 6.27T 918G none none none none standard lz4 off n.a. special full_set ACL off

If I click on folder ACL I see 4 listed. user:root, owner@, group@, everyone@. I set all of these to full_set and I changed all inheritance to file,dir. Originally I tried changing just everyone@ but it didn't work so I changed them all, it still doesn't work.

Next I looked at share-ACL. In this one I deleted all ACL excepct ACL 0 which is now everyone@, full_set.

0 everyone@ rwxpdDaARWcCos full_set rd(acl,att,xatt) wr(acl,att,xatt,own) del(yes,child) x, s folder only allow delete

It still doesn't work, so I don't know what to do. All the ACL I can find should permit "everyone" to do everything and the normal unix permissions are 777 which should do the same thing.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
What you should know with the Solaris kernelbased SMB server:

- Only nfs4 ACl are relevant (not Unix permissions) for best Windows ntfs compatibility
- use /usr/bin/ls -v /folder if you want to check ACL via console
- main difference of ACL to traditional Unix permissions are a finer permission control and inheritance

Especially the inheritance settings are a problem if you do a chmod to Unix permissions
as this will delete all inheritance settings as they are not available there.

-If you modify folder ACL settings, this will affect new files but not current files.


What you should do now
- If you create a filesystem, set the folder ACL to fully open with everyone@=modify
with inheritance enabled (This is the default behaviour if you create a filesystem in napp-it)

In your case, reset permissions recursively to everyone@=modify
- use menu ZFS filesystems > Folder-ACL > reset ACL (below ACL listing)

After this, you should be able to edit all folders/files with your guest enabled shares.
 

seijirou

n00b
Joined
Feb 28, 2012
Messages
34
Bingo. It just didn't occur to me that the settings weren't recursive. I didn't see the reset ACL link but I see it now, and access is working. I understand much better now, thank you Gea.
 

enigmah

n00b
Joined
May 13, 2010
Messages
23
OK, I got it. It's simpler that I expected.
Everything written to STDOUT or STDERR is checked if a condition matches.
The following script with "other job" set to "condition" and "noerr=ok"...

Code:
#!/bin/sh

printf '%s\n' "My Message A" | tee --append myscript.log
printf '%s\n' "My Message B"
printf '%s\n' "My Message C"
printf '%s\n' "ok"

... will return in nappit log "info: sh myscript.sh: My Message A" with status "OK".
It doesn't matter if the "ok" is printed first or later for a successful check of the condition as you can see in the example.
So for larger scripts with multiple log prints to files it makes sense to print a dummy message for the nappit log at the beginning of the script. Further using a distinct condition is essential to prevent other log prints from generating an OK status in nappit.

@Gea: Would it be possible to make the condition be dependent on the exit status for the script? In that case one would write the exit code in the condition. Completed by an input text field with the message of exit code for the nappit log.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
@Gea: Would it be possible to make the condition be dependent on the exit status for the script? In that case one would write the exit code in the condition. Completed by an input text field with the message of exit code for the nappit log.

possible, yes
You may modify the script /var/web-gui/data/napp-it/zfsos/_lib/scripts/job-other.pl
But as you can use any success string the current behaviour should be flexible enough.
 

seijirou

n00b
Joined
Feb 28, 2012
Messages
34
Need a little help with clarification on replication job settings.

I see when configuring the job that you can "keep" shapshots via regex, and that you can "hold unkept snaps for days: otherwise only last two are hold".

For "otherwise only last two are hold". Does this mean that the default is 2 days?

I thought it meant the default is 2 snapshots but now I can see many more than 2 snapshots are still around so I'm wondering, are they going to clear themselves up when they're 2 days old?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
If you do not enter anything in keep or hold,
you will see two snaps per filesystem.

if you enter for ex 00 in keep, any snap that ends with 00
ex ..100, ..200 are kept unless you destroy manually

If you enter for ex 14 in hold, you will hold all snaps
(for all filesystems) for 2 weeks. Older replication snaps are
deleted.
 

seijirou

n00b
Joined
Feb 28, 2012
Messages
34
If you do not enter anything in keep or hold,
you will see two snaps per filesystem.

if you enter for ex 00 in keep, any snap that ends with 00
ex ..100, ..200 are kept unless you destroy manually

If you enter for ex 14 in hold, you will hold all snaps
(for all filesystems) for 2 weeks. Older replication snaps are
deleted.

Thanks Gea. Unfortunately the snapshots are piling up on my system and I've left the field blank. Do you think this is because I'm running on OI?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
Thanks Gea. Unfortunately the snapshots are piling up on my system and I've left the field blank. Do you think this is because I'm running on OI?

I do not test the newer releases with OI.
If you use an older napp-it release, you may try the current release.
 

seijirou

n00b
Joined
Feb 28, 2012
Messages
34
I do not test the newer releases with OI.
If you use an older napp-it release, you may try the current release.

AFAIK I'm running the latest release (16.01). I had some trouble getting omnios to boot but I'll give it another go.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
I have done some OS tuning tests with the new OmniOS 151017 beta with SMB 2.1 support on 10G Ethernet.
With some tunings, I got performance improvements up to 20-30% compared to the base config.

I can now achieve 800-900 MB/s writes and 600-700 MB/s reads from Windows with SMB 2.1 to a pool from a single Intel P750.
To make tunings easier, I added a tuningpanel to napp-it 16.1p (Menu System > Tuning) with a base tuning setup and up to
10 settings that you can edit together with an option to create a bootable systemsnap and an immediate reboot.

I have updated the pdf
http://napp-it.org/doc/downloads/performance_smb2.pdf
 

enigmah

n00b
Joined
May 13, 2010
Messages
23
Does somebody have an idea why the following script prints all lines to the log file when executed in the console window but not when executed by the nappit "other job"?

OmniOS r151014

Code:
#!/bin/sh
pathlog="./test.log"
> "$pathlog"
printf '%s\n' "$(date +%Y-%m-%dT%H:%M:%S%Z)" | tee --append "$pathlog" # works in console and "other job"
printf '%s\n' "$(date -u +%s)" | tee --append "$pathlog"  # works in console and "other job"
printf '%s\n' "$(date --utc +%s)" | tee --append "$pathlog" # works in console only
printf '%s\n' "$(date -d @1452193692)" | tee --append "$pathlog"  # works in console only
printf '%s\n' "$(date --date='@1452193692')" | tee --append "$pathlog"  # works in console only
printf '%s\n' "$(date --version)" | tee --append "$pathlog"  # works in console only

When executed by nappit it doesn't like the double hyphens -- in front of the options and the @ for the timestamp.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
napp-it starts the shellscript from Perl.

You can start a script from console via Perl like
perl -e ' system("sh /path/yourscript.sh") ;'

for testings
 

natkin

n00b
Joined
Mar 31, 2014
Messages
30
For online backups of my rpool filesystems (e.g., https://docs.oracle.com/cd/E23824_01/html/821-1448/recover-3.html "Creating a ZFS Snapshot Archive for Recovery), it would be convenient if napp-it would let me set up a snapshot job for rpool. I.e., if napp-it would manage the recursive snapshot of rpool and the automatic deletion of the dump and swap filesystems—and of course if napp-it would manage the lifetime of the recursive snapshots per the regular napp-it snapshot job parameters.

Right now, napp-it only allows me to snapshot-job for non-rpool filesystems. May I request this feature?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
If you want to create a file from a rpool snap for recovery purpose you can create one or two
shellscripts with the three needed commands and execute them with an "other job" to have
always the last or the two last states of rpool on your datapool.

But this is a very complicated way to do a recovery.
I prefer the following

- create bootable snaps (BE) on system changes what allows a go back to the former states
This is also snap based but creates a writeable and bootable clone that you can select during boot

- do not add any services to a barebone setup that requires a complicated setup.
Napp-it settings can be easily restored especially if you use the napp-it backup job

- if you want to add services like databases, mediaservers or webserver, use virtualizing
and store the VMs on secure ZFS storage with snaps, see napp-in-one based on ESXi.

- the easiest way if you really need a fast recovery option of your system disk:
- use clonezilla from an USB stick to clone/restore the systemdisk (usb disk or NFS/SMB share)
This requires a power down but imaging is done in a very short time.
- use a 3,5" hardware mirror enclosure for 2 x 2,5" disks that can mirror you systemdisk.
You can hot remove/ add the second disk as a coldspare for your systemdisk,
 

enigmah

n00b
Joined
May 13, 2010
Messages
23
You can start a script from console via Perl like
perl -e ' system("sh /path/yourscript.sh") ;'

Doing it the way you proposed it works.
Could it be that nappit is executing the shell script in a different way?
I found that in the job-other.pl:

Code:
my $r=`$action 2>&1`;

But I haven't any perl knowledge to understand in detail what's going on there.
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
using Backticks or system is quite similar
instead of
Code:
perl -e ' system("sh /path/yourscript.sh") ;'

you can also use
Code:
perl -e ' `sh /path/yourscript.sh` ;'

btw
$action is a placeholder ex
$action="sh /path/yourscript.sh";

Then you can run it from Perl like
Code:
my $r=`$action 2>&1`;

where $r contains the output/returnvalue of the script
2>&1 means that errormessages are redirected to stdout

http://www.perlhowto.com/executing_external_commands
 

enigmah

n00b
Joined
May 13, 2010
Messages
23
Code:
perl -e ' system("sh /path/yourscript.sh") ;'
perl -e ' `sh /path/yourscript.sh` ;'

Executing the script either way works from the console window.
So this is really weird cause nappit calls it the same way with the backticks.
Is there anything else in nappit filtering the script?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
The only remaining difference is, that jobs are executed as a root cron job
what means that the environment like paths may be not the same.

btw
If you need a more complex script, its much easier to use a Perl script instead a shell script.
You call it via `perl script.pl`.

http://qntm.org/perl
 

enigmah

n00b
Joined
May 13, 2010
Messages
23
I got it!
Your hint with the environment variable solved it.
console is executing /usr/gnu/bin/date
nappit is executing /usr/bin/date

Tanks a lot!
 

natkin

n00b
Joined
Mar 31, 2014
Messages
30
Thank you. I will try your "other job" recommendation. I do use bootable snaps for on-site recovery, but I'm wanting to do regular (and online) backups of my omnios rpools to iDrive encrypted cloud storage.

It would have been nice to use an actual napp-it snap-job for this, as it manages very well a set of recent snapshots per the "keep n; hold x days" parameters. I rely on this for my "tank" pool, and then I just have to script sending the daily deltas (zfs send -i ...) to iDrive encrypted storage.

If you want to create a file from a rpool snap for recovery purpose you can create one or two
shellscripts with the three needed commands and execute them with an "other job" to have
always the last or the two last states of rpool on your datapool.

But this is a very complicated way to do a recovery.
 

chune

Weaksauce
Joined
Nov 2, 2013
Messages
71
I have done some tests with OpenIndiana Hipster
(20151003 from http://dlc.openindiana.org/isos/hipster/) and found the following

- gcc not installed
pkg install gcc-48

- Perl Module Carp not installed, you must install from CPAN or
edit /var/web-gui/data/wwwroot/cgi-bin/admin.pl and comment out
use CGI::Carp at line 4

- Perl Module Expect with Tty.pm not working
(ex menu user). You must recompile/ install Expect

I will add these modules to current 0.9f6 and the bugfix edition 0.9f7
so OpenIndiana Hipster should run again - maybe tomorrow


update:
OI Hipster (October) is supported from napp-it 0.9f6 (Dec 22, 2015)
http://napp-it.org/downloads/changelog.html

Thanks Gea! I have napp-it installed and working on hipster! I do see a weird zpool parsing bug. It looks like the "last login" section is corrupting the zpool list output:
iJ6zACq.png
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
I will check that tomorrow and hope that there are not too many surprises in Hipster and it behaves basically like OpenIndana or OmniOS as I only do very basic tests with Hipster.
 
Top