OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Captainquark said:
On an unrelated subject, I have recently created a new ZFS folder on my datapool, and then moved some 100 Gigs of data from one folder to the other (within the same pool, that is). I just realized that it did not free up the space of the moved away data on the source folder. Is that expected behaviour, and is there something I can do about it?'
Click to expand...

...you happen to have a snap on that pool or source-folder?
 
I have a problem again with TLS email. TLS email test works ok, but when I try enable TLS mail nothing happens. Cant find anything on _log directory that could indicate what is happening.
Enable regular mail says "regular smtp is used". On the email section smtp test fails. TLS test mail works ok.

Cant create new TLS email jobs since the enable doesnt work. And the old jobs didnt work, so I tried deleting them but it didnt work. Had to remove the files from _log/jobs and after that they vanished from the jobs menu.

Any clues how I could enable the TLS?
 
mooniz said:
I have a problem again with TLS email. TLS email test works ok, but when I try enable TLS mail nothing happens. Cant find anything on _log directory that could indicate what is happening.
Enable regular mail says "regular smtp is used". On the email section smtp test fails. TLS test mail works ok.

Cant create new TLS email jobs since the enable doesnt work. And the old jobs didnt work, so I tried deleting them but it didnt work. Had to remove the files from _log/jobs and after that they vanished from the jobs menu.

Any clues how I could enable the TLS?
Click to expand...

napp-it use the module /var/web-gui/data/napp-it/zfsos/_lib/scripts/job-email.pl for email.
When you switch from regular to tls or vice versa, you overwrites this module from defaults
job-tls-email.pl or job-regular-email.pl (They are identical beside the tls/smtp send in function my_send)

In job-overview under Opt3 you have then either smtp port 25 or TLS port 587
(only jobs that are created with 0.9)

the test-email function is hardware coded either to use tls or smtp (depends on menu).
If you want to compare or check behaviours, compare these two scripts.

and/or
recheck email settings (about setup)
use most current napp-it, recreate jobs or try with another mailserver
 
Has anyone gotten FTP properly set up on OmniOS? I have found zero documentation in the weeks I have been googling and messing around (with omnios, nexenta, freebsd, s11.1, all over the board). S11.1 I had FTP up and running (chroots and all) pretty fast - proFTPd is pretty nice built-in like it is. However, for a production environment I think we might be steering toward OmniOS - that is, if I can ever get FTP working...

Basically just want my SMB users (no shell access) to be able to login and get chrooted into the main share, and the guest login (again, no shell access) to get chrooted into the public folder underneath that share. Also would prefer to have the ACLs work properly, which they seem to on S11.1.
 
taishan said:
Has anyone gotten FTP properly set up on OmniOS? I have found zero documentation in the weeks I have been googling and messing around (with omnios, nexenta, freebsd, s11.1, all over the board). S11.1 I had FTP up and running (chroots and all) pretty fast - proFTPd is pretty nice built-in like it is. However, for a production environment I think we might be steering toward OmniOS - that is, if I can ever get FTP working...

Basically just want my SMB users (no shell access) to be able to login and get chrooted into the main share, and the guest login (again, no shell access) to get chrooted into the public folder underneath that share. Also would prefer to have the ACLs work properly, which they seem to on S11.1.
Click to expand...

As long as you have SSH setup on your OmniOS host, you could use sftp. Of course if the clients are on WIndows boxes they would need to have an ssh client (with sftp). I'm sure there are plenty, but we use cygwin.
 
more replication trouble. Hoping you find these reports useful, not inteded to pile on.


1368039969 error
job-replicate 1035

20.56:57
end log

initial remote replication finished (time: 27974 s, size: MB, performance: 0.0 MB/s) error





noid

20.56:57
glib 942

repli ask remote host 192.168.1.186

-> grouplib_ask_remote_destroy_snap
nc -w 60 192.168.1.186 81
do=request_destroy_snap&hostname=nappit1&snap_name=r10/B1@1368039969_repli_zfs_nappit1_nr_1&snap_rec=

answer=




1368039969 error
job-replicate 352

20.56:55
destination snap not created



missing
rz2/B1@1368039969_repli_zfs_nappit1_nr_1



1368039969
job-replicate 325

20.56:55
receiver finished with mess
 
When you start a remote replication, the script checks
- source and target ZFS and snaps
- delete sporce snaps that are newer than target snaps
this seems to work, this happens on target machine -> port 81 on source

next step is to initiate zfs receive and then a remote controlled zfs send over netcat on a port > 50000
(port see job overview). When the zfs is transferred successfully, a target snap is created

The last point does not happen. You may check
- receiver started (target ZFS created?)
- size of target ZFS. If it is zero, no data is transferred (i.e. firewall blocks transfer),
if it is > zero data transfer was interrupted.

beside trivial problems
- disk full
- hidden clone from a formet transfer (delete, menu snap)
- other problems (try a reboot)
 
new OmniOS stable r511006 (build 8d266aa) is available
ReleaseNotes

How to Update:

1. check publisher
pkg publisher

You have either
pkg.omniti.com/omnios/bloody or pkg.omniti.com/omnios/release (stable)

If you have bloody, remove the bloody publisher and set the stable release one:
pkg unset-publisher omnios
pkg set-publisher -g http://pkg.omniti.com/omnios/release omnios

2. Now you can update:
pkg update
reboot

On next reboot, you should have OmniOS stable release r511006 build 8d266aa
 
Last edited:
Hey _Gea, I have a question based on something on your tuning page. It says to use as many vdevs as possible and as many disks as possible but then it says avoid large vdevs. (Just a little confusing).

In my production server (which has been awesome BTW) I am running mirrored vdevs so is there an issue with continuing to add to a large pool? My plan was to just keep adding mirrored disks into the pool so I can get the most spindles in there since we are running our databases on the NFS store as well.

I'd hate to create a new pool with just a couple of disks based on our recent performance benchmarks against our Dell MD3000i. Our virtualized Napp-It install ran between 50% and 300% faster than our Dell SAN and we put our current AIO system together for half the price. Nice.

So, is there a practical limit on mirrored vdevs? It makes sense for raid-z2 or raid-z3 due to rebuild issues, but with the mirrors it doesn't seem like it's a problem.

Can you clarify?
 
brianmat said:
Hey _Gea, I have a question based on something on your tuning page. It says to use as many vdevs as possible and as many disks as possible but then it says avoid large vdevs. (Just a little confusing).

In my production server (which has been awesome BTW) I am running mirrored vdevs so is there an issue with continuing to add to a large pool? My plan was to just keep adding mirrored disks into the pool so I can get the most spindles in there since we are running our databases on the NFS store as well.

I'd hate to create a new pool with just a couple of disks based on our recent performance benchmarks against our Dell MD3000i. Our virtualized Napp-It install ran between 50% and 300% faster than our Dell SAN and we put our current AIO system together for half the price. Nice.

So, is there a practical limit on mirrored vdevs? It makes sense for raid-z2 or raid-z3 due to rebuild issues, but with the mirrors it doesn't seem like it's a problem.

Can you clarify?
Click to expand...

This is a question of your goal.

Example:
If you have ten disks and if you look at theoretical sequential write performance:
- with a Raid-Z2 of 10 disks, you can write simulaniously to 8 datadisks
wheras with a 5 x 2 mirror, your write performance is like 5 disks.

I would expect the Raid-Z2 to be faster.

If you look at I/O performance, then your Raid-Z2 is equal to one disk
because for every datablock, every head on every disk must be positioned
wheras the 5 x 2 mirror is equal to 5 disks.

In your case (database) you should look only at I/O so mirror is the best.
It has also the shortest resilver time. There is also no limit in number of mirrors
beside the problem that at some point your server is not capable to handle
more transfers.
 
_Gea said:
The last point does not happen.
QUOTE]

I left out the useful bit!

20.56:55
receiver finished with mess

cannot receive new filesystem stream: invalid stream (checksum mismatch)


This replication attempt was a reseed back on the LAN. All the data seemed to have transferd - pool allocation and zvol seemed to have grown to full size. Any idea what can cause a checksum mismatch. Wondering if it's necessary to quiesce the source volume during replication, or even just initial replication?

I have 2 zvols and succesfully replicated the second last night while writes to the source volume were ongoing. retrying the first now. Have noticed during these 3 replications that network throughput periodiclaly drops to almost nothing for a few seconds at a time.
Click to expand...
 
jb33 said:
_Gea said:
The last point does not happen.
QUOTE]

I left out the useful bit!

20.56:55
receiver finished with mess

cannot receive new filesystem stream: invalid stream (checksum mismatch)


This replication attempt was a reseed back on the LAN. All the data seemed to have transferd - pool allocation and zvol seemed to have grown to full size. Any idea what can cause a checksum mismatch. Wondering if it's necessary to quiesce the source volume during replication, or even just initial replication?

I have 2 zvols and succesfully replicated the second last night while writes to the source volume were ongoing. retrying the first now. Have noticed during these 3 replications that network throughput periodiclaly drops to almost nothing for a few seconds at a time.
Click to expand...

This message (checksum mismatch) is an error from zfs receive that detects a datastream failure for whatever reason during network transfers (This is checksummed as well). You can only retry and check for network problems like cabling, bad nics or switches.
Click to expand...
 
Hi,
I need some advice.
I have a pool consisting of two raidz vdev's.
raidz1-0, five WD WD20EARS, ashift=9
raidz1-1, five SEAGATE ST2000DM001, ashift=12
+ one spare ST2000DM001.

The goal is to replace the WD20EARS disks with ST2000DM001 disks.
Is this possible without recreating the pool? Replacing 512b disk's with 4k in a ashift=9 vdev is not possible as I understand.



regards
Jeken

napp-it v0.9a9, OI 151a7
 
jeken said:
Hi,
I need some advice.
I have a pool consisting of two raidz vdev's.
raidz1-0, five WD WD20EARS, ashift=9
raidz1-1, five SEAGATE ST2000DM001, ashift=12
+ one spare ST2000DM001.

The goal is to replace the WD20EARS disks with ST2000DM001 disks.
Is this possible without recreating the pool? Replacing 512b disk's with 4k in a ashift=9 vdev is not possible as I understand.

regards
Jeken

napp-it v0.9a9, OI 151a7
Click to expand...

Although the Ears is a 4k disks, I would expect that this is not possible.
If possible, I would create a new Pool from the new disks, copy files over,
destroy the old pool and add the 4k vdevs to the new pool.

Info
If you mix 512 and 4k disks in a vdev, this should result in a ashift=12 vdev too
 
jeken said:
Hi,
I need some advice.
I have a pool consisting of two raidz vdev's.
raidz1-0, five WD WD20EARS, ashift=9
raidz1-1, five SEAGATE ST2000DM001, ashift=12
+ one spare ST2000DM001.

The goal is to replace the WD20EARS disks with ST2000DM001 disks.
Is this possible without recreating the pool? Replacing 512b disk's with 4k in a ashift=9 vdev is not possible as I understand.
Click to expand...

Yes, the new Seagate disks will not fit in your old pool.
Also, replacing the old ones is a one-at-a-time operation...time consuming and potentially dangerous (giving unnecessary stress to the old disks during resilvering of each and every new one).

If you can mount all five new disks in your box, I'd create a new pool, do a zfs-send/receive, take the old pool offline and rename the new one to the old name.

In fact I'd rather put in an additional controller for this or even dismantle the other existing array in order get access to the required number of ports until procedure is done.
 
I have ~10T of data so creating a new pool using the new disks could not hold all data.
I do have an other pool (raidz 5x3T) that I could backup to.
So I will backup the "old" pool, destroy it and create a new one using only the ST2000MD001 disks.


/jeken
 
Oh, right...you're using 2 vdevs in a single pool...sorry, I misread your info.

Good that you have another pool with enough space to hold your data.
IMHO a zfs send/receive from your old pool to a new folder in your backup pool and then back to yoiur newly created pool should do the trick.
 
Is there any news on the current health/roadmap/development of the various ZFS OS forks?

I currently use openIndiana, but I worry that it has not been updated significantly in a while.

Do you know if there is any significant health in that fork? Any big updates planned soon?
 
lordsegan said:
Is there any news on the current health/roadmap/development of the various ZFS OS forks?

I currently use openIndiana, but I worry that it has not been updated significantly in a while.

Do you know if there is any significant health in that fork? Any big updates planned soon?
Click to expand...

OI is a pure community project. No one is working full time on it.
You can hope for a new dev version from time to time.

If you like newest Illumos or Solaris features and need bugfixes within a short time,
you should look at either Oracle Solaris (closed source) or OmniOS as a free option (both stable with payed support option).

For my own, I switched to OmniOS
 
_Gea said:
napp-it update is not a problem
(update per wget, reboot, recreate jobs beside replication)

The problem:
Oracle changed some network and share basics with 11.1
Napp-it 0.9 cares only about Solaris 11.1 so I would suggest:
stay at napp-it 0.8 unless you are ready to 11.1
Click to expand...

_Gea,
I just saw here that Oracle supports updating 11 11/11 to 11.1...:
If you are currently running Oracle Solaris 11 11/11 (or have applied any Support Repository Updates), there is no need to reinstall. You can update to Oracle Solaris 11.1 using the network repository (pkg.oracle.com)
Click to expand...
Do you know if this works reliably? What do you recommend - updating or reinstalling from scratch?

Thanks,
Cap'
 
_Gea said:
OI is a pure community project. No one is working full time on it.
You can hope for a new dev version from time to time.

If you like newest Illumos or Solaris features and need bugfixes within a short time,
you should look at either Oracle Solaris (closed source) or OmniOS as a free option (both stable with payed support option).

For my own, I switched to OmniOS
Click to expand...
Hi _Gea,
For those of us who currently run Openindiana, do you have any suggestions for the least painful way of converting to OmniOS? Obviously our ZFS pools will still be there, but all of our configuration that is part of the kernel rpool will not.

Thank you.
 
Captainquark said:
_Gea,
I just saw here that Oracle supports updating 11 11/11 to 11.1...:

Do you know if this works reliably? What do you recommend - updating or reinstalling from scratch?

Thanks,
Cap'
Click to expand...

Yes. It works reliably.

Some people (including me) with unusual networking configurations using VLANs and multi-link LAGs had to remove and rebuild the network configs after upgrade, but since then they have been reliable (you have to do this on a scratch install anyway, so its not really a benefit to load from scratch because of this).

Solaris 11.1 DID break the iSCSI target software, so if you are serving iSCSI block devices from your machine you should think twice about upgrading to 11.1 at all. They do have a patch but it is only available with support. It will be fixed in the next public "release" (11.1.1, 11.2 or 12.0 - whatever they actually release, probably late this year).
 
turquoisewords said:
Hi _Gea,
For those of us who currently run Openindiana, do you have any suggestions for the least painful way of converting to OmniOS? Obviously our ZFS pools will still be there, but all of our configuration that is part of the kernel rpool will not.

Thank you.
Click to expand...

1.
You must install OmniOS from scratch and import your pools
(If you cannot use a preconfigured napp-it To Go USB stick)
Follow my setup instructions at http://napp-it.org/downloads/omnios_en.html

2.
A lot of settings are in the pool (sharing, ZFS properies, ACL)
Other settings must be redone, in some cases you can use a copy of config files
 
PigLover said:
Solaris 11.1 DID break the iSCSI target software, so if you are serving iSCSI block devices from your machine you should think twice about upgrading to 11.1 at all. They do have a patch but it is only available with support. It will be fixed in the next public "release" (11.1.1, 11.2 or 12.0 - whatever they actually release, probably late this year).
Click to expand...

Hey,
Thanks for the info. I don't have any fancy config, it's very basic, really. No LAG's and also no iSCSI targets (NFS only), so I guess I'll do a backup and give it a try soon.
Do you know if I have to upgrade TLS, too? I am using it right now and it took me a while to make it running, so I wouldn't like to have to touch this, either...
Cheers,
Cap'
 
Hi,

Anyone here with experience with nic teaming?
I just bought a Netgear GS110TP smart switch and want to use both nics on my SM x9scmf for nic teaming. I added the second nic in Esxi 5.1 as an active nic and selected nic teaming with IP hash.
Now when I bind the 2 ports on my netgear in the LAD section I use static and I get message Link Up!
So,my guess is it's working! Only problem is I can't login with vsphere anymore,files remain reachable and when I reboot my server doesn't get an ip!
So what am I doing wrong?

Need some help!TY
 
added support in zfs for aclmode=restricted

https://www.illumos.org/issues/3254

"ZFS ACL's are quite powerful and very useful; however, there is currently no way to protect them from being destroyed or corrupted by a drive-by chmod. There is virtually no way to simply avoid chmod. Whether it's a user or junior admin who simply doesn't know any better, or a closed binary application that is not ACL aware and uses chmod under the hood, or even the issue with NFS exclusive open that stomps on inherited ACL's, one way or another at some point your carefully constructed ACL is going to get mangled.

To prevent this, I propose adding an additional aclmode "restricted", which would restrict any attempt to chmod a zfs object with a nontrivial ACL. "
Click to expand...

This new feature is included in OmniOS.
I support this in current napp-it and use it as a default value during creating a filesystem in OmniOS
 
_Gea : is there documentation on the parameters you use to create pools/FS with napp-it ? So far I've done things manually with CLI but since my next pool will have many drives I'm considering using the napp-it GUI (already installed), I'd like to know what it does though.
 
Aesma said:
_Gea : is there documentation on the parameters you use to create pools/FS with napp-it ? So far I've done things manually with CLI but since my next pool will have many drives I'm considering using the napp-it GUI (already installed), I'd like to know what it does though.
Click to expand...

All sources are open.
check the menu-action in "/var/web-gui/data/napp-it/zfsos/06_pools/01_create pool/action.pl" and
"/var/web-gui/data/napp-it/zfsos/07_zfsfolder and shares/01_create/action.pl"

If you activate Edit (top-menu), you can even check the script within napp-it.
It is some Perl calling the CLI commands.
 
PigLover said:
Yes. It works reliably.

Some people (including me) with unusual networking configurations using VLANs and multi-link LAGs had to remove and rebuild the network configs after upgrade, but since then they have been reliable (you have to do this on a scratch install anyway, so its not really a benefit to load from scratch because of this).
Click to expand...

So I did the upgrade, and it went super smooth, also the napp-it update to 0.9 was easy and there was no problem - almost. One little thing is that TLS Email is not working (again), and I can't seem to find the source of the problem. When I test TLS Email, I get this error message
admin.pl: invalid SSL_version specified at /usr/perl5/site_perl/5.12/IO/Socket/SSL.pm line 308
Click to expand...
however, line 308 in the SSL.pm file is empty... any help here would be appreciated.

Second, one little question remained, and that is about the mirrored boot disk... now that I have upgraded to Sol11.1, is this already (still!) mirrored correctly, or do I have to "upgrade" something there, too? I did do the mirroring using Constantin's article here, but I'm not sure if this is still good - sorry for the noob question, I am admittedly not so much into Solaris... thanks for any help!
 
Currently perl-TLS mail is only tested and working in OI
There are also problems with OmniOS
Workaround: use a free forwarder, that accepts unencrypted mail

regarding mirror:
have you tried napp-it menu Disks - mirror bootdisk (based on the same howto)
 
I have TLS running on Solaris 11.1

If I remember correctly, I had to downgrade Socket SSL package by running "install S/SU/SULLR/IO-Socket-SSL-1.68.tar.gz" in the CPAN shell
 
TCM said:
The joys of "enterprise-grade" software :rolleyes:
Click to expand...

???

Sending encrypted emails to Google without email server or client application like
thunderbird is not part of any OS nor part of napp-it.

It can be done on any platform for example with Perl and TLS modules from CPAN.
But they have a lot of dependencies and are not updated on a regularly base.

Using the older Socket is also needed in OI and noted in the howto.
Only on current stable of OmniOS, i could no get it working.
 
Has anyone tried purchasing Solaris 11.1 support for a supermicro based white box? I have 25 servers we have to move off Nexanta and so far I am being told since it's not on the HCL, we can not buy support. Their suggestion was to buy hardware off the HCL list and then order support...
 
Last edited:
The Spyder said:
Has anyone tried purchasing Solaris 11.1 support for a supermicro based white box? I have 25 servers we have to move off Nexanta and so far I am being sold since it's not on the HCL, we are SOL.
Click to expand...

Why do you need to move off Nexenta?

Support for Solaris 11 is 12% of hardware cost per year, so that's a big chunk of change to get support for hardware you already have. Maybe it'd make sense to purchase a smaller number of newer systems directly from Oracle?

The upshot is Oracle only provides support to the machines they want to support, and Supermicro isn't jumping through their hoops. See the HCL definitions for more details.
 
If you need OS support, you may ask OmniTI as well (OmniOS).
It is the same OS like Nexenta (without Nexenta Web-UI) and stable based
on newest Illumos bits (like NexentaStor 4 someday)

For hardware support look for a solid distributor and buy two spare boxes -
its always cheaper
 
You must log in or register to reply here.
Back
Top