OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

no, as I had laying around an Sata drive and an iso cdr but IPMI should work

btw
I am working on a preconfigured and ready to use storage server image with OmniOS and napp-it that you can download and clone to a/any mainboard with any supported nic and an Sata bootdisk > 60GB from a 16 GB USB stick with Clonezillea or import as an ESXi template in case of the ESXi VM. This may be my future default distribution method (napp-it toGo)
 
Info

OmniOS 151018 stable is available

Main advantage: many improvements regarding SMB2, see http://napp-it.org/doc/downloads/performance_smb2.pdf and ReleaseNotes/r151018

I am working on preconfigured images for a napp-it ZFS storage server
beginning with the SuperMicro Boards X11-SS where you cannot install OmniOS from USB sticks.

The ready to use images are cloned with CloneZilla. You can also use this method for backup and disaster recovery, see http://www.napp-it.org/doc/downloads/napp-it.pdf
 
About a year ago, I was writing about my problems with ZFS pool freezing and iSCSI target dying.

On random and towards the end, at the same time every week, our iSCSI target died (svc reported service running, but there was nothing listening on 3260 port) and we were unable to write to pools (issuing touch /zpool/a hung for 1min-2h). There was also no chance to restart iSCSI, since svcadm restart target resulted in svc going to online* state and waiting for service to shutdown. Reboot was the only solution.

Since we were running unsupported configuration with Supermicro 28bay case with SAS expander and SATA enterprise drives, we decided to ditch this configuration and go with all HCL hardware from Nexenta list.

After 3 months of successful running and slowly moving users from old storage to the new one, target started to crash again when we reached 100 iSCSI clients.

Since we were unable do find the solution, we turned to OmniTI. With the help of Dan (maintainer of OmniOS) and some memory dumps, we discovered 2 problems:
  • One was a rare case of broken iSCSI packet, that resulted in system to panic and reboot. Dan found a Nexenta's patch, we applied it and one problem solved. Patch has also, as far as I know, been upstreamed to Illumos and latest OmniOS.
  • Second problem was the iSCSI target crashing. Dan discovered that when target crashed, a thread was waiting in kmem allocator for mpt_sas. It looks like we hit a case of memory exhaustion. I remembered that we didn't limit the size of ARC and that the system ate all the memory for ARC and could not freed it fast enough. I limited the max size of ARC and we didn't have any problems with iSCSI target for the last month.
So now it looks like we have a stable system and we couldn't have done it without Dan's help. Sometimes it really helps to have someone on the call, who can produce a fix in a few hours instead of trying to find the problem yourself.

I hope my experience will help someone.

Matej
 
Thanks for that Info. I have seen that already in the omnios-discuss mailing list
and added this info to my tuning tipps.

One question.
What is your ARC limit or more exact how many RAM should be left free?
 
Glad I can be of help.

Currently I did "over the finger" limit and just limited my ARC to 200GB.

Looking at my graphs, I usually had around 10GB of free memory and now I have around 18GB after ARC is filled.

# echo ::memstat | mdb -k
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 8443847 32983 13%
ZFS File Data 53644741 209549 80%
Anon 67460 263 0%
Exec and libs 1788 6 0%
Page cache 6731 26 0%
Free (cachelist) 16775 65 0%
Free (freelist) 4893633 19115 7%

Total 67074975 262011
Physical 67074974 262011
Click to expand...

Matej
 
thanks,
You have 256 GB RAM with ARC limited to 200 GB?
This gives a first thumbrule of limit ARC to 80% on systems with a lot of RAM

I suppose the problem grows with more RAM.
The ZFS defaults are 75% of memory on systems with less than 4GB RAM
and physical Memory - 1 GB if you have more memory
This may become a problem with a lot of RAM as freeing RAM costs time
ZFS ARC Parameters - Oracle Solaris 11.1 Tunable Parameters Reference Manual


btw
The default timeout values of ZFS for disks or initiators are way too high
Have you limited disk timeout or Initiator timeout to a smaller value like the 7s of TLER?

IIn napp-it I have added a tuning panel in menu System > Tunings to set system values
and an Initiator option in menu Comstar > Initiator > Settings to set system values.
I will add an option in my default tuning settings to limit max arc to 80% of physical RAM.
 
Interesting limits. Wonder why there isn't a bigger default limit on systems with more memory. In my case, time that was needed to free ram cost us crash of a service. Unfortunately my graphs are already aggregated and I can't look back at them to see if there was a momentary lack of free memory. It's also possible collectd didn't catch the lack of memory.

What timeouts are we talking about?
On initiator or target?

I did change the timeout values on initiators according to AWS recommendations. I had some clients having problems with timeouts and windows dropping drives (event IDs 7,20,47) (no problems on Linux clients though). Since update, there has been no disconnects, but every now and then I see a drop of iSCSI traffic for up to a minute.

According to Dan, COMSTAR is a little fragile. Nexenta did a lot of work on COMSTAR, but there is no one out there that would upstream the fixes to Illumos and later to OmniOS.

Also related: on our old server, where we used to have problems with target dying and ZFS locking up, since we moved the majority of users to the new server, we hadn't had a target crash for the last 2 months, which tells me it really could be ARCs fault.

Matej
 
I didn't change any settings on the server (target), but I must set higher timeouts on clients (initiators) because of the way we use iSCSI, which is not ideal (remote clients, sometimes high latency,...) and lower timeout lead to iSCSI drives dropping. I know iSCSI wasn't made for that kind of use, but currently it works for us.

Will need to switch to something else in the future, but for now, this is what we have and it works for our purpose.

Matej
 
Hey,

What's currently the best distro? I'm using the first build of OI based on Illumos!! Just backing up my data to do a complete fresh install.

Also, I currently have 2 raid-z2 spanned with 12 disks (+ 2 hot spares), so effectively raid 60. I'm thinking of moving to 3 way mirrors for a bit better data protection. I don't need copious amounts of space anymore as not going to bother with dvd/blu-ray rips as streaming has come a long way. So just pictures, music and home videos.

One thing I was thinking, if it's possible with any of the distros: can you have say a pool with 3 3 way mirrors, so 9 disks, then a 3 disk raid-z1 and have data sync'd between the 2 pools automatically. For some backup facility. If so, I could save a bit by using 2 way mirrors instead of 3 way. So 4 2 way mirrors and a 4 disk raid-z1 to use all 12 disks (and keep the 2 hot spares).
 
ldoodle said:
can you have say a pool with 3 3 way mirrors, so 9 disks, then a 3 disk raid-z1 and have data sync'd between the 2 pools automatically. For some backup facility.
Click to expand...
You want to set up a 1-direction replication (i.e. zfs send/receive) to mirror the live pool to its backup.
 
I think OmniOS is the way to go if you want stable server OS.

Regarding my post about iSCSI target dying. I guess I spoke too soon. It just crashed 4 times today. Will report when I'll have some more info.

Matej
 
Based on advice from Gea a few months back, I've decided to have a go at running my file server for my home network of Macs on OmniOS, now that r151018 with SMB 2.1 is out. My goal is to do the simplest config possible, with no unnecessary steps and if possible no added software. It's gone well so far but I've got a few questions that my digging has not found answers for yet.

1. Bonjour
I want to advertise my SMB shares via Bonjour. I enabled the built-in mDNSResponder by running
svcadm enable dns/multicast​
To configure, I ran
dns-sd -R "MyServer" _device-info._tcp. local 445 model=Xserve &
dns-sd -R "MyServer" _smb._tcp local. 445 &​

This works perfectly but, and I could be wrong, it doesn't seem to be persistent. Is that correct? If so, where would you recommend putting the commands such that they get executed on each boot?

2. PAM
Once I had an SMB share running, I tried to connect to it from my Mac using a pre-existing user on the OmniOS box. I received an error whose exact text I've forgotten but which basically said that I didn't have permission. I had seen mention on some pages of having to configure PAM, yet other guides didn't mention it. So I put the following line at the end of /etc/pam.conf:
other password required pam_smb_passwd.so.1 no warn​
Then I set the password for the user I was trying to connect with. I noticed that there seemed to be password complexity requirements that weren't there before (but I could be wrong about that). After that, I was able to connect from the Mac. The question is: Should I have needed to do these steps? I only ask because they seem to be omitted from some online guides.

3. ZFS Filesystem Creation Parameters
I created a zpool with no parameters, opting to set the options during the creation of the filesystem(s). For the pool I simply did:
zpool create Pool mirror d1 d2 (forgot the actual identifiers)​
Then my confusion was which parameters to use for the filesystem. I've used two different ZFS variants on the Mac (Zevo and OpenZFS) and in both cases I always used certain parameters because I had heard Macs liked them. What I don't know is whether I should use those same parameters making this filesystem knowing that it will be accessed exclusively by Macs. Anyway, I did:
zfs create -o compression=lz4 -o utf8only=on -o normalization=formD -o casesensitivity=insensitive -o nbmand=on -o atime=off Pool/FileSystem​
I guess the utf8only is implied by the normalization so that could be left out. But should I be setting normalization=formD? Any comments on the other parameters, appropriate or not?
 
Yes, you must adjust pam.conf as SMB does its own authentication with an additional password file to the Unix one.
Have you installed OmniOS manually without napp-it as napp-it comes as a ready to use systemimage for a webmanaged
appliance that you only need to restore to your system or an online installer that configures these settings automatically.
see http://www.napp-it.org/doc/downloads/napp-it.pdf chapter 4

Regarding Bonjour
You can call these settings either with a bootinit script or cron
When you use napp-it, you can place them in menu Services > Bonjour and Autostart

Regarding filesystem parameters
Wit napp-it, pools are created automatically with your settings.

The normalization=formD is a default suggestion for OSX to (try to) keep national characters in filenames
identical over services and platforms.

see
Zpool - OpenZFS on OS X
[URL="https://blogs.oracle.com/nico/entry/normalization_insensitivity_should_be_the"]On Unicode Normalization -- or why normalization insensitivity should be rule (Nico's blog)[/URL]

Regarding password restrictions
You can set them in napp-it menu User > pw restrictions
 
Infos for the Napp-it ToGo ZFS Storage Server
I have uploaded new images (freely available)

New OmniOS 151018 stable with SMB2.1 and napp-it 16.02f,
Base-Tuning, TLS Alert Emails ex to Gmail. Alle Nics are autoconfigured for DHCP.


Napp-it ToGo Sata
These are images for a barebone/ hardware setup to an Sata disks.
After Cloning the image to Sata, your appliance is ready to use

The images are intended for a Kingston V300-60GB or an enterprise class
Intel S3510-80 with powerloss protection.
HowTo: http://www.napp-it.org/doc/downloads/napp-it.pdf

The images were created on an SuperMicro X11-SSH-CTF but are running on
other mainboards and systems. If you have installed them successfully on
other systems ex Dell or HP, please send an email to [email protected]


Napp-it ToGo VM
These are images for a virtualized SAN/NAS under ESXi.
You can import the image ready to use (deploy a template)
HowTo: http://www.napp-it.org/doc/downloads/napp-in-one.pdf

Current Image 2016.04 for ESXi 5.5U2 - 6.0U2
Open-VM-Tools, vmxnet3s and MTU9000

Open-VM-Tools are now integrated in OmniOS 151018
 
Thanks Gea.

While I know Napp-it would solve these problems for me, I have a desire to do it myself for gaining more experience and insight. I may use Napp-it anyway because of the web interface. And, thank you for the product and your expertise freely shared with all who ask.

Regarding PAM, my curiosity was whether some other step that I skipped would normally have added the line automatically. For example, seems that in some tutorials people say to enable SMB sharing for the OS (don't remember the command at the moment). I didn't do that, just used a zfs command to create my share and nothing more. Should you still "officially" enable SMB sharing and if so might that have added the PAM line? I guess no need to ask here, I can just try it. I guess in the end it's just a goal of mine to use the fewest steps necessary to get to the end goal.

Regarding mDNS, so you affirm that the commands are NOT permanent and that they do need to run for each boot?

Thanks!
 
SMB settings
Depend if you use SAMBA or the into the kernel and ZFS integrated Solaris SMB server.
With the last, you must edit pam.conf and enable sharing as a ZFS property - no other settings-

With SAMBA this is different as SAMBA does not know anything about ZFS
and the smbshare property is there only for ZFS compatibility but without the usual SAMBA behaviour

mDNS
The settings are not persistent over a reboot
 
Hi Gea,

Can you please help me out with enabling guest smb shares. (Sorry I'm a bit lost)

Up from OmniOS 151018 guest access requires
-a user guest
-smbadm enable-user guest

I'm using the Napp-it ToGo VM

Thanks!
 
I got it to work, ran these commands:

useradd guest
smbadm enable-user guest

Recreated the zfs filesystem with smbguest on.

Is there anything I might have missed?
 
I have not tried myself but you should
- create a user named guest in menu user
- enable guest to login without a password
enter the command at console or the napp-it cmd field
Code: 
smbadm enable-user guest

see the mail from Gordon Ross in the OmniOS-discuss maillist
I can take a guess what this might be about.

There were several bugs fixed as part of the "extended security" work:
1122 smbsrv should use SPNEGO (inbound authentication)

One of those was that we used to give a client a "guest" logon
if they tried to logon to SMB with _any_ unrecognized account.
No, that was never a good idea. Not only was it questionable
for security, but it confused issues about failed logon. Example:
Windows user does NOT get the expected pop-up dialog asking
for new credentials when they try to connect to a share using
an invalid user name. Instead, they would get connected,
but would fail to have access to anything in the share.

So with that bug fixed, one can logon as "guest" only if:
(1) you actually ask for guest in your logon request,
(2) a local Unix account named "guest" exists, and
(3) the guest account is enabled for SMB

Therefore, if you were using guest access before 1122 was fixed,
(and were depending on accidental guest access working),
you'll need to do the following to re-enable guest access:

useradd (options] guest
smbadm enable-user guest

The guest account password is ignored by SMB, so
all that matters to SMB is whether that account is
marked as enabled in /var/smb/smbpasswd

To keep Unix users from using guest for login, you can
set the Unix password hash to something invalid, etc.
Click to expand...

more
omnios-discuss
 
Last edited:
_Gea said:
Regarding Bonjour
You can call these settings either with a bootinit script or cron
When you use napp-it, you can place them in menu Services > Bonjour and Autostart
Click to expand...

You mentioned that you could use cron for the mDNS commands. In your experience, do these commands need to be continually run on some regular interval or once set do they persist until the next reboot, no matter how long that is?
 
You start the commands as a background task so they run until reboot or until you manually kill them.
If you start them via cron or at, you should run them only once ex on reboot or you must use a script that takes care about.

The usual way to start services/daemons at boottime is a service manifest or a startscript in /etc/init.d
 
HammerSandwich said:
You want to set up a 1-direction replication (i.e. zfs send/receive) to mirror the live pool to its backup.
Click to expand...

_Gea said:
I would suggest the new OmniOS 151018 stable as it offers some huge SMB improvements
and support for newer hardware ex the new Intel i219 nics

ReleaseNotes/r151018
http://napp-it.de/doc/downloads/performance_smb2.pdf
Click to expand...

Thanks both for the replies.

However..... we've just upgraded some tin at work to Windows Server 2012 R2 and I've seen the ability to use Windows storage spaces in a Hyper-V Failover Cluster (!!!) so it's got me thinking about moving my home NAS to Windows Server! I can't play with the ones at work because they're in production and we use SANs with their own RAID so storage spaces is redundant.

But I'd like to know if Windows storage spaces can replicate what ZFS can do in regards to grouping raid sets. My current setup is 2X 6-drive RAID-Z2 for one pool. Could I do that in Windows, or even group 7X 2-way mirrors for one pool? Or would it be 7X distinct 2-way mirrors?

Thanks.
 
I cannot comment the Windows options as I am happy that I was able to switch all my storage away from Windows to ZFS and the Storage spaces concept with ntfs or even ReFS iseems far away to change my mind.

What you can do, is to offer Storage for Windows via iSCSI and this is a serious option with ZFS.
maybe you can follow the discussion at (german, may translate with Google):
[Sammelthread] Microsoft Hyper-V Stammtisch
 
ldoodle said:
ut I'd like to know if Windows storage spaces can replicate what ZFS can do in regards to grouping raid sets. My current setup is 2X 6-drive RAID-Z2 for one pool. Could I do that in Windows, or even group 7X 2-way mirrors for one pool? Or would it be 7X distinct 2-way mirrors?
Click to expand...

I have some limited experience with Storage Spaces and ReFS. First, if you're using their equivalent of RAIDZ/2, which they call "parity spaces" I think, the performance is awful. This is well-known. Their equivalent of a mirror is fine. Also, dealing with Storage Spaces is very convoluted compared to ZFS. I tried to implement it for our server at work but I just couldn't bring myself to. It was just so clunky compared to ZFS. I ended up with a standard hardware RAID controller (gasp!). If I were doing it over now I'd do either OmniOS running an iSCSI target or have a go at directly sharing via the built-in SMB 2.1 server, assuming that can be properly and seamlessly integrated into an AD domain, which I have no experience or information about.
 
The Solaish kernelbased SMB server works perfectly in an AD.
As it supports Windows SMB groups and stores Windows SID as extended ZFS attributes you can even replicate or move a pool to another AD member server without loosing Permissions, unique for a Unix SMB server on a Unix filesystem.
 
_Gea said:
The Solaish kernelbased SMB server works perfectly in an AD.
As it supports Windows SMB groups and stores Windows SID as extended ZFS attributes you can even replicate or move a pool to another AD member server without loosing Permissions, unique for a Unix SMB server on a Unix filesystem.
Click to expand...

That's really awesome. Next time I redo our work setup that's how I'll go.
 
_Gea said:
You start the commands as a background task so they run until reboot or until you manually kill them.
If you start them via cron or at, you should run them only once ex on reboot or you must use a script that takes care about.

The usual way to start services/daemons at boottime is a service manifest or a startscript in /etc/init.d
Click to expand...

I think I'm missing something. I didn't think those commands stayed running, I thought they were just to configure the already-running mDNSResponder. That's not correct?
 
ypo must find a way to start your following commands at boottime as a background task
(thiis is what the & at the end means)

Code: 
dns-sd -R "MyServer" _device-info._tcp. local 445 model=Xserve &
dns-sd -R "MyServer" _smb._tcp local. 445 &
 
_Gea said:
ypo must find a way to start your following commands at boottime as a background task
(thiis is what the & at the end means)

Code: 
dns-sd -R "MyServer" _device-info._tcp. local 445 model=Xserve &
dns-sd -R "MyServer" _smb._tcp local. 445 &
Click to expand...

Ah, didn't know that about the &. Based on that, I assume it should be fine to put both commands in one shell script, right?

For my first attempt, which works, I put each command in its own script and have those linked into rc3.d per the Oracle article Run Control Scripts (System Administration Guide: Basic Administration)

If they will reliably run from the same script, I'll do that. It just seemed from my experience running them "by hand" that there might be timing issues in that they didn't quite seem to return right away and that I didn't see the shell prompt after running them until I hit Enter again.
 
_Gea said:
No problem, you can add all commands in a single script
Click to expand...

That's working, thanks. Only odd thing is that once today my Mac told me when I clicked on the server name in the sidebar that the object was no longer valid or something. I relaunched Finder and all was well. Not sure what happened.

Anyway, after experimenting all week with a VM, I installed and configured my real server today. Aside from the hiccup above (which may be the Mac's fault), it's running well, and seems very quick.

I have a few picky questions. The first one is:

Is there a way to use uppercase letters in an (Illumos kernel based) SMB share name? I can put uppercase letters in the name specified in the zfs command but they show up all lowercase on the Mac (not sure about Windows).
 
The lowercase naming is the same on Windows.
I cannot say why. If you want to know, ask Gordon Ross (he is maintaining the SMB server) at #illumos irc or the illumos-discuss maillist.

regarding performance
the OmniOS defaults are quite conservative and optimized for 1G and less RAM

For faster networks or RAM > 1-2GB, you can increase tcp buffers or MTU settings,
see http://napp-it.de/doc/downloads/performance_smb2.pdf
 
Wow, that's a great write-up and similar in a lot of ways to my setup since I'm also running 10GbE with an X540, although my Mac is a self-upgraded 2010 with an internal ATTO NT12 so that's different. I haven't run any benchmarks yet.

Regarding jumbo frames: If I'm reading correctly, it's enabled on the server always and you just vary it on the clients to see the result. True? Aside from speed, any other issues with an MTU "mismatch"? What actually happens, does the server see that the client can't handle 9000 and drop back to 1500?

Mostly unrelated question: How would you compare built-in SMB to Samba? Clearly SMB is easier, don't need to install it and configuration is super easy.

So ease is on the side of SMB. What about performance and stability? Any other reasons to pick one over the other?
 
If you want Jumbo Frames, you must enable on server, clients and switches.
If a client is set to mtu 1500 it should work.

In the Solaris world, you can use the Solaris kernelbases, multithreaded SMB server or SAMBA that is available on any X system.

regarding stability
The Solaris SMB server is very stable as it is part of the OS and ZFS and maintained by the OS provider, either Oracle for Solaris or Illumos.

regarding performance
The Solaris SMB server is multithreaded and very fast

regarding Windows compatibility
The Solaris SMB server offers SMB groups and uses Windows SID als extended ZFS attributes. This keeps permissions intact even when you move or replicate diles to another Solaris AD member-server. The Solaris NFS4 ACL are nearly identical to ntfs permissions. As Shares and snaps are a ZFS property, Snaps as Windows "Previous versions" work out of the box without any problems.
This is unique in the Unix/ Linux world.

But there are features that only SAMBA offers, ex
- special features ex veto files, AD and Master Broser server
- sharing not a ZFS filesystem property but based on the regular folders
- using the SMB server in zones (nor possible in Solaris, possible in OmniOS from 151016 up)
- SMB3 (Solaris SMB is currently on 2.1) but mostly you should use currently SMB3 only on Windows
 
What about Mac extended attributes on files? At first glance it appears they're not coming through. I say this because an example JPEG which, when stored locally on the Mac, shows lots of info in Finder Get Info, such as camera model, dimensions, etc, but when copied to the server (OmniOS/kernel SMB), does not display any of that info.
 
Another question. What about Spotlight support for netatalk on Illumos-based OSs like OmniOS?

I know this requires Gnome Tracker being installed and netatalk being compiled with support for it. This seems possible on FreeBSD although my half-hearted efforts at making it work weren't fruitful.

Would it work on Illumos?
 
You must log in or register to reply here.
Back
Top