OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Discussion in 'SSDs & Data Storage' started by _Gea, Dec 30, 2010.

  1. Jon S

    Jon S n00b

    Messages:
    8
    Joined:
    Apr 29, 2011
    Gea,

    Did you try a remote CD/DVD ISO via the Supermicro IPMI Virtual Media to install OmniOS ?

    - Jon
     
  2. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    no, as I had laying around an Sata drive and an iso cdr but IPMI should work

    btw
    I am working on a preconfigured and ready to use storage server image with OmniOS and napp-it that you can download and clone to a/any mainboard with any supported nic and an Sata bootdisk > 60GB from a 16 GB USB stick with Clonezillea or import as an ESXi template in case of the ESXi VM. This may be my future default distribution method (napp-it toGo)
     
  3. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    Info

    OmniOS 151018 stable is available

    Main advantage: many improvements regarding SMB2, see http://napp-it.org/doc/downloads/performance_smb2.pdf and ReleaseNotes/r151018

    I am working on preconfigured images for a napp-it ZFS storage server
    beginning with the SuperMicro Boards X11-SS where you cannot install OmniOS from USB sticks.

    The ready to use images are cloned with CloneZilla. You can also use this method for backup and disaster recovery, see http://www.napp-it.org/doc/downloads/napp-it.pdf
     
  4. levak

    levak Limp Gawd

    Messages:
    386
    Joined:
    Mar 27, 2011
    About a year ago, I was writing about my problems with ZFS pool freezing and iSCSI target dying.

    On random and towards the end, at the same time every week, our iSCSI target died (svc reported service running, but there was nothing listening on 3260 port) and we were unable to write to pools (issuing touch /zpool/a hung for 1min-2h). There was also no chance to restart iSCSI, since svcadm restart target resulted in svc going to online* state and waiting for service to shutdown. Reboot was the only solution.

    Since we were running unsupported configuration with Supermicro 28bay case with SAS expander and SATA enterprise drives, we decided to ditch this configuration and go with all HCL hardware from Nexenta list.

    After 3 months of successful running and slowly moving users from old storage to the new one, target started to crash again when we reached 100 iSCSI clients.

    Since we were unable do find the solution, we turned to OmniTI. With the help of Dan (maintainer of OmniOS) and some memory dumps, we discovered 2 problems:
    • One was a rare case of broken iSCSI packet, that resulted in system to panic and reboot. Dan found a Nexenta's patch, we applied it and one problem solved. Patch has also, as far as I know, been upstreamed to Illumos and latest OmniOS.
    • Second problem was the iSCSI target crashing. Dan discovered that when target crashed, a thread was waiting in kmem allocator for mpt_sas. It looks like we hit a case of memory exhaustion. I remembered that we didn't limit the size of ARC and that the system ate all the memory for ARC and could not freed it fast enough. I limited the max size of ARC and we didn't have any problems with iSCSI target for the last month.
    So now it looks like we have a stable system and we couldn't have done it without Dan's help. Sometimes it really helps to have someone on the call, who can produce a fix in a few hours instead of trying to find the problem yourself.

    I hope my experience will help someone.

    Matej
     
  5. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    Thanks for that Info. I have seen that already in the omnios-discuss mailing list
    and added this info to my tuning tipps.

    One question.
    What is your ARC limit or more exact how many RAM should be left free?
     
  6. levak

    levak Limp Gawd

    Messages:
    386
    Joined:
    Mar 27, 2011
    Glad I can be of help.

    Currently I did "over the finger" limit and just limited my ARC to 200GB.

    Looking at my graphs, I usually had around 10GB of free memory and now I have around 18GB after ARC is filled.

    Matej
     
  7. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    thanks,
    You have 256 GB RAM with ARC limited to 200 GB?
    This gives a first thumbrule of limit ARC to 80% on systems with a lot of RAM

    I suppose the problem grows with more RAM.
    The ZFS defaults are 75% of memory on systems with less than 4GB RAM
    and physical Memory - 1 GB if you have more memory
    This may become a problem with a lot of RAM as freeing RAM costs time
    ZFS ARC Parameters - Oracle Solaris 11.1 Tunable Parameters Reference Manual


    btw
    The default timeout values of ZFS for disks or initiators are way too high
    Have you limited disk timeout or Initiator timeout to a smaller value like the 7s of TLER?

    IIn napp-it I have added a tuning panel in menu System > Tunings to set system values
    and an Initiator option in menu Comstar > Initiator > Settings to set system values.
    I will add an option in my default tuning settings to limit max arc to 80% of physical RAM.
     
  8. levak

    levak Limp Gawd

    Messages:
    386
    Joined:
    Mar 27, 2011
    Interesting limits. Wonder why there isn't a bigger default limit on systems with more memory. In my case, time that was needed to free ram cost us crash of a service. Unfortunately my graphs are already aggregated and I can't look back at them to see if there was a momentary lack of free memory. It's also possible collectd didn't catch the lack of memory.

    What timeouts are we talking about?
    On initiator or target?

    I did change the timeout values on initiators according to AWS recommendations. I had some clients having problems with timeouts and windows dropping drives (event IDs 7,20,47) (no problems on Linux clients though). Since update, there has been no disconnects, but every now and then I see a drop of iSCSI traffic for up to a minute.

    According to Dan, COMSTAR is a little fragile. Nexenta did a lot of work on COMSTAR, but there is no one out there that would upstream the fixes to Illumos and later to OmniOS.

    Also related: on our old server, where we used to have problems with target dying and ZFS locking up, since we moved the majority of users to the new server, we hadn't had a target crash for the last 2 months, which tells me it really could be ARCs fault.

    Matej
     
  9. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
  10. levak

    levak Limp Gawd

    Messages:
    386
    Joined:
    Mar 27, 2011
    I didn't change any settings on the server (target), but I must set higher timeouts on clients (initiators) because of the way we use iSCSI, which is not ideal (remote clients, sometimes high latency,...) and lower timeout lead to iSCSI drives dropping. I know iSCSI wasn't made for that kind of use, but currently it works for us.

    Will need to switch to something else in the future, but for now, this is what we have and it works for our purpose.

    Matej
     
  11. ldoodle

    ldoodle Limp Gawd

    Messages:
    161
    Joined:
    Jun 29, 2011
    Hey,

    What's currently the best distro? I'm using the first build of OI based on Illumos!! Just backing up my data to do a complete fresh install.

    Also, I currently have 2 raid-z2 spanned with 12 disks (+ 2 hot spares), so effectively raid 60. I'm thinking of moving to 3 way mirrors for a bit better data protection. I don't need copious amounts of space anymore as not going to bother with dvd/blu-ray rips as streaming has come a long way. So just pictures, music and home videos.

    One thing I was thinking, if it's possible with any of the distros: can you have say a pool with 3 3 way mirrors, so 9 disks, then a 3 disk raid-z1 and have data sync'd between the 2 pools automatically. For some backup facility. If so, I could save a bit by using 2 way mirrors instead of 3 way. So 4 2 way mirrors and a 4 disk raid-z1 to use all 12 disks (and keep the 2 hot spares).
     
  12. HammerSandwich

    HammerSandwich [H]ard|Gawd

    Messages:
    1,112
    Joined:
    Nov 18, 2004
    You want to set up a 1-direction replication (i.e. zfs send/receive) to mirror the live pool to its backup.
     
    ldoodle likes this.
  13. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    I would suggest the new OmniOS 151018 stable as it offers some huge SMB improvements
    and support for newer hardware ex the new Intel i219 nics

    ReleaseNotes/r151018
    http://napp-it.de/doc/downloads/performance_smb2.pdf
     
    ldoodle likes this.
  14. levak

    levak Limp Gawd

    Messages:
    386
    Joined:
    Mar 27, 2011
    I think OmniOS is the way to go if you want stable server OS.

    Regarding my post about iSCSI target dying. I guess I spoke too soon. It just crashed 4 times today. Will report when I'll have some more info.

    Matej
     
  15. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    Random problems without a clear reason are really annoying.
     
  16. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Based on advice from Gea a few months back, I've decided to have a go at running my file server for my home network of Macs on OmniOS, now that r151018 with SMB 2.1 is out. My goal is to do the simplest config possible, with no unnecessary steps and if possible no added software. It's gone well so far but I've got a few questions that my digging has not found answers for yet.

    1. Bonjour
    I want to advertise my SMB shares via Bonjour. I enabled the built-in mDNSResponder by running
    svcadm enable dns/multicast
    To configure, I ran
    dns-sd -R "MyServer" _device-info._tcp. local 445 model=Xserve &
    dns-sd -R "MyServer" _smb._tcp local. 445 &

    This works perfectly but, and I could be wrong, it doesn't seem to be persistent. Is that correct? If so, where would you recommend putting the commands such that they get executed on each boot?

    2. PAM
    Once I had an SMB share running, I tried to connect to it from my Mac using a pre-existing user on the OmniOS box. I received an error whose exact text I've forgotten but which basically said that I didn't have permission. I had seen mention on some pages of having to configure PAM, yet other guides didn't mention it. So I put the following line at the end of /etc/pam.conf:
    other password required pam_smb_passwd.so.1 no warn
    Then I set the password for the user I was trying to connect with. I noticed that there seemed to be password complexity requirements that weren't there before (but I could be wrong about that). After that, I was able to connect from the Mac. The question is: Should I have needed to do these steps? I only ask because they seem to be omitted from some online guides.

    3. ZFS Filesystem Creation Parameters
    I created a zpool with no parameters, opting to set the options during the creation of the filesystem(s). For the pool I simply did:
    zpool create Pool mirror d1 d2 (forgot the actual identifiers)
    Then my confusion was which parameters to use for the filesystem. I've used two different ZFS variants on the Mac (Zevo and OpenZFS) and in both cases I always used certain parameters because I had heard Macs liked them. What I don't know is whether I should use those same parameters making this filesystem knowing that it will be accessed exclusively by Macs. Anyway, I did:
    zfs create -o compression=lz4 -o utf8only=on -o normalization=formD -o casesensitivity=insensitive -o nbmand=on -o atime=off Pool/FileSystem
    I guess the utf8only is implied by the normalization so that could be left out. But should I be setting normalization=formD? Any comments on the other parameters, appropriate or not?
     
  17. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    Yes, you must adjust pam.conf as SMB does its own authentication with an additional password file to the Unix one.
    Have you installed OmniOS manually without napp-it as napp-it comes as a ready to use systemimage for a webmanaged
    appliance that you only need to restore to your system or an online installer that configures these settings automatically.
    see http://www.napp-it.org/doc/downloads/napp-it.pdf chapter 4

    Regarding Bonjour
    You can call these settings either with a bootinit script or cron
    When you use napp-it, you can place them in menu Services > Bonjour and Autostart

    Regarding filesystem parameters
    Wit napp-it, pools are created automatically with your settings.

    The normalization=formD is a default suggestion for OSX to (try to) keep national characters in filenames
    identical over services and platforms.

    see
    Zpool - OpenZFS on OS X
    On Unicode Normalization -- or why normalization insensitivity should be rule (Nico's blog)

    Regarding password restrictions
    You can set them in napp-it menu User > pw restrictions
     
  18. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    Infos for the Napp-it ToGo ZFS Storage Server
    I have uploaded new images (freely available)

    New OmniOS 151018 stable with SMB2.1 and napp-it 16.02f,
    Base-Tuning, TLS Alert Emails ex to Gmail. Alle Nics are autoconfigured for DHCP.


    Napp-it ToGo Sata
    These are images for a barebone/ hardware setup to an Sata disks.
    After Cloning the image to Sata, your appliance is ready to use

    The images are intended for a Kingston V300-60GB or an enterprise class
    Intel S3510-80 with powerloss protection.
    HowTo: http://www.napp-it.org/doc/downloads/napp-it.pdf

    The images were created on an SuperMicro X11-SSH-CTF but are running on
    other mainboards and systems. If you have installed them successfully on
    other systems ex Dell or HP, please send an email to community@napp-it.org


    Napp-it ToGo VM
    These are images for a virtualized SAN/NAS under ESXi.
    You can import the image ready to use (deploy a template)
    HowTo: http://www.napp-it.org/doc/downloads/napp-in-one.pdf

    Current Image 2016.04 for ESXi 5.5U2 - 6.0U2
    Open-VM-Tools, vmxnet3s and MTU9000

    Open-VM-Tools are now integrated in OmniOS 151018
     
  19. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Thanks Gea.

    While I know Napp-it would solve these problems for me, I have a desire to do it myself for gaining more experience and insight. I may use Napp-it anyway because of the web interface. And, thank you for the product and your expertise freely shared with all who ask.

    Regarding PAM, my curiosity was whether some other step that I skipped would normally have added the line automatically. For example, seems that in some tutorials people say to enable SMB sharing for the OS (don't remember the command at the moment). I didn't do that, just used a zfs command to create my share and nothing more. Should you still "officially" enable SMB sharing and if so might that have added the PAM line? I guess no need to ask here, I can just try it. I guess in the end it's just a goal of mine to use the fewest steps necessary to get to the end goal.

    Regarding mDNS, so you affirm that the commands are NOT permanent and that they do need to run for each boot?

    Thanks!
     
  20. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    SMB settings
    Depend if you use SAMBA or the into the kernel and ZFS integrated Solaris SMB server.
    With the last, you must edit pam.conf and enable sharing as a ZFS property - no other settings-

    With SAMBA this is different as SAMBA does not know anything about ZFS
    and the smbshare property is there only for ZFS compatibility but without the usual SAMBA behaviour

    mDNS
    The settings are not persistent over a reboot
     
  21. crept2k

    crept2k n00b

    Messages:
    2
    Joined:
    Apr 21, 2016
    Hi Gea,

    Can you please help me out with enabling guest smb shares. (Sorry I'm a bit lost)

    Up from OmniOS 151018 guest access requires
    -a user guest
    -smbadm enable-user guest


    I'm using the Napp-it ToGo VM

    Thanks!
     
  22. crept2k

    crept2k n00b

    Messages:
    2
    Joined:
    Apr 21, 2016
    I got it to work, ran these commands:

    useradd guest
    smbadm enable-user guest

    Recreated the zfs filesystem with smbguest on.

    Is there anything I might have missed?
     
  23. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    I have not tried myself but you should
    - create a user named guest in menu user
    - enable guest to login without a password
    enter the command at console or the napp-it cmd field
    Code:
    smbadm enable-user guest
    see the mail from Gordon Ross in the OmniOS-discuss maillist
    more
    omnios-discuss
     
    Last edited: Apr 22, 2016
  24. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    You mentioned that you could use cron for the mDNS commands. In your experience, do these commands need to be continually run on some regular interval or once set do they persist until the next reboot, no matter how long that is?
     
  25. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    You start the commands as a background task so they run until reboot or until you manually kill them.
    If you start them via cron or at, you should run them only once ex on reboot or you must use a script that takes care about.

    The usual way to start services/daemons at boottime is a service manifest or a startscript in /etc/init.d
     
  26. ldoodle

    ldoodle Limp Gawd

    Messages:
    161
    Joined:
    Jun 29, 2011
    Thanks both for the replies.

    However..... we've just upgraded some tin at work to Windows Server 2012 R2 and I've seen the ability to use Windows storage spaces in a Hyper-V Failover Cluster (!!!) so it's got me thinking about moving my home NAS to Windows Server! I can't play with the ones at work because they're in production and we use SANs with their own RAID so storage spaces is redundant.

    But I'd like to know if Windows storage spaces can replicate what ZFS can do in regards to grouping raid sets. My current setup is 2X 6-drive RAID-Z2 for one pool. Could I do that in Windows, or even group 7X 2-way mirrors for one pool? Or would it be 7X distinct 2-way mirrors?

    Thanks.
     
  27. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    I cannot comment the Windows options as I am happy that I was able to switch all my storage away from Windows to ZFS and the Storage spaces concept with ntfs or even ReFS iseems far away to change my mind.

    What you can do, is to offer Storage for Windows via iSCSI and this is a serious option with ZFS.
    maybe you can follow the discussion at (german, may translate with Google):
    [Sammelthread] Microsoft Hyper-V Stammtisch
     
  28. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    I have some limited experience with Storage Spaces and ReFS. First, if you're using their equivalent of RAIDZ/2, which they call "parity spaces" I think, the performance is awful. This is well-known. Their equivalent of a mirror is fine. Also, dealing with Storage Spaces is very convoluted compared to ZFS. I tried to implement it for our server at work but I just couldn't bring myself to. It was just so clunky compared to ZFS. I ended up with a standard hardware RAID controller (gasp!). If I were doing it over now I'd do either OmniOS running an iSCSI target or have a go at directly sharing via the built-in SMB 2.1 server, assuming that can be properly and seamlessly integrated into an AD domain, which I have no experience or information about.
     
    ldoodle likes this.
  29. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    The Solaish kernelbased SMB server works perfectly in an AD.
    As it supports Windows SMB groups and stores Windows SID as extended ZFS attributes you can even replicate or move a pool to another AD member server without loosing Permissions, unique for a Unix SMB server on a Unix filesystem.
     
  30. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    That's really awesome. Next time I redo our work setup that's how I'll go.
     
  31. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    I think I'm missing something. I didn't think those commands stayed running, I thought they were just to configure the already-running mDNSResponder. That's not correct?
     
  32. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    ypo must find a way to start your following commands at boottime as a background task
    (thiis is what the & at the end means)

    Code:
    dns-sd -R "MyServer" _device-info._tcp. local 445 model=Xserve &
    dns-sd -R "MyServer" _smb._tcp local. 445 &
     
  33. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Ah, didn't know that about the &. Based on that, I assume it should be fine to put both commands in one shell script, right?

    For my first attempt, which works, I put each command in its own script and have those linked into rc3.d per the Oracle article Run Control Scripts (System Administration Guide: Basic Administration)

    If they will reliably run from the same script, I'll do that. It just seemed from my experience running them "by hand" that there might be timing issues in that they didn't quite seem to return right away and that I didn't see the shell prompt after running them until I hit Enter again.
     
  34. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    No problem, you can add all commands in a single script
     
  35. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    That's working, thanks. Only odd thing is that once today my Mac told me when I clicked on the server name in the sidebar that the object was no longer valid or something. I relaunched Finder and all was well. Not sure what happened.

    Anyway, after experimenting all week with a VM, I installed and configured my real server today. Aside from the hiccup above (which may be the Mac's fault), it's running well, and seems very quick.

    I have a few picky questions. The first one is:

    Is there a way to use uppercase letters in an (Illumos kernel based) SMB share name? I can put uppercase letters in the name specified in the zfs command but they show up all lowercase on the Mac (not sure about Windows).
     
  36. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    The lowercase naming is the same on Windows.
    I cannot say why. If you want to know, ask Gordon Ross (he is maintaining the SMB server) at #illumos irc or the illumos-discuss maillist.

    regarding performance
    the OmniOS defaults are quite conservative and optimized for 1G and less RAM

    For faster networks or RAM > 1-2GB, you can increase tcp buffers or MTU settings,
    see http://napp-it.de/doc/downloads/performance_smb2.pdf
     
  37. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Wow, that's a great write-up and similar in a lot of ways to my setup since I'm also running 10GbE with an X540, although my Mac is a self-upgraded 2010 with an internal ATTO NT12 so that's different. I haven't run any benchmarks yet.

    Regarding jumbo frames: If I'm reading correctly, it's enabled on the server always and you just vary it on the clients to see the result. True? Aside from speed, any other issues with an MTU "mismatch"? What actually happens, does the server see that the client can't handle 9000 and drop back to 1500?

    Mostly unrelated question: How would you compare built-in SMB to Samba? Clearly SMB is easier, don't need to install it and configuration is super easy.

    So ease is on the side of SMB. What about performance and stability? Any other reasons to pick one over the other?
     
  38. _Gea

    _Gea 2[H]4U

    Messages:
    3,834
    Joined:
    Dec 5, 2010
    If you want Jumbo Frames, you must enable on server, clients and switches.
    If a client is set to mtu 1500 it should work.

    In the Solaris world, you can use the Solaris kernelbases, multithreaded SMB server or SAMBA that is available on any X system.

    regarding stability
    The Solaris SMB server is very stable as it is part of the OS and ZFS and maintained by the OS provider, either Oracle for Solaris or Illumos.

    regarding performance
    The Solaris SMB server is multithreaded and very fast

    regarding Windows compatibility
    The Solaris SMB server offers SMB groups and uses Windows SID als extended ZFS attributes. This keeps permissions intact even when you move or replicate diles to another Solaris AD member-server. The Solaris NFS4 ACL are nearly identical to ntfs permissions. As Shares and snaps are a ZFS property, Snaps as Windows "Previous versions" work out of the box without any problems.
    This is unique in the Unix/ Linux world.

    But there are features that only SAMBA offers, ex
    - special features ex veto files, AD and Master Broser server
    - sharing not a ZFS filesystem property but based on the regular folders
    - using the SMB server in zones (nor possible in Solaris, possible in OmniOS from 151016 up)
    - SMB3 (Solaris SMB is currently on 2.1) but mostly you should use currently SMB3 only on Windows
     
  39. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    What about Mac extended attributes on files? At first glance it appears they're not coming through. I say this because an example JPEG which, when stored locally on the Mac, shows lots of info in Finder Get Info, such as camera model, dimensions, etc, but when copied to the server (OmniOS/kernel SMB), does not display any of that info.
     
  40. mkush

    mkush n00b

    Messages:
    36
    Joined:
    Aug 2, 2012
    Another question. What about Spotlight support for netatalk on Illumos-based OSs like OmniOS?

    I know this requires Gnome Tracker being installed and netatalk being compiled with support for it. This seems possible on FreeBSD although my half-hearted efforts at making it work weren't fruitful.

    Would it work on Illumos?