OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Discussion in 'SSDs & Data Storage' started by _Gea, Dec 30, 2010.

  1. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    How to setup a ready to run OpenSolaris derived ZFS Storage Server
    +configure it + add a WEB-GUI within minutes


    more: napp-it mini HowTo 1 - setup a ZFS Server


    How to setup a All-In-One Server -
    VMware ESXi based + virtualized ZFS-SAN-Server + virtualized network Switch + other VM's like Windows, Linux..


    more: napp-it mini HowTo 2 - setup a all-in one Server


    with main focus on free systems
    i will keep this initial post up to date, please re-read from time to time !!


    ZFS is a revolutionary file-system with nearly unlimited capacity and superior data security thanks to copy on write, checksums with error-autofix, raid z1-3 without the raid5/6 write hole problem, with a online filecheck/ refresh feature and the capability to create nearly unlimited data snapshots without delay or initial space consumption. ZFS Boot snapshots are the way to go back to former OS-states. ZFS is stable, availabe since 2005 and used in Enterprise Storage Systems from Sun/ Oracle and Nexenta.

    Features like Deduplication, online Encryption (from ZFS V.31), Triple Parity Raid and Hybrid Storage with SSD Read/ Write Cache Drives are State of the Art and just a included ZFS property. Volume- Raid and Storage Management are Part of any ZFS System and used with just two commands zfs and zpool.

    ZFS is now part not only in Solaris derived Systems, but also in Free-BSD. There are efforts to include it in LInux ( ZoL). Even Apple (The days before they transformed from Apple Computers to the current iPod company) had planned to integrate ZFS but cancelled it just like they have done with their Xserve systems.

    But Solaris derived Systems are more. ZFS is not just a file-system-add-on. Sun, inventor of ZFS, had developed a complete Enterprise Operating System with a unique integration around ZFS with other services like a Kernel-integrated AD Windows ACL compatible and fast SMB and a fast NFS Server as a ZFS property. Comstar, the included iSCSI Framework is fast and flexible, usable for complex SAN configurations. With Crossbow, the virtual Switch Framework you could build complex virtual network switches in software, Dtrace helps to analyse the system. Service-management is done the via the unique svcadm-tool. Virtualisation can be done on application level with Zones. All of these fetaures are from Sun- now Oracle, perfectly integrated into the os with the same handling and without compatibility problems between them - the main reason why i prefer ZFS on solaris derived systems-.

    [​IMG]

    In 2010 Oracle bought SUN and closed the free OpenSolaris project:
    There are now the following Operating Systems based on (the last free) OpenSolaris


    1. commercial options

    - Oracle Solaris 11 (available 2011)
    - Oracle Solaris 11 Express (free for non-commercial and evaluation use) wth GUI and time-slider
    download http://www.oracle.com/technetwork/server-storage/solaris11/downloads/index.html

    the most advanced ZFS at the moment, the only one with encryption
    (based on "OpenSolaris" Build 151). I support it with my free napp-it Web-Gui
    download http://www.napp-it.org/downloads/index_en.html
    Solaris Express 11 testinstallation http://www.napp-it.org/pop11.html



    -NexentaStor Enterprise Storage-Appliance
    stable, based on OpenSolaris Build 134 with a lot of backported fixes
    storage-only-use with the Nexenta Web-Gui
    download trial at http://www.nexenta.com

    -based on NexentaStor, there is a free Nexentastor Community-Edition
    (=NexentaStor, with the same GUI but without Service, without some add-ons, limited to 18 TB RAW capacity and storage-only use)
    NOT allowed for production use - home use only -
    download at http://www.nexentastor.org

    Compare different Nexenta Versions at:
    http://www.nexenta.com/corp/nexentastor-overview/nexentastor-versions


    2. free options

    best suited for a napp-it system:

    -OmniOS a minimal server distribution mainly for NAS and SAN usage, stable release with weekly betas
    free with optional commercial support, very close and up to date with Illumos kernel development
    download http://omnios.omniti.com/

    -OpenIndiana (the free successor of OpenSolaris), based on OpenSolaris Build 151 with GUI and time-slider
    currently beta, usable for Desktop or Server use.
    download http://openindiana.org/download/


    other options:

    -NexentaCore 3, end of live
    was based on a OpenSolaris Kernel Build 134 with a lot of backportet fixes from newer builds. CLI-only (without the NexentaStor Web-Gui) but running with my free napp-it Web-Gui up to napp-it 0.8

    -Illumian, base of NexentaStor 4 is available.
    http://www.illumian.org
    Illumian is based on Illumos (like OpenIndiana and uses the same binaries) but with Debian-like software installation and packaging

    -other distributions like Bellenix, Eon or Schillix


    All of the above are based originally on OpenSolaris. Further development is done either by Oracle.
    It was told, that they will freely publish the results after final releases If so, we may see new Oracle-features also in the free options with a delay.

    On the other side, there is the Illumos Project. It is a free community driven fork of the OpenSolaris Kernel with the Base-OS-functions and some base tools. They will be as close as possible to Oracle Solaris, use their new features, when they become available but also do their own development and offer Oracle to eventually include them. Illumos is mainly financed by Nexenta and others with a focus on free ZFS based Systems. Illumos is already base for Schillix and will become the base of next Eon, Bellenix, Nexenta* and OpenIndiana. Illumos is intended to be completely free and open source.


    3. Use cases:

    Although there is a desktop-option with OpenIndiana, Solaris was developed by Sun to be a Enterprise Server OS with stability and performance at first place, best for:

    NAS
    -Kernel based high speed Windows Cifs/SMB Fileserver (AD, ACL kompatible, Snaps via previous version )
    -opt. also via Samba
    -Apple File-Server with TimeMaschine and ACL-support via netatalk

    SAN
    -NFS and iSCSI Storage for ESXi and XEN

    Webserver
    -AMP Stack (Apache, mySQL, PHP)

    Backup and Archiv
    -Snapshots , Online File-Checks with Data Refresh , Checksum

    Software based Virtualization via Zones

    and of course Enterprise Databases
    -one of the reason, Oracle bought Sun


    Appliances
    -All OpenSolaris derived systems could be used as appliances and managed remotely via SSH or Browser and Web-Gui (Nexenta/ Nexentastor or napp-it on the others).


    3.1 All in One Solutions

    You could run OpenSolaris based systems not only on real hardware but also virtualized, best on (also free-version) ESXi with pci-passthrough to SAS Controller and disks. With ESXi 4.x you will have quite the same performance like you would have on real hardware - if you could add enough RAM to this VM. It is possible to integrate the three usual parts of a professional solution like Virtual-Server, virtualized NAS/SAN-Server and virtualized LAN/SAN/vlan-network-switch build on ESXi features in one box.

    If you need hot-snaps of your VM's
    -create snaps within ESXi, then with ZFS; delete ESXi snap
    In case of problems, restore ZFS snap. You could then restore hot-snap within ESXi.

    If you need higher availabilty, use two all-in one-boxes and replicate your vm's from the first to the second box via smb-copy from snaps or ZFS-send

    Links about all-in-one:
    http://wiki.xensource.com/xenwiki/VTdHowTo
    http://www.servethehome.com/configure-passthrough-vmdirectpath-vmware-esxi-raid-hba-usb-drive/
    http://www.napp-it.org/napp-it/all-in-one/index_en.html


    3.2 short how for my All-In-One concept:

    -use a mainboard with Intel 3420 or 5520 chipset + one or two Xeons
    -you need a second Disk-Controller, best are LSI 1068 or 2008 based SAS/SATA in IT-mode
    (flash it to IT-mode, if you have a IR/Raid flashed one)
    -Install esxi (onto a >= 32 GB boot drive, connected to sata in ahci mode)
    -activate vti-d in mainboard bios
    -activate pci-passthrough for this SAS controller in esxi, reboot
    -use the remaining space of your boot drive as a local datastor, install a virtualized Nexenta/Solaris on this local datastore; add/ pass-through your SAS Controller to this VM
    -connect all other disks to your SAS/Sata Controller. Nexenta/ Solaris will have real access to these disks due to pass-through.
    -set this VM to autostart first and install vmware-tools
    -create a pool, share it via NFS and CIFS for easy clone/move/save/backup/snap access from Windows or from console/midnight commander.
    -import this NFS-Share as a NFS datastore within ESXi and store all other VMs like any Windows, Linux, Solaris or BSD on it
    -share it also via CIFS for easy cloning/ backup of a VM's from your ZFS folder or via Windows previous version.



    4. Hardware:

    Opensolaris derived Systems may run on a lot of current systems. But often it’s not a trouble-free experience. But there are a few always trouble-free options, mostly based on current Intel Server Chipsets, Intel Nics and LSI 1068 (ex Intel SASUC8I) or LSI 2008 based SAS/ Sata HBA (JBOD) Controller. Avoid Hardware-Raid at all and 4k drives if possible (They will work but may have lower performance, TLER disks are not needed).


    See
    http://www.sun.com/bigadmin/hcl/
    http://nexenta.org/projects/site/wiki/About_suggested_NAS_SAN_Hardware


    Cheap and low power Intel Atom or the new HP Proliant Micro Server N36 (4 HD bays, max 8 GB ECC Ram, pcie 16x und 1x slots) with 1 GB+ RAM are ok but are running with reduced performance. But you should not use some goodies like deduplication, ZFS Z1-3 or encryption or it will become really slow. But it is suggested to add as much RAM as possible.

    If possible, I would always prefer server-class hardware best are the X8 series from SuperMicro with 3420 or 5520 chipset like a micro-ATX Supermicro X8-SIL-F. It is relatively cheap with i3 CPU, fast but low power with L-Class Xeons (40W Quads), can hold up to 32 GB ECC Ram with fast Intel Nics and 3 x pci-e 8x slots and ipmi (browser based remote keyboard and console access with remote on/off/reset - even when ih power-off state). It also supports vti-d, so you may use it as a ESXi virtual server with pci-passthrough to a embedded, virtualized Nexenta or Solaris-based zfs storage server.
    read about that mainboard http://www.servethehome.com/supermicro-x8silf-motherboard-v102-review-whs-v2-diy-server/

    minimum is 4 GB Bootdrive + 1 GB RAM + 32 Bit CPU (Nexenta)
    best is 64 Bit Dual/ QuadCore, 8 GB ECC Ram+ (all used for caching)
    If you want to use deduplication, you need 2-3 GB Ram per TB Data


    Usually i would say: Trouble free =

    - X8 Series from SuperMicro with 3420 or 5520 Intel server-chipsets (up from 150Euro),
    - X-Class or L-Class low power Xeons (Dual, better Quads)
    - Onboard SATA with Intel ICH10 (ahci)
    - use always Intel Nics
    - LSI Controller based on 1068e (always the best) or SAS2 LSI 2008 with IT firmware

    about onboard LSI1068 or LSI 2008 controller
    they are per default flashed in Raid-mode,
    think about reflash to IT mode (native SAS without Raid functions)
    see http://www.servethehome.com/flashing-intel-sasuc8i-lsi-firmware-guide/


    - Avoid expander when not needed (you need it for external enclosures or if you have more disks than you can supply with additional controller) and hardware raid at all for datapools, avoid other controller or use only if others have reported success
    Use mainboards with enough 4x or 8x slots for needed NICs (1Gb now /10Gb next) and needed Storage Controller.

    If you follow these simple rules, you will have fun.

    I would avoid SAS2 Controller ex LSI 2008 when not needed because of the WWN problem.
    (LSI 2008 will report disk related WWN instead of controller based ID)
    read http://www.nexenta.org/boards/1/topics/823


    4b. some general suggestions:

    -keep it as simple as possible
    -performance improvements below 20% are not worth to think about more than a moment
    -if you have the choice, avoid parts that may have problems (ex. expander, 4k disks,hardware-raid, realtek nic etc.)
    -do not buy old hardware
    -do not buy very new hardware
    -do not buy it too small
    -do not buy to satisfy "forever needs", think only up to 2 years but think about growing data, backups or if you need virtualization

    follwow the best to use suggestions whenever possible
    (with ZFS there are not too many)

    or if i should say it in german:
    "Geiz ist geil" ist genauso blöd wie
    "Mann gönnt sich ja sonst nichts - wobei ich schon weiß, wie man Mann schreibt"



    5. ZFS Server-OS Installation (Nexenta.*, Solaris Express 11 or OpenIndiana with suggested easy to use Live edition or with text-installer)

    NexentaCore is CLI only. With OpenIndiana or Solaris Express i would prefer the GUI-Live version because they are much more user friendly and you will get the time-slider feature
    (You can select a folder and go back in time, really nice to have)

    Download ISO, boot from it and install OS to a separate opt. mirrored boot-drive (use always ahci with onboard sata)
    You will always use the whole boot disk, so installation is ultra easy. just answer questions about your time-zone and keyboard and preferred network settings
    On Nexenta3 there is currently a bug with dhcp. After Installation with dhcp, you have enable auto-config via: svcadm enable nwam

    To configure a Ready ro Run ZFS-Server with all tools and extras like Web-Gui AMP and AFP, you could use my online installer script


    6. First time configuration or update your NAS with the napp-it web-gui, toolss and additional NAS/www-services

    [​IMG]

    Install NexentaCore, OpenIndiana or Solaris Express 11, reboot and
    login as root (or as user, enter su for root access from your home-directory) and enter:
    wget -O - www.napp-it.org/nappit | perl

    The nappit online installer will configure your os (user, permissions and smb-settings), add a web-gui + minihttpd, add/ compile tools like netcat, mc, iperf, bonnie and smartmontools


    Thats all, your NAS is ready to run.
    Manage it via browser http://ip:81

    You could setup most things via web-gui, some like acl-settings
    or snapshot access via previous version are more comfortable within Windows.


    7. add-ons:

    On top of a running napp-it installation you could add AMP webserver stack via:
    wget -O - www.napp-it.org/amp | perl

    The amp online-installer will setup Apache, mySQL and PHP


    or add Apple AFP+Time Machine support via:
    wget -O - www.napp-it.org/afp | perl

    The afp installer will compile and configure a netatalk AFP Server
    with Time-Machine support. Share settings are included in the napp-it
    web-gui in menue folders.


    8.Some useful Links about ZFS:

    excellent summary on practical tipps about ZFS
    http://www.nex7.com/readme1st

    newest docs from Oracle
    http://docs.oracle.com/cd/E23824_01/ Manuals from Oracle Solaris 11

    http://www.c0t0d0s0.org/archives/6639-Recommended-SolarisSun-Books.html (recommended books)
    http://hub.opensolaris.org/bin/view/Main/documentation (overview about docs)
    http://www.c0t0d0s0.org/pages/lksfbook.html (less known features)
    http://dlc.sun.com/pdf/819-5461/819-5461.pdf (ZFS manual)
    http://www.solarisinternals.com/ (best source of Howtos)
    http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide (best practice guide)
    http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide (best tuning guide)
    http://www.oracle.com/us/products/servers-storage/storage/storage-software/031857.htm (oracle info)
    http://hub.opensolaris.org/bin/view/Project+cifs-server/docs (cifs server)
    http://nexenta.org/projects/site/wiki (Nexenta wiki)
    http://www.zfsbuild.com/ (howto and performance tests)
    ten-ways-easily-improve-oracle-solaris-zfs-filesystem-performance (10 ways to improve performance)
    favorite-oracle-solaris-performance-analysis-commands (analysis tools)
    http://download.oracle.com/docs/cd/E19963-01/index.html (Oracle Solaris 11 Express Information Library )
    http://constantin.glez.de/blog/2011/03/how-set-zfs-root-pool-mirror-oracle-solaris-11-express (mirrored bootdrive on Solaris/OpenIndiana)

    german ZFS manual (Handbuch)
    http://dlc.sun.com/pdf/820-2313/820-2313.pdf (ZFS Handbuch)

    about silent data corruption (schleichende Datenfehler)
    (german, translate via google if needed)

    http://www.solarisday.de/downloads/storageserver0609nau.pdf
    http://www.fh-wedel.de/~si/seminare/ws08/Ausarbeitung/02.zfs/funktionen.html#n17


    Help forums, wiki and IRC
    http://echelog.matzon.dk/logs/browse/openindiana/
    http://echelog.matzon.dk/logs/browse/nexenta/
    http://echelog.matzon.dk/logs/browse/illumos

    https://forums.oracle.com/forums/forum.jspa?forumID=1321
    http://wiki.openindiana.org/oi/OpenIndiana+Wiki+Home
    https://www.illumos.org/projects/site/boards


    FAQ

    Q What are the three biggest advantages of ZFS?
    A1 Data security due to real data checksums, no raid-write hole problem, online scrubbing against silent corruption
    A2 nearly unlimited datasnaps + systemsnaps and writable clones without initial space consumption due to copy on write
    A3 easyness, it just works and is stable without special raid-controller or complex setup

    Q Can I expand/ shrink a pool
    A You cannot shrink (who wants?)
    A A pool is build from one or more vdevs/ RaidZ
    A You can expand a pool by adding more RaidZ/vdevs but you cannot expand a single Raidz of a pool

    Q Is ZFS faster than other systems?
    A ZFS was developped mainly to satisfy needs in datacenters with higest data security in mind
    A With a single disk its mostly slower than other systems with much better data security
    A It scales perfectly up to hundreds of disks in Petabyte ranges. Performance becomes better with any vdev you add to a pool.

    Q On which platform can I use ZFS
    A Newest features are from Oracle Solaris*, but you must pay for a support contract if you need patches or use it commercially
    A Best free alternative is OpenIndiana or Nexentacore (dead see -> Illumos iCore project) or commercial NexentaStor
    A On non-Solaris platforms you can use FreeBSD, I would not use current implementations on Linux
    A Newest or all ZFS features are only available on Solaris based systems



    Please read my miniHowTo's
    napp-it mini HowTo 1 - setup a ZFS Server
    napp-it mini HowTo 2 - setup a all-in one Server

    [​IMG]

    Gea
     
    Last edited: Jan 22, 2016
  2. john4200

    john4200 [H]ard|Gawd

    Messages:
    1,537
    Joined:
    Oct 16, 2009
    Good to see you back, Gea. Hopefully no over-ambitious moderator bans you this time!

    By the way, what are your thoughts on ZFS with drives that have 4KB sectors but emulate 512B sectors (and report 512B sectors to the OS)? How is the write performance in Solaris-based ZFS systems?
     
    Last edited: Dec 30, 2010
  3. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    i do not use any of them but i know a lot of people use them because they are are cheap.

    i expect them to work on any of the above systems with better support on newer ZFS releasea:
    Solaris Express 11 > OpenIndiana > Nexenta 3

    I expect 4k drives to be standard in the near future, so the 4k problem should be obsolete with newer os-versions. any benchmarking is only valid for a few weeks or months. so if you could buy, non 4k drives, its the best at the moment or you must be aware about some performance problems up to the next major os-updates.
     
  4. john4200

    john4200 [H]ard|Gawd

    Messages:
    1,537
    Joined:
    Oct 16, 2009
    sub.mesa has implemented a workaround for 4KB/512B emulation drives in his GUI for FreeBSD ZFS (I don't have a reference, but I'm sure sub.mesa can supply it if you need one).

    I was wondering if you are aware of any temporary workarounds for 4KB drives with Solaris-based ZFS systems, while we await a more permanent solution.
     
  5. Ruroni

    Ruroni [H]Lite

    Messages:
    89
    Joined:
    Jul 12, 2007
    Gave that Napp-IT GUI a shot on Solaris 11 Express in VirtualBox. Pleasantly surprised. Enabled encryption and created SMB share/user. I did find that I could take snapshots, but there was no way to rollback via the gui. I had to use cmd line, which worked of course.

    Between this and sub.mesa's project, ZFS NAS opportunities are looking good.
     
  6. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    I've been lurking Hard Forums for about a year now, so I thought it was time to register and share my goodies for some fun and advice.

    Using Bonnie I got the following benchmarks of a 5 disk, Hitachi HDS721010CLA332 raidz1-0 with the included hardware:

    Code:
    System release: SunOS openindiana 5.11 oi_148
    Memory size: 3840 Megabytes
    System Configuration: MSI MS-7623
    BIOS Configuration: American Megatrends Inc. V11.5 08/10/2010
    Processor: AMD Athlon(tm) II X2 250 Processor CPU1
    
    ------Sequential Output------ --Sequential Input-      --Random-
        --Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
    Size  K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
    7672M  71814 94 221845 59 111435 31 54921  97 424945 51 408.8   2
    
    
    And a Crystal Mark test result over SMB/CIFS from W7 running in Hyper-V over a cheap "green" gigabit switch:

    [​IMG]

    I'll have to research it some more to see if this is any good before I move any of my data over, or go with another solution (FreeBSD.) ZFS is interesting though. Maybe a beefer chip and some more RAM will boost results.
     
  7. john4200

    john4200 [H]ard|Gawd

    Messages:
    1,537
    Joined:
    Oct 16, 2009
    That CDM over SMB sequential read looks horrible. I would have expected 100+ MB/s.

    Did your virtualized Win7 boot from the SMB volume? Because I don't know how to set CDM to use a Samba share unless it was the boot volume. Anyway, if it was the boot disk, then maybe that is affecting the speeds?
     
    Last edited: Dec 30, 2010
  8. odditory

    odditory 2[H]4U

    Messages:
    3,887
    Joined:
    Dec 23, 2007
    Welcome back Gea. Napp-It on Solaris Express 11 working awesome. Vielen dank! Your GUI is actually helping me to learn command line, by removing the "barrier to entry" to set everything up initially, and then finding out which cmdline options i need to use to maintain it. If I had to set it all up from cmdline to start with I would have never bothered.

    @Ruroni: re: snapshots. You do realize if you're using a Windows client to connect to an SMB share on your ZFS pool, you can right-click a folder that's been snapshotted and use the "Restore previous versions" function? Works great. Works just like Volume Shadow Copy Service on windows server.
     
    Last edited: Dec 31, 2010
  9. Ruroni

    Ruroni [H]Lite

    Messages:
    89
    Joined:
    Jul 12, 2007
    Actually, I tried the previous version thing on the share itself rather than a folder contained within. Didn't do anything (I snapshotted before adding files, then added a single file), but the previous versions functionality was enabled (buttons not greyed out). Now that you say this, perhaps if I had folders within that share this would've worked. I'll give it another shot. :)
     
  10. adi

    adi Limp Gawd

    Messages:
    399
    Joined:
    Aug 1, 2002
    For comparison's sake, here are my CDM results. They aren't going to be perfect, as I had other network traffic going during the benchmark, but should give you an idea, and I wanted to benchmark them before my new hardware arrives.

    iscsi : 4 drive WD 7200rpm RAID10
    [​IMG]

    smb: 5 drive Seagage 7200rpm RAIDZ1
    [​IMG]
     
  11. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    Here is another test, on dedicated hardware:

    [​IMG]

    Looks no better, so my guess is the switch (DGS-2205 5-PORT 10/100/1000) being "green" is causing some performance issues. Gonna try and upgrade it this weekend and see if I can get some better results before moving everything over.
     
  12. adi

    adi Limp Gawd

    Messages:
    399
    Joined:
    Aug 1, 2002
    It could also be 1) crappy nic/ bus bottleneck, 2) SATA controller bus bottleneck.
     
  13. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    I went with a Intel PWLA8391GT and all drives are connected to the onboard sata, operating as "IDE", not "ACHI," so maybe i'll change that first.
     
  14. Ruroni

    Ruroni [H]Lite

    Messages:
    89
    Joined:
    Jul 12, 2007
    Alright. I report success. :) Too awesome.
     
  15. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    some things like snapshot access or acl settings are so comfortable within windows.
    not worth to add a menue item in napp-it now.

    but i know there are people without windows (ex san usage or mac only) so i will add these items some day - when the more important things are done, like menue-driven async ZFS-replication between host, i am currently working on.

    gea
     
  16. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    Changed onboard from IDE to ACHI, won't boot now.. :mad: Gonna re-install and see.
     
    Last edited: Jan 1, 2011
  17. lightp2

    lightp2 Gawd

    Messages:
    672
    Joined:
    Oct 28, 2009
    Hi, my apology in case it is present but I miss it. Suggest an easy menu-option for CIFS sharing to allow guestok=true by default, or default value during initial setup. Maybe this is helpful for home users completely new to Solaris ZFS filer so that they do not need to wonder about security mapping between Windows client and the ZFS host and can go straight to file sharing.

    Cheers
     
  18. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    there is a menue option for guest access in menue folder, either on menue folder - create or
    on the folder overview if you share a folder via smb.

    if you want to have guest access per default, you may edit that menu item.
    you will only need minimal perl/php or html knowledge to adjust the html form.
    napp-it web-gui is fully user editable or extendable - see also mernue extend.

    -activate menue editing (menue napp-it-setup, set editing to on or always)
    -in editmode=on, there is a edit-link in the right top, klick on it
    -klick on the "e" right of the menue item, you want to edit/modify

    if something goes wrong, there is a 3-step undo in editing or just do a update via wget..
    the old menues are still available after a update (selectable via napp-it setup from folder /zfs_date)

    on the other hand, Solaris-mapping is quite easy. create a local user administrator, add him to smb-group administrators.
    now verify that you have a mapping winuser:administrator=unixuser:root (should already be created from napp-it)
    see http://www.napp-it.org/pop_en.html

    now its quite the same handling like a real wndows-box.
    connect as administrator and set permssions from windows.

    gea
     
  19. fmaster

    fmaster n00bie

    Messages:
    18
    Joined:
    Mar 1, 2010
    Thanks so much for creating this thread, it will be very helpful.

    Maybe it is just me but, this link isn't working for me?


    -based on NexentaStor, there is a free Community-Edition
    (=NexentaStor without Service, without some add-ons, limited to 18 TB and storage-only use )
    download at http://www.nexenastor.org

    I think you might have forgotten a 't'?
    Should it have been http://www.nexentastor.org/ instead?
     
  20. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    thanks,
    fixed the typo
     
  21. lightp2

    lightp2 Gawd

    Messages:
    672
    Joined:
    Oct 28, 2009
    Hi _Gea,

    Thank you for your kind pointer. The instructions are easy to understand and I created the necessary environment based on your direction.

    I also re-read your 1st post several times plus the ESXi all-in-one description. For convenient sake, I did the test on existing VMware player environment and found the CDM scores very encouraging. This info is for some considering quick test but find full setup of ESXi demanding. My test VM runs OpenIndiana with single vcpu and 4GB RAM with ZFS mirror for CDM test. (VMware Tools installed)

    Edit : I tested OpenSolaris briefly last time and was primarily interested in Time Slider (still inside OpenIndiana). It appears now this can be done in web-gui as well (though not 100% sure because I recall reading complex blog entries by developers about putting the TimeSlider stuff in various usage scenarios.)

    Cheers
     
    Last edited: Jan 2, 2011
  22. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    Got a pair of Intel PCI-E nics, and heres the results coming from my netbook:

    Code:
    -----------------------------------------------------------------------
    CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
                               Crystal Dew World : http://crystalmark.info/
    -----------------------------------------------------------------------
    * MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]
    
               Sequential Read :    65.569 MB/s
              Sequential Write :   100.084 MB/s
             Random Read 512KB :    65.763 MB/s
            Random Write 512KB :    90.673 MB/s
        Random Read 4KB (QD=1) :     9.110 MB/s [  2224.1 IOPS]
       Random Write 4KB (QD=1) :     4.679 MB/s [  1142.3 IOPS]
       Random Read 4KB (QD=32) :    38.719 MB/s [  9452.8 IOPS]
      Random Write 4KB (QD=32) :     7.392 MB/s [  1804.7 IOPS]
    
      Test : 1000 MB [Z: 0.0% (0.0/3648.6 GB)] (x5)
      Date : 2011/01/03 23:59:35
        OS : Windows 7  [6.1 Build 7600] (x64)
      
    Gonna put another one of the Intel's in my Hyper-V setup and see how it does.
     
  23. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    Your current values seems to be nearly 30% better than your first results. Did you only switched from IDE to AHCI mode or did you also changed nics?

    But its clear, intel Nics are the best and AHCI is always better than IDE-mode, not only for speed but also because AHCI is hotpluggable.

    Your write speed is now at Gbit-level, your other values especially on concurrent access may be only better if you add more RAM .
     
  24. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    Both, a Intel EXPI9301CTBLK, and did a reinstall with AHCI active. I was thinking, my Hyper-V install runs some VMs off a Raid1, one being a WHS install with a pool of 4x Seagate Barracuda LP ST32000542AS 2TB 5900 RPM. I'm moving away from WHS to ZFS, so I want to use these drives.

    Should I create a mirror of 3 drives to add to the pool? And how does that work?
     
  25. adi

    adi Limp Gawd

    Messages:
    399
    Joined:
    Aug 1, 2002
    I'm moving from Nexenta Core+ napp-it to Solaris Express 11 with napp-it when I get my hardware on Thurs. Been playing with it in vmware, no big issues so far. Just wanted to say thanks for your work.
     
  26. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    if you have 4 disks for zfs-datapools
    you have the following options

    create a n-way-mirror example (3 or 4 way)
    very good performance, extremely safe
    up to any of 3 drives may fail without dataloss

    create two mirrored pools, one for data, one
    for backups

    create a raid-z1/(z2)
    slow, but max capacity, good for
    backup or a mediaserver
    one (two) disk may fail without dataloss

    if you need performance,
    i would buy a 5th disk and
    create a raid 10 (stripe of two mirrors)
    very good performance, safe
    up to two disks may fail if they are in different mirrors
    you shoukd add a 5th disk as hot spare

    but be aware, your disks are good for media or backups,
    they are not high speed or have a lot of iops
    you may add at least a 50GB ssd as read-cache.
    (hybrid storage)

    gea
     
  27. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    I have the following to use:

    1x SSD 32g (OS install)
    4x Seagate Barracuda LP ST32000542AS 2TB 5900 RPM
    5x HITACHI Deskstar 7K1000.C HDS721010CLA332 1TB 7200 RPM
     
  28. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    os disk is nearly performance irrelevant

    i would prefer:

    use a small and cheap bootdisk
    ex buy a wd 160gb, 2,5" 24/7
    or use any other disk

    create a 2tb high speed pool of your hitachi disks
    raid 10 + hotfix + ssd as read cache drive

    create a raid-z backup-pool of your seagate disks
     
  29. nicka

    nicka n00bie

    Messages:
    27
    Joined:
    Dec 30, 2010
    Code:
    -----------------------------------------------------------------------
    CrystalDiskMark 3.0 x64 (C) 2007-2010 hiyohiyo
                               Crystal Dew World : http://crystalmark.info/
    -----------------------------------------------------------------------
    * MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]
    
               Sequential Read :    65.503 MB/s
              Sequential Write :    85.153 MB/s
             Random Read 512KB :    65.504 MB/s
            Random Write 512KB :    84.163 MB/s
        Random Read 4KB (QD=1) :     9.174 MB/s [  2239.6 IOPS]
       Random Write 4KB (QD=1) :     3.536 MB/s [   863.4 IOPS]
       Random Read 4KB (QD=32) :    37.474 MB/s [  9148.9 IOPS]
      Random Write 4KB (QD=32) :     6.190 MB/s [  1511.2 IOPS]
    
      Test : 1000 MB [Z: 0.0% (0.0/1827.0 GB)] (x5)
      Date : 2011/01/05 21:42:57
        OS : Windows 7  [6.1 Build 7600] (x64)
      
    Seems the RaidZ was faster on the write side.
     
  30. odditory

    odditory 2[H]4U

    Messages:
    3,887
    Joined:
    Dec 23, 2007
    @gea: any thoughts on creating a ZFS pool spindown script and GUI options in napp-it? Looks like kfoda wrote his own for FreeBSD 8, wonder if it can be adapted for Solaris.

    The ability to spin down a ZFS pool is really the last remaining obstacle keeping me from migrating some of my storage off of hardware raid cards.

    Check out his script here.

    http://forums.servethehome.com/showthread.php?36-FreeBSD-vault&p=163&viewfull=1#post163
     
  31. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    With OpenSolaris & Co, you do not need such a script.
    Power-Management is part of the OS

    Basically you have to do only the following:

    -set desired cpu and disk threshold
    edit /etc/power.conf

    -enable powermanagement service via Service-Management
    svcadm enable power


    If you use napp-it, you can set it (for all disks) via menu
    system - power mgmt

    Problems about:
    You have to care, that there is no service or task, regularly accessing disks like logging or ZFS-health check services or clients, regularly checking shares.


    Optional:
    Simple way, but always working (instead or additional to powermanagement) with max. power saving:

    For my home-server i have done it much simpler.
    I know, i do not need my server past 2:00 am so i have set
    a power-off job via napp-it autojob.

    If you also have one of these nice serverboards with ipmi, you can power-on remotely via browser, even from power-off state or you can use a simple power-timer to switch it on. (and off past shutdown)

    Some links about powermanagement:
    http://www.unix.com/man-page/opensolaris/4/power.conf/
    http://www.nexenta.org/boards/1/topics/105
    http://constantin.glez.de/blog/2010/03/opensolaris-home-server-scripting-2-setting-power-management
     
  32. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    about low power solutions:
    HP Proliant Microserver N36

    There is a thread in http://forums.overclockers.co.uk/showthread.php?t=18216324&page=14 about the the cheap and quiet HP Proliant Microserver N36
    with a very good bonnie benchmark result with raid-1 and raid-z on Solaris Express 11.

    These new HP microservers with amd's new low-power dualcore cpu's seems to be a pretty nice home or backup ZFS-solution
    with 4 x 3,5" disk-bays and up to 8 Gb ECC RAM. They seems to work nicely with Solaris & Co.

    Due to its 16x and 1x pci-e slots, it may also be possible to add a 4 x 2,5" enclosure in the optical 5,25" slot for 2,5" disks or ssd's.
    Does anyone tried a extra SAS 1068 Controller in the pci 16s slot yet? You will need a CD/DVD drive only for OS-install.
     
  33. odditory

    odditory 2[H]4U

    Messages:
    3,887
    Joined:
    Dec 23, 2007
    @Gea, thanks for the info. I'm aware of the ability to power on and off the entire server on a timer, I'm just trying to find the closest way of replicating my current storage scheme which is an Areca raid card powering down idle arrays after 60 minutes of inactivity, aka MAID. I don't care about saving $$ on power, I care about extending the life of the drives. I have a many arrays and many drives and it is mostly archival data so due to the infrequency of access I don't want those disks spinning 24x7. Also I have VM's running on some of the servers with arrays (i.e. Win2008 R2 with Hyper-V) so powering down the whole server isn't an option.

    I think a simple GUI option to power down the ZFS pool after X minutes of inactivity is something a lot of people would be interested in, so they wouldn't have to edit conf files and deal with cmdline. I realize there are complexities to that - and you can't just "power down the pool" due to other background processes being hooked into ZFS that might wake the pool up unnecessarily, so I at least wanted to start a dialog on the best way to implement.
     
  34. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    set disk threshold in system powermanagement to 3600 and your disk should go down 60min after last access. But i heave heard of poblems due to ZFS-health checks on Nexenta. (Regular health check to discover disk failures prior to a failed access). I have not tried it by myself yet.

    i have also a lot of servers with many disks. but i do not expect less failures due to regular power-down. i expect a higher failure rate. i expect most failures are on power up cycles.
     
  35. odditory

    odditory 2[H]4U

    Messages:
    3,887
    Joined:
    Dec 23, 2007
    Understood, I'm just putting the idea out there, that you might consider making a GUI option in the future. I'm more interested in how to accomplish it than the pros and cons of spinning down disks as it relates to lifespan - I realize there are two schools of thought on this. People like me that live in an earthquake zone are served well to keep one copy of their data spun down or on tape.

    I assume that excessive disk wakeups due to zfs health checks would be a similar problem on Solaris Express 11. I might play around with FreeBSD 8 on account of the guy I linked that has his zfs pool spindown script working for this exact purpose.
     
    Last edited: Jan 6, 2011
  36. odditory

    odditory 2[H]4U

    Messages:
    3,887
    Joined:
    Dec 23, 2007
    Somewhat related, my dream is a multi-level, nearline approach to archival data storage, be it based on ZFS or anything else, where you've got one large pool of write-infrequently/read-infrequently data (say 16 x 2TB in raidz2), which can stay spun down or low-power-idled most of the time, and then a smaller pool (2 x 2TB in raidz, or multiple SSD's) that acts as a buffer/cache and flushes to the larger pool at a user defined interval, say every 24 hours, or every week, or month, or when its X% full. A "landing pool" basically. This process is transparent to the client, all that's seen is one pool. This functionality is obviously beyond the way L2ARC and ZIL function right now. A few commercial third parties have attempted this with ZFS, but obviously it would require getting into the source for that kind of ability. Hopefully this is something we'll see in the coming years.
     
  37. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
  38. _Gea

    _Gea 2[H]4U

    Messages:
    3,530
    Joined:
    Dec 5, 2010
    Just saw this thread from Zarf at http://forums.overclockers.co.uk/showthread.php?t=18221307 about HP's new HP Proliant Microserver and Solaris 11

    101 MB/s via SMB would be a ultra excellent result with such a low power, low cost server. Could anyone with a HP Microserver and Solaris 11 verify such high values?


    Zarf wrote
    "Using Solaris Express 11. I was on OpenIndiana, but switched over to the dark side to see if it made any difference (which it didn't). Now its working I cba to go back.
    I sorted things out, still not exactly sure what did it but I get 101MB/s sustained reads and writes over SMB now. There's a great web configuration interface for ZFS called napp-it if you are interested. It's easily installed via perl script from the internet.

    Among the things I did were upgrading to 4GB RAM, disabling sync on the smb shared zfs folder.."
     
  39. jonnyjl

    jonnyjl Limp Gawd

    Messages:
    195
    Joined:
    Apr 12, 2009
    Hmm I wonder how risky disabling sync (I imagine he means the zfs sync= property). I briefly read about it.

    Honestly I average around 90+MB/sec with smb, I have 12GB of ram (I also have a couple of VirtualBox VMs running inside OS), so probably not worth it, but maybe something I could try on a Share I don't care too much about.

    I find most of the time my slowdowns are my workstation and not the server. (oh and the network... man I need a beefier network...)

    Not to hijack... but has anyone done the OpenSolaris to Solaris 11 Express? I've been reading the blogs, just wondering if anyone has some first hand experience and any unexpected hiccups. I would hope that the upgrade would be smooth if you're running b134... of course unless you're running PostgreSQL :) I'd probably keep my zpool/zfs levels as I'd want the option to go back to b134.

    PS Keep up the good work Gea. I don't personally use Napp-it, but impressive project. I should give it a whirl some time. I'm unclear do you utilize time-slider for the snapshots or your own scripts/cron jobs?
     
    Last edited: Jan 10, 2011
  40. cymon

    cymon Limp Gawd

    Messages:
    453
    Joined:
    Apr 16, 2009
    Hierarchial storage, huh?

    It looks like OpenArchive might be able to do something like this. OpenArchive can use NFS/CIFS as a backend. It can maintain the most frequently used data on one array, then push out the less-used data to a larger archive pool.

    Here's a PDF on it:

    http://www.openarchive.net/sites/default/files/file/0309-OA-QG-E-350.pdf

    EDIT: Looks like I was wrong, it doesn't look like you can use NFS as a storage backend. However, it will accept RAID (so hey, you can keep those Arecas!)