Linux/OmniOS/Solaris for ZFS?

farscapesg1

2[H]4U
Joined
Aug 4, 2004
Messages
2,648
My OpenIndiana server is driving me nuts and on it's way out. Something got corrupted during an update and it has been limping along since then. For whatever reason, the server rebooted last night or this morning and anytime it restarts, if I don't select a previous boot snapshot, Napp-it, Comstar, and other services are broken. Unfortunately, my knowledge of Solaris is extremely limited, so trying to troubleshoot has taken me forever and gotten nowhere.

At this point I feel my best option is to rebuild using a spare Xeon X3440 and Supermicro X8SIL-F motherboard (slight upgrade to my current X3430). Will also be going from 16Gig RAM to 32Gig. My original plan was to switch over to Linux because Crashplan has a supported client. However, in the meantime I installed the client on a windows system and have it backing up, so that is less of a priority at the this point.

At this point, I need to decide on a new OS and I'm not really sure what the benefits/costs are for each of them. My current server is connected to my ESXi hosts via iSCSI using fiber cards, to provide storage for VMs. It also houses 6 3TB drives that serve for file storage. When it comes to any of the OS's, I'm equally incompetent with all of them ;) Solaris seems to have the newest features (improved resilvering), but technically is only to be used for development/testing unless purchased. OmniOS seems to be the most streamlined/simplified OS for ZFS at the moment. Linux gives me the flexibility of being able to install additional services on the server also (torrent/usenet software, Plex media server, SageTV PVR, Ubiquiti Unifi controller, etc.).

Obviously, I'm leaning more towards Linux but I just wanted to make sure I got a good list of pros/cons between the options. The decision to keep the storage server separate or to make it a VM on a 3rd R710 host I have as a spare will come later. Especially since that decision depends on spending money on external HBAs, a SAS expander, and cables.
 
If you go linux use debian. They've recently mainlined zfs into the repository which should help keep it up to date and in sync with the kernel.

Edit: Also my vote is for linux :)
 
I just setup ZFS on Linux Mint today... just because that's the OS I had installed, and I figured why not? (It works fine btw) I'm just testing some large SSD pools on it before deployment... but likely will go with Ubuntu. My main issue with OmniOS or some other is I need IB drivers to work and ideally NVME too, which pretty much they fall behind Ubuntu.
 
I had a standalone openindiana with nappit for years but it was getting buggy after each update. So I decided to move another solution.

I first try freenas again still on freebsd 9 (10 should be release real soon):
pro: - lots of new features, (so many supported plugins now owncloud too) huge community, lots of guides, works in VM well
cons: i really didn't like the gui, i just found it hard to navigate and I didn't the resource the gui takes up

nas4free on freebsd 10 now (even have ARM build for raspberry pi)
pro - nice simple interface, lots of feature and plug in since i last try it,
cons: loading driver still bit tricky, the wiki isn't documented well, had some VM issues with nas4free

OmniOS
pros - very easy to setup because of _gea senpai well documented pdf and community support here.
cons - i kind of missed the desktop since I don't know many cli and always have to google them

I ended up with OmniOS because for other needs like owncloud I can always setup a VM, OmniOS is very well zfs OS! so far been transferring data over for 2 days now over Ethernet, I wish i had fiber cards like you :D

but for linux zfs, I hear many goods things with debain + zfs. Maybe i will test with my old openindiana box after all the datas been transfer
 
Each platform has advantages regarding features, hardware support or available extra services/apps.

Solaris bases systems for example have the best integration of ZFS and storage services into the OS. Suns own NFS and SMB services are fully integrated in ZFS and offer some unique features regarding Windows permission compatibility. Combined with Comstar/iSCSI and Crossbow network virtualisation it offers all storage coreservices integrated by Sun/Oracle as a just enough server OS with all services from one vendor.

There are the typical services like amp etc around (mainly via pkgin from Joyent) but compared to Linux this is far below. This is mainly an item if you want to add additional barebone services but this is not what I would do if you have virtualisation. I would keep the storage server as small as possible otherwise you must care of backup and recovery. If you use VMs you have easy backup, recovery and snaps. On problems with the storage server, you just need to reinstall, import the pool and you are done. This paired withe the bootable ZFS root filesystem with bootenvironments where you can go back to a former system state makes recovery easy.
 
Last edited:
If you go linux use debian. They've recently mainlined zfs into the repository which should help keep it up to date and in sync with the kernel.

Edit: Also my vote is for linux :)

What? Where is this on the Debian repos?
 
Thanks for the input everyone. Gea brings up a good point that I usually try to stick to.. keeping server roles separate. However, this is a home lab/prod environment that is spreading out of control according to the family :p When the wife calls me at work to say the TV isn't working, I have to track down if it is Plex, SageTV, Unifi (for wireless devices), etc. I'll have to rethink some of my consolidation ideas.

How is iSCSI support on Linux? I'm currently using iSCSI for my vmware hosts and one VM (my SageTV PVR). I'm not dead-set on it, just what I'm comfortable with and my performance seemed to be better when I first set up the server, compared to NFS. That is probably due to having sync disabled on the pools (no L2ARC or SLOG devices). Yes risky, but with a UPS and backups, if I have to rebuild I can. I remember trying out FreeNas back when I originally built this and there were iSCSI issues with VMWare at the time so I moved on.
 
@ farscapesg1
I'm going to vouch for FreeBSD, good ZFS support and iSCSI. If you're looking for additional services you have jails if you're paranoid and also bhyve if you want to run virtual machines. 10.2 works fine but -HEAD works even better and noticably faster overall. I've been using it at home for years now at home, I use 10.2 in production however. =)
If you need addtional software you have the ports tree which is very easy to maintain.
 
...
but for linux zfs, I hear many goods things with debain + zfs. Maybe i will test with my old openindiana box after all the datas been transfer

ZOL should be good on all abroad linux distro on my understanding

I am running ZoL on centos 6.X and 7.X....
 
I'd be very careful about ZoL since it diverges from OpenZFS unless those changes have been reverted.
 
I'd be very careful about ZoL since it diverges from OpenZFS unless those changes have been reverted.

please look on http://open-zfs.org/wiki/Main_Page
OpenZFS was announced in September 2013 as the truly open source successor to the ZFS project. Our community brings together developers from the illumos, FreeBSD, Linux, and OS X platforms, and a wide range of companies that build products on top of OpenZFS.

since ZoL running on DKMS, there are some mechanism needed to "make work" on internal DKMS

ZoL no need to worry...

what changes?
 
Here you go. It got added within the last few weeks.
https://packages.debian.org/wheezy/zfs-fuse

Uh, you definitely don't want to use that.

That's an ancient FUSE version of ZFS which is not compatible with the standard OpenZFS that Illumos, BSD, ZoL (kernel module), and OS X use.

I'd be very careful about ZoL since it diverges from OpenZFS unless those changes have been reverted.


There were never any changes to ZFS on Linux that diverged from OpenZFS. Maybe I can see you talking about a bug or something that broke pools. ZFS on Linux at worst case was sometimes behind in implementing the latest on-disk feature changes that were developed in OpenZFS, but it has been all caught up for some time now and remains completely compatible with OpenZFS on the other platforms.

You can create pools with all features enabled in Illumos or BSD for instance and use them on Linux and vice versa. OpenZFS project exists to reduce the number of differences between the codebases between the platforms so there only exist less and less differences as time goes on. I always see people merging in Illumos changes to ZoL from upstream.

https://github.com/zfsonlinux/zfs/commits/master



Both of these were bugs that could have existed in any implementation of OpenZFS. They were fixed years ago and there have not been any such big bugs (compatibility breaking) like that since. I mean it wasn't that big of a deal as they fixed the bug and made sure the code in the bugfixed release fixed the pool (via a scrub) to make it re-compatible with OpenZFS again as it was always supposed to be.
 
Last edited:
I've seen numerous reports of importing Linux ZFS pool into other OSes so I'd still be very careful about relying on that feature. That said, I wouldn't run Linux in the first place if I were to use ZFS.
 
Well you don't sound like you have actually used it yourself so I'm not sure why you are giving recommendations on it.

I've actually been running ZFS On Linux myself for over 2 years and have not had any issues. I've imported my pool on both FreeBSD and Illumos too (OmniOS and SmartOS).

I'll take my own actual experiences with it over someone just seeing some reports.
 
Last edited:
Well you don't sound like you have actually used it yourself so I'm not sure why you are giving recommendations on it.

I've actually been running ZFS On Linux myself for over 2 years and have not had any issues. I've imported my pool on both FreeBSD and Illumos too (OmniOS and SmartOS).

I'll take my own actual experiences with it over someone just seeing some reports.


we should move on, should'n we :)

just ignore that. this is useless and wasted your time by arguing with non-sense knowledge

I am,
Running ZoL since 2010..
first started with OmniOS :D... in early 2010.


I honestly don't remember the exact commit but it was either some compression algorithm or a flag that wasn't standard and Linux only. Oh well, you're most like better off with btrfs if you're going with Linux.
http://zfsonlinux.org/msg/ZFS-8000-ER/ <-- Yey ;-)

are you honestly? :p heheheh
yey, already fixed!!!!
:p
please reread my prev posting again :D
 
As I said, there has been numerous reports about this so don't take it for granted. That just proved my point...
 
Oh boy.. it's almost like I started a Ubiquiti thread :p

Back to my original intent of my questions.. what features, if any, would I be missing going with Linux vs OmniOS for example? Or does it really just boil down to true Solaris having the new resilvering features.

How is LIO stability/features vs Comstar? Any driver/support issues for the QLogic 2400 hba series since I'm already using them, without a fiber switch? NFS is still on the possibility list, just deciding if a Samsung 830 would be any more "risky" than just running with sync disabled which is what I have been doing for 2 years now.

Since FreeBSD was mentioned, how is FreeNas nowadays? I've read a couple articles so far and CIFS/SMB performance was listed as subpar compared to OmniOS for example. Not to mention NFS performance.

Just to throw another wrench in the mix, I'll probably be converting one of my ESXi hosts over to Hyper-V due to a shift in outlook for hypervisors in the future at the company I work for. Need to start familiarizing myself to fit in with the plan going forward :(
 
For me the biggest advantage of a Solaris based system is that you get all from Sun/Oracle/Illumos what is needed for a storage server . ZFS is the only available filesyestem and services like NFS and SMB are fully integrated in ZFS and the kernel with included services like Comstar and Crossbow virtual networking. Its a just enough storage server OS. Do a minimal setup like OmniOS and everything is included. Only one solution for one problem that just works without 3rd party software needed. Same with disk detection. Disks have only and always WWN identification with current HBAs that stay the same - does not matter what controller, server or OS or hot spare support that just work as well as pervious version support from Windows. You do not need to setup anything.

Especially the fast multithreaded SMB server with support for Windows SMB groups, Windows SID and Windows ntfs alike nfs4 Permissions was the final argument for me to switch away completely from Windows to ZFS. This is a unique feature of the Solaris SMB server compared to SAMBA the other SMB server option for any Linux/Unix.

Especially on Linux you have a wider support of hardware and more applications. Its more a toolbox where you always find one or mostly some solutions for any problem but not the "just work" experience.

Regarding hardware this mean that you must select the correct hardware on Solaris based systems while on Linux you may find drivers support for nearly any hardware in a server.
 
@ farscapesg1
No idea about FreeNAS but FreeBSD 10.2 and newer (-CURRENT has better overall however) works just fine using Samba 4.3 and has very good performance. Samba also does NFSv4 permissions nowdays...

As for application support you have a much wider range than what Solaris offers and most what yo'll every use is already in ports so it's not an issue. You can give http://www.freshports.org/ a quick search if there's anything in particular you want.
SageTV just good ported too... :)
http://forums.sagetv.com/forums/showthread.php?t=62629
 
Well, crap! My storage box took a complete dump tonight and refuses to come back up. Can't get a desktop on Openindiana, can't even SSH into a console session. Even previous boot environments won't load.
Looks like I'll be in the process of rebuilding tomorrow. Unfortunately, I was still in the process of evaluating options, both software and hardware. Software was down to Linux or OmniOS, leaning towards Linux because I'm not making the X3430 I was using even blink, let alone sweat. Just makes sense to me to throw a few other things on the box since it is on 24/7 anyways.
Hardware side, I could leave it as a standalone... Or I have an extra Dell R710 with a X5550 and 64GB. At idle, there is about 50W difference. The thought was to possibly replace the CPU with a L5640 or similar, replace one M1015 controller (currently using two) with an external hba, and connect the old Norco case as an external SAN box until I can replace it with a UNAS or similar 8-bay case.
Oh well, guess the hardware is decided for now. Don't want to run off my backup HP microserver any longer than I have to...
 
Honestly, it is just lack of knowledge. I've installed and managed a couple Linux systems and been running my Openindiana server for a couple years, so I feel a tad more comfortable with them. Considering I need to get this back up and running ASAP (all my TV/movies/family photos/etc.), I just don't feel like this is the best time to try learning a new OS from scratch. I'm not ruling it out long term though...
 
Good news.. after sleeping on it and having more time this morning to investigate (without the family screaming "why can't we watch Flash/XFiles/etc.") I've got it limping along again. Pulled out all the drives, rebooted and it came up!. Put the drives back in and restarted, one array at a time until I tracked it down to my mass storage array. Then had to use the same process one disk at a time until I found the offending drive. Once that was removed... the system boots up and the array is running in a degraded mode. The drive never threw any errors.. just crapped out. My desktop sees it in an external dock, but at the wrong capacity and it times out. Looks like the hba was continuously trying to connect and instead of getting errors it would just time out and crash the boot process. Stupid Seagate drives...

Gives me time to play with FreeBSD though...
 
It's actually very easy, if you want some pointers I have small guide I did for some ppl out basic setup that's short and consistent ;-)
 
Back
Top