Storage Server: If it were you

[auxiliary]

Good Evening.

I'm looking for some advice on some new hardware we just picked up and the best setup for our environment. We are an IT department for 6 school districts and are deploying 2 new SuperMicro 6047R-E1R24N chassis. Found Here.

The servers are populated with 24 Seagate Constellation ES3 1 TB NL SAS2 drives connected to a SuperMicro badged LSI 2108 controller. Each server has 32 gigs of system memory and dual E5-2603 Xeons.

The servers will be used as shared storage for 2 XenServers connected via 10GbE. NFS is the preferred protocol. The data will be VM VHD's of Windows 2008R2 and 2012 servers, Exchange, SQL 2012 and other random data. Although we use SQL, we do not need the typical IOPS because our usage is low (typically less than 15 users accessing 1 to 2 databases)

Some of the questions I have...

1. If you were building this setup, what route would you take in terms of the operating system and file system? FreeBSD, OpenSuse, FreeNAS?
2. Hardware RAID or Software RAID?
3. Would you build one server for the storage, and then use something like OpenSuse as a cluster for failover to the 2nd server or just go on a OBR10 model where the reliability is high considering controllers rarely fail?
4. Probably a lot more but it's late and I'm tired...

Additional Notes:
We do not have any additional funds for SSD's or other drives to put towards L2ARC/ZIL.
It's unlikely we could purchase 16 gig USB drives for mirrored OS installation either.

I've probably left a ton out so feel free to ask any additional information you need. I'm just looking for some advice since I'm been banging my head for a couple weeks on the options. I have a feeling I'm just making it more complicated than need be.

Thanks in advance.

 

[_Gea]

1. use ZFS Software Raid - nothing else
2. decide between BSD, Linux and Solaris variants

- Solaris/ OmniOS: best performance and proven stability, this is where ZFS and ZFS development comes from
- BSD (FreeNAS etc), best alternative when you need programms, not available on Solaris
- Linux/ZoL: would not use yet


Question:

why do you use a expander solution where you need expensive SAS disks
A ZFS solution with larger Sata disks and several HBA controllerws would be cheaper and maybee faster

why do you use dualcore boards for storage when you only plan to use 32 GB RAM
a single (cheapest) Hexcore would be much cheaper, you produce nothing than idle CPUs


My top-deal would be: (If you just own the hardware, skip that part)
- same case but without expander (maybee you can use a 16 bay or SuperMicro 2,5" cases with SSD only pools or 2,5" spindels)
- 3 HBA controller (LSI 9211, IBM 1015 etc)
- X9SRH-7TF mainboard (single 2011, hexcore, 10 Gbe onboard, SAS HBA onboard)
- 64 GB RAM or more

- less 4 TB Sata disks (I would use a slow Raid-Z2/3 backup pool and a fast SSD only datastore pool for VMs)
- add a really fast ZIL (ex Intel 200 GB SSD S3700) for performance with secure sync writes
- Solaris based OS (I prefer OmniOS stable)

- add a cheap Netgear XS708 switch (8 x 10 GBe)

last:
keep it simple,
If you need real HA, your complexity grows enormously.
Mabyee a async replication (sync pools every 5 minutes) between the two boxes is "good enough"

not to forget:
LSI 2108 is a hardware raid, really bad choice for ZFS (you loose all self-healing ZFS features)
use one of the following: http://www.lsi.com/channel//products/storagecomponents/Pages/HBAs.aspx
ex LSI 9211, IBM 1015 (cheapest) or LSI 9207

 

[MrVining]

1. use ZFS Software Raid - nothing else

why do you use dualcore boards for storage when you only plan to use 32 GB RAM
a single (cheapest) Hexcore would be much cheaper, you produce nothing than idle CPUs

I agree, ZFS all the way. If you care about the data then it should be on ZFS, nothing else even comes close.

Depending on how you configure the drives you're just barely within best practices. You'd get more use out of more RAM than a 2nd CPU.

As far as configuration of the raid:
RAID-Z with a hot spare (if you like to walk on the wild side)

RAID-Z2 with someone ready to replace the bad drive ASAP (you have more than enough CPU for this)

RAID-Z2 with hot spare (nice and safe data aside from a fire or something)

I wouldn't bother with mirror + stripe for what you are describing, you are building general storage not database storage. IMO

 

[spazoid]

I don't ever see raidz or raidz2 with hotspares as making sense.. Might as well just put it in the vdev and have it already "resilvered"..

Obviously, if you have more vdevs and it's not possible to add another disk to each vdev, then it can make sense, but that doesn't seem to be the case here..

 

[cantalup]

do you prefer software or Hardware RAID?...

mostly on budget wise.. software RAID is a good pick.,

current favorable in this forum is ZFS
some using mdraid (linux). or can use windows flavor..

before jump . read as much as possible to strengthen you decision.
------------
on mine
1) running Hardware Raid, HP RAID ( and old adaptec RAID series 3).. some people hate it, I am OK....
2) running mdraid. yeah, stable... but will move to btrfs (or could running btrfs Raid0 on mdraid..)
3) runnging ZFS on OI,, had some issue with OI, slowing down( did want to debugging in hours or weeks) on NIC.. kind of stalling. I move to ZFS on Linux, ZoL. running Good, since I do not use funky functions on ZoL. one recent thread on ZoL release -> http://hardforum.com/showthread.php?t=1753806&highlight=zol . I use ZoL as a backup server and planning to add one more server with ZoL. (honestly, I prefer btrfs raid5/raid6, but still on going/development state. still waiting...)



if you need to stick with GUI style, stick with ZFS on OI or solaris flavor. winks to Gea

 

[dilidolo]

Take a look at Tegile. I'd use them rather than build my own.

 

[auxiliary]

Thanks for all the replies and information. After more research, I think I'm going to stay away from ZFS and just run OpenSuse with LVM and ext4 on top of the RAID10 array from the LSI controller. I'll retain true hotswap and not have to mess with learning ZFS.

Since I don't want to throw more ram at it, and don't have SSD's and other equipment for caching, ext4/LVM makes the most sense.

 

[JoeComp]

Since I don't want to throw more ram at it, and don't have SSD's and other equipment for caching, ext4/LVM makes the most sense.

You may want to consider XFS instead of ext4. For a lot of workloads, XFS scales better.

 

[twistacatz]

Thanks for all the replies and information. After more research, I think I'm going to stay away from ZFS and just run OpenSuse with LVM and ext4 on top of the RAID10 array from the LSI controller. I'll retain true hotswap and not have to mess with learning ZFS.

Since I don't want to throw more ram at it, and don't have SSD's and other equipment for caching, ext4/LVM makes the most sense.

ZFS is not complicated to run and the benefits are overwhelming. I would do a little more research. Hot swap is no problem with the equipment you have and SSD's and more RAM are not needed.

 

[TheDingy]

auxiliary: I used to be a hardware raid fan, big time, a while back I started using a thumper that ran ZFS, now I wouldn't have hardware raid for anything. I recently grew a system from 48TB to 96TB without ANY downtime. I do iSCSI exports, NFS, SAMBA, AFP all from a single system(actually a primary and a secondary) and it works flawless. I do not run FreeNAS, I did start with that, but I now run a combination of Nas4Free and Solaris. If you aren't willing to learn new technology then I think that you may want to look for something out of IT, because if you aren't learning you are failing and doing a disservice to your taxpayers(Which are your employers) by not looking for the best and most cost effective solution but the easiest way out.

BTW: I run over 40TB on a single i5 processor with 32GB of ram and a hotswap backplane, I can hotswap and it runs flawless under a huge load at all times so saying you need to spend more $'s is a cop out.

 

[bAMtan2]

Thanks for all the replies and information. After more research, I think I'm going to stay away from ZFS and just run OpenSuse with LVM and ext4 on top of the RAID10 array from the LSI controller. I'll retain true hotswap and not have to mess with learning ZFS.

Since I don't want to throw more ram at it, and don't have SSD's and other equipment for caching, ext4/LVM makes the most sense.

you came to the wrong place unfortunately, but I think you made the right decision. zfs is the best file system, but it's not better enough to beat the extra risk introduced by taking on something totally new in a production environment. don't listen to these yahoos.

 

[TheDingy]

you came to the wrong place unfortunately, but I think you made the right decision. zfs is the best file system, but it's not better enough to beat the extra risk introduced by taking on something totally new in a production environment. don't listen to these yahoos.

I am a yahoo I am sure, when did you start? My first cert was a CNE on Novell 3.11. I have also ran multiple racks with each rack running 800 amps of power(Think high performance and high availability). I am currently working on a project that has a large number of InfiniBand systems running in a cluster with a total power draw in the many megawatt range. Yes, I am a yahoo that doesn't know what he is talking about. A competent admin should be able to learn ZFS in a couple of hours.

 

[twistacatz]

I am a yahoo I am sure, when did you start? My first cert was a CNE on Novell 3.11. I have also ran multiple racks with each rack running 800 amps of power(Think high performance and high availability). I am currently working on a project that has a large number of InfiniBand systems running in a cluster with a total power draw in the many megawatt range. Yes, I am a yahoo that doesn't know what he is talking about. A competent admin should be able to learn ZFS in a couple of hours.

I could not agree more. While I would never say ZFS is simple it's really not complicated at all. In the end of the day auxiliary is the one who is responsible for the array so if ZFS is over his head by all means he should pick something that hes comfortable with and can be accountable for.

 

[_Gea]

The whole ZFS storage management is based on only two CLI commands zfs and zpool

http://docs.oracle.com/cd/E19082-01/819-2240/zfs-1m/index.html
http://docs.oracle.com/cd/E19082-01/819-2240/zpool-1m/index.html

With these two commands, ZFS storage is easier to handle than any other solution I am aware.
If you dislike CLI commands or if you want easy management, you can select one of the Web-based ZFS appliance solutions:

BSD based
FreeNas
Nas4free
ZFSGuru

Solaris based:
napp-it + [ a regular Unix server OS: OmniOS/OpenIndiana/Oracle Solaris ]
NexentaStor

 

[JayG30]

Not to derail the thread, but it seems the general consensuses is that Solaris based systems are the way to go over FreeBSD. I already know the history and lineage of these systems and how they relate to ZFS.

As of today, the illumos based options (OmniOS) aren't really providing a more "up-to-date ZFS" then FreeBSD correct? The only way you are getting a newer ZFS version is Solaris 11 commercial. The issue is simply the underlying OS, Solaris vs FreeBSD. And in this regard Solaris tends to be more optimized for storage servers and ZFS, correct? With the GUI based tools available in Solaris, I'm confused why more people don't go this route (or at least that I've read/seen). I'm reading that you might have a need for "FreeBSD applications not available to Solaris". Now, I've worked with Solaris in enterprise before and dabbled in FreeBSD in the home world, but I'm unclear of what type of application someone might want/need in FreeBSD and what they would be "giving up". Perhaps someone can enlighten me.

Also, is it a possible/simple task to run the illumos based OS'es (lets say OmniOS) and all other necessary features from a 4/8/16GB USB stick? I know this is something I really like about FreeNAS, and that it's optimized and expected to be run this way.

 

[_Gea]

If we talk about storage, and I am in this job since 25 years,
I have learned that quality of storage is mainly a filesystem item:

Filesystems up to some Gigabytes are well with FAT or HFS filesystems,
up to several hundred Gigabytes EXTn, HFS+, NTFS or XFS are ok.
Above that, they are a nightmare if you look at needed filechecks (offline, mostly useless, last days),
silent errors (cannot detect or repair) or modern features like unlimited snapshots (require CoW).

This is where ZFS come in. A filesystem, developped by Sun, called last word in filesystems.
No other FS offers similar features or data security - maybee Btrfs or Win ReFs some day.

Yes I agree, ZFS is best at Solaris & Co where ZFS comes from and where development is done,
either at Oracle (commercial tree) or Illumos (free fork). It is a pity that we now have two independent
development trees with Oracle Solaris/ ZFS v. 30+ and ZFS encryption as a unique feature and
Illumos with ZFS v. 5000/feature flags and LZ4 compression as a unique feature (already available in OmniOS).

I think, we have two development lines in future (unless Oracle reopens Solaris, I do not expect)

You ask about applications.
Solaris is superiour in ZFS, iSCSI (Comstar), Networking (Crossbow), Virtualization (KVM), SMB Server (Solaris CIFS)
or SMF service management. All from one enterprise and perfectly integrated.

If you look at a special mediaserver at home, a special webserver with a special module or similar, it can happen
that this can be a nightmare to compile yourself. Solaris is definitely nr 1 especially in storage.

about USB sticks
Solaris is a datacenter OS. But with a minimal server distribution like OmniOS and disabled atime, it runs perfectly from
fast 16 GB USB sticks. USB sticks do not affect fileserver performance on Solaris but reactiveness of a GUI or Web-UI.

With newer sticks, USB + OmniOS is a perfect duo - I would only mirror two of them for reliabilioty. (ZFS boot mirror)-
like i do with my USB distribution napp-it to Go.

 

[JayG30]

Ah, ok. When people start talking about applications I start to wonder since from what I know Solaris runs pretty much anything I can think of. MySQL, OpenVPN, nginx/apache/etc., bitTorrent (transmission), etc.. It just seems like this shouldn't really be an issue.

It does seems like, although it is an option, Solaris on a USB stick isn't quite as evolved as FreeNAS. OmniOS+napp-it wouldn't be a memory based install would it? The only thing I've seen on the Solaris side that looks comparable to FreeNAS would be EON. Or am I incorrect on this?

Thanks

 

[TheDingy]

JayG30: Look at Nas4Free over FreeNAS FreeNAS is going for the $'s where as Nas4Free is going for reliability and current updates/patches over the $'s.

 

[auxiliary]

I should have been more clear in my response. I just didn't want to create a large thread that eventually might not lead to where I was going and waste people's time.

It's not learning ZFS that is any issue, I'm familiar with it and understand how to set it up. It's the fact that there are 5 other techs who work in this department and it's not always going to be myself managing the setup. I have to provide something that monkey-brained techs can walk into the server room at another site, see an amber light and pop in a new disk that rebuilds automatically and doesn't cause an issue. Those techs wouldn't necessarily be trained in what ZFS is let alone the commands to remove the disk before replacement and get a resilver kicked off.

To me, already owning the LSI controller and the hardware makes sense to use it even though I know that ZFS would be nice for bit rot and it's other features. Having a simple RAID10 with good performance that I know will be reliable and any of our staff could go swap a drive for is worth it.

Edit - I even considered running the hardware RAID10 but with ZFS still as the filesystem. But that means I'm running Solaris or BSD and again, it puts the system into the "only I can run it" category.

 

[_Gea]

I

Edit - I even considered running the hardware RAID10 but with ZFS still as the filesystem. But that means I'm running Solaris or BSD and again, it puts the system into the "only I can run it" category.

Even on Linux/ZoL, ZFS is the most secure solution,
but avoid hardware raid. Only OS without ZFS is Windows and ReFS is miles behind.

With hardware raid, ZFS can report every error that no other filesystem can detect -
but cannot repair. You miss all self-healing ZFS features.

ZFS (and descendants like Btrfs and REFS) are a break-through for high capacity storage technology.

 

[auxiliary]

Well, I just found out the 2108 can't be flashed to an IT firmware, so if I wanted to go the ZFS route I would have to get the vendor to replace the controllers now.

Guess we'll see. Thanks for all information.

 

[TheDingy]

auxiliary: You can easily teach any "monkey" to learn zfs and bsd, if you can't then maybe you should interview better. To replace a ZFS disk it is as simple as zpool replace MainStorage /dev/ada2 /dev/ada3 also you can grow your filesystem on the fly with ZFS. My 96TB system started as 6TB years ago and is now at 96TB with no problems.

 

[bAMtan2]

auxiliary: You can easily teach any "monkey" to learn zfs and bsd, if you can't then maybe you should interview better.

typical zfs cult post.

he's a public servant with limited money and staff, and he basically just needs any NAS. and the great idea from hardforum is what?

take on a totally new work load to prevent a bit rot risk that you can't even quantify because it is so small? sorry folks. you're wrong. real talk

 

[GeorgeHR]

It's the fact that there are 5 other techs who work in this department and it's not always going to be myself managing the setup.

That consideration is usually all that is important.

Since you know your staff, do something they can handle.

 

[cantalup]

Thanks for all the replies and information. After more research, I think I'm going to stay away from ZFS and just run OpenSuse with LVM and ext4 on top of the RAID10 array from the LSI controller. I'll retain true hotswap and not have to mess with learning ZFS.

Since I don't want to throw more ram at it, and don't have SSD's and other equipment for caching, ext4/LVM makes the most sense.

pick what is the best for your situation, some are good for you, but not good for others.

I think, you already dig a lot of information

LSI (Hardware RAID) is reliable as I know, not like current Adaptec HW raid :|.

HW RAID is simple, unplug a bad HD, and replug a fresh HD, the Hardware would sync automatically

on SW RAID, mostly we deal with extra step to resyc/silvering.., not many steps , just extra...
-------------------------------------------------------------------------------
I use HW and SW RAID at home and work, I would pick the best solution for me(home) and for my works/projects; ZoL, HW Raid, and mdraid(would be btrfs on top),

good luck!!!

 

[TheDingy]

typical zfs cult post.

he's a public servant with limited money and staff, and he basically just needs any NAS. and the great idea from hardforum is what?

take on a totally new work load to prevent a bit rot risk that you can't even quantify because it is so small? sorry folks. you're wrong. real talk

Not ZFS cult, but the best/lowest cost solution post on this. Where we need it, we use NetApp but a public servant with limited money/staff etc should look at ZFS. We also use GFS regularly but with what it sounds he is using it for ZFS is the best fit. It isn't about bit rot, but long term solutions, not a short vision type of solution etc. what happens when that great hardware raid solution runs out of space?

 

[Billy_nnn]

typical zfs cult post.

he's a public servant with limited money and staff, and he basically just needs any NAS. and the great idea from hardforum is what?

take on a totally new work load to prevent a bit rot risk that you can't even quantify because it is so small? sorry folks. you're wrong. real talk

To be fair, the advice to go ZFS was certainly reasonable given the information in the original post - however the goalposts were later moved when "provide something that monkey-brained techs can walk into the server room at another site, see an amber light and pop in a new disk that rebuilds automatically and doesn't cause an issue" became a must-have feature.
You could argue till the cows come home about how easy/hard disk replacement is on ZFS, but the fact is that it doesn't offer simple hot-swap with auto-rebuild - if that's the main decider, then HW raid is the only viable option (though even then it's not always as easy as "amber light on...pop in new disk...walk away")

 

[Rectal Prolapse]

Is there really a solution that you can look at the amber light, swap the hdd, and walk away? Without having to log in to the system and barely even knowing what a hard drive is for?

I think fast food restaurants require more brain power than that! I would hate to have mindless staff like that. Ugh, the OP is in a bad spot. Too bad he didn't specify the low IQ requirement in the first place.

 

[_Gea]

You could argue till the cows come home about how easy/hard disk replacement is on ZFS, but the fact is that it doesn't offer simple hot-swap with auto-rebuild - if that's the main decider, then HW raid is the only viable option

Why do you think that hotswap and autorebuild is a hardware-raid feature?
For hotswap you need a hotswap capable controller in both cases. With ZFS naarly any LSI HBA supports this. A cheap IBM 1015 is a perfect hotswap capable controller.

The same with auto-rebuild. You need no more settings than to add a hotspare disk to your ZFS pool. In case of a failure, a rebuild starts automatically. For a manual replace, you only need to hot-insert a new disk and start a disk-replace faulted -> new. After that you can hot-unplug the faulted disk without problems.

ZFS is very fault resistant. You even can hot-unplug some disks from a running pool while saving a file. The pool goes to offline. If you reinsert the disks, the pool is online again without any problem.

I would not try this with hardware raid. The Raid mostly needs a rebuild and without a battery on your raid-controller, you should do a filecheck because your filesystem may got corrupted (write hole problem with hardware raid).

 

[Billy_nnn]

Why do you think that hotswap and autorebuild is a hardware-raid feature?
For hotswap you need a hotswap capable controller in both cases. With ZFS naarly any LSI HBA supports this. A cheap IBM 1015 is a perfect hotswap capable controller.


The same with auto-rebuild. You need no more settings than to add a hotspare disk to your ZFS pool. In case of a failure, a rebuild starts automatically. For a manual replace, you only need to hot-insert a new disk and start a disk-replace faulted -> new. After that you can hot-unplug the faulted disk without problems.

In my experience, hot-plug and hot-swap are not quite the same thing - though I suppose it's semantics at the end of the day.

Hot-swap (as many LSI raid controllers support) literally means pull one disk out (amber light lit or not), insert another and walk away - the array takes all necessary recovery steps automatically (unless something goes wrong of course )

Hot-plug, which the vast majority of modern controllers support, means you can swap the disk live, but you then have to complete the recovery process manually.

ZFS falls into the latter - yes, you can configure hotspares, and you are halfway there, but the original failure still has to be dealt with, and at some stage it requires manual intervention.

The OP specifically mentioned

"provide something that monkey-brained techs can walk into the server room at another site, see an amber light and pop in a new disk that rebuilds automatically and doesn't cause an issue"

......and ZFS doesn't really have that!
I agree, swapping a disk under ZFS is a relatively painless affair, but it's not remove/insert and walk away!

ZFS is very fault resistant. You even can hot-unplug some disks from a running pool while saving a file. The pool goes to offline. If you reinsert the disks, the pool is online again without any problem.

I would not try this with hardware raid. The Raid mostly needs a rebuild and without a battery on your raid-controller, you should do a filecheck because your filesystem may got corrupted (write hole problem with hardware raid).

I'm not sure why you think I'm saying ZFS is not fault resilient - it is.
All I'm saying is that disk failure recovery is not hot-swap in the way some hw raid arrays are.

 

[Billy_nnn]

Is there really a solution that you can look at the amber light, swap the hdd, and walk away? Without having to log in to the system and barely even knowing what a hard drive is for?

I think fast food restaurants require more brain power than that! I would hate to have mindless staff like that. Ugh, the OP is in a bad spot. Too bad he didn't specify the low IQ requirement in the first place.

Yes, such solutions do exist, and aren't that uncommon TBH.

 

[_Gea]

Hot-plug, which the vast majority of modern controllers support, means you can swap the disk live, but you then have to complete the recovery process manually.

ZFS falls into the latter - yes, you can configure hotspares, and you are halfway there, but the original failure still has to be dealt with, and at some stage it requires manual intervention.

.

Maybee we move to technical details too much but this is not a question of ZFS but a question of the used controller. There is a ZFS pool property "autoreplace". If it is set to on, a faulted disk with a special id is replaced automatically whenever a new disk with the same id appear. This is the case with all controllers that offers disk with a controller-port based id like c0t0d0 where you can remove a faulted disk an insert a new in the same slot and a rebuild starts automatically.

This will not work with controllers that use disk-unique WWNs to identify disks like modern SAS2 controller. First I dislike these WWNs due to this fact. But with many disks (think about 10, 20 or 100 disks in a server) this is a feature I would not miss any longer.

 

[Shockey]

auxiliary: You can easily teach any "monkey" to learn zfs and bsd, if you can't then maybe you should interview better. To replace a ZFS disk it is as simple as zpool replace MainStorage /dev/ada2 /dev/ada3 also you can grow your filesystem on the fly with ZFS. My 96TB system started as 6TB years ago and is now at 96TB with no problems.

I know Napp-it GUI (ZFS Feature) has an option you can set to auto replace disk when new one is inserted in same spot.


Gea beat me too it