• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Raid for 24 drive server?

F

fishie

n00b
Joined
May 30, 2013
Messages
42
Hi,

Im currently trying to solve a few things, this setup is for a shared hosting provider, that requires lots and lots of IOPS, mostly reads but on some servers writes are fairly heavy.

They wan't to try and find a working solution that works on all of their servers, and offer them the best possible performance.

First thing, RAID-10 is not an option, as they will loose too much space compared to their needs, also they do not wish to use hardware raid, so mdadm is whats going to be used, and what they have been using for years, do respect that :D

Read pattern is highly random, while writes will be mostly sequential, but for some servers they might have 3-4 concurrent write "session" from clients.

They have been testing out two setups right now, one is a 2 array RAID6, 12 disks in each array, this has been fairly okay for them, but they have also been testing a 2x RAID50 setup each comprised of 3 raid5's.

The case here, is that they are unable to actually benchmark the servers with clients on them to see which would fit better, would a 12 drive raid6 be faster than a raid50 (3xraid5)?

Are there any things that is a must to tune for mdadm to optimize for random reads?

I hope someone in here's smart enough to shed some light on this! :D
 
RAID 50 would be better for sure inregards to performance. What size drives are they using? RAID50 and RAID 6 might be a no no because of UREs. They might want to consider RAID60 if they are not going to go ZFS.
 
Raid 50 would NOT be better at all and never is, Raid 10, or if needed raid 60

if they need IOPS then they need Raid 10 or something else and if they want to do it write need to suck up the cost.

Are you able to use SSD for cache or anything?
 
SomeGuy133 said:
RAID 50 would be better for sure inregards to performance. What size drives are they using? RAID50 and RAID 6 might be a no no because of UREs. They might want to consider RAID60 if they are not going to go ZFS.
Click to expand...

Took the words right out of my mouth.

I'm going to lodge my vote for RAID60. You'll see slightly worse random i/o performance when compared to RAID50, but the likelihood of a URE goes down significantly.

I'm also going to put forth a recommendation of looking at ZFS on Linux for testing purposes.
two RAIDz2 VDEVs would be the equivalent of a RAID60, while adding the capacity for ARC/L2ARC caching.

Requires lots and lots of IOPS, mostly reads but on some servers writes are fairly heavy.
Click to expand...

Does the configuration have to be the same across all servers?
Are you using SAS 10k drives? What capacity are you dealing with?
What types of controllers / SAS Expanders are you using?
How much memory do you have in these servers? mdadm does some decent caching in system ram for reads.
 
MrGuvernment said:
Raid 50 would NOT be better at all and never is, Raid 10, or if needed raid 60

if they need IOPS then they need Raid 10 or something else and if they want to do it write need to suck up the cost.

Are you able to use SSD for cache or anything?
Click to expand...

RAID 50 beats Raid 6 in performance...RAID 10 is not an options

Also as iamwhoiamtoday, we need much more info.
 
and with URE's you want to trust raid 50? a rebuild you will almost be guaranteed to lose your array..

Raid 5 / raid 50 has no use cases this day in age with Mechanical drives. With SSD's that it another story.

https://community.spiceworks.com/topic/273364-what-about-raid-50

http://www.smbitjournal.com/2012/11/choosing-a-raid-level-by-drive-count/

Even Dell acknowledged it
https://www.reddit.com/r/sysadmin/comments/ydi6i/dell_raid_5_is_no_longer_recommended_for_any/

What is the end use for this array?

What will be backing up this array?

Can you afford to have abysmal performance during a rebuild with parity raid resulting in the end user experience being pure crap-tastic
 
Many solutions are possible, I would use
- as many RAM as possible as readcache

Intelligent blockbased solutions based on last/most used with a prefetch like the ZFS Arc ramcache
deliver most reads from RAM. If RAM is at a limit, you can extend the readcache with SSDs,
like L2Arc on ZFS.

- use ZFS for faster writes as the ZFS RAM based write cache will collect small random writes to a fast
sequential write.

- use software raid like ZFS with CopyOnWrite as it does not suffer from the write whole problem of
traditional raid 5/6. It is also crash resistent unlike ext or ntfs solutions.

- use multiple raid-10 for best of all iops as iops scale on reads with number of disks and on writes with number of mirrors. but 50% of capacity is for redundancy

With Raid 5/6/ZFS Raid Z every single disk must be positioned on every read/ write what means that such an array has the same iops like a single disk.

A Raid 50/60 or a ZFS pool from two Raid Z1/2/3 has the same iops like two disks. A ZFS pool with 24 disks build from 4 x 6 disks Raid Z2 has the iops of 4 disks with 30% capacity for redundancy. This would be the best compromise between iops and capacity with 24 disks.

look at next gen filesystems, look at ZFS
 
Hi guys!

Sorry for the lack of information, this was written very late:(

The servers are running with the following specs:
E5-2650v2
128GB RAM
12 drives connected to a HP P420i controller which is in HBA mode
12 drives connected to a HP P220 controller, which is from an external enclose (HP MSA).
All drives are HGST 7k4000 enterprise 4TB

Currently some servers are split into two arrays, 3x4 drive raid5 striped to a raid50 on each of the 12 drives.
There's a system running raid50 hardware, same setup.
And the third build is running two 12 drive raid6's.

It should be said, that there's no backup strategy, and there's no real emphasis to setup one as the data isnt mission critical, so right now they are looking for the best possible performance and not exactly the best possible data redundancy while still keeping as much storage space as possible, which is really why the RAID10 is out of the questions as the loss of storage space is too great.

ZFS is definately not out of the picture at all, but i haven't really thought of ZFS as a high performance setup, more a resilliant setup that can handle URE, and drive loss without any issues.

Also, ZFS was tested about a year ago with not that great results, it might however have been due to me not fully understanding it :p

SSD caching might not be possible, as from what i can gather the chassis's are filled completely as of now i will check up on this.

Is it possible to have ZFS perform on par with a raid50 if there's a ~500GB SSD Cache that are dedicated purely for writes?

It's hard for the system to actually cache any reads as they are 99% random reads.
 
With 99% random reads you're in trouble.

If I had 99% random reads on a huge dataset over one Petabyte, I would really have no idea how I could improve performance in a simple way, I would have to analyse everything. Looking into several cache strategies, using a Bloom filter, tuning the queries, distributed cache, and so on....

There is just no simple answer.
 
99% might be over the top, but it's highly highly random reads, also quite typical a combined read from a 24 drive server in raid50, is maxing out at around 250MB\s..

However, random reads and lots of clients hitting the drives, i know some servers have quite a few users the run torrent clients, but i have no idea how to tune the array to handle things like this, or if changing to ZFS is better for this usage.

Would more RAM help in this case? Can we optimise how it caches data?
 
BTW SSDs are small. You could simply tape velcro to the back of an SSD and velcro it to part of the case if there are no bays free.
 
SomeGuy133 said:
BTW SSDs are small. You could simply tape velcro to the back of an SSD and velcro it to part of the case if there are no bays free.
Click to expand...

Hi,

Im afraid that the server is rented equipment, and i will have to get a confirmation from the provider, although im not sure how much of a difference this will give.

As said, ZFS have been tested before with a read cache, but since it so highly random data, the cache isnt really that effecttive.

However, if it can be setup so that it writes files to an SSD cache and then slowly moves the files over it might be worth looking into..
 
fishie said:
Hi,

Im afraid that the server is rented equipment, and i will have to get a confirmation from the provider, although im not sure how much of a difference this will give.

As said, ZFS have been tested before with a read cache, but since it so highly random data, the cache isnt really that effecttive.

However, if it can be setup so that it writes files to an SSD cache and then slowly moves the files over it might be worth looking into..
Click to expand...

just dont tell them -_- It is Velcro with adhesive. You can easily remove it before returning server. :D
 
With 128 GB of RAM I would expect that a ZFS Arc cache will deliver > 80% of reads from RAM after some time of learning. Without such a cache all reads must be delivered from disks what will slow down the whole server extremely.

An SSD as write cache is problematic as an SSD is much slower as your raid regarding sequential writes and may only help on small randow writes. But in this case a smaller rambased write cache is more efficient and you do not need to care about writing the cache back to the pool what is not a trivial task under load.

At the end, if performance is not good enough with the help of a massive readcache in the area > 100 GB, you need a faster array with SSD > Multiple Raid 10 > Multiple Raid 5/6/Z (Raid 50/60/multiple vdev with ZFS) > Single Raid 5/6/Z
 
Thanks.

I will pass it on, are there any thing that i should be aware of to try and tune to setup as it is now? readhead, stripe_cache_size etc?

Can someone shed some light on that, as i believe i have been hunting improvements for so long that im no longer able to find any new ideas..

It is to be said, that its 80/20 scenario, 80% read with 20% write, on some servers its even less writes..
 
Last edited:
_Gea said:
With 128 GB of RAM I would expect that a ZFS Arc cache will deliver > 80% of reads from RAM after some time of learning. Without such a cache all reads must be delivered from disks what will slow down the whole server extremely.

An SSD as write cache is problematic as an SSD is much slower as your raid regarding sequential writes and may only help on small randow writes. But in this case a smaller rambased write cache is more efficient and you do not need to care about writing the cache back to the pool what is not a trivial task under load.

At the end, if performance is not good enough with the help of a massive readcache in the area > 100 GB, you need a faster array with SSD > Multiple Raid 10 > Multiple Raid 5/6/Z (Raid 50/60/multiple vdev with ZFS) > Single Raid 5/6/Z
Click to expand...

a single SSD might actually be faster then those pools but even if its slower this isn't

2-3x SSDs in RAID 0 will greatly out perform it.
 
Lets try and look away from the SSD part, as thats not entirely possible without hacks.

Whats the best option out there? Will ZFS tuned correctly with 128GB ram, we can allocate around 90GB RAM to caching if some sort be faster than mdadm?

This is purely from a performance perspective, not from a realiability perspective.
 
mdadm is a tool to manage software raid on Linux, nothing more, not the performance relevant part.

ZFS is also a tool to manage a software raid but includes much more functionality like a volume manager, included cache and share management, blockbased zvols, all based on a crash resistent CopyOnWrite filesystem without raid write hole problems like traditional raid 1/5/6.

ZFS was developed for Solaris about 10 years ago, is on BSD available for some years and now also on Linux in a state where it can be considered as ready for production use. At least in the past, performance was often Solaris > BSD > Linux.

On Solaris, you do not need a special tuning to use all RAM as ZFS readcache. For Linux others may give hints about tuning settings.
 
We would really love to avoid changing the backend system, so we really wan't to stay on linux, the entire system is build upon Debian stable..

Rebuilding everything for Solaris would be quite of a hazzle.
 
No problem.
Install ZoL on Debian and use ZFS instead of ext4 + mdadm
 
Is it faster to run 4 x raidz2 (6 drives per vdev) in a single pool or to split it out in pools of 2 vdevs and then balancing the load between the two pools?

Do I need to tinker with the caching strategies at all?
 
fishie said:
Is it faster to run 4 x raidz2 (6 drives per vdev) in a single pool or to split it out in pools of 2 vdevs and then balancing the load between the two pools?

Do I need to tinker with the caching strategies at all?
Click to expand...

The summarized sequential performance and iops is the same. Under low load the first is faster. With high load it may happen that one pool is "overloaded" while the other is fine.

If mass load is on pool 1 and critical load is on pool 2, this may be desired. If all requests have the same priority , 4 x raid-z2 will be faster (this is something like a stripe of two raid 60) plus any two disks may fail with up to 4 disks overall without a dataloss.

regarding ZFS readcache
Only setting is the amount of RAM available for caching. For a pure filer, reserve a minimum of 2-3 GB for the OS. The rest can be Arc cache unless you need RAM for other services.
 
After some careful reading, im not fully convinced that i should go the ZFS route.

The workload that the servers handle, where it's highly random reads, requires lots and lots of IOPS, in this case im not quite sure that ZFS is the right way to go.

In a pure iops setup, looking away from raid10 on a 24 drive server, what would be the best option?

8xraid5's? Not considering data integrity or the danger of URE..
 
Regarding raw performance, ZFS adds checksums so you need to read/write more data than a checksum free filesystem. With CopyOnWrite (probably always) a filesystem becomes slower when filled over say 80% due fragmentation effects.

Count against, that reading from RAM gives you iops in the million area, reading from a good SSD up to 100.000 iops and reading from a disk only 100 iops.

So if you need iops, you must deliver as much as possible from RAM then SSD and then disks.
If you need iops from disks, count on reads (10k disks)

1 disk = 100 iops
1 mirror = 200 iops
1 raid 5/6/Z1 = 100 iops

If you build a pool from 24 disks from 8 x 3 raid5/Z1: 800 iops
If you use 12 x mirror: 2400 iops
 
A beast of a box like that and such poor decisions made on the I/O config that they don't want to use Raid 10 but want great performance..

Wow...
 
MrGuvernment said:
A beast of a box like that and such poor decisions made on the I/O config that they don't want to use Raid 10 but want great performance..

Wow...
Click to expand...

The reason for this is quite simple.

They need the storage, there's too much lost storage on raid10 for it to be a factor, im aware that raid10 is the fastest and the one that shuold be picked in a setup like this..

Does IOPS scale differently on mdadm compared to ZFS?
 
fishie said:
The reason for this is quite simple.

They need the storage, there's too much lost storage on raid10 for it to be a factor, im aware that raid10 is the fastest and the one that shuold be picked in a setup like this..

Does IOPS scale differently on mdadm compared to ZFS?
Click to expand...

is cost an issue? You could just get a second case to house the extra 24 drives.\ need for space

They really need to loo at their budget and see if the extra money is worth the time savings? How much is their time? How much money will be saved in regards to time being saved with a better set up?

An article to consider...not sure if your workflow is transaction based

http://www.vm.ibm.com/devpages/jelliott/evrrt.html
 
You must log in or register to reply here.
Back
Top