Considering moving from RAID5 to Storage Spaces

F

Flapjack

2[H]4U
2FA
Joined
Apr 29, 2000
Messages
3,217
I currently have a 2 x 2TB RAID5 array connected to my Windows 8.1 HTPC. After the stripe, it's 4TB. I am constantly filling it up with the stuff I'm ripping to it (trying to go fully disc-less). As many are aware, I cannot just add a disc to like I would on my FreeNAS machines, but I want to keep the data local to the HTPC, so it has to be something that will work with Windows.

I've been looking into Storage Spaces, and while it is no ZFS, it seems like it would do the trick for me. I currently have a TRM5 tower enclosure housing my 5 WD Reds, connected via eSATA and port replication to a RocketRaid 2314 SATA/eSATA RAID controller.

I have a 5TB external drive that could house all the data needed for the migration, so I'm not worried about trying to preserve the current array. I'm actually thinking of getting a TRM8 or a SANS Digital 8-Bay enclosure. I can even use the cheapo RAID card that comes with it for another project I'm working on.

Any thoughts? Can Storage Spaces actually do what I need? I want to be able to add drives to the pool as needed. It seems like it does, but I'm curious what everyone else thinks.
 
i can not offer any info on storage spaces but i can say that going dic-less, utilizing network storage, and even removing windows from the front end of the HTPC is quite easy and might save you time and money.

sorry i am of no direct help, however if you would like to try a slightly different yet still fully featured HTPC config i would be happy to help.
 
Parity vDisks, the Storage Spaces equivalent of RAID 5, suffers from TREMENDOUSLY slow write speeds. Starting out, you may only get 20MB/s write speeds and even changing some advanced settings you may get up to 50MB/s reliably, however writes will still hammer the disks and cause high latency.

Parity vDisks keep a journal of every write before committing them and that journal is stored on the disk pool itself. You can add SSDs to the pool and dedicate them as journal drives only, but that obviously costs money and eats up drive slots.
 
Thanks for the replies. At this point, I'm not interested in changing from Windows for the front end. I run a wide variety of Linux machines for servers on my network... but for my HTPC, Windows has been painless and rock solid. Even if I did switch to something *nix-based, it would have to be a BSD variant, as I've had nothing but bad luck with ZoL. Imagining trying to get ZoL working nicely along with a system configured as a HTPC sounds to me like more headache than I care for. I've never even been able to get surround sound working on the last Linux type HTPC options. That crap just works in Windows.

As for Storage Spaces, am I to understand that it does not offer a non-RAID array option (similar, if not the same as ZFS)?

The primary motivating factor here is not having to tear down an array and rebuild it every time I want to add a disc. If Storage Spaces can't do this, I will probably build a separate FreeNAS machine, which I usually get 100MB/s out of. I would much, MUCH rather have the storage local to the HTPC, though.
 
Last edited:
I like SnapRaid, and it's free. Made for media storage that doesn't change much, and no speed penalty.
 
Only two downsides to Snapraid, otherwise you're golden.

1) Not good for files constantly open or changing. Don't use it on your C: drive.
2) Risk of losing ONLY the new files you've added since the last sync, since it's an offline RAID calc.
 
Windows users probably will also be disappointed in the pooling capabilities of SnapRAID.

I use SnapRAID on linux however my pvr software (mythtv) handles multiple storage devices without issue so I do not need or want pooling at the filesystem level.
 
SS is slow. VERY SLOW. I'm using it in a work environment where long term archival storage is what's needed, so ReFS is perfect. But for a NAS, nope nope NOPE.
 
I use ZFS on FreeNAS and Solaris both, and it is very fast. I use SSD's for L2ARC & ZIL's where I need the speed. I have a few of these systems, and most of my storage space is on an iSCSI mount. A couple of things that I really like are snapshots and the fact that I don't have to redo the entire system to go from 1tb to where I am at now with 12x4tb disks on my home media system.
 
CombatChrisNC said:
SS is slow. VERY SLOW. I'm using it in a work environment where long term archival storage is what's needed, so ReFS is perfect. But for a NAS, nope nope NOPE.
Click to expand...
In what setup? Seems reasonably ok for a v1 microsoft effort with mirrors and ssd tier.
 
Valnar said:
I like SnapRaid, and it's free. Made for media storage that doesn't change much, and no speed penalty.
Click to expand...
mrwizardno2 said:
I agree with this post. I use SnapRaid in a 30TB 'array' and it's blazing fast!
Click to expand...
Valnar said:
Only two downsides to Snapraid, otherwise you're golden.

1) Not good for files constantly open or changing. Don't use it on your C: drive.
2) Risk of losing ONLY the new files you've added since the last sync, since it's an offline RAID calc.
Click to expand...
SnapRAID sounds very viable. It really doesn't have to be super reliable. While I add media occasionally, what is on there typically doesn't change much. It also seems to run inside of windows. It is definitely at the top of my list now.

drescherjm said:
Windows users probably will also be disappointed in the pooling capabilities of SnapRAID.
Click to expand...
Can you elaborate? Really, the only pool capability I need is to be able to add drives as the array fills up. It sounds like I don't even need to use the same size/type of disk (though I probably will).

CombatChrisNC said:
SS is slow. VERY SLOW. I'm using it in a work environment where long term archival storage is what's needed, so ReFS is perfect. But for a NAS, nope nope NOPE.
Click to expand...
bmh.01 said:
In what setup? Seems reasonably ok for a v1 microsoft effort with mirrors and ssd tier.
Click to expand...
How slow can it be? It would have to be slower than accessing the media over GigE, which is what I would do if I setup a separate FreeNAS setup with a CIFS share.
 
SnapRAID doesn't do pooling (making multiple drives look like one letter). For that, people tend to pair it with DrivePool for a nominal fee. There are some special setup parameters you need to make it work, but from what I hear they work well together. I don't use it personally as all my media apps (SageTV, Plex, etc) allow me to define multiple folders or drives anyway.

There is a more expensive commercial app called FlexRAID which does both, but there have been some complaints as to its stability in other forums, so I stayed away from it. ymmv

I love my Snapraid setup. I have it run a job to sync the parity drive every night. It's all I need. Doesn't run as a service, and has a very little footprint. For a media storage server, its great. Don't have to worry about RAID-5, hardware RAID cards or any of that. Everything is just as protected after a sync, and has the bonus of being able to recover accidentally deleted files between syncs.
 
Valnar said:
SnapRAID doesn't do pooling (making multiple drives look like one letter). For that, people tend to pair it with DrivePool for a nominal fee. There are some special setup parameters you need to make it work, but from what I hear they work well together. I don't use it personally as all my media apps (SageTV, Plex, etc) allow me to define multiple folders or drives anyway.

There is a more expensive commercial app called FlexRAID which does both, but there have been some complaints as to its stability in other forums, so I stayed away from it. ymmv

I love my Snapraid setup. I have it run a job to sync the parity drive every night. It's all I need. Doesn't run as a service, and has a very little footprint. For a media storage server, its great. Don't have to worry about RAID-5, hardware RAID cards or any of that. Everything is just as protected after a sync, and has the bonus of being able to recover accidentally deleted files between syncs.
Click to expand...
Hmm... I didn't realize there was any cost associated with snapraid.

I'm torn now. I think what I'll do is migrate the data off the array and play around with a few solutions.

Other than SnapRAID and SS, is there any other windows-based solutions you can think of? I have an extra Dell 2950 sitting and collecting dust. If this doesn't work out, maybe I'll just build another FreeNAS machine.
 
Windows 2012 R2, LSI 9211 IT mode. 9x 1TB 7200 RPM disks of mixed brands.

9 disk pool, speed shown with Zero Parity, Single Parity (RAID5), Mirror, Dual Parity (RAID6).


how to take a screen shoth

As for my observed results, I backed up ~230GB of data over a 100Mb network from identical systems. Using robocopy I specified 100 threads. I was able to keep the link saturated fully the whole time and the READ system had a queue in the 1-5 range. The WRITE system had a queue in the range of 10-50. If I were able to scale up to 1Gb I have no idea what either system would look like.

I'm sure I would get much faster results if I had SSD tiering, sure. I also would have faster writes if the write journal could be offloaded to an SSD, which is possible. If anyone can post similar benches from ZFS for an 8 or so disk setup, I would be interested.
 
Here you go. The terminal is directly off the machine and I didn't turn anything off VirtualBox has 4 Windows 2012 machines in it, There were 3 1080P streams coming from the machine. This is my small storage server with 6x4TB Greendrives running RAIDZ2, no SSD ZIL or SSD L2ARC. It is running an a i5-4460 desktop class machine. You can see where I used CIFS and iSCSI as a mount. The performance isn't too bad considering that I can drop any two drives at once, and get a snap shot every five minutes.

zfsperformance.jpg
 
You mentioned using freenas, why are you stuck on local storage? 10Gb point to point is cheap and you could do iscsi for block-level access. Using omnios+napp-it you could do FC also.
 
bds: The tests that I showed you were my freenas system. The local storage was on the server and the client that I had I was just using a 1GB connection. I have 10GB connections on the bigger stuff, I just don't need it right now, performance is plenty with what I have. The only local storage I have is the SSD's for the OS etc. That D drive that you see that is 150GiB is a 24TB array made up of just cheap green drives.
 
DaPayne said:
bds: The tests that I showed you were my freenas system. The local storage was on the server and the client that I had I was just using a 1GB connection. I have 10GB connections on the bigger stuff, I just don't need it right now, performance is plenty with what I have. The only local storage I have is the SSD's for the OS etc. That D drive that you see that is 150GiB is a 24TB array made up of just cheap green drives.
Click to expand...

I understand that, the op mentioned freenas, suggesting he had some experience with it. He also said he wanted to keep the storage local.

The comment was not meant for you. It also appears that your pool is set on "sync=standard". Without a SSD ZIL you would experience higher performance on iSCSI would be similar to your CIFS numbers if you set it to "sync=disabled".

To continue my point to the OP, even if your application requires block level access you can still achieve this with ZFS. If you have a good UPS you may be able to get away with no SSD ZIL and get great numbers over point-to-point 10Gb or point-to-point FC. If you are interested in perusing this route, let me know, I can elaborate.

Just for an example, here is my pool served to one of my ESXi boxes on 4Gb FC with 2x STEC Mach16 50GB SLC SSD's (mirrored) for a ZIL, pool set on "sync=always". Keep in mine that there are 25 VM's running on that while the benchmark is running. Setting the pool to "sync=disabled" results in maxing out the 4Gb connection both ways.

poolbench.jpg


My point: Even if you need block-level access you can still use ZFS, have transfers that max out the pool and still be affordable.
 
Last edited:
Please stop using raid 5 with such large drives!

So, now that has been said, keeping an eye on this thread.
 
I say that the number of disks are more a factor than the size. I find the URE failure articles sound believable however they are very wrong in practice. I mean you will not have a URE every 12TB of data read if you have tested working drives (for me this means any drive put in an array must pass the 4 pass badblocks test before going into the array without a single bit lost). From my weekly scrubs of around 70TB (for years) I see URE rates much lower than 1 URE per 12TB and more like 1 corrupt sector per several hundred TB of data read.
 
Flapjack said:
Even if I did switch to something *nix-based, it would have to be a BSD variant, as I've had nothing but bad luck with ZoL. Imagining trying to get ZoL working nicely along with a system configured as a HTPC sounds to me like more headache than I care for.
Click to expand...
Can you elaborate? I use ZoL at work and home since 0.6.1 in a variety of roles and apart from a few minor issues it has been nothing but joy.
 
CombatChrisNC said:
Windows 2012 R2, LSI 9211 IT mode. 9x 1TB 7200 RPM disks of mixed brands.

9 disk pool, speed shown with Zero Parity, Single Parity (RAID5), Mirror, Dual Parity (RAID6).

As for my observed results, I backed up ~230GB of data over a 100Mb network from identical systems. Using robocopy I specified 100 threads. I was able to keep the link saturated fully the whole time and the READ system had a queue in the 1-5 range. The WRITE system had a queue in the range of 10-50. If I were able to scale up to 1Gb I have no idea what either system would look like.

I'm sure I would get much faster results if I had SSD tiering, sure. I also would have faster writes if the write journal could be offloaded to an SSD, which is possible. If anyone can post similar benches from ZFS for an 8 or so disk setup, I would be interested.
Click to expand...
Unfortunately, my work blocks your pics. I'll have to check it out when I get home. So you can do the equivalent of ZIL/L2ARC in SS too? It makes sense... I've just never set it up, so I'm not sure what all the options are.

DaPayne said:
Here you go. The terminal is directly off the machine and I didn't turn anything off VirtualBox has 4 Windows 2012 machines in it, There were 3 1080P streams coming from the machine. This is my small storage server with 6x4TB Greendrives running RAIDZ2, no SSD ZIL or SSD L2ARC. It is running an a i5-4460 desktop class machine. You can see where I used CIFS and iSCSI as a mount. The performance isn't too bad considering that I can drop any two drives at once, and get a snap shot every five minutes.
Click to expand...
My FreeNAS machine is setup with RAIDZ2, with iSCSI and CIFS shares. Over a single NIC, I can get 100+ MB/s over CIFS with just L2ARC. I'm still working on tuning the iSCSI stuff (which is what my Xen servers connect to), but I have quad port NICs in the FreeNAS and Xen servers with an MPIO setup through a managed switch. I can't seem to get full utilization on all the ports, though. I'm thinking of switching to an NFS share and doing LACP.

bds1904 said:
You mentioned using freenas, why are you stuck on local storage? 10Gb point to point is cheap and you could do iscsi for block-level access. Using omnios+napp-it you could do FC also.
Click to expand...
I never really considered 10Gb point-to-point, but that could work. I just really don't want to manage a separate machine for that. I could use on of the 2950s in my rack, but it's not collocated with the HTPC... I'd have to run a separate CAT6 cable (Cat5e throughout the house). I currently have a TMR5 RAID enclosure and a decent RocketRAID card connected to the HTPC, which is why I was leaning towards a local array. I still have to look at what options the RAID BIOS gives me, but I had hoped to break up the current RAID5 array and present the disks individual to Windows. If I could get a more flexible pool working, I would then buy an 8TB array and add disks as I needed. I currently have five 2TB WD Reds in the enclosure. My 2950 has 6 SATA/SAS slots, so I would be limited to adding one more disk, since FreeNAS boots off a thumb drive... unless I switched to larger disks, which is something I don't have the cash for right now.

bds1904 said:
To continue my point to the OP, even if your application requires block level access you can still achieve this with ZFS. If you have a good UPS you may be able to get away with no SSD ZIL and get great numbers over point-to-point 10Gb or point-to-point FC. If you are interested in perusing this route, let me know, I can elaborate.
Click to expand...
I'd definitely be interested in setting up a point-to-point 10GB ethernet connection, as I can run that cable fairly easily. I don't really care to mess with running fiber through existing walls. I'm assuming there are some fairly decent 10GigE cards that are both Windows and BSD-compatible that won't break the bank?

Just for an example, here is my pool served to one of my ESXi boxes on 4Gb FC with 2x STEC Mach16 50GB SLC SSD's (mirrored) for a ZIL, pool set on "sync=always". Keep in mine that there are 25 VM's running on that while the benchmark is running. Setting the pool to "sync=disabled" results in maxing out the 4Gb connection both ways

My point: Even if you need block-level access you can still use ZFS, have transfers that max out the pool and still be affordable.
Click to expand...
I can't see the pic, but I've seen what FreeNAS can do, so I'm sure I could get decent results. Again, it was more of not wanting a separate system than local-vs-remote performance concerns. No more than 2 people use the media shares at any given time.

While I've been using FreeNAS for years, I've never really spent much time "tuning" it. I'm not familiar with the "sync=enabled/disabled" setting. I'll have to look it up.
 
Flapjack said:
I believe it was explained that the syncs only happen on a schedule, so any changes made prior to a sync can be lost. I may have that wrong, though...
Click to expand...

how often can syncs be done? I plan on using SnapRAID for a NAS/mass data but is it like sync on a daily thing or on an hourly or min thing?
 
I use storage spaces at home, and I'm using it for storing primarily two things - video, and backup images. I have 3x3tb drives in what is basically a software raid 5 array. The drives are internally installed in a computer in my basement, which runs Server 2012 R2. The main htpc is upstairs running Ubuntu 14.04, and I have everything connected through an 8 port 1gb switch. The clients that back up to the server are a mix of Windows (7 & 10), and linux (mostly Ubuntu but I dabble in CentOS too).

To answer your original questions, storage spaces does allow for adding a disk or disks and expanding the space on an as-needed basis. If the storage space you create is simple, then you just add disks as you wish and there's nothing else to consider. Windows stripes the data behind the scenes. Now, if you create a storage space with parity then you must add drives in the same ratio as the existing space. In my case, I would have to add drives in trios because that's the configuration I used when I created it.

As far as performance, this is a home NAS... how much speed to I need? Movies play back flawlessly, even when multiple clients are reading from the drives at the same time. I can play movies while copying data to the drives, or play movies while ripping or encoding. I have multiple clients back up to the drives simultaneously overnight.

If I'm copying a large chunk of data to the server from a client, the first 4-5gb goes very quickly. I suspect it is because that's how much memory there is on the server that can be used to cache. Beyond that size the throughput does noticeably diminish, but it is not terrible. How often am I copying more than say 20-30GB at once? Not very, and that only takes minutes to complete. The backups do write much more, but those run overnight and are fully automated. Speed is a non-factor there.

Anyway, if you want to play around with storage spaces I suggest giving it a look. It's very capable and works well for storing media.
 
doublejack said:
I use storage spaces at home, and I'm using it for storing primarily two things - video, and backup images. I have 3x3tb drives in what is basically a software raid 5 array. The drives are internally installed in a computer in my basement, which runs Server 2012 R2. The main htpc is upstairs running Ubuntu 14.04, and I have everything connected through an 8 port 1gb switch. The clients that back up to the server are a mix of Windows (7 & 10), and linux (mostly Ubuntu but I dabble in CentOS too).

To answer your original questions, storage spaces does allow for adding a disk or disks and expanding the space on an as-needed basis. If the storage space you create is simple, then you just add disks as you wish and there's nothing else to consider. Windows stripes the data behind the scenes. Now, if you create a storage space with parity then you must add drives in the same ratio as the existing space. In my case, I would have to add drives in trios because that's the configuration I used when I created it.

As far as performance, this is a home NAS... how much speed to I need? Movies play back flawlessly, even when multiple clients are reading from the drives at the same time. I can play movies while copying data to the drives, or play movies while ripping or encoding. I have multiple clients back up to the drives simultaneously overnight.

If I'm copying a large chunk of data to the server from a client, the first 4-5gb goes very quickly. I suspect it is because that's how much memory there is on the server that can be used to cache. Beyond that size the throughput does noticeably diminish, but it is not terrible. How often am I copying more than say 20-30GB at once? Not very, and that only takes minutes to complete. The backups do write much more, but those run overnight and are fully automated. Speed is a non-factor there.

Anyway, if you want to play around with storage spaces I suggest giving it a look. It's very capable and works well for storing media.
Click to expand...
Thanks for the awesome reply.

So when you say "windows is striping" in the background, you're just talking about some sort of RAID0/JBOD action going on... no fault tolerance?

In ZFS, I can add disks to the pool as needed. If StorageSpaces can't do that, then I may just go with an external box.

My major requirement is wanting enough parity to support at least one drive failure, and the ability to add to the pool as it fills up. Right now, the RAID5 setup gives me as much fault tolerance as I need, but it's a giant PITA to add space to the array.
 
That's is correct. The default, or "simple" configuration option of storage spaces is a raid0/jbod setup. Windows will stripe data for better performance, but only in some situations. If you add multiple drives of the same size to the pool, it will default to striping them. If you add them one at a time or they are different sizes you get jbod.

You can get both parity / fault tolerance and the ability to add to the array with storage spaces, but the caveat is you can't get that by adding one drive at a time. You could go with a mirror storage pool, but you lose 50% of your space and you have to add drives in pairs. Or you could go with the parity option, but that requires 3 or more drives to start with and you must add additional drives in whichever multiple you started with.

Basically storage spaces won't let you start with 5 drives with parity, and add two more later and have everything protected against loss. It can't rebuild the storage pool on the fly to handle that kind of configuration change.
 
------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
MADFILESRV 16000M 63 99 295319 83 181426 72 145 99 596663 88 243.4
Click to expand...

As you can see I get 596 MB/s on reads and 295MB/s on writes. This is on RAIDz2 no L2ARC on WD Greens.
 
doublejack said:
That's is correct. The default, or "simple" configuration option of storage spaces is a raid0/jbod setup. Windows will stripe data for better performance, but only in some situations. If you add multiple drives of the same size to the pool, it will default to striping them. If you add them one at a time or they are different sizes you get jbod.

You can get both parity / fault tolerance and the ability to add to the array with storage spaces, but the caveat is you can't get that by adding one drive at a time. You could go with a mirror storage pool, but you lose 50% of your space and you have to add drives in pairs. Or you could go with the parity option, but that requires 3 or more drives to start with and you must add additional drives in whichever multiple you started with.

Basically storage spaces won't let you start with 5 drives with parity, and add two more later and have everything protected against loss. It can't rebuild the storage pool on the fly to handle that kind of configuration change.
Click to expand...
Thank you. That was exactly the answer I was looking for.

So at this point, I'll probably just end up building another NAS. Sigh...
 
You must log in or register to reply here.
Back
Top