New RAID server

C

carlmart

Gawd
Joined
Sep 17, 2006
Messages
687
I need to assemble a video server network, which will be between 15 to 20TB sized. It will be a music server too, but those files are much smaller.

The alternatives I have been recommended have been a NAS with RAID 1 or 6. Opinions are welcome.

Unfortunately I do not have access to low priced HDD vendors or even long warranties, as there is in the USA.

I live in Brazil, where you can't even trust the HDDs you are buying are not refurbished sold as new, so I usually buy all this stuff from Amazon in the USA. Newegg won't sell to me because my credit card is not from the USA.

So these are my starting limitations.

From a previous research I did here, the most recommended HDD was WD Red, so I started my search in Amazon.

At that time you could also buy an external 3.5" WD and use its internal HDD for a good savings, but don't know if that's still an option.

So please tell me which would be the basic guide lines to begin this project.

Thanks!
 
Multiple Raid 10 or Raid-6 behave quite similar regarding sequential performance like writing a large file to an empty array. The main difference is iops where a Raid-6 array has the iops of a single disk (say 100 iops) whereas on a multiple Raid-10 the write iops is equal to the number of mirrors with read iops twice the value.

Example
6 disks in a Raid-6: read and write iops around 100
6 disks in Raid-10 from three mirrors: read iops around 600 and write iops around 300

This can make a huge difference in a multiuser environment with concurrent read/write
As you see, performance, mainly iops scale with number of disks, prefer many disks over large disks for a multiuser video server. Avoid MSR disks and prefer 7k rpm disks suited for NAS (24/7).

If you care about disk quality and data security, use ZFS as a filesystem as it guarantees your data and report any data error due checksums on data and metadata. It is also crash resistent (no corrupted filesystems on a crash during write). The fastest ZFS server is Oracle Solaris with native ZFS from the inventor of ZFS (commercial OS) . From the free Open-ZFS options I prefer OmniOS, a free Solaris fork for production storage with a stable, a long term stable, regular security fixes and even a commercial support option. Next best is Free-BSD wher ZFS was added years ago. ZFS is now available on Linux but without the "it just works" especially from the Solarish world or the Free-BSD. Newest ZFS features like encryption or special vdevs to increase performance based on data type is available on OmniOS and Linux not yet on Free-BSD. If you want to use SMB to access storage, the Solarish internal multithreaded SMB server is a unique feature from the Solaris world (much simpler, mostly faster and with better ntfs ACL support than SAMBA).

From hardware, look at SuperMicro. From disks WD is ok but recheck if disks are not MSR, avoid in any case for storage. Best of all is WD Ultrastar but more expensive. With many disks in a case prefer them as they are suited to withold the vibrations. Normal NAS disks become slow when in a case with more than one, two or four disks.

I have collected some manuals for ZFS at https://napp-it.org/manuals/index_en.html (design principle, build example etc).
 
WD Red NAS drives are still the go-to for projects like this. Be weary of buying external drives and removing them from the enclosure, SOME of them have odd powering issues using a standard SATA power cable. Not saying you can't get good deals doing that, just make sure to research and don't be the first one to try a specific drive.
 
Avoid newer WD Red with MSR in any case for a NAS.
Either use WD Red Plus or the WD red versions without MSR (or the datacenter types WD Ultrastar for production/non home use)

https://www.servethehome.com/wd-red-plus-launched-with-cmr/
https://www.servethehome.com/wd-red-dm-smr-update-3-vendors-bail-and-wd-knew-of-zfs-issues/

And to be clear. This is not a ZFS problem, this is a MSR in a Raid/ NAS problem. Even Synology (NAS without ZFS) does not list them as compatible. SMR is not suitable with a NAS and Raid!
 
So which list in Servethehome is not MSR?

It looks as if the model I'm picking (WD80EFAX) is not on the SMR list, am I right?
 
Last edited:
Sorry again, but can you explain what is MSR and why I should avoid MSR disks? This is the first time I heard about this term.
 
SMR is poorly suited for RAIDs. Best case scenario they can result in a significantly worse performance, worst case they can randomly drop from arrays or fail to rebuild. https://arstechnica.com/gadgets/202...-for-sneaking-smr-disks-into-its-nas-channel/

It's mostly a problem because WD lulled users into not checking this garbage by giving us handy color coded drives. green, blue, black, red, gold. all with specific uses and features.... easy. Now that consumers are used to the system they started slipping in drives not suited for the "colors" indicated purpose.
 
Last edited:
WD80EFAX is cmr and ok
https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/

To inform about msr, read the whole STH article.
Due a large ramcache and a cmr type mediacache area newer msr disks are as fast as the cmr models but only for a quite short time until the mediacache is full. After this their write performance go down dramatically. In a normal benchmark this behaviour is not visible. In a nas or multiuser environment a nogo.

Their "advantage" is a higher capacity with same disk technology. But as they are not cheap enough, this advantage is only for WD as they can sell higher capacity with same disks. You as a user have no advantage only disadvantages.
 
All HDDs bought, six of them. Took some time because of Brazilian customs.

Now I would like to know where to start. I have the motherboard (Asus H61M-A/BR), Intel CPU, memory and power supply. I need to buy a case, one where I can also fit the six 8TB WDs.

Will need a PCI board to plug the HDDs in. Something perhaps better than this, but similar I think.

https://www.amazon.com/MZHOU-Contro...PCI+Express+to+SATA+3.0&qid=1613485876&sr=8-9

Now I have format the HDDs to ZFS, as recommended by _GEA up above, though first I have to decide on which Linux program to use for the server.

What should I do next? What is my following step?
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
For ZFS you should never use a Raid 5/50/6/60 board as ZFS is software raid.
Best is to use onboard Sata/AHCI or a BroadCom/LSI HBA with a 2008, 2307 or 3008 chipset.

These HBA boards are available with IT firmware (best, raidless) or IR firmware (good, raid 1/10) or raid 5 (do not use). Mostly you can crossflash them. OEM models see https://forums.servethehome.com/ind...and-hba-complete-listing-plus-oem-models.599/

I prefer Solarish based (Unix) operating systems for ZFS as OS integration is best especially with the ZFS integrated multithreaded SMB server, mostly OmniOS, https://omnios.org/. If I would not use Solarish where ZFS comes from, I would use Free-BSD as base OS.
 
So considering my hardware options, can you guide me with the rest of the hardware I need and the software, including what to use to format the HDDs.

Which RAID would you recommend?

Linux or Unix? Which RAID to use?
 
For max performance, use multiple mirrors, for max capacity a raid-Z .
For 20 TB usable you can use 4 x 10 TB disks in a raid-10 (30 TB in a Z1)

I use ZFS since around 13 years, first on Solaris, then NexentaCore and now mainly on OmniOS (Unix) and still prefer it in its native environment.
 
carlmart said:
So considering my hardware options, can you guide me with the rest of the hardware I need and the software, including what to use to format the HDDs.

Which RAID would you recommend?

Linux or Unix? Which RAID to use?
Click to expand...

I would highly recommend using ECC memory if possible, in which case you want an Intel Xeon process or AMD Ryzen (no APUs). CPU power depends on what you're doing, if mainly a NAS then you don't need anything beefy but if you're going to be running Plex transcoding, other software and/or virtual machines you might want something with some more cores/threads. Memory I would go with 32GB if possible for ZFS caching, if not you can get away with 16GB.

Depending on the size of disk you buy you could go about it different ways. A RAIDZ1 (parity with no loss of data if only ONE hard drive fails) of three 8 TB hard drives would give you 16TB usable storage. I recommend playing with this calculator: https://wintelguy.com/zfs-calc.pl If going with more smaller capacity drives you might want to do multiple vdevs. Here is a good primer on ZFS including zpools and vdevs: https://arstechnica.com/information...01-understanding-zfs-storage-and-performance/

I would highly recommend using TrueNAS Core as the base operating system. It's based on FreeBSD which has really good OpenZFS support. The GUI makes it easy to create SMB/NFS shares, automate S.M.A.R.T testing, automate scrubs, send email alerts, make iocage jails (plugins suck, avoid those and learn how to use FreeBSD pkg or ports), configure Bhyve virtual machines, and much much more. Linux varies with support of ZFS, but it's getting there. If you're more familiar with Linux you could go that route, just pick a good NAS based distro with solid support for ZFS. I recommend against UnRAID.
 
carlmart said:
I need to assemble a video server network, which will be between 15 to 20TB sized. It will be a music server too, but those files are much smaller.

The alternatives I have been recommended have been a NAS with RAID 1 or 6. Opinions are welcome.

Unfortunately I do not have access to low priced HDD vendors or even long warranties, as there is in the USA.

I live in Brazil, where you can't even trust the HDDs you are buying are not refurbished sold as new, so I usually buy all this stuff from Amazon in the USA. Newegg won't sell to me because my credit card is not from the USA.

So these are my starting limitations.

From a previous research I did here, the most recommended HDD was WD Red, so I started my search in Amazon.

At that time you could also buy an external 3.5" WD and use its internal HDD for a good savings, but don't know if that's still an option.

So please tell me which would be the basic guide lines to begin this project.

Thanks!
Click to expand...
so you just need a file server

if you have not had any experience and don't trust the hdd get a 5 bay (or higher) Synology NAS and use SHR2 or just straight RAID6 (RAID6 all disk have to be replaced one at a time once the final disk is rebuilt you can gain he new space, SHR2 you have to replace 4 disks before space is available)

use 5 8TB disks (RAID6 needs 2 disks for redundancy so approx. 20TB will be available for space (due to overhead) and well 8TB not actually been 8TB in bytes

buy enterprise or NAS type HDDs (WD enterprise/Seagate ES (enterprise) but make sure they are Not SMR type google model or consumer NAS hdds > Toshiba N300, Seagate ironwolf or WD red Plus or pro, the old WD red are also fine at 8TB or higher as currently they are CMR below 8TB you need to check the full model if its SMR or CMR/PMR)

at the time of posting all disks above 8TB are CMR based (no SMR yet on them sizes and not on NAS based ones) But enterprise or non Nas HDDs can be SMR (if not all Seagate consumer disks are now SMR but fail to have the SMR labelled on them, and never buy archive HDDs as they are SMR)

i would avoid Qnap (price is good) as seem to have many security problems just does not seem worth it and they lack BTRFS and any type of hybrid raid (but still don't poke holes in your router to open your Synology to the internet unless you know what your doing)
 
Last edited:
I agree with likeman. If you just want something that works out of the box and you're using this mainly as a file server, pickup whatever Synology NAS meets your needs and be done with it. Setup is extremely simple, the web interface will walk you through it step by step.

If you really want to keep things simple you can even purchase Synology branded hard drives.
 
likeman said:
if you have not had any experience and don't trust the hdd get a 5 bay (or higher) Synology NAS and use SHR2 or just straight RAID6 (RAID6 all disk have to be replaced one at a time once the final disk is rebuilt you can gain he new space, SHR2 you have to replace 4 disks before space is available)
Click to expand...

As I said at the beginning I will be using a PC, with Asus H61M-A MB with Intel CPU as the base for my server, with 5 x 8TB WD Red NAS HDDs assembled in RAID6. All the supply and the case I already have, was my previous computer, so it works fine. I will need to buy a PCI board with 8 SATA sockets at least, and to do so apparently I will need freenas to run it all through software, with Linux OS. This seems more straightforward and simpler than assembling the server through hardware.

I will only run video and audio files through it.
 
carlmart said:
As I said at the beginning I will be using a PC, with Asus H61M-A MB with Intel CPU as the base for my server, with 5 x 8TB WD Red NAS HDDs assembled in RAID6. All the supply and the case I already have, was my previous computer, so it works fine. I will need to buy a PCI board with 8 SATA sockets at least, and to do so apparently I will need freenas to run it all through software, with Linux OS. This seems more straightforward and simpler than assembling the server through hardware.

I will only run video and audio files through it.
Click to expand...
don't go cheap (well you cant with 8 sata ports any way) get a 8 port SAS card in IT mode (Loads of LSI ones to get)
 
carlmart said:
Would this LSI serve my purposes:

https://www.amazon.com/dp/B002IT4YG...ge=es_US&ref_=psdcmw_3015424011_t1_B008NF6TKY

I know this board would allow to assemble a hardware controlled system, but maybe that's an option I would like to try in the future. Or it would be spending more money than I need?
Click to expand...

If you want to do a hardware RAID, then that's the card to buy. But if you want to use TrueNAS or another OS with ZFS you'll want an HBA, something like a LSI 9211-8i or one of the Dell or IBM equivalents.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
Yeah, that card should work, Amazon looks to have cheaper variants as well but it's really hard to tell what you're getting when they come from third party vendors.

Your motherboard doesn't appear to have an m.2 slot, so you'll want to go with a normal sata ssd to boot off of since your motherboard only has sata ports.
 
Well, I do have my first SSD, a 256GB Crucial that I eventually use externally. Crystal shows a 92% health, but it's not specific SSD testing program, so it doesn't show life expectancy, like SSDLife shows for SSDs.
Well, I could use that.
Aren't there adapters that allow using m.2 boards in normal SATA sockets?
 
You must log in or register to reply here.
Back
Top