• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Thinking of downsizing my NAS server

B

black6spdz

n00b
Joined
Sep 25, 2010
Messages
54
Hi all, my currect NAS has gone relatively untouched and trouble free for a couple years. Its currently running:

Win 8 - with server 2012 dedup,
C2Q 2.4 GHz,
8GB RAM,
8x Samsung HD204UI "Originals" 2TB
IBM M5015 with Raid6 key "Raid6".
Large Supermicro 8-hotswap bay 4U chassis.

I am currently only using 2.54TB with 152GB saved in deduplication. Since my data hasn't grown in quite some time I've tossed around the idea of downgrading. I have a small cube case with a C2D 3.3GHz and 4GB DDR3. So I've been thinking about downsizing to it with a couple mirrored 4TB drives. This would use a lot less power and less heat. What's the latest and greatest NAS OS if I went that route? How about Nas4Free with a 60GB Vertex 3 L2ARC and 8 or 16GB of RAM? The new LZ4 compression peaks my interest too.
 
Last edited:
Two things that spring to mind:
if you're not going with ZFS one doesn't need anywhere near that much memory for a NAS device. Think like 1-2Gig tops. Around 4 if you're doing ZFS. So you're good on memory there.
Far as L2ARC goes: why? in a home environment there's every chance you'll never need it.

I've also heard good things about FreeNAS. Far as OSes go there isn't a single "latest and greatest", it's all down to your tastes and what you want.

also: formatting, please use some next time. Just a giant block of text doesn't read well
 
The old FreeNAS and first iterations of ZFS were down quite a bit in performance, that's mainly the reason I decided to go with a hardware raid controller and windows. I'm sure it has matured in 3-4 years and a pair of 4TB mirrored drives shouldn't take much hardware at all.
 
Operating ZFS with less than a lot of memory actually works OKish as long as nothing else is running on there and as long as you have single access to the filesystem (one thing at a time). ZFS can deliver single files streaming or random seeking in many files as long as it's one request at a time at very good speeds even if you remove its ability to buffer as much as it wants (keep in mind it is double buffering on FreeBSD and Linux).

The main problem is that if you have a ZFS in a machine that you -say- run your web browser on the web browser will suffer.

If your browser is Chrome then you have two things on your machine both of which behave like they own all the RAM for themselves.

If you have multiple client processes at the same time then there will be a performance difference. But the question is - do you really care?
 
This will again be a dedicated NAS Samba server, no other services running. I have an extra SSD lying around so I was wondering what improvements if any would there be as a L2ARC/ZIL cache? or is that more so used with RAIDZ and not ZFS mirroring?
 
I did my homework and read about ZIL and L2ARC, implementing ZIL seems too risky for marginal gains, L2ARC is not too beneficial unless you have many clients. Key seems to put as much RAM as you can. I'm a bit afraid of the consequences of using non-ecc memory though, as they say scrubs with bad memory can corrupt your entire z-pool. Maybe I shouldn't use ZFS at all and stick with GEOM Mirror "software raid"?
 
black6spdz said:
I did my homework and read about ZIL and L2ARC, implementing ZIL seems too risky for marginal gains, L2ARC is not too beneficial unless you have many clients. Key seems to put as much RAM as you can. I'm a bit afraid of the consequences of using non-ecc memory though, as they say scrubs with bad memory can corrupt your entire z-pool. Maybe I shouldn't use ZFS at all and stick with GEOM Mirror "software raid"?
Click to expand...

You would be better off with ZFS even with non ECC memory. No doubt the risk of a bad memory block would wreck havoc during a scrub, but the probability of that happening is too low to lose sleep over.

If you are building a new machine, sure, go with ECC, otherwise, it would be fine for a home environment.
 
black6spdz said:
I did my homework and read about ZIL and L2ARC, implementing ZIL seems too risky for marginal gains, L2ARC is not too beneficial unless you have many clients. Key seems to put as much RAM as you can. I'm a bit afraid of the consequences of using non-ecc memory though, as they say scrubs with bad memory can corrupt your entire z-pool. Maybe I shouldn't use ZFS at all and stick with GEOM Mirror "software raid"?
Click to expand...

Bad memory can always corrupt all data on your filesystem. I rate ECC as essential. I do use non-ECC machines such as laptops but they are generally behind NFS so that corruption is limited to files they actually access.

ZFS doesn't make it worse, though, it makes it a tiny bit better.

I don't think I would base a new large install on GEOM raid. If you pick ZFS you can move the whole shebang between operating systems now. Think about accessing the data in 8 years or so.
 
black6spdz said:

Win 8 - with server 2012 dedup,
C2Q 2.4 GHz,
8GB RAM,
8x Samsung HD204UI "Originals" 2TB
IBM M5015 with Raid6 key "Raid6".
Large Supermicro 8-hotswap bay 4U chassis.
Click to expand...

Whats with the ZFS tangent being ridden away with here?

The OP has made it clear that he is a Windows user and is looking for nothing more than a mirrored pair. No need for ZFS or even NAS OS's!

OP, How about running a simple load of Windows 7, 8 or Server 2K8R2, 2012 and just run either on-board RAID (Intel ICHR's will do this well) or make the volumes dynamic mirrors in the OS. Add simple shares, have RDP access and be able to use it for other things like P2P and VM's....?
 
Lost-Benji said:
Whats with the ZFS tangent being ridden away with here?

The OP has made it clear that he is a Windows user and is looking for nothing more than a mirrored pair. No need for ZFS or even NAS OS's!

OP, How about running a simple load of Windows 7, 8 or Server 2K8R2, 2012 and just run either on-board RAID (Intel ICHR's will do this well) or make the volumes dynamic mirrors in the OS. Add simple shares, have RDP access and be able to use it for other things like P2P and VM's....?
Click to expand...

That's what I have now.. lets go back about 4 years when I first put together the idea of creating my own home NAS for the purpose of a central file share.. desktop, laptops, TV media players, etc. 2TB drives were dirt cheap, bought 8 of them at roughly $80 a pop. I had a Perc 6/i and hardware Raid controllers are nothing new to me.. been working with servers for years. I tried FreeNAS 7 and some other PC based NAS OSs of the time, ZFSGuru, NexentaStore, and I liked what ZFS was selling. To make this growing story short the performance wasn't up to par "CIFS and ZFS" on the free NAS OSs vs just a basic Windows install with the hardware Raid 6. Step-forward to today.. My data isn't growing much and I can easily fit what I have now on a 4TB disk, mirror it onto another for some redundancy and put it all into a nice small micro atx cube case that I had lying around. Less heat/noise/power, newer platters and the opportunity to revisit more matured dedicated NAS based OS such as NAS4Free with ZFS again.
 
It's a backup. It should have snapshots and it should be readable many years from now. ZFS excels in both right now. And I have been a ZFS basher until very recently.

If the space is sufficient any form of mirroring is highly preferable over raid or zraid since you can just read (some) data from any single drive that might have survived.
 
black6spdz said:
What changed your opinion about it?
Click to expand...

For me the main advantage is that I finally have a filesystem, and a redundancy layer, that I can share between FreeBSD and Linux and I expect that I can easily read today's filesystems in 15 years or whenever. Using geom or md and UFS2 or ext4 will force you do then have that particular OS around.

I like the faster resync on filesystems that are not full.

I looked up in detail how snapshots in UFS2 work and they are not scalable. Sure you can use them as tactical snapshots while you edit pictures or whatever, but you can't keep hundreds forever.

I still have significant concerns, especially around RAM consumption in general, the double caching in RAM in particular, the naive automatic mechanisms and the like. It also mixed badly with the current high RAM prices. I will probably still do some backups using rsnapshot, the main reason being that if I decide a certain large file needs to go from all snapshots I can just do it whereas in ZFS I would have to recreate all snapshots. (I think an unlink through ZFS snapshots might be in the technical cards in the future, though)

I tested performance, especially around zraid, and I tested resync behavior by randomly pulling disks and it's kosher.
 
should I be worried about only 4GB of RAM and ZFS mirroring? I know every recommendation says 8GB but this mini itx board I have only supports 4GB max "G41 chipset".. maybe I'll get lucky and find some 4GB DIMMs that will work. I just got my pair of Seagate ST4000DM000 drives in, running a good burn-in scrub on both of them before I put them to use. So far the first 20% of the drive averages 158 MB/s read.. not too bad for 5900RPM spinners.
 
black6spdz said:
should I be worried about only 4GB of RAM and ZFS mirroring? I know every recommendation says 8GB but this mini itx board I have only supports 4GB max "G41 chipset".. maybe I'll get lucky and find some 4GB DIMMs that will work. I just got my pair of Seagate ST4000DM000 drives in, running a good burn-in scrub on both of them before I put them to use. So far the first 20% of the drive averages 158 MB/s read.. not too bad for 5900RPM spinners.
Click to expand...

1-2 GB are enough for a stable and reactive OS.
With 4 TB you have a 2-3 GB readcache - more than enough for a single user or with smaller files. If you for ex have more users that want to view a video concurrently you may have drop outs on slow disks as performance fall back to pure disk performance. With more RAM most data is delivered from RAM after first read from disks.

But 4 GB is really performant for a usual workload.
 
Do you have some other use for the existing Windows 8 license that you're currently using on this servver? If not, just use that. Why make it more complicated that it needs to be for less that 3TB of data? I'd also further decomplicate things by skipping the RAID and just backing up the data periodically to an external drive.
 
black6spdz said:
I did my homework and read about ZIL and L2ARC, implementing ZIL seems too risky for marginal gains, L2ARC is not too beneficial unless you have many clients. Key seems to put as much RAM as you can. I'm a bit afraid of the consequences of using non-ecc memory though, as they say scrubs with bad memory can corrupt your entire z-pool. Maybe I shouldn't use ZFS at all and stick with GEOM Mirror "software raid"?
Click to expand...

A common misunderstanding
A Zil is always and only used on secure sync writes - either on pool or on a dedicated device. For a filer ex with CIFS it is not used at all

A L2ARC extends the fast rambased readcache (ARC) with a slower SSD. Can be usefull if you cannot add more RAM for the faster ARC cache.

RAM errors can corrupt data on every filesystem beside a ton of other reasons. These errors are mostly undetected unless you read the file or when it affects metadata.

The reason everyone suggests ECC together with ZFS in a special way is that this is the most propable way for undetected data errors or data loss on ZFS. Nearly all other errors that are possible on other filesystems and are there undetected will be detected on ZFS due to checksums and repaired automatically due the self healing filesystem or does not occur due to CopyOnWrite.

Compare that to a car. If you have an old car, why should you care of an airbag if all other modern security features are missing. If you have them all you won't miss that extra.
 
Last edited:
black6spdz said:
should I be worried about only 4GB of RAM and ZFS mirroring? I know every recommendation says 8GB but this mini itx board I have only supports 4GB max "G41 chipset".. maybe I'll get lucky and find some 4GB DIMMs that will work. I just got my pair of Seagate ST4000DM000 drives in, running a good burn-in scrub on both of them before I put them to use. So far the first 20% of the drive averages 158 MB/s read.. not too bad for 5900RPM spinners.
Click to expand...

I agree with the above that this RAM is sufficient as long as there are single clients such as a computer interactively used by one user accessing the array of the network. It will get tricky if you try interactive programs with files based on it and at the same time copy large video files around.

The contiguous throughput won't be affected anyway by low RAM.

With those seagates don't forget to scrub regularly.
 
I may occasionally stream a movie from it and I run a couple basic VMs from it most of the time, besides it being the download location and my over all dump space for everything. I think I'll take this unused 60GB Agility 3 and dedicate it to an L2ARC, this should ease the load when I stream some movies. Besides my initial MHDD scans of these drives and several passes of memtest, are there any other pre-deployment shakedown tests to ensure the hardware and setup is good to go?
 
Both drives scrubbed several times.. no problems out of the box. I've decided to upgrade my hardware a bit though. The 4GB max and onboard realtek GigNIC will ultimately hurt performance. I'm taking a hit on storage space downsizing but want to try to keep the performance on par with my old setup. I just ordered a new LGA 1155 main board and I already have a pair of 4GB mem sticks , an i5-3450 and a dual-port PCI-E Intel server NIC.
 
I'm starting to remember why I moved away from ZFS in all its glory on paper.. I get several seconds of >100MB/s throughput then several seconds of pausing.. and this repeats over and over. Why can't I copy large files and sustain high throughput? And this is just a mirror, no high CPU utilization calculating parity on RaidZ arrays. Sure hope the upgraded hardware will fix this.
 
Last edited:
black6spdz said:
I'm starting to remember why I moved away from ZFS in all its glory on paper.. I get several seconds of >100MB/s throughput then several seconds of pausing.. and this repeats over and over. Why can't I copy large files and sustain high throughput? And this is just a mirror, no high CPU utilization calculating parity on RaidZ arrays. Sure hope the upgraded hardware will fix this.
Click to expand...

This is how ZFS works on regular writes and this is not a bug but one of the best ZFS features.
All writes go to RAM for about 5s and are then written as one large sequential write to disk while the RAM collects the next 5s in the meantime. This massively improves performance with many small writes.

With a 1Gb (=about 100 MB/s) network, this write cache can go up to about 500 MB in 5s. If your pool is capable to write 500MB in 5s (100MB/s) this is nothing that slow down large writes while it improves small writes.

If you have concurrent read/writes on a quite slow pool (slow single disk or raid-1), your pool may not be capable of such a streaming write performance with the effect of a slower write performance. On a fast pool it jumps as well but the average write performance is equal the datastream form network.

But the average write speed for large files in such a case would not be better without this "many small random writes to one large sequential write" conversion.
 
Last edited:
_Gea said:
This is how ZFS works on regular writes and this is not a bug but one of the best ZFS features.
All writes go to RAM for about 5s and are then written as one large sequential write to disk while the RAM collects the next 5s in the meantime. This massively improves performance with many small writes.

With a 1Gb (=about 100 MB/s) network, this write cache can go up to about 500 MB in 5s. If your pool is capable to write 500MB in 5s (100MB/s) this is nothing that slow down large writes while it improves small writes.

If you have concurrent read/writes on a quite slow pool (slow single disk or raid-1), your pool may not be capable of such a streaming write performance with the effect of a slower write performance. On a fast pool it jumps as well but the average write performance is equal the datastream form network.

But the average write speed for large files in such a case would not be better without this "many small random writes to one large sequential write" conversion.
Click to expand...

Every OS works like you describe but ZFS goes quite a bit further taking up more RAM in more buffers. On most (all?) OSes there also is double buffering for ZFS since ZFS has its own but the OS'es VM cache is still operating normally. That's why you need a lot more RAM when running ZFS.

Of course it is unrealistic to expect the initial performance to hold up since it doesn't hit the disks. The only thing that really matters is sustained performance up until an fsync(2) call, unless you happen to unlink the file before it's committed.
 
why does it halt the network buffer while it writes out to disk? If the network is capable of 100MB/s transfer and read/writes are also capable of 100MB/s I don't see why there are any substantial pauses in data flow. I'm going to experiment a little while I wait for my new hardware gets here.. when I throw over twice the CPU, twice the RAM and an intel server nick on it the gig-e link better be the only bottleneck.
 
Last edited:
black6spdz said:
why does it halt the network buffer while it writes out to disk? If the network is capable of 100MB/s transfer and read/writes are also capable of 100MB/s I don't see why there are any substantial pauses in data flow. I'm going to experiment a little while I wait for my new hardware gets here.. when I throw over twice the CPU, twice the RAM and an intel server nick on it the gig-e link better be the only bottleneck.
Click to expand...

This happens if the disk is not fast enough. A modern disk is capable of reading/writing >100 MB/s but only on pure streamings. A single concurrent read/write or a fragmented filesystem and you are gone because then I/O performance is the limiting factor.

You can share a Zvol via iSCSI (the fastest way to share) and do a Crystalmark benchmark that also checks I/O and smaller or concurrent (qd32) writes. Disk performance can go down under these conditions to 10%. If you enable secure sync write this may be even worser.

Compare http://napp-it.org/doc/manuals/benchmarks.pdf

Avoiding ZFS with its checksumming and Copy on Write may be a faster option. But then you loose the always consistent filesystem, the security that data is valid, the self healing filesysten, the snaps and many other features. For what should that be good.
 
black6spdz said:
why does it halt the network buffer while it writes out to disk? If the network is capable of 100MB/s transfer and read/writes are also capable of 100MB/s I don't see why there are any substantial pauses in data flow. I'm going to experiment a little while I wait for my new hardware gets here.. when I throw over twice the CPU, twice the RAM and an intel server nick on it the gig-e link better be the only bottleneck.
Click to expand...

It's hard to remote diagnose but in general the OS will build up write buffer cache like mad, in the case of ZFS double, and then stall while sorting out the buffers at some point, even if throughput on the plain disks would have been sufficient to satisfy the incoming stream. In this situation having more RAM can actually make things worse. It's one of the reasons why RAM-hungry solutions like Linux in general or ZFS (FreeBSD without ZFS is better) are not universally popular. ZFS also tunnels normal control. The Linux kernel can be tuned to behave better here but that only affects the OS'es own VM cache. ZFS's own RAM usage ignores it.

If you have a high degree of control over the program that does the writing you could switch to writing with O_DIRECT.
 
If all you want is a simple two-way mirror of two 4tb drives then there is no reason to do anything special at all. You don't need hardware raid and you don't need a sophisticated OS or file system. Since you seem to be comfortable with windows then stick to what you know. Win 8.1 with a storage spaces mirror will do just fine for what you are proposing.
 
I'm not opposed to using ZFS, this isn't my first run around with it at all going back several years. I just never could understand while the "bloated" windows OS could maintain much faster network throughput on the same hardware. Is it all because of ZFS's overhead?
 
black6spdz said:
I'm not opposed to using ZFS, this isn't my first run around with it at all going back several years. I just never could understand while the "bloated" windows OS could maintain much faster network throughput on the same hardware. Is it all because of ZFS's overhead?
Click to expand...

Partly true.
Its not Windows vs ZFS its ext/ntfs vs ZFS.
Windows with ReFS is also slower as Windows with ntfs.

With realtime checksumming you have to write more data. With Copy On Write you do not update only modified data but you rewrite all the data. The same with fragmentation. This is a problem on all filesystem but Copy On Write filesystems like btrfs, ReFS or ZFS are more sensitive.

So Raw sequential write performance on ZFS can be slower on single disks/mirrors compared to older filesystems. But ZFS scales very well over disks/vdevs. This and the intelligent memory management can deliver nearly any desired performance.
 
Storage is a bug I tell you! Being worried about write performance of a single drive mirror setup I decided to buy two more 4TB drives.. so much for downsizing eh? So no I have 16TB in 4 drives instead of 8... but I think I can squeeze them all in this mini itx case. My dilemma now is RaidZ2 or Raid10? Another minor dilemma is with only a single pci-e slot on this board, should I use it for the intel server nic or a two-port sata card so I can utilize my SSD for L2ARC
 
black6spdz said:
I'm not opposed to using ZFS, this isn't my first run around with it at all going back several years. I just never could understand while the "bloated" windows OS could maintain much faster network throughput on the same hardware. Is it all because of ZFS's overhead?
Click to expand...

No. You need to look at the specifics.

Linux caches like mad when writing because it wants to keep the data around in case you use it soon, or in case you erase the file before fully writing it. This is being criticized, FreeBSD is not doing it that way an you can tell Linux to cut it out, although not really in a flexible enough way.

Then there is ZFS's own caching which is not integrated with the VM buffer cache. It's overhead for you on a linear write but you wouldn't call it overhead if it later delivers your data in no time :)
 
I'll start this new machine off with 8GB of RAM but may put 16GB in the future.. I'm leaning towards RaidZ2 for the slightly better redundancy odds, and I should have more than enough CPU to handle parity.. I'm not sure which would be more benificial, SSD L2ARC or Intel server NIC instead of the built-in realtek
 
black6spdz said:
I'll start this new machine off with 8GB of RAM but may put 16GB in the future.. I'm leaning towards RaidZ2 for the slightly better redundancy odds, and I should have more than enough CPU to handle parity.. I'm not sure which would be more benificial, SSD L2ARC or Intel server NIC instead of the built-in realtek
Click to expand...

I vote Intel nic, onboard Realtek chips generally suck in my opinion.
 
yea that's been my experience in the past as well.. but I thought I read the realtek's are just bad on low powered CPUs
 
black6spdz said:
Hi all, my currect NAS has gone relatively untouched and trouble free for a couple years. Its currently running:

Win 8 - with server 2012 dedup,
C2Q 2.4 GHz,
8GB RAM,
8x Samsung HD204UI "Originals" 2TB
IBM M5015 with Raid6 key "Raid6".
Large Supermicro 8-hotswap bay 4U chassis.

I am currently only using 2.54TB with 152GB saved in deduplication. Since my data hasn't grown in quite some time I've tossed around the idea of downgrading. I have a small cube case with a C2D 3.3GHz and 4GB DDR3. So I've been thinking about downsizing to it with a couple mirrored 4TB drives. This would use a lot less power and less heat. What's the latest and greatest NAS OS if I went that route? How about Nas4Free with a 60GB Vertex 3 L2ARC and 8 or 16GB of RAM? The new LZ4 compression peaks my interest too.
Click to expand...

I'm doing something similar with the drives right now only going from 11 drives to 3 (might add a couple more of the old drives back to increase space), but upgraded the size of the case to fit some water cooling so I didn't have to buy a new cooler (I like big cases because my gaming PC this year will likely become my server in 4 years).

For OS I'm going to try Windows Server Essentials and DrivePool or Driver bender to create the giant virtual drive.
This will enable me to do nightly incremental backups on both my HTPC's and Laptops if they are sleeping, as well as backup my main rig. It enables me to have folder level redundancy. Allows me to run my security cameras and blue iris to view them on my phone.

Not sure what a NAS OS has to offer over such things, maybe easier setup?
 
I am running a very light install of Win8 with Server 2012 dedup right now on 8x 2TB F4s.. its a server case with redundant PSUs, lots of fans, not a bad thing for a NAS but overkill really. I want a new simple setup in this small cube case. I like the classic FreeNAS "NAS4Free now" interface and its built on FreeBSD 9.2 with latest ZFS v5000.. the project is pretty mature and they still do fairly frequent updates which I like. With two of these 5900 RPM 4TB drives it only draws 55w idle, I think it will be around ~80-90w with 4 drives. I want to say my current rig is ~300w if I remember correctly but I'll plug in the watt-meter and check again for comparison.
 
You must log in or register to reply here.
Back
Top