ZFS RAM Intensive?

Joust

Supreme [H]ardness
Joined
Nov 30, 2017
Messages
4,913
I'm using ZFS ala FreeNAS. In ZFS2 (RAID 6 type).

Hardware: MSI Z77 Mobo, 3570k, 16(2x8) ripjaws 1600, 2 Dell/lsi HBAs, 16X4tb 7.2k Seagate Constellations.

I'm seeing very high wired RAM usage all the time - without any stress on it other than writing data. What's the wisdom here? Need more RAM? The only thing the box does is Plex.
 

IdiotInCharge

NVIDIA SHILL
Joined
Jun 13, 2003
Messages
14,679
Need?

Nope. ZFS will use all of the RAM you give it, but it's for caching, and that's mostly unnecessary for just file-server usage.

More for a SAN, where the storage is being used for VMs and other services. Just local stuff at home, you need the 8GB that FreeNAS requires, and anything else is cake.
 

Joust

Supreme [H]ardness
Joined
Nov 30, 2017
Messages
4,913
Need?

Nope. ZFS will use all of the RAM you give it, but it's for caching, and that's mostly unnecessary for just file-server usage.

More for a SAN, where the storage is being used for VMs and other services. Just local stuff at home, you need the 8GB that FreeNAS requires, and anything else is cake.

Well. It does transcode. But I haven't put this config through paces yet. The new install on 11.2 gave me fits. Took me hours to figure out why the hell metadata wasn't working.
 

extide

2[H]4U
Joined
Dec 19, 2008
Messages
3,494
The ARC (memory read cache in ZFS) uses prefetching and will pull data into the cache, hopefully, before it is needed.
It is designed to pretty much use all of the available memory, but can be tuned. Check out this article for more information:
https://wiki.freebsd.org/ZFSTuningGuide

See vfs.zfs.arc_max
 

ChadD

Supreme [H]ardness
Joined
Feb 8, 2016
Messages
5,391
Simple answer 16gb is lite for 64tb of drive.

For 64TB of storage ideally you would run 64gb of ram. (you could perhaps get by with 32, if your really not writing a ton of data at once... parity data writes add a good bit of overhead0 With raid6 your writing a lot of parity data and ZFS is hungry.
 

BlueLineSwinger

[H]ard|Gawd
Joined
Dec 1, 2011
Messages
1,052
This is true if you're running a production appliance and actually taxing the array(s). Just using it as a home NAS? It'll run fine on 8GB. I give mine 12GB and call it done.


Exactly. I have a 6 GB x10 RAID-2Z FreeNAS system running on 16 GB RAM for home use. That's more than enough for such a setting.
 

mwarps

Supreme [H]ardness
Joined
Oct 6, 2002
Messages
7,045
ZFS ARC counts as wired memory. This is normal, expected, and fine.
You do not need more memory.
 

DeathFromBelow

Supreme [H]ardness
Joined
Jul 15, 2005
Messages
7,316
Simple answer 16gb is lite for 64tb of drive.

For 64TB of storage ideally you would run 64gb of ram.

The 1GB per TB thing is a good rule of thumb but it really depends on the workload and number of users. I mean, I've run a 30TB FreeNAS box with 10GBe on 8GB and it actually performed fine, but I was the only user. For most people 16GB is more than adequate even with a large pool.
 

Joust

Supreme [H]ardness
Joined
Nov 30, 2017
Messages
4,913
Two completely different answers here. Heh. I hope the guys saying I'm good to go are right...
 

Dew

2[H]4U
Joined
Jun 23, 2003
Messages
3,852
Exactly. I have a 6 GB x10 RAID-2Z FreeNAS system running on 16 GB RAM for home use. That's more than enough for such a setting.

To pile on. 16GB is more than sufficient for NAS service, especially in a home scenario.

The more RAM, the better. I try to run 128GB as a minimum on all my filers, but that is because I depend heavily on that RAM cache for performance and do not use L2ARC. I also run sync=always across the board, but that is overkill and me being extra paranoid for the filers.

One of the legacy filers here that I inherited has over 80TB of data on it and does just fine with 16GB of RAM serving 10x clients.
 

Joust

Supreme [H]ardness
Joined
Nov 30, 2017
Messages
4,913
For what it's worth, I thought I'd provide an update for anyone who comes across this later in life:

I was doing a gazillion reads/writes on a ZFS2 array of 16 drives. After the memory leak in freebsd, I kept having server crashes which looked like this:

First mapped drives (smb) failed.
Second, FreeNAS GUI failed.
Third, SSH and console would fail.

Hard reset / power cycle required.

That was with an 3570k and 16 gb ram in dual channel. I doubled it to 32 gb and have much, much increased stability.

YMMV.
 

Joust

Supreme [H]ardness
Joined
Nov 30, 2017
Messages
4,913
Update 2:

The FreeNAS machine doesn't crash, but does drop mapped drives. Currently attributing this to the onboard realtek nic, Intel nic arrives Monday. Going to get this working.
 
Top