Freenas Build

[parlotti]

Below is a parts list for a freenas build. I need a fresh pair of eyes on these components. I was originally going to go with the Asrock C2750D4I as my mainboard but I have heard too many who suffer with the marvel controllers on that board when doing a build above six HDDs. Has anyone any experience with a similar set of parts? Any insight is welcome.

PS: I was originally going to go with the E3-1230v3 (socket 1150) processor but the one below is supported by the board and on promo now.

- X2 SanDisk USB 3.0 Ultra Fit 32GB
- Fractal Design Define R5 (Black)
- Fractal Design Newton R3 600W
- X2 Crucial DDR3 PC12800/1600MHz ECC 2x8GB
- X8 WD Red WD60EFRX 64MB 6TB
-Intel Xeon E3-1241v3 3,5GHz
-Supermicro X10SL7-F

 

[Deleted member 3081]

Overall it looks good. Only things to keep in mind is that FreeNAS still doesn't like USB3.0 devices (well... most USB 3.0 controllers that is), so try to stick to USB2.0 devices if you can. A 32GB flash drive is also overkill too; 16GB is more than big enough to house the OS.

The other thing is you *NEED* to flash the LSI controller on that board to P16 IT mode. You can get the P16 IT firmware from Supermicro at this link:

ftp://ftp.supermicro.com/Driver/SAS/LSI/2308/Firmware/IT/Previous Releases/

Instructions on how to do it:

http://blog.widodh.nl/2014/10/flash-lsi-2308-to-it-mode-on-a-supermicro-x10sl7-f-mainboard/

The only thing that concerns me is the number of HDD's. Depending on the RAIDZ level you may want to go with nine (an odd number) drives instead for performance reasons. I forgot where I read that, but supposedly ZFS works better with an odd number of drives. Unless you're running in a production environment for a multi-user business and the system is being used to house a database, it probably won't make much of a difference though. I'm sure someone with more experience will comment on the topic here though.

 

[parlotti]

Overall it looks good. Only things to keep in mind is that FreeNAS still doesn't like USB3.0 devices (well... most USB 3.0 controllers that is), so try to stick to USB2.0 devices if you can. A 32GB flash drive is also overkill too; 16GB is more than big enough to house the OS.

The other thing is you *NEED* to flash the LSI controller on that board to P16 IT mode. You can get the P16 IT firmware from Supermicro at this link:

ftp://ftp.supermicro.com/Driver/SAS/LSI/2308/Firmware/IT/Previous Releases/

Instructions on how to do it:

http://blog.widodh.nl/2014/10/flash-lsi-2308-to-it-mode-on-a-supermicro-x10sl7-f-mainboard/

The only thing that concerns me is the number of HDD's. Depending on the RAIDZ level you may want to go with nine (an odd number) drives instead for performance reasons. I forgot where I read that, but supposedly ZFS works better with an odd number of drives. Unless you're running in a production environment for a multi-user business and the system is being used to house a database, it probably won't make much of a difference though. I'm sure someone with more experience will comment on the topic here though.

Thanks for the great pointers and links for the LSI firmware. There is nothing like a fresh pair of eyes to help make us see what is wrong or can be improved.

I shall do the following based on your insight:

I shall drop the USB drives to 16GB USB 2.0 drives (Happen to have some handy so that saves money)

I shall also look into getting a sata DOM and dropping the USB route all together (Hopefully they can be found at a good price)

As for the even and odd number ZFS, I shall sort that out during testing by startting off with 7 drives. Its worth learning more about that first hand on this build.

Thanks again for your suggestions, they helped

 

[Deleted member 3081]

Skip the SATA DOM, it's completely unnecessary. FreeNAS is designed to be run from a USB flash drive so you have the max amount of free SATA ports available for your build. Going the SATA DOM route will only eat up a SATA port and won't give you any performance benefits.

You're better off spending the money you'd put towards a SATA DOM towards either a UPS if you don't have one, more RAM (preferably), two SSD's for the ZIL and one as an L2ARC cache drive, and/or a managed gigabit switch.

Honestly though? You'd have to be doing some specific workloads to see the benefits of adding SSD's to the system. Here's my current setup for comparison:

Old socket AM2 MSI board (Nforce 980 based if memory serves me right)
AMD Phenom X6 1100T
16GB RAM (not ECC)
two 128GB SSD's mirrored for the ZIL cache
one 32GB SSD for the L2ARC cache (doesn't need to be mirrored so only went with a spare 32GB drive I had kicking around)
Intel dual-port server NIC (previous generation), set up as a 2GB LAGG connection
ZyXel managed gigabit switch
Seven 2TB HDD's in RAIDZ2 config (about 8.2TB of space after provisioning).
IBM M1015 crossflashed to LSI (running P16 IT firmware w/o the management rom flash so it doesn't slow down the bootup)
8GB Sandisk Cruiser Fit hosting the FreeNAS install.


The system was actually my old gaming system that I re-purposed after I upgraded to a new gaming rig and a separate workstation that I use for my photography work. The FreeNAS system is what holds all my photographic work as well as other software I've accumulated over the years. I started off with just FreeNAS 8 without any of the SSD's in the system when I first built it. In the past year I upgraded it to FreeNAS 9 and then afterwards added the SSD's to see if I could squeeze more performance out of the system.

Truth be told, it's a hair faster with the SSD caching, but not enough where I'd have spent the money to buy the SSD's specifically for this. FreeNAS already buffers the writes in ram so there's not going to be a huge gains to be had in my usage scenario. I happened to have those drives laying around after doing upgrades on other computers I owned so instead of letting them sit collecting dust I put them to some use.

I don't know your usage scenario but for mine (one user/high burst throughput when writing large .PSD and camera raw files) the SSD's didn't make any significant difference. If you have multiple users hitting the box at the same time and they're writing a large amount of data, the SSD's will make a difference.

If it's just you using the box I'd say stick with what you spec'ed out for now. It's already a really nice setup and is going to perform great.

 

[parlotti]

Skip the SATA DOM, it's completely unnecessary. FreeNAS is designed to be run from a USB flash drive so you have the max amount of free SATA ports available for your build. Going the SATA DOM route will only eat up a SATA port and won't give you any performance benefits.

You're better off spending the money you'd put towards a SATA DOM towards either a UPS if you don't have one, more RAM (preferably), two SSD's for the ZIL and one as an L2ARC cache drive, and/or a managed gigabit switch.

Honestly though? You'd have to be doing some specific workloads to see the benefits of adding SSD's to the system. Here's my current setup for comparison:

Old socket AM2 MSI board (Nforce 980 based if memory serves me right)
AMD Phenom X6 1100T
16GB RAM (not ECC)
two 128GB SSD's mirrored for the ZIL cache
one 32GB SSD for the L2ARC cache (doesn't need to be mirrored so only went with a spare 32GB drive I had kicking around)
Intel dual-port server NIC (previous generation), set up as a 2GB LAGG connection
ZyXel managed gigabit switch
Seven 2TB HDD's in RAIDZ2 config (about 8.2TB of space after provisioning).
IBM M1015 crossflashed to LSI (running P16 IT firmware w/o the management rom flash so it doesn't slow down the bootup)
8GB Sandisk Cruiser Fit hosting the FreeNAS install.


The system was actually my old gaming system that I re-purposed after I upgraded to a new gaming rig and a separate workstation that I use for my photography work. The FreeNAS system is what holds all my photographic work as well as other software I've accumulated over the years. I started off with just FreeNAS 8 without any of the SSD's in the system when I first built it. In the past year I upgraded it to FreeNAS 9 and then afterwards added the SSD's to see if I could squeeze more performance out of the system.

Truth be told, it's a hair faster with the SSD caching, but not enough where I'd have spent the money to buy the SSD's specifically for this. FreeNAS already buffers the writes in ram so there's not going to be a huge gains to be had in my usage scenario. I happened to have those drives laying around after doing upgrades on other computers I owned so instead of letting them sit collecting dust I put them to some use.

I don't know your usage scenario but for mine (one user/high burst throughput when writing large .PSD and camera raw files) the SSD's didn't make any significant difference. If you have multiple users hitting the box at the same time and they're writing a large amount of data, the SSD's will make a difference.

If it's just you using the box I'd say stick with what you spec'ed out for now. It's already a really nice setup and is going to perform great.

Thanks a lot for weighing in. I see your point about the DOM. I think I shall just go with what I have and adjust as need be. Also thanks for sharing your specs. It really helps one gauge what is absolutely essential for such a build.

This build shall house all NFS mounts for a 3 workstation LAN, pictures and a few jails (for Plex etc). And maybe light ISCSI for my proxmox box

Do the specs look good for that load?

 

[BlueLineSwinger]

I have a similar setup. That mainboard works very well. As others mentioned be sure to flash the LSI HBA to the proper firmware version.

The CPU should be just fine. Were this simply a storage build I'd say it's very much overkill. With Plex and other jails in play it looks to be a good match. I think you should be able to get at least 2-3 simultaneous streams out of it.

RAM is fine for a storage build. I'm not sure how Plex and the other jails will cut into that, however. You may have to experiment/monitor. Be sure to start with 2x8 GB sticks to leave the other pair of slots open in case you need to upgrade later.

As mentioned, SATA DOM is unnecessary and eats a port. The current plan to use a USB2 flash drive is correct. Even better, the installer allows you to install the OS to a mirrored pair, so if one flash drive fails you're still up and running.

I'm currently using a Fractal Arc Midi R2 for the case. It works well, and is well-ventilated. WD Reds don't put out too much heat, which helps.

Definitely get a UPS. Losing your data due to an abrupt power outage would suck. A UPS will let you ride out power glitches and perform a graceful shutdown in the event of an outage.

The only thing that concerns me is the number of HDD's. Depending on the RAIDZ level you may want to go with nine (an odd number) drives instead for performance reasons. I forgot where I read that, but supposedly ZFS works better with an odd number of drives. Unless you're running in a production environment for a multi-user business and the system is being used to house a database, it probably won't make much of a difference though. I'm sure someone with more experience will comment on the topic here though.

The "rule" for RAIDZ for the number of drives is the total should be a power of 2 plus the RAIDZ level. i.e., for RAIDZ the recommended number of drives is 3, 5, or 9. For RAIDZ2 it's 4, 6, 10, and RAIDZ3 5, 7, 11. As you noted though, for this particular deployment it probably won't matter. I'd skip RAIDZ and go for RAIDZ2.

 

[parlotti]

I have a similar setup. That mainboard works very well. As others mentioned be sure to flash the LSI HBA to the proper firmware version.

The CPU should be just fine. Were this simply a storage build I'd say it's very much overkill. With Plex and other jails in play it looks to be a good match. I think you should be able to get at least 2-3 simultaneous streams out of it.

RAM is fine for a storage build. I'm not sure how Plex and the other jails will cut into that, however. You may have to experiment/monitor. Be sure to start with 2x8 GB sticks to leave the other pair of slots open in case you need to upgrade later.

As mentioned, SATA DOM is unnecessary and eats a port. The current plan to use a USB2 flash drive is correct. Even better, the installer allows you to install the OS to a mirrored pair, so if one flash drive fails you're still up and running.

I'm currently using a Fractal Arc Midi R2 for the case. It works well, and is well-ventilated. WD Reds don't put out too much heat, which helps.

Definitely get a UPS. Losing your data due to an abrupt power outage would suck. A UPS will let you ride out power glitches and perform a graceful shutdown in the event of an outage.



The "rule" for RAIDZ for the number of drives is the total should be a square of 2 plus the RAIDZ level. i.e., for RAIDZ the recommended number of drives is 3, 5, or 9. For RAIDZ2 it's 4, 6, 10, and RAIDZ3 5, 7, 11. As you noted though, for this particular deployment it probably won't matter. I'd skip RAIDZ and go for RAIDZ2.

Thanks a lot for your insight and useful suggestions. A UPS would buy me good protection and they dont cost that much either when compared with the risk of losing data. I think this build is a go. Shall post back here how it went.

Thank you all for helping.

 

[ThreeDee]

the more ECC ram the merrier with FreeNAS .. I run Plex off of my setup with myself, the wife and our little rug rat streaming simultaneously and a friend that I share my library with watching stuff occasionally without a hitch .. my setup via copy and paste from my signature over in the FreeNAS forums:

| Intel(R) Xeon(R) CPU E31230 | 1155 Intel S1200BTL Motherboard | 2 x 8GB/2 x 4GB ECC DDR3 UDIMM's | 5 x 2TB HDD Raidz | 750wtt 80Plus PSU | Generic Full Tower Case | FreeNAS-9.3 | Installed on 64GB PNY USB Flash Drive (Thanks SEF!) | PLEX Media Server | 700wtt Battery Back-up |

 

[Netwerkz101]

<snipped>
- X2 Crucial DDR3 PC12800/1600MHz ECC 2x8GB
-Intel Xeon E3-1241v3 3,5GHz
-Supermicro X10SL7-F
</snipped>

Don't know your budget or if you are interested in lightly used equipment.

Check my sig / for sale thread. (Item #3)

I have the motherboard and RAM - and I believe I left the controller flashed
at version 16 because I was testing FreeNAS 9.x as well.

The processor I have in it currently may/may not meet your needs.

Fractal Design R4/R5 are nice but NZXT Source 210 Elite is capable for lot less ..again not knowing your budget
(I have an R4 and Source 210 myself - should have went Elite on the NZXT)

 

[parlotti]

the more ECC ram the merrier with FreeNAS .. I run Plex off of my setup with myself, the wife and our little rug rat streaming simultaneously and a friend that I share my library with watching stuff occasionally without a hitch .. my setup via copy and paste from my signature over in the FreeNAS forums:

| Intel(R) Xeon(R) CPU E31230 | 1155 Intel S1200BTL Motherboard | 2 x 8GB/2 x 4GB ECC DDR3 UDIMM's | 5 x 2TB HDD Raidz | 750wtt 80Plus PSU | Generic Full Tower Case | FreeNAS-9.3 | Installed on 64GB PNY USB Flash Drive (Thanks SEF!) | PLEX Media Server | 700wtt Battery Back-up |

Wow. That's reassuring. I shall be doing more or less the same and hope to get good results like you have.

 

[parlotti]

Don't know your budget or if you are interested in lightly used equipment.

Check my sig / for sale thread. (Item #3)

I have the motherboard and RAM - and I believe I left the controller flashed
at version 16 because I was testing FreeNAS 9.x as well.

The processor I have in it currently may/may not meet your needs.

Fractal Design R4/R5 are nice but NZXT Source 210 Elite is capable for lot less ..again not knowing your budget
(I have an R4 and Source 210 myself - should have went Elite on the NZXT)

I had not set any budget restraints on this project. But I'm a practical person and would much rather pay less than more for parts. That said, I have pulled the trigger on this build and the parts are mostly here. I hope I get to catch an offer like yours earlier on my next build.

Cheers

 

[parlotti]

The build went perfectly guys. Thanks for all your help. Everything matches and fits together. Now Im just testing the memory. So far so good. When that is done, I shall test the drives then do the final install and move my data. LSI 2308 flashing to IT mode went great and Im currently testing with freenas.