• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Pre-ZFS Build info

R

Ripley

Limp Gawd
Joined
Nov 4, 2004
Messages
240
I've been planning a ZFS system for a while. I have 24 1TB drives for the system. I'm going to use the new Norco RPC-4224(http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038). I am planning on using an AMD processor, something like this AMD Phenom II X4 925(http://www.newegg.com/Product/Product.aspx?Item=N82E16819103916) or this AMD Phenom II X6 1055T(http://www.newegg.com/Product/Product.aspx?Item=N82E16819103851) along with ECC Unbuffered RAM. What I'm stuck on are the following:

1.) What motherboard.
What's improtant for a build like this? What features do I need/want? Specific recommendations welcome.

2.) Multiple HBAs or an expander.
Where's the cost/performance split on this?

3.) Which HBA.

4.) Intel NICs
I see a lot of people saying to go with Intel NICs. With an AMD board that's not an onboard option. Do I need to consider a PCIe card?

5.) Power supply
How do I size a power supply for this?

6.) OS drive
I was thinking of this for the OS drive (http://www.newegg.com/Product/Product.aspx?Item=N82E16817997008). Any other ideas? I am planning on using a 2.5" drive for the OS.

7.) Anything else I'm not thinking of.
 
Ripley said:
1.) What motherboard.
What's improtant for a build like this? What features do I need/want? Specific recommendations welcome.

2.) Multiple HBAs or an expander.
Where's the cost/performance split on this?

3.) Which HBA.

4.) Intel NICs
I see a lot of people saying to go with Intel NICs. With an AMD board that's not an onboard option. Do I need to consider a PCIe card?

5.) Power supply
How do I size a power supply for this?

6.) OS drive
I was thinking of this for the OS drive (http://www.newegg.com/Product/Product.aspx?Item=N82E16817997008). Any other ideas? I am planning on using a 2.5" drive for the OS.

7.) Anything else I'm not thinking of.
Click to expand...


if you want to build a trouble-free system, i would suggest:

- Mainboard: should have at least 3 x pci-e 8x slots with Intel Nics (avoid Realtek)
good to have: Option for a lot of RAM, ECC-Ram, virtualization features

Personally, i would currently prefer Intel mainboards with 3420 or 5520 server chipsets.
One of the cheapest and most used with ZFS is the SuperMicro X8-SIL-F
http://www.servethehome.com/supermicro-x8silf-motherboard-v102-review-whs-v2-diy-server/

- HBA, avoid Hardware Raid-Controller. Simply the best are the LSI 1068e based ones
flashed with IT-mode firmware example Intel SASUC8I with 8 ports

- avoid Expander solutions when not needed, 3 x Intel SASUC8I (24 Ports) are about the same price like 1 x Intel SASUC8I + Expander but faster with less problems.

-Power Supply, depends on your drives, look at the needed power at start-up. that could be up to 20 or 30W per drive, reducing to about 6 w during operation.
In case of problems, you may otherwise need to unplug some drives on power-on cycles.
With 24 x 1TB drives look at dual PSU or ones with 700W+

OS-Drive, uncritical
i usually use cheap 160 GB WD 2,5" 24/7

Gea
 
_Gea said:
-Power Supply, depends on your drives, look at the needed power at start-up. that could be up to 20 or 30W per drive, reducing to about 6 w during operation.
In case of problems, you may otherwise need to unplug some drives on power-on cycles.
With 24 x 1TB drives look at dual PSU or ones with 700W+

OS-Drive, uncritical
i usually use cheap 160 GB WD 2,5" 24/7

Gea
Click to expand...
Just to note, if you do go with a 1068e solution, you should be able to do a staggered spin up to reduce the load.

Still, high wattage, quality PSUs aren't expensive... assuming you're doing a standard ATX power supply. heck if your case can support dual/redundant PSes go for it.

I also use laptop WDs 2.5" drives for my OS. WD mainly because I want the advanced warranty option.
 
While staggered spin-up is nice, my 2 TB Samsung F3EG drives seem to completely ignore the staggered spin-up setting on the backplane in contrast to the F1EG drives that adhere to the setting.

Can someone comment on if that works again with the F4EG drives?
 
I recall there is a pin on either the SATA power connector (i think this one) or the SATA data connector, that indicates whether the drive should spin up immediately upon receiving power. I can remember people drilling holes in their SATA (power?) cables just to achieve the effect of them not spinning up at once, and thus obeying the staggered spinup their controller initiates. But i'm puzzled by this as to how staggered setup would usually work; certainly not by drilling holes anywhere. :D
 
i have a lot of SuperMicro and Chenbro Server Cases and Powering on with a lot of Sata has been always a problem with some PSU. (I already have the problem with one of them; unplug some drives, power on and replug them prior to initilization)

Best is to buy a PSU big enough or use a dual PSU

@sub.mesa
its poor sata design. old scsi drives have had such a feature always
 
Blue Fox
Any specific boards? I haven't found any yet.

_Gea
The SuperMicro X8-SIL-F looks like it only has 2 PCIe x8 slots. Any other boards that have 3 sots? Also, why Intel over AMD?

adi
That HBA looks good. It'd be cheaper than buying three 1068e. What OS are you using it with?


Is there an easy dual PSU setup for the 4224 case?
 
about X8SIL-F

this board has 2 x 8x and 1 x 4x pci-e slots
http://www.supermicro.com/products/motherboard/xeon3000/3400/X8SIL.cfm

the 4x slot (using 8x slot) has 1 (2 in Duplex mode) GB/s Bandwith,
more than enough see
http://www.tomshardware.com/reviews/pci-express-scaling-analysis,1572-2.html

intel vs amd
my impression is, that it is cheaper to buy a amd with a special degree of cpu-power
but on the higher end, its currently intel who leads the market

some features like vt-d is known to work without problem with intel server-chipsets.
some amd boards may work also, but i do not know and there are not too much infos about.

Gea
 
Last edited:
adi
That HBA looks good. It'd be cheaper than buying three 1068e. What OS are you using it with?
Click to expand...

I'm running it under Solaris 11 Express on a Supermicro X8SIA-F-O.I upgraded from a 9211-8i because my old system ran out of PCI-e slots and couldn't just pop in another controller.
 
I prefer this one for a serious ZFS setup:

X8ST3-F_spec.jpg

SuperMicro X8ST3-F product webpage

Load it up with 24GiB DDR3 and ready to rock!

Advantages:
- very moderate power consumption given its features
- 6x full-speed SATA/300 ports
- 8x full-speed SAS/300 ports via LSI 1068E controller (fully compatible with BSD/Solaris)
- four PCI-express slots
- six DIMM slots (up to 24GiB non-ECC DDR3 RAM)
- IPMI on dedicated LAN port
 
The only problem with that board is the lack of ECC memory support.

sub.mesa said:
I prefer this one for a serious ZFS setup:

X8ST3-F_spec.jpg

SuperMicro X8ST3-F product webpage

Load it up with 24GiB DDR3 and ready to rock!

Advantages:
- very moderate power consumption given its features
- 6x full-speed SATA/300 ports
- 8x full-speed SAS/300 ports via LSI 1068E controller (fully compatible with BSD/Solaris)
- four PCI-express slots
- six DIMM slots (up to 24GiB non-ECC DDR3 RAM)
- IPMI on dedicated LAN port
Click to expand...
 
The only problem with that board is the lack of ECC memory support.
Click to expand...

I can't access the manual to get exact details on ECC support for Xeon vs I7, but it does support ECC. It appears to not support registered ECC though.

"Supports up to 24 GB 1333 / 1066 / 800MHz DDR3 ECC / non-ECC Un-Buffered memory"
 
I see that I read that wrong. What's the difference between registered and unregistered, buffered and unbuffered?
 
sub.mesa that board looks great but its a bit out of my budget. One of the main reasons I was looking at AMD was price. With a board that's $310+ and then a CPU on top of that, while I'm sure it would perform very well, its more than I can comfortably spend. Anyone got recommendations for an AMD solution?
 
-2x Intel® 82574L Gigabit Ethernet
-IPMI 2.0
-LSI SAS1068E
Now look for an AMD board and add the costs of those things which you would have to buy as addon cards. Still too expensive? ;)
 
Ripley said:
sub.mesa that board looks great but its a bit out of my budget. One of the main reasons I was looking at AMD was price. With a board that's $310+ and then a CPU on top of that, while I'm sure it would perform very well, its more than I can comfortably spend. Anyone got recommendations for an AMD solution?
Click to expand...

You need to keep in mind the fact that board already has 8xSAS ports integrated [along with the 6 onboard SATA], so that's a lot less money you need to spend on HBA's.

Once you actually factor that in - even forgetting the rest of the features - the board works out to be an absolute bargin!
 
While you can use ECC sticks with specific i3 chips, the ECC function will not be enabled.
 
nexxium said:
While you can use ECC sticks with specific i3 chips, the ECC function will not be enabled.
Click to expand...

Are you sure? The i3 datasheet definitely states support for ECC memory if used together with a server chipset. Even the ECC signals are stated as present in the pin-out list. I booted my X8SIA-F with an i3 once (also with an i5) and the BIOS stated ECC as active.
 
I am interested in this also as it would lower the cost of my build - which, by the way, is probably going to be based off of this thread - Thanks!

I see in the 2nd gen i3 spec doc that it says the type of memory is based on the PCH chipset 1.2.1.

http://download.intel.com/design/processor/datashts/324641.pdf

That would be very cool if it does support ECC. I have a i3 in my lab, but no supermicro board to lab it yet.
 
Can anyone confirm that ECC works with i3 procs? If so I'd think heavily about going that way. Get the good board and the cheaper processor with ECC RAM. Then if I need more power down the line I can swap in a more powerful chip.
 
Ripley said:
Can anyone confirm that ECC works with i3 procs? If so I'd think heavily about going that way. Get the good board and the cheaper processor with ECC RAM. Then if I need more power down the line I can swap in a more powerful chip.
Click to expand...

I posted this in a similar thread: http://www.intel.com/support/processors/corei3/sb/CS-031175.htm

I am pretty sure you can use an i3 with ECC Ram, however, you won't have the ECC functionality. I really don't see why Intel would allow the i3 to support ECC, because who would then buy the more expensive Xeon?
 
According to the thread you posted Intel said explicitly that ECC RAM will work with ECC functionality with an i3 and a "server class" motherboard.
 
Thank you for finding that thread. So my board was actually right as it reported ECC as active. :)
Also, don't trust Intel Ark to much, as it contains a lot of errors.
 
Does the Intel® Core™ i3 desktop processor support Error Correction Code (ECC) memory?

The Intel® Core™ i3 desktop processor does not support ECC memory. Typically ECC memory is used on servers and workstations, rather than on desktop platforms. This is due to the price premium and extremely low likelihood of a data error occurring even on memory not utilizing ECC.

Taken from http://www.intel.com/support/processors/corei3/sb/CS-031175.htm

Now maybe it indeed does work, and Intel just does not want that splashed up so we buy Xeon?


Edit: Looks like cnick79 already posted that link. Now, I can't seem to find any posts where someone has stated if memtest listed the status of ECC, using a i3+ecc ram+server class mobo.

Perhaps omniscence could perform that test, if possible? Would be nice to finally put this issue to rest.
 
Last edited:
So here's what I've got so far:


NORCO RPC-4224
http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038


(2) Intel SASUC8I PCI-Express x8
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117157


Supermicro X8ST3-F
http://www.provantage.com/supermicro-x8st3-f~7SUPM2FT.htm


Intel Xeon E5506 Nehalem-EP 2.13GHz LGA 1366
http://www.newegg.com/Product/Product.aspx?Item=N82E16819117186


(2) Kingston ValueRAM 4GB 240-Pin DDR3 SDRAM DDR3 1333 ECC Unbuffered
http://www.newegg.com/Product/Product.aspx?Item=N82E16820139077


Scythe Slot Rafter
http://www.newegg.com/Product/Product.aspx?Item=N82E16817984008

I have all the hard drives I need, both OS, and storage. The only thing I really need now is a power supply. Is there a dual power supply option for this thing? Also, what's a good single power supply?

Anything else that I am missing?
 
Last edited:
Ripley said:
So here's what I've got so far:


NORCO RPC-4224
http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038


(2) Intel SASUC8I PCI-Express x8
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117157


Supermicro X8ST3-F
http://www.provantage.com/supermicro-x8st3-f~7SUPM2FT.htm


Intel Xeon E5506 Nehalem-EP 2.13GHz LGA 1366
http://www.newegg.com/Product/Product.aspx?Item=N82E16819117186


(2) Kingston ValueRAM 4GB 240-Pin DDR3 SDRAM DDR3 1333 ECC Unbuffered
http://www.newegg.com/Product/Product.aspx?Item=N82E16820139077


Scythe Slot Rafter
http://www.newegg.com/Product/Product.aspx?Item=N82E16817984008

I have all the hard drives I need, both OS, and storage. The only thing I really need now is a power supply. Is there a dual power supply option for this thing? Also, what's a good single power supply?

Anything else that I am missing?
Click to expand...

Apologies for the n00b question but why are you getting the Intel HBA (correct?) when mobo comes with 8 SAS ports already?
 
e2g said:
Apologies for the n00b question but why are you getting the Intel HBA (correct?) when mobo comes with 8 SAS ports already?
Click to expand...

He wrote (2) before the HBA. Probably because the RPC-4224 has a 24-port backplane.
 
john4200 said:
He wrote (2) before the HBA. Probably because the RPC-4224 has a 24-port backplane.
Click to expand...

Yes, I know. If I understand the whole SAS thing, if it has 8 SAS ports, each capable of serving 4 SATA connections, that would be 32 SATA ports. I am obviously missing something/getting it wrong.


It seems SAS ports allows you to connect up to 128 devices, so I am guess on this motherboard, the assumed 4 SATA connections on each SAS port may be false. Hence, why OP is getting (2) HBA's??

It says on the product page ==> 8 SAS / SATA ports

Does that mean these SAS connections support (only) one SATA connection each and the total number of SATA drives with this board alone is 8 + 6 = 14?
 
e2g said:
Yes, I know. If I understand the whole SAS thing, if it has 8 SAS ports, each capable of serving 4 SATA connections, that would be 32 SATA ports. I am obviously missing something/getting it wrong.


It seems SAS ports allows you to connect up to 128 devices, so I am guess on this motherboard, the assumed 4 SATA connections on each SAS port may be false. Hence, why OP is getting (2) HBA's??

It says on the product page ==> 8 SAS / SATA ports

Does that mean these SAS connections support (only) one SATA connection each and the total number of SATA drives with this board alone is 8 + 6 = 14?
Click to expand...

about sas controller
on a 8 port sas-controller (internal or external) you can plugin no more than 8 drives
either SAS or SATA. for 24 drives you need 3 controller (3 x 8 port)

about backplanes
if your backplane has mini-SAS connectors, you usually cannot connect them to onboard Sata Ports ex via a fanout cable,
connect them only to controllers with mini-SAS connectors or use a backplanes with 24 separate SAS/Sata connectors.

about expanders
if your SAS controller is a HBA type (JBOD, with IT firmware), you can connect a expander to your SAS controller.
It works similar to a network switch but for SAS/Sata drives. Only with expanders you can connect up to 128 drives.

Gea
 
e2g said:
Yes, I know. If I understand the whole SAS thing, if it has 8 SAS ports, each capable of serving 4 SATA connections, that would be 32 SATA ports. I am obviously missing something/getting it wrong.


It seems SAS ports allows you to connect up to 128 devices, so I am guess on this motherboard, the assumed 4 SATA connections on each SAS port may be false. Hence, why OP is getting (2) HBA's??

It says on the product page ==> 8 SAS / SATA ports

Does that mean these SAS connections support (only) one SATA connection each and the total number of SATA drives with this board alone is 8 + 6 = 14?
Click to expand...

The board has 8 SAS channels, which are available via 2 mini-SAS ports on the board.

Each mini-SAS connecter carries 4 channels. That's why if you get a break-out cable they break out to 4 SATA ports.

Assuming you don't use expanders you will need one channel for each drive, or said another way, one mini-SAS connector for every 4 drives. The case in question here has 24 drives, so will need 6 mini-SAS ports. Two are on board, and there are two on each add-in card.
 
apnar said:
The board has 8 SAS channels, which are available via 2 mini-SAS ports on the board.

Each mini-SAS connecter carries 4 channels. That's why if you get a break-out cable they break out to 4 SATA ports.

Assuming you don't use expanders you will need one channel for each drive, or said another way, one mini-SAS connector for every 4 drives. The case in question here has 24 drives, so will need 6 mini-SAS ports. Two are on board, and there are two on each add-in card.
Click to expand...

Thank you! Much more clear. It took a while earlier today for me to reach this same conclusion as I am new to all of this. I hate to ask one more question but the case in question, the backplane it comes with, do you call that as an expander as well, or is it equivalent to a mini-SAS to 4-SATA ports cable? I see it has been generally said to avoid expanders due to performance, reliability issues?
 
_Gea said:
about the case
http://www.newegg.com/Product/Produ...-_-na-_-na&AID=10521304&PID=3891137&SID=194r0

This case has a backplane with 24 separate SAS/ Sata connectors. There is no build-in Expander. You can connect the backplane to onboard-Sata via regular SATA cables. If you have a SAS/Sata Adapter with SFF 8087 miniSAS-port (see http://en.wikipedia.org/wiki/Serial_Attached_SCSI ), you need a breakout cable SFF 8087 -> 4 x sata
Click to expand...

Gea,

You must be confusing the RPC-4224 case with one of their other models. The RPC-4224 has 6 back planes in it, each has a single mini-SAS connector (SFF-8087). So if you are using it with the configuration listed above you wouldn't need any break-out cables, just mini-SAS to mini-SAS

Case details here: http://www.norcotek.com/item_detail.php?modelno=RPC-4224

You are correct though that there are no built in expanders. The back plane act as simple pass throughs to the 4 drives they connect to, while also providing power distribution to the drives.
 
The Norco has Mini-SAS connectors as far as i know that can be connected to your SAS HBA with a special cable (mini-SAS to mini-SAS; not the reverse breakout). This is different from expanders, which you should not use, because each disk has its own signal/cable you just let one cable carry multiple signals.

So make sure you got the right cables, but otherwise you should be fine.

Some other stuff:

- how much RAM you have? This is a crucial factor for ZFS performance scaling. I prefer 24GiB DDR3 for 200 euro due to its insane low price and ZFS could use that memory very well! All of it!

- Intel NICs work fine, but if you don't have a problem with your onboard gigabit NIC i would stick with it; little reason to upgrade to Intel the performance of the cheapest realtek and an Intel PCIe NIC is less than 1%. But if the OS doesn't have a good driver for your onboard NIC and you have problems then buying an Intel PRO/1000 CT PCIe would be useful. I would try to avoid it!

- Os drive; depending on your OS, you could consider a simple USB stick. Easy to backup as well. If your OS does not/cannot use ZFS or another CoW-filesystem on the USB stick then i would not recommend going through with this option but rather sticking to your mirror of 2,5" notebook HDD; but it's alot of money to store just one gigabyte of system files which rarely get accessed once the system is running!
 
As to the proposed build. I know everyone loves the 1068E controller, but if you're starting from scratch I'd take a look at the SAS2008 controller as a possible alternative, which bumps you from SAS (3Gbps) to SAS II (6Gbps). The major issues is there isn't currently support in FreeNAS, but most everything else supports it. You can find them on some of the SuperMicro boards like the X8SI6-F which you could pair with a X3440 or even a X8DT6-F which would give you the ability to drop in a second processor and more memory down the road.

Obviously 3Gbps is going to be enough for your normal drive channels, but if you later add some new SATA III drives, really fast SSDs, or use some expanders it may make a difference.
 
You must log in or register to reply here.
Back
Top