Best 24+ Drive Setup

[scuppasteve]

Ok, right now i have an amalgamation of 30+ drives over three servers i will be migrating into one 24bay Norco case. I am looking at Intel E3-1270V2 or V3. The problem lies with how to connect the drives. I plan to use windows and Flexraid, as it has worked well for me. Right now i am looking at needing 4 PCIe slots and i can only find one V2 board that has that. I need one for NIC and 3 for SUPERMICRO AOC-SASLP-MV8 as that is my plan for connecting the drives.

Does anyone have advice for best way to do this on a cost/performance basis. I know there are cheaper options than what i have but i would like to go with the V3 i just cant find a board that works. I know the raid cards arent the best but as i am doing software raid, i dont exactly need that function.

Thanks

 

[omniscence]

There are currently no (IPMI-enabled) Haswell server boards with 4 PCIe slots except the Intel S1200V3RPS/S1200V3RPL and ASUS P9D-M. The last slot on the Intel boards is only x1, however. The ASUS Board has 1 x8 and 3 x4 slots. Why do you need a separate NIC?

 

[scuppasteve]

I want (4) gigabit ports for streaming and transferring files through the house hold, I have 6 PCs that at any given time may be streaming or transferring files off the server. Is the 3 supermicro cards a better option than say one 24 drive Areca, i mean then i would just need 2 pcie ports

 

[omniscence]

Ever heard of ethernet switches?

I'm not sure what type of workload you have, but for media streaming and file transfers in a home environment a single gigabit link is usually sufficient.

 

[Lost-Benji]

Buying a board for nothing more than on-board quad-port NIC is a little silly.

• Norco chassis (SAS Back-planes)
• M1015 (flashed to LSI IT Firmware)
• Chenbro 36-port expander
• Quad-port PCI-E NIC card
• MoBo that has at least two PCI-E slots

Doubt the need for the network speed, especially with FlexRAID or that it is a home server.

 

[scuppasteve]

I already have the NIC cards, and i have them in place now and i use them to move stuff around on the network, it really doesnt matter.

Is there a guide for how to flash the m1015, i had looked at this option before i thought and it was a pain in the butt. If i understand properly, one of the M1015 sas ports will go to the in-port on the Chembro expander and it has 6 internal sas ports for connecting to the norco backplane?

 

[scuppasteve]

Do i even need to flash the m1015 as i will not be using hw raid? secondly can i use one of the external ports on the chenbro 36 port expander to go to input on another chenbro expander? or can i just use both m1015 ports to goto 2 chenbro expanders? will that card really support 48 drives?

 

[Philmatic]

Going with a M1015 + SAS Expander will reduce your SAS PCIe slots requirement from 3 to 2.

Also, you can just get a mobo that already has QUAD Intel NICs, there are a few Haswell based motherboards that offer that.

 

[staticlag]

one m1015 will control up to 256 drives.

You can connect as many expanders as necessary to do this.

 

[Lost-Benji]

I already have the NIC cards, and i have them in place now and i use them to move stuff around on the network, it really doesnt matter.

Is there a guide for how to flash the m1015, i had looked at this option before i thought and it was a pain in the butt. If i understand properly, one of the M1015 sas ports will go to the in-port on the Chembro expander and it has 6 internal sas ports for connecting to the norco backplane?

There are plenty of good and simple guides and it only takes a few minutes.

Do i even need to flash the m1015 as i will not be using hw raid? secondly can i use one of the external ports on the chenbro 36 port expander to go to input on another chenbro expander? or can i just use both m1015 ports to goto 2 chenbro expanders? will that card really support 48 drives?

The idea of flashing to the LSI IT firmware is to allow removal of the BIOS boot ROM (Slows down booting) but also to make sure that it passes the drives through like a proper HBA. There is also a drive count limit of 16 from memory on the stock and will be opened out to 64.
http://www.redbooks.ibm.com/abstracts/tips0740.html

Going with a M1015 + SAS Expander will reduce your SAS PCIe slots requirement from 3 to 2.

.....and it will allow easy expansion later without changes to system.

Also, you can just get a mobo that already has QUAD Intel NICs, there are a few Haswell based motherboards that offer that.

Please read the entire thread as you have missed the boat.

one m1015 will control up to 256 drives.

You can connect as many expanders as necessary to do this.

Might pay to list your source as the advertised support in the past has been 64 drives on LSI IT firmware.

 

[dandragonrage]

Ever heard of ethernet switches?

I'm not sure what type of workload you have, but for media streaming and file transfers in a home environment a single gigabit link is usually sufficient.

For 6 clients? I couldn't disagree more.

 

[Lost-Benji]

For 6 clients? I couldn't disagree more.

WTF?

Please state how much traffic and what type of traffic for HOME use is going to choke GbE?

That's a fark-load of streaming.....

 

[dandragonrage]

WTF?

Please state how much traffic and what type of traffic for HOME use is going to choke GbE?

That's a fark-load of streaming.....

He said file transfers man. Why do all of you come in every time and say "BUT STREAMING IS ONLY LIKE 2MBPS AND WE NEVER DO FILE TRANSFERS." I used my network at my previous place (with more people) for a hell of a lot more than streaming. And with only one 1Gbps connection and without QoS, file transfers did sometimes cause lags in streaming (and also cause each other to go slower). Eventually upgraded to 2x1Gbps bonded, and picked up a cheap switch to go 10Gbps (server only, since each client would then have 1Gbps available) but never got any NICs for it. Moved before I had the chance.

 

[Rectal Prolapse]

As soon as someone starts a file copy over the network, kiss your streaming goodbye! (unless they are very low bitrate)

EDIT: Will QOS help?

 

[dandragonrage]

As soon as someone starts a file copy over the network, kiss your streaming goodbye! (unless they are very low bitrate)

EDIT: Will QOS help?

QoS can help yes, but it can be a pain to get QoS running in your LAN. You'll either need a central router with a network interface per machine that needs QoS or you'd need to set up QoS on each client machine that you want it for.

 

[hotcrandel]

You could hook your supermicro HBA into a SAS expander, and then plug all your drives into that. It would let you get by with a motherboard with a PCIE slot for the HBA, and another for the NIC(s).

You need a manageable and feature rich networking switch to take advantage of anything like LACP/port channeling or QOS properly, so if you don't have that, you can configure NIC's independently without gateways, and have two computers share single IP's of each NIC, partially isolating the traffic.

If you do any switch daisy-chaining this is pretty much pointless, every device must be plugged into the same switch or your traffic will be bottlenecked by having to traverse to the gateway/L3 device for routing.

Keep in mind with FlexRAID your maximum throughput is pretty much the RW of any single drive in your pool. Even with six computers running a single gigabit connection I'd think should be sufficient, unless your house is full of people doing video editing or something.

There are many guides to flashing the IBM M1015 but for flex raid you probably don't need too. I've used this guide many times to success, but you can't (easily) flash on a motherboard with EFI.

Flashing is helpful when you're utilizing certain softwares that don't have drivers for the native IBM card. (VMware ESXi, in particular with VT-D doesn't work reliably with much but it does with flashed M1015 cards). If you're just using windows or linux you can just straight away use the cards.

 

[Lost-Benji]

He said file transfers man. Why do all of you come in every time and say "BUT STREAMING IS ONLY LIKE 2MBPS AND WE NEVER DO FILE TRANSFERS." I used my network at my previous place (with more people) for a hell of a lot more than streaming. And with only one 1Gbps connection and without QoS, file transfers did sometimes cause lags in streaming (and also cause each other to go slower). Eventually upgraded to 2x1Gbps bonded, and picked up a cheap switch to go 10Gbps (server only, since each client would then have 1Gbps available) but never got any NICs for it. Moved before I had the chance.

As soon as someone starts a file copy over the network, kiss your streaming goodbye! (unless they are very low bitrate)

I will address these as one as they are both in same area of limited opinions.
Poorly designed network possible but more likely crappy performance of the RAID setup and system. A single GbE NIC (I am also refering to a half-decent NIC, Intel, RealTek or Broadcom) can handle all the streaming ever wanted. It will also handle "File Transfers" concurrently. What WILL choke is the RAID and drives, it can be banked on. Add into the mix crappy network media players with no buffering, cheap switches that use store-n-forward and/or have limited switching backplane, cascaded switches and messy networks; you now have a shit fight.
The OP has already given away a big hint that headaches are coming and hasn't figured out where the issues lay. FlexRAID is not known for stellar speeds of R/W's or I/O. Single drive performance will suck in an instant when several streams are started and/or a file transfer is dropped into the mix.
If you wish to keep arguing opinions, screenshots of the NIC getting smashed and while at it, go to Perf-Mon and grab the disks I/O graphs (OP has already said windows environment).

As for LACP and bonding, it is simply not needed nor is the requirements for managed switches in this scenario. Windows 8 and server 2012 now run SMB 3.0 and also support NIC teaming with or WITHOUT a need for anything more than a dumb switch. Simply hook all NICs to a switch and let it go. Do alll shares and file transfers to the SMB share name and not an IP. Simple as shitting on neighbours lawn.

You could hook your supermicro HBA into a SAS expander, and then plug all your drives into that. It would let you get by with a motherboard with a PCIE slot for the HBA, and another for the NIC(s).

Old and rather limited card, the M1015 smashes it out of the park.


As for everything that the thread wanted, keep it simple. No need for a expensive board and on-board quad-port NIC. No need for multiple HBA's when an expander allows all and a lot more.

Better options that bloody FlexRAID too, especially with a Win environment. Storage Spaces will see good R/W (stop reading forums with opinions from limited people) and plenty of I/O to do streams and file transfers with multi-GbE networks.

 

[Rectal Prolapse]

Well, my solution would be to not allow file copies to happen while streaming. Should be easy to do in a single household. Then you can use whatever solution you want.