Supermicro 847BE1C Build - Need Help

[BecauseScience]

I'm migrating my NAS from a Rosewill RSV-L4500 to a Supermicro 847BE1C chassis. This is the one I will use:

Supermicro CSE-847BE1C-R1K28LPB

I will reuse the the cpu, memory, SAS HBA, and disks from the current system.
Motherboard: Supermicro X10SLH-F
HBA: IBM M1015

Requirements
R1) Connect the M1015 to both front and rear backplanes.
R2) Install two 2.5" drives inside the chassis using the internal drive bay(s).
R3) Add the optional dual 2.5" disk hotswap cages on the rear of the chassis (immediately to the right of the power supplies).
R4) Have "enclosure features" on the backplanes working so that I can identify drives via LED.
R5) All cabling up to SAS3 standards. I plan to upgrade to an SAS3 HBA at some point.
R6) All fans that are PWM should be controlled by the motherboard in variable speed mode.

Plan
Connect two ports of the HBA to the front backplane using two SFF-8087 to SAS HD cables.
Connect the remaining two ports from the front backplane to the rear backplane using two SAS HD to SAS HD cables.
M1015 -> front backplane -> rear backplane

Open Questions
Q1) Is my plan to connect the HBA and backplanes the right way to do it?
Q2) What length SAS cables do I need?
Q2) Do I need Supermicro MCP-220-84701-0N to mount two 2.5" drives in the internal bay?
Q3) Is Supermicro MCP-220-84701-0N the correct accessory to add the rear 2.5" hotswap bays?
Q4) What do I have to do to enable enclosure management features, specifically the drive identification LED's? Is it possible with my HBA? Do I need SAS cables with sideband?
Q5) Do I need to buy any additional cabling/splitters to connect all of the chassis fans to my motherboard?
Q6) Do I need any miscellaneous cables for power to motherboard, backplanes, 2.5" rear hotswap, etc. Any cables that I missed anywhere?

It would be great if I could get exact recommendations (with part numbers) for the SAS cables.

 

[izx]

Plan
Connect two ports of the HBA to the front backplane using two SFF-8087 to SAS HD cables.
Connect the remaining two ports from the front backplane to the rear backplane using two SAS HD to SAS HD cables.
M1015 -> front backplane -> rear backplane

The -E1C chassis you're purchasing has only TWO SAS ports on each backplane. You can cascade the rear backplane to the front using one port, but you'll get better performance by simply connecting the M1015 directly to each backplane. If you have multiple HBAs, both backplanes do support "dual-linking" for double the bandwidth (i.e. connect each backplane to a single HBA using both ports).

Q2) What length SAS cables do I need?

The 80cm cables (CBL-SAST-0531) are recommended and should be sufficient.

Q2) Do I need Supermicro MCP-220-84701-0N to mount two 2.5" drives in the internal bay?
Q3) Is Supermicro MCP-220-84701-0N the correct accessory to add the rear 2.5" hotswap bays?

This chassis includes the 2x2.5 rear hotswap drive cage & trays (that go adjacent to the power supplies).

I don't see any reference to any "internal" bays (I know some of the 846 cases supported slinging a 3.5" drive under the power supplies, etc.) If you mean using 2.5 drives in some of the 36 regular bays, the accessory you need is tray MCP-220-00043-0N.

Q4) What do I have to do to enable enclosure management features, specifically the drive identification LED's? Is it possible with my HBA? Do I need SAS cables with sideband?

I know that the M1015, flashed with LSI firmware, will communicate via SES-2 with the backplanes with no additional modifications or special cables. You don't need special sideband cables with these backplanes.

Q5) Do I need to buy any additional cabling/splitters to connect all of the chassis fans to my motherboard?

There are 7 fans total, 7k RPM screamers. They come with decent length PWM cables, but you may want extensions. They are rated at 0.6A each, so make sure your motherboard headers support that much current. If you need to, use the splitters that have a separate molex for power and a PWM connector only for speed control.

Q6) Do I need any miscellaneous cables for power to motherboard, backplanes, 2.5" rear hotswap, etc. Any cables that I missed anywhere?

Check Appendix A in the manual.

 

[BecauseScience]

The chassis arrived yesterday. It sure looks like there are four SFF-8643 ports on both backplanes. Maybe my definition of a "port" is wrong.

This is a picture of the ports on my rear expander. The front one has the same ports. I didn't take a picture of it because I would have had to remove it from the chassis to do so.

These are the cable I bought:

LSI LSI00405 1.0 meter internal cable SFF8643 to SFF8643 (mini SAS HD to mini SAS HD)--Avago Technologies

LSI LSI00402 1 meter internal cable SFF8643 to SFF8087 (mini SAS HD to mini SAS)--Avago Technologies

It sure looks like I could plug four of those SFF8643 connectors into each backplane. Am I missing something?

I plan on posting a build thread once this is all together but here's a quick summary of the other questions I had:

The rear 2.5" hotswap was not included with the chassis. I had to buy it separately.

There are no internal bays. I had seen pictures of some 847 chassis with internal bays below the motherboard tray, between the rear expander and fan wall. My chassis has empty space with a bunch of metal standoffs down there but they don't look like they're for drive cages.

The advice about powered fan splitters was spot on. Supermicro motherboards have a strange fan header setup. There are typically a bunch of fan headers labeled FAN1, FAN2, FAN3 ... and then one header labeled FANA. The ones with numbers all run at the same speed and are controlled by CPU temperate. The FANA header is controlled by system temperature. Much thanks to this thread over at freenas.org for the insight.

Usually, systems like this use a passive heatsink on the CPU and rely on chassis fans to blow the air that cools the CPU heatsink. In that case, you would want the system fans on FAN1, FAN2, ...

I'm using a desktop style fan/heatsink combo so I'll put that on FAN1 to be controlled by CPU temperature.

I bought a powered PWM fan splitter to control the seven chassis fans from one motherboard header. I'll plug that on the motherboard FANA header. The chassis fans should (hopefully) be controlled by system temperature.

 

[izx]

The chassis arrived yesterday. It sure looks like there are four SFF-8643 ports on both backplanes. Maybe my definition of a "port" is wrong.

No, you're right -- there are four ports on there. Good to see Supermicro eliminated the ambiguous "three-port mixed I/O" structure in the new SAS3 backplanes, and that the shorter backplane has four ports too.

The first two ports (PRI_J1 & PRI_J2) are designated a input ports. The second two (PRI_J3 & PRI_J4) are designated as output ports (for connecting that backplane to another backplane). As BlueFox pointed out below, the designation doesn't matter -- the SAS expander is like a switch and will auto-determine what a port is connected to.

The reason two input ports are provided is so that a user with only one backplane can connect both HBA ports to the inputs, providing a "dual-link" at 2x48 = 96 Gbps.

In your case, I think the simplest and fastest solution is the most direct:

M1015 Port 1 ---> PRI_J1 Front Backplane
M1015 Port 2 ---> PRI_J1 Rear Backplane

Because the M1015 is SAS2, that means you'll get a dedicated 24GBps link to each backplane. Unless you're going with multiple SSDs in the front (or rear), there's no point in dual-linking one and cascading it to the other.

 

[Blue Fox]

The first two ports (PRI_J1 & PRI_J2) are ALWAYS input ports. The second two (PRI_J3 & PRI_J4) are ALWAYS output ports (for connecting that backplane to another backplane).

It actually doesn't matter what ports one uses. There are no designated "in" or "out" ports when it comes to SAS expanders.

 

[izx]

It actually doesn't matter what ports one uses. There are no designated "in" or "out" ports when it comes to SAS expanders.

Thanks! I had thought so but was wary of using the the third port on my 846-SAS2-EL1 for any input. Will connecting to any two ports enable dual-linking (works with the first two ports for now)? If you had a dual SAS-chip controller like the LSI -16e series cards with 8088->8087 adapters, could youpossibly triple or even quad-link to the backplanes (if the standard even supports that)?

 

[Blue Fox]

Yes, any port should work. In theory, extra links would be supported without issue either. A single link is already four lanes and I don't think there's a limit. Problem is, all the controllers I've ever encountered support 8 lanes from the ASIC and anything past that is through SAS expanders (which often wind up on the card itself). Are the new crow of LSI cards any different in that regard?

 

[izx]

A single link is already four lanes and I don't think there's a limit. Problem is, all the controllers I've ever encountered support 8 lanes from the ASIC and anything past that is through SAS expanders (which often wind up on the card itself). Are the new crow of LSI cards any different in that regard?

LSI has SAS2 HBAs with 16 lanes using either two SAS2008 or one SAS2116 chip. They also have SAS3 HBAs with 16 lanes using two SAS3008 chips. Extremely recently--it looks like--they've come up with SAS3 16 & 24 lane HBA chips (SAS3216, SAS3224) and the associated 9305-xx HBAs (pdf)

Their SAS2/3 RAID chips/controllers still appear to be limited to 8 lanes, with an onboard expander for more.

 

[Blue Fox]

The ones with two chips definitely wouldn't work.

 

[BecauseScience]

I moved the guts of my old nas into the sc847. I connected two ports of my m1015 to the front backplane, then two cables from front to back backplane. I've tried disks in both backplanes and they're working great. Drive identification lights work too. I have no idea which backplane ports I used for what. I couldn't see the markings and was too lazy to pull the backplanes.

PSU fan noise is zero. I've yet to hear a peep from them. I'm not even sure they're running. The psu's in this chassis are the PWS-1K28P-SQ (1kw platinum).

My next project is to do something about the chassis fan noise. I wouldn't say they're loud but they have a very annoying sound profile even at low rpms. I have another supermicro fan on order as a test. It's not listed as being compatible with the sc847 but from pictures it looks like it will fit the existing fan hotswap brackets in the chassis. It's from one of their SQ "super quiet" chassis so I'm hopeful that it'll sound ok.

I ended up with the chassis fans on the FAN2 header. That syncs them to the pwm speed of the cpu fan. I tried the FANA header but it seems to have a faster minimum speed than the cpu thermal zone (FAN1, FAN2, ...) The downside is that every time I run something the chassis fans spin up higher. I use this box for more than just a nas so that's a problem. I hope to move the chassis fans back to FANA once I've replaced the current fans.

I'm also running a low profile GTX750TI in this box. It's working great too.

 

[XxDaRkReAp3rxXdOtCoM]

Hell yeah, sick build man. I moved my server from a Norco 4020 to a SM CSE846 and it was miles better. Dual PSUs, expander backplane, kickass Nidec fans. How you liking a real chassis now?

 

[izx]

My next project is to do something about the chassis fan noise. I wouldn't say they're loud but they have a very annoying sound profile even at low rpms. I have another supermicro fan on order as a test. It's not listed as being compatible with the sc847 but from pictures it looks like it will fit the existing fan hotswap brackets in the chassis. It's from one of their SQ "super quiet" chassis so I'm hopeful that it'll sound ok.

Very good to hear your 847 build is working great!

Could you do me a favor please? Pull out one of the fanwall fans and post both the Supermicro part number (FAN-NNNN-L4) on the sticker, and the actual fan manufacturer/part number/current rating on the fan hub? Thanks!

 

[BecauseScience]

Thanks. I'm loving the Supermicro even with the fan noise.

I suffered with a crappy Rosewill chassis for years. It didn't have hotswap. Replacing a disk meant disassembling 50% of the rig. I lost a bunch of 3tb Seagates over a period of two months earlier this year. It was hell. I was out of bays anyway but that experience underscored the need for a Supermicro.

BTW: The whole "raid rebuild can kill disks" thing is real. I lost one of my previously good 3tb Seagates during a rebuild. Nothing like tearing the crappy Rosewill chassis apart twice in a day.

 

[BecauseScience]

Very good to hear your 847 build is working great!

Could you do me a favor please? Pull out one of the fanwall fans and post both the Supermicro part number (FAN-NNNN-L4) on the sticker, and the actual fan manufacturer/part number/current rating on the fan hub? Thanks!

FAN-0126L4
Nidec UltraFlo
V80E12BHA5-57
0.6A @ 12VDC

There are seven of them in the chassis...SEVEN!

 

[BecauseScience]

I forgot to mention my first chassis showed up with serious damage. The box was fine but when I opened it I found the motherboard tray was bent and the I/O area as pushed in at least an inch. The way it was packed, there's no way it happened during shipment. The I/O area that was crushed in is recessed and there was nothing in the box for it to hit.

I contacted my vendor and Supermicro OVERNIGHTED a replacement. The box for these things is huge and they have to weigh 80-100lbs. I can only imagine the freight bill...

Thumbs up to Supermicro for amazing rma service.

 

[BecauseScience]

I've finally managed to quiet the fan noise of this chassis to an acceptable level.

My first try was to order a Supermicro FAN-0104L4. The fan is quiet but it doesn't fit the fan brackets on the 847BE1C.

I searched for a quiet 80x38mm fan to replace the Nidec UltraFlo's that came in my 847BE1C. No luck.

I removed one of the fans from the chassis and attached it to an Arduino to play around with low speed PWM control. I found that Nidec UltraFlo's can run reliably at very low rpms. Not all fans can run at such a low PWM duty cycle. The sweet spot for me is 10% duty cycle on the PWM line. One "notch" up to 12% and the fan takes on the annoying sound profile I mentioned in a previous post.

I built a small board with an AVR microcontroller, two 7407 open collector hex buffer chips, and eight individually driven 4 pin fan headers. I programmed the AVR to output a PWM signal at 10% duty cycle and installed the board into the bottom of my 847BE1C in the empty space in front of the rear backplane. Now my 847BE1C is as quiet as my old Rosewill with 120mm fans.

I would have liked to buy an off the shelf PWM fan controller (Corsair Link, etc) but I couldn't find one that fits the bill:

• can drive fans down to 10%
• Linux support or a "set and forget" solution
• support for seven fans