HAF XB Server - Dual Xeon Sandy Bridge-EP LGA 2011

The need is for a small form-factor server/workstation that's easy to service where space is limited and a rack is not available. The platform is primarily intended for virtual machine sandbox software development shared by a few developers. There will also be occasional modeling, effects and video editing, with background video encoding/rendering work.


SOFTWARE

Windows 7 Professional 64-bit is the boot OS running Adobe's entire Creative Suite CS6 for post-production work, LightWave for 3D modeling and ZBrush for sculpting. Oracle VirtualBox will manage five sandbox CentOS UNIX virtual machines to continually run the following server software in separate silos:

1. Web Service VM: Tomcat server running several Spring RESTful web services
2. Web Application VM: Tomcat and Apache web servers running several applications
3. Cache/Queue VM: MongoDB server with data cache and queued collections
4. Database VM: Full Oracle database server and Fusion stack
5. Search VM: Tomcat server running full Solr search engine stack


HARDWARE

The build centers around a standard ATX motherboard with Sandy Bridge-EP dual Xeon LGA 2011 sockets. The open-bench inspired Cooler Master HAF XB case will be used in an air-cooled configuration with several Noctua fans. Highlights include two Xeon E5-2670 processors, 64 GB ECC registered quad-channel server memory, and a Quadro K5000 engineering graphics adapter.

Cooler Master HAF XB ATX Computer Case
ASUS Z9PA-D8 dual LGA 2011 ATX mainboard with PIKE 2008 daughter RAID adapter
KingWin LZP-1000, 80 PLUS Platinum Modular Power Supply
2 x Intel Xeon E5-2670 8-Core 2.6GHz Sandy Bridge-EP Processor, 3.3GHz Turbo Boost, 20MB L3 Cache
64GB 8 x Samsung 8GB DDR3 1600 MHz ECC Registered Server Memory M393B1K70DH0-CK0
nVidia Quadro K5000 Graphics Card 1536 CUDA cores 4GB GDDR5 Memory VCQK5000-PBN
2 x Sedna PCIE 4-Port USB 3.0 Adapter (2 x External, 2 x Internal)
ASUS XONAR_EONE Digital-to-Anal​og Converter (External USB sound)
Boot Array: 2 x Samsung 840 Series 250GB Solid State Drive (SSD) in RAID-0 (500GB)
1.5TB VM Array A: 2 x Seagate Momentus XT 750GB SATA 6.0Gbs Solid State Hybrid in RAID-0
1.5TB VM Array B: 2 x Seagate Momentus XT 750GB SATA 6.0Gbs Solid State Hybrid in RAID-0
LG Blu-Ray Burner SATA 14X BD-R 2X BD-RE 16X DVD+R 5X DVD-RAM 12X BD-ROM 4MB Cache
AFT PRO-57U All-in-one USB 3.0 5.25" Media Card Reader
2 x Noctua NF-A14 ULN 140x140x25mm Fan, 800/650 RPM, SSO2 Bearing (front fans)
Noctua NF-P12 PWM 120mm SSO2 Bearing (upper-rear fan)
2 x Noctua NF-R8 80mm Case Fan (lower-rear fans)
BitFenix Spectre 200mm Case Fan (top fan)
2 x Artic Freezer i30 CPU Cooler with four direct-contact heat pipes
2 x Noctua NF-F12 PWM 120mm Focused Flow Fan, 1500/1200 RPM, SSO2 Bearing (CPU fans)


DESIGN

This build may disappoint some, because I am rather plain and conservative when it comes to computer customization. I am a true believer in the KISS principle while holding function over form in almost every instance.

Therefore, there will be no:

• Exotic liquid cooling
• Radical case customization
• Anal retentive color coordination
• Fancy cable sleeves
• Garish lighting effects

I actually prefer a stock build. For example, I do relish the ugly Noctua fans with their odd brown color scheme, all the while recognizing that cooling performance will be top notch. In the end, the only thing I want to see is a stable platform, good operating temperatures, and a blinking motherboard heart beat LED!


CONFIGURATION

Somehow, the ASUS Z9PA-D8 squeezes two LGA 2011 sockets, eight DDR3 DIMM memory slots and five PCIE expansion slots onto a standard ATX form factor motherboard! The server will pack a heavily threaded punch with two Xeon E5-2670 processors, 16 cores, 32 threads and 64GB of ECC DDR3 server memory. To cool both volcanoes down, each processor will be fitted with an Artic Freezer i30 CPU heat sink mounted in a push/pull configuration; the stock fans will be replaced by superior Noctua NF-F12 PWM 120mm Focused Flow fans. The ASUS server motherboard does not offer enough integrated USB3 ports, so two Sedna 4-Port USB3 PCIE adapters (4 x external ports, 2 x 20-pin internal jacks) provide plenty of USB3 connectivity both inside and outside the HAF XB case.


In front, for a slight increase in CFM along with a notable decrease in noise, both stock 120mm Cooler Master fans will be replaced by two Noctua NF-A14 ULN 140mm fans. For media input, a fast LG Blu-Ray burner and AFT all-in-one USB 3 reader will be installed in the front 5.25" bays. Both 3.5" hot-swap bays will contain Samsung 840 Series 250GB solid state drives in RAID-0 (500GB). Windows 7 Professional 64-bit will boot from this Samsung RAID-0 array, with VirtualBox, Adobe Creative Suite CS6, and other related video/modeling applications, including large scratch and temporary file areas. Internally, encoding space and all five UNIX virtual machines will reside in two 1.5TB RAID-0 arrays, with the image files periodically backed up to external network storage. Each array set consists of two Seagate Momentus XT 750GB SATA 6.0Gbs Solid State Hybrid drives.


In back, one Noctua NF-P12 120mm PWM fan will be installed in the upper corner position. Two Noctua NF-R8 80mm fans will be placed in the lower corner location near the internal drive cage. Network connectivity is provided by ASUS via two on-board Gigabit Ethernet RJ45 ports. Two integrated USB3 ports are located in back, along with four external USB3 ports from two Sedna PCIE adapters. Power is provided by a very quiet KingWin 1000W power supply.


On top, a thin BitFenix Spectre 200mm case fan will be fitted to exhaust heat upward.

Sub'd! I've always wanted to see someone stuff 10 pounds of potatoes into a 3 lb bag!

Most of the parts have been delivered, although four Seagate Momentum 750GB hybrid drives and two PCIE USB 3 adapters are still in the delivery pipeline. The build can proceed, however, since those missing parts do not prevent the system from bootstrapping.

After a rather extended delivery process, the Seagate hardware has finally arrived. It will be fun stuffing the entire internal HAF XB drive cage with 3TB of relatively quick RAID-0 storage. However, the two USB3 adapters must be on a slow boat from China (literally), since they are both sitting in San Francisco still, obviously mired in a US Postal twilight zone...


A number of factors led to a rather "momentous" decision, and the numbers provided by this StorageReview.com analysis swayed me away from my habit of relying on traditional WD RE4 disks for data RAID sets:

http://www.storagereview.com/seagate_momentus_xt_750gb_review

1. Cost Effective Storage

Pricing has dropped for the Momentus which is available now for $120 with 750GB of capacity, providing a huge cost-per-gigabyte advantage over SSD hardware.

2. Higher Capacity

The latest Momentus drive offers 50 percent more capacity over its prior version, 750GB, and 50 percent more storage than most SSDs today-- unless your budget allows for $3K to purchase a 1GB SSD.

3. Good Performance

Scaling two Seagate Momentus 750GB XTs in RAID-0 just about doubles its single-drive performance, delivering nearly 240 MB/s of sequential read throughput. This matches the sequential performance I see using WD RE4 in RAID-0, although neither drive is anywhere near the 980 MB/s capability of two Samsung SSDs in RAID-0.

4. Low Power

Power consumption is lower than most SSDs and traditional disk drives, peaking at 3.7 watts during sequential reads, compared to 9+ watts for the WD RE4 2TB drive and 4.3 watts for the Samsung 840 SSD.

5. Small Size

The Momentus is a 2.5" drive, an important factor in smaller cases like the Cooler Master HAF XB, which only ships with an internal 2.5" drive cage, thereby eliminating all larger 3.5" drives. Using four Momentus XT disks maximizes limited internal capacity, providing 3TB of cage storage.

6. Nice Warranty

Seagate is offering a 5-year warranty, something rather rare in the SSD marketplace. Another advantage of the hybrid design is whenever the high-density 8GB flash chip eventually does decide to fail, the Momentus remains operational as a traditional 7K RPM hard drive, unlike SSD hardware that degrades and ultimately fails altogether.

The HAF XB case has been disassembled, including the removal of both stock fans in front; all five Noctua case fans were installed thereafter. The motherboard tray was removed next and the SuperMicro mainboard was mounted. Two of eight motherboard mount points did not align with available, pre-drilled holes on the Cooler Master tray, so a trusty Dremel was deployed to drill out both missing holes. Installing the Xeon processors was a snap thanks to a very clean LGA 2011 socket design.

The "Artic Squeeze Play" worked according to plan, with both heat sinks in perfect back-to-back alignment. Clearances are good between the cooler and the top of each server DRAM module. Stock Artic Freezer fans were replaced with Noctua Focused Flow hardware, offering better cooling performance and higher top-end CFM numbers at an equivalent noise level. I am very eager to see how the push-pull configuration performs under load.

The Samsung 840 SSDs were installed in the front 3.5" bays, ready to be configured in a RAID-0 boot configuration. The LG BluRay burner and AFT All-in-one media reader were also installed in the front 5.25" bays, completing all front-side setup work.

Looks like a cool build. I love the unique form-factor! I haven't ever actually seen that case before. I like it. Now I want to do a build in that case, lol.

Love it when a plan comes together. Those heat sinks look sweet mounted back to back.

64GB of replacement DRAM arrived today: eight sticks of Samsung 8GB DDR3 1600 MHz ECC Registered Server Memory (M393B1K70DH0-CK0). The two Kingston 16GB DRAM modules were exchanged for this Samsung memory, since the Kingston sticks were not on the SuperMicro compatibility list. Also, further investigation indicated the Xeon 2600 series memory controller operates at maximum bandwidth only when four channels are populated. Moreover, moving from 32GB to 64GB allows each UNIX VM more room for larger JVM heaps, and should allow all four VMs to remain alive and responsive even while video tasks are running as well.

Wow, that's alot of computer in such a tiny box! What are you shooting with?

Cool looking build, neat to see what all people are using this case for, it's definitely not as tiny as I thought it was, saw it at MicroCenter the other day and thought "damn that won't even fit on my desk!".

Looks nice but good gawd the back ground blur is annoying as fuck!

Do i dare ask what was the final price tag on this rig...

greywarden said:

Wow, that's alot of computer in such a tiny box! What are you shooting with?

Click to expand...

Space is limited in my little den where I code the day away. My main workstation is in a Cosmos II case and that behemoth takes all the floor space beside my desk. This HAF XB case will rest atop an Armoire-- so very thankful for a 10-foot ceiling!

The camera is a Canon EOS 5D and the lens is an EF 50mm 1:1.2 - hence the short focal range and background blur which apparently greatly annoys some viewers...

MrGuvernment said:

Do i dare ask what was the final price tag on this rig...

Click to expand...

Just over $3K...

The slowest boat from China has finally arrived with two USB v3 PCIE adapters that provide a total of 4 external ports and two 20-pin internal jacks to satisfy all on-board USB 3 requirements. Unfortunately, the SuperMicro mainboard only provides USB v2 ports.

One of the finest aspects of the HAF XB is its flexibility and total accessibility, a direct byproduct of its test-bench design origin. Need to remove the top or both sides? No problem. Need to remove the motherboard? No problem. On the other hand, after gaining quick access to the guts a nasty reminder just how cramped the quarters are becomes the main impression. Space it tight, making cable management even tougher than normal.

In order to cleanly install all four Seagate hybrid drives, the Enermax power supply was nudged halfway out for better clearance and easier access behind the 2.5" drive cage:


Thankfully, cable management in the lower compartment is complete at this point; hopefully, both 80mm Noctua fans will perform well and provide sufficient airflow in the bottom chamber:

I get the whole big aperture thing, but there is a time and a place. Some of these shots would probably be a little more useful if you stopped that sucker down.

Is that a 3.5" hot swap bay? I see the 2.5" drive bays, then what looks like an optical drive bay, and another compartment, what is that?

Just built one of these for my gf, the case is an excellent design.
Keep up the good work, it's looking good so far!

extide said:

I get the whole big aperture thing, but there is a time and a place. Some of these shots would probably be a little more useful if you stopped that sucker down.

Is that a 3.5" hot swap bay? I see the 2.5" drive bays, then what looks like an optical drive bay, and another compartment, what is that?

Click to expand...

Overall, the HAF XB provides two 3.5" external bays, two 5.25" external bays and an internal cage with four 2.5" drive mounts.

The front panel features two 3.5" hot-swap bays on the right when facing the case. The other two bays are standard 5.25" bays on the left. I placed the RAID-0 SSD boot drives in the hot-swap bays for easier access. The two larger bays contain an LG BluRay burner and a USB v3 multi-function card reader. The internal 2.5" drive cage contains four Seagate hybrid drives.

SonataSys said:

Overall, the HAF XB provides two 3.5" external bays, two 5.25" external bays and an internal cage with four 2.5" drive mounts.

The front panel features two 3.5" hot-swap bays on the right when facing the case. The other two bays are standard 5.25" bays on the left. I placed the RAID-0 SSD boot drives in the hot-swap bays for easier access. The two larger bays contain an LG BluRay burner and a USB v3 multi-function card reader. The internal 2.5" drive cage contains four Seagate hybrid drives.

Click to expand...

Ok, so it is the 3.5" bays then. Cool!

SonataSys said:

The slowest boat from China has finally arrived with two USB v3 PCIE adapters that provide a total of 4 external ports and two 20-pin internal jacks to satisfy all on-board USB 3 requirements. Unfortunately, the SuperMicro mainboard only provides USB v2 ports.

Click to expand...

Do you have a link to that card? Can't seem to find any with internal USB 3 headers.

You can find the adapter here: SEDNA - PCI Express USB 3.0 4 Port Adapter (2 external + 2 port internal with 20 Pin connector)

SonataSys said:

You can find the adapter here: SEDNA - PCI Express USB 3.0 4 Port Adapter (2 external + 2 port internal with 20 Pin connector)

Click to expand...

Nice little card for someone in your situation..I like the build choices so far, but I have to ask..Did you consider an AMD 4P G34 system? I know the case would be larger, but they are offer some [b[SERIOUS[/b] bang for the buck..4x 12 cores would be nice, and I would imagine give those Xeons a spanking depending on which model you chose (clock speed wise)..

ccityinstaller said:

Did you consider an AMD 4P G34 system? I know the case would be larger, but they are offer some [b[SERIOUS[/b] bang for the buck..4x 12 cores would be nice, and I would imagine give those Xeons a spanking depending on which model you chose (clock speed wise)..

Click to expand...

I really didn't consider going with a 4P system for two reasons:

1. Since this is just a local development sandbox server (used by a few developers, including myself) and not a production server with heavy user loading, the advantages and need for greater scaling aren't present.

2. Space in our room is very limited, hence the need for a small case and standard ATX form-factor-- which basically rules out larger 4P motherboards.

Although clearances between both Arctic Freezers and the memory slots are good, there is not enough wiggle room to replace every stick of DDR3 memory with the CPU heat-pipes mounted, especially the two slots nearest each CPU socket.

Unfortunately, both coolers had to be removed in order to install all eight sticks of 8GB Samsung memory. This also required removal of the prior Arctic Silver 5 thermal compound, cleaning all the surfaces and repeating the mounting process after installing all 64GB of server memory.

You have ton of horse power sitting at your fingertips. It's cool that you put it to good use.

raptorxrx said:

You have ton of horse power sitting at your fingertips. It's cool that you put it to good use.

Click to expand...

It is a good thing the HAF XB is basically a test bench, because requirements and hardware keep changing even before the server has been deployed!

Due to an increase in expected scope/workload, both Xeon E5-2620 processors have been returned for an even bigger punch: two Xeon E5-2670 2.6GHz processors with 3.3GHz Turbo Boost and 20MB L3 Cache.

The number of developers is expected to increase from two to four later this year, a fifth VM needs to be spun up to support a CPU-intensive search engine (Solr), and the server has also been nominated for initial load-testing and baseline benchmark duty. An increase in both thread count and clock rate was deemed appropriate to maintain good performance.

Compared to the E5-2620, the E5-2670 provides eight cores instead of six. The 2670 also clocks at 2.6GHz instead of 2GHz, with a temporary boost up to 3.3GHz instead of 2.5GHz. The 2670 also provides 5GB of additional L3 cache, 20MB instead of 15MB. However, the 2670 does consume more power, 115W instead of 95W.

The new requirements and resulting changes just won't stop. The initial plan was to offload just the background encoding and image processing to a server. However, the need to also support modeling, effects and video editing work has emerged. Therefore, the little Radeon HD 6450 passive video card is not going to cut it; some sort of Quadro/Tesla adapter will be needed, although the actual video card choice has not been made-- any suggestions?

The main modeling/effects/editing applications are:

• Adobe Premiere Pro CS6
• Adobe After Effects CS6
• Adobe Photoshop CS6
• LightWave 3D

Video card suggestions offered today by various developers that fit within the budgeted $2,500 usually fell into two camps, one group advocating more expensive Quadro hardware and the other camp suggesting the use of less costly desktop gaming hardware:

• GTX 680
• GTX 690
• GeForce "Titan"
• Quadro 4000
• Quadro 4000 + Tesla C2075
• Quadro K5000

Regardless, the original SuperMicro ATX server motherboard does not provide a PCI-E x16 Gen3 X16 slot, so it will be exchanged this week for an ASUS Z9PA-D8 Dual LGA 2011 mainboard that does support high-end workstation graphics. The new ASUS has the same dimensions, a standard 12" x 9.6" ATX form factor. However, the ASUS board does not provide enough SATA ports without adding a daughter RAID adapter, so a PIKE 2008 RAID interface will be used to enable 8 additional SATA/SAS ports with support for RAID 0/1/10 configurations.

The upgraded Xeon processors arrived today, two E5-2670 CPUs that produce 32 rather happy threads playing at 2.6GHz with boosts up to 3.3GHz for short periods of added intensity. The build remains on hold until the replacement ASUS motherboard and RAID adapter are both delivered. The only other open item is the graphics adapter. A decision doesn't appear to be near at hand, especially with some wanting to wait for NVidia's "Titan" release next month. I'm staying out of the debate, since it's not my area of expertise. The build can proceed using the original Radeon passive video card until a final decision is made and the actual workstation card shows up.

The ASUS ASUS Z9PA-D8 server motherboard and its daughter Pike RAID adapter arrived today:

That is the question: Will this ASUS motherboard operate properly with two Xeon processors installed? Thus far, it has refused to POST with both processors installed. The on-board diagnostic LED always displays "6A". Either Xeon works great in the first socket and the board POSTs just fine. Both sockets look good with no bent pins. ASUS technical support suggests one more test before doing an RMA: Try a newer "platinum" power supply with SSI v2.9 EPS 12V support. Therefore, the build is on hold once again until a replacement power supply arrives.

The current Enermax unit delivers EPS power but the documentation doesn't mention SSI or any version number. After a few days of analysis, an order was placed for the KINGWIN LZP-1000. It meets all of the power requirements and standards, has garnered excellent reviews, offers modular cabling, and is very reasonably priced.

The Kingwin LZP-1000 arrived yesterday. After a brief moment of terror, the unit is actually working very well. At first, I thought the power supply was DOA because the unit was totally silent and the PSU fan was not spinning when I first turned it on, unlike the Enermax unit. However, the motherboard and case fans all powered up and BIOS posted cleanly with a single Xeon in socket 1.

Unfortunately, the issue with socket 2 remains; this new supply with SSI EPS 12v power didn't resolve a dead socket 2, so an RMA has been issued for the ASUS board.


Reading through the manual indicates the unit operates passively until the 140mm fan is needed based on internal temperature levels. Besides very quiet operation, other practical benefits include a smaller overall size and ribbon-style cables that are much easier to manage.

Now that you've had a lot of time with the HAF XB, would you use it again?

valve1138 said:

Now that you've had a lot of time with the HAF XB, would you use it again?

Click to expand...

Yes, it's been a savior given the amount of changes taking place. It's the next best thing to an open bench environment. The appreciation level rises as the situation becomes more volatile. Most tasks can take place quickly without even removing the motherboard tray.

The only downsides I see are its ATX restriction and limited internal storage space. I would prefer Cooler Master to have added a few inches in case width or length to accommodate slightly larger mainboards and one or two 3.5" internal drive cages.

interesting build indeed ... sub`d

A decision was made yesterday regarding the graphics card: Quadro K5000. This adapter provides a nice combination of features, including Adobe CS6 compatibility and strong rendering capacity with nVidia's latest Kepler architecture that provides 1,500+ CUDA cores and 4GB of GPU memory for larger 3D models.

The order is in and delivery should take place before the weekend arrives...

SonataSys said:

A decision was made yesterday regarding the graphics card: Quadro K5000. This adapter provides a nice combination of features, including Adobe CS6 compatibility and strong rendering capacity with nVidia's latest Kepler architecture that provides 1,500+ CUDA cores and 4GB of GPU memory for larger 3D models.

The order is in and delivery should take place before the weekend arrives...

Click to expand...

Drool.

The Quadro K5000 landed today. This sucker packs 1,500+ Kepler cores and 4GB of GPU memory to enable work with larger models in a responsive viewport. Ray tracing in After Effects should get a nice boost, and editing with Premiere Pro will enjoy some serious hardware assist...

Epic

The RMA process for the replacement ASUS motherboard went smoothly, for which I am most thankful. I'm very pleased to report that both CPUs posted cleanly, and the BIOS recognized all 64-GB of ECC memory as well.

Needless to say, every precaution was followed to ensure things went as smoothly as possible:


I should have taken a shot of the thermal (Artic Silver 5) compound application, but I was totally focused on mounting the cooler and forgot to take a picture. I applied three lines of thermal paste, centered proportionally and aligned with the direction of the heat pipes.


Nylon standoffs were installed to avoid any possibility of a short with the motherboard tray. In addition, red insulating fiber washers were used when screwing the board down. This should prevent short circuit issues.


The PIKE RAID controller, an optional daughter card that really shouldn't be optional-- sort of a hidden cost if a build requires more than 4 SATA ports.


The KingWin PSU was brought into the process early on. First, the PSU itself was tested. Next, the bare motherboard was tested. Thereafter, the first CPU in socket 0 was tested with 32-GB of ECC memory. Lastly, both CPUs were tested together with 64-GB of memory. Thankfully, everything posted cleanly with no suspicious ASUS Q-Codes, beeps or ominous LED indicators.