Project: Alters - VMware ESX Server, 56K warning - lots of pics

axan

[H]ard|Gawd
Joined
Nov 5, 2005
Messages
1,935
Project name: Alters - is a name for alternate egos or personalities in people diagnosed with multiple personality disorder, now given that virtualization allows one to run multiple systems with multiple distinct operating systems all one one server I though the name was fitting.

Goal of the project: The goal is to expand my working knowledge of VMware ESX infrastructure as well as to consolidate and virtualize a bunch of existing servers I have in the house such as torrent client box, smoothwall firewall, Active directory etc.

Hardware:

2x Intel Xeon e5405 2.0Ghz Quad cpus
Asus DSBF-DE mobo
4 x Kingston 2x2 4GB memory kit
Adaptec 5805 SAS Raid Controller with battery backup
9 x Hitachi Ultrastar 36GB 15K RPM SAS drives, 8 in Raid 10 and 1 as a cold spare
1 x Samsung Spinpoint F1 750GB Sata drive to hold vm templates, iso images and serve as storage for torrent client.
Pioneer slim slot loading dvd drive + supermicro slim ide to sata adapter
2x Intel Dual Port Server network cards
Supermicro 825TQ 2U Rackmountable case with 8x hot swap bays

Software:


The server will run VMware ESX Foundation edition

VMs will consist of:

Server 2008 for Ms stuff such as active directory etc
Gentoo linux distro for torrent client, game server
Smoothwall 3 to act as a router/firewall

Possibly others to be added as need arises.

Some pictures:

You can see full resolution pics HERE

Overall shot of all the hardware that will be used in this build
PICT0886.JPG
 
Close up of various components:

Hitachi SAS Drives

PICT0887.jpg


PICT0890.jpg


PICT0897.jpg


Pioneer slot loading dvd

PICT0891.jpg

PICT0899.JPG


Supermicro slim ide to sata adapter
PICT0898.JPG


16GB of kingston fb-dimms
PICT0892.JPG


Intel Dual Port Server NICs
PICT0893.JPG


Adaptec Raid Card
PICT0894.JPG

With the battery backup mounted
PICT0900.JPG
 
Interior of the supermicro case

PICT0905.jpg


PICT0906.jpg


PICT0909.JPG


SAS Backplane

PICT0907.JPG


Xeon E5405 with retail active heatsink

PICT0908.JPG
 
Mobo installed:
PICT0910.JPG


CPUs and Ram, also hooked up the power connectors

PICT0911.JPG


PICT0914.JPG


The Single sata drive
PICT0915.JPG


PICT0916.JPG


Supermicro HDD trays, very nice build quality

PICT0917.jpg


Hitachi SAS drives mounted in trays
PICT0918.jpg

PICT0919.jpg
 
Slim DVD

PICT0920.JPG


PICT0921.JPG


Fans used in the case( 3 of those)

PICT0922.JPG


Raid card and network cards installed

PICT0924.JPG


PICT0925.JPG


Not much space to work with but I tried to keep it clean

PICT0926.JPG


PICT0927.JPG


PICT0928.JPG
 
Shot of the back

PICT0929.jpg


Front shot with hdd trays removed:

PICT0930.jpg


Drives installed and system powered on, it's Alive!

PICT0931.jpg


Some overall shots of finished system:

PICT0932.jpg


The cable for front connection (power buttons, leds etc) is too short so I had to route it in a ghetto way

PICT0933.jpg


PICT0934.jpg


PICT0935.jpg


PICT0936.jpg


PICT0937.jpg
 
Yes the case is very nice, the psu is bit loud and I might replace it with something quieter but I'll wait till after I move to the new house and mount the case in the rack.
 
Shiny...looks like it wasn't cheap. I bet you got a good deal on the drives though.
 
Got the drives on eBay, I think after shipping and everything it was like $40 per drive so not bad for 15K SAS drives.
 
That's a lot of horsepower. You should look at hosting it so you have the bandwidth and expanded possibilities ;)
 
That's a lot of horsepower. You should look at hosting it so you have the bandwidth and expanded possibilities ;)

LOL, I bet I know who could find him a great deal on hosting too right? Very nice build too axan.
 
I would like to setup a box at coloc but ockie is not willing to walk on the dark side :p
 
Love this, especially the case. Seems like a lot of horsepower for what you're doing but I suppose it's better to be [H]arder than necessary :)
 
makes all my boxes in my living room for my upcoming WHS build look like a pile of poop :(


nice build man
 
Very nice setup!

I'm looking at the supermicro 825tq as a chassis for a rack mounted SAN/NAS, with the adaptec 5805 RAID card.

Of course I don't need the power of dual Quad processor motherboard so I'm looking at some ATX candidates. (Most EATX motherboards provide dual processor support.)

Do you know if the 825tq provides a mount pattern that accepts an ATX motherboard? It looks like the standoffs can be moved - do all of those holes on the motherboard tray accept standoffs?

And did you have any issues with the lengths of the provided cables?
 
The chassis will let you install atx mobo; however the psu cables are pretty short if your atx plug is not along the top edge of the motherboard you might need an extension. Also the cable for power/reset, status leds is bit too short for my liking. It will reach pretty much anywhere on the mobo but there isn't enough slack to route it neatly. Another thing is that the case was designed for redundant psu and even the single psu models use 1u psus and have an empty bay for 2nd psu. Those 1U psus are pretty loud, it's not an issue if you have a dedicated server room/rack but you don't want to be in the same room with the case when it's on.
 
Imagining all the hot air from the CPUs blown over the RAM and chipset makes me cringe a bit :(

Further a really nice build :) Thanks for making us all feel jealous ;)
 
yes 8 fbdimms next to each other got pretty hot so I added an asus fbdimm cooler, now the ram stays under 60 C which is well under max operating temp for fbdimms.
 
yes 8 fbdimms next to each other got pretty hot so I added an asus fbdimm cooler, now the ram stays under 60 C which is well under max operating temp for fbdimms.

Ah yes, they're also FD-DIMMs... I've heard that they get really warm and will generally overheat if there's not enough airflow over them :eek: To add even more heat to that is a risky business :)

Good job on solving it :) although I must say that 60 C for RAM still sounds insanely warm to me. Then again I'm only use to regular DDR2 :p
 
it's perfectly safe, usual fb-dimm chip tjmax temp spec is between 85-125° C
 
You could turn an egg to charcoal with those temperatures :eek:

But I guess that the buffering serves a purpose :)
 
Noticed you're using a Adaptec 5805 card. We just had an issue with our similarly configured ESX box... a drive developed a bad block, the 5805 locked up, and the ESX server followed a moment later. It did this twice.

I updated the firmware to the 16343 release and it has been stable, despite a few more bad blocks on a dying drive.

Just a heads up.
 
thanx for the heads up lizardking, so far everything is running smooth as butter. I did the adaptec firmware update anyway, I like to keep current :p
 
Quick question about the cables you used from the 5805 to the backplane. The site says that it comes with;
Cables: two internal Mini SAS x4 to (4x) SATA fanout cables w/ sideband (2247000-R), 0.5 meter
Is this what you used?
 
ya i think i used the original cables, can't recall 100% they were mini sas to 4x sata + sgpio
 
Project name: Alters - is a name for alternate egos or personalities in people diagnosed with multiple personality disorder, now given that virtualization allows one to run multiple systems with multiple distinct operating systems all one one server I though the name was fitting.

Goal of the project: The goal is to expand my working knowledge of VMware ESX infrastructure as well as to consolidate and virtualize a bunch of existing servers I have in the house such as torrent client box, smoothwall firewall, Active directory etc.
]

Axan,

Great thread and awesome job with the custom server build... if you were to do this again, what would you do differently? Would you still use same hardware for this project?

Dannye
 
If i had to do it all over I would use same hardware for the most part, I would probably swap the hdds for ssds. I'm very happy with this server
 
Back
Top