40TB Build Log

E

edlin303

n00b
Joined
Apr 6, 2011
Messages
43
I decided to start a new thread based on my ramblings in http://hardforum.com/showthread.php?t=1599279. This will document my build, and any issues I run across. Ironically, after a lot of back and forth on all the parts I found I chose almost the same as this person did last year. http://www.natecarlson.com/2010/05/07/review-supermicros-sc847a-4u-chassis-with-36-drive-bays/

This is just a shell now. I will fill it in over the next couple of weeks as the hardware arrives and I get it all fired up.

Hardware:
  • CASE: SuperMicro 847A-R1400LPB
  • MOBO: Supermicro X8DTH-6F
  • CPU: (2) Intel E5645 (Hex-core)
  • HEATSINKS: (2) SNK-P0038P
  • RAM: 48GB Crucial in 4GB DIMMs (DDR3 PC3-10600 Unbuffered, ECC, DDR3-1333, 1.5V, 512Meg x 72)
  • SAS: (1) On-board LSI 2008, (2) IBM M1015
  • HDDs: (18) Hitachi 5k3000 *likely to increase as I start installing
  • SYSTEM DRIVE(S): Kingston 96GB SSDNow V+100
  • OS: ESXi All-In-One, SE11 host with passthrough of SAS controllers likely
  • NIC: Build-in Intel Dual-Port 1GE



Software:
  • OS: ESXi
  • NAS: TBD, probably SE11 or OI with hardware passthrough of SAS controllers.
  • Clients: The parent ESXi server as well as 2 physical servers will use this as their primary datastore via NFS. Other PCs will use it as CIFS/NFS share and for backups.
  • Other services: iTunes share, remote backup for remote users to back up data to offsite.
 
Last edited:
I'll update with OS info shortly. I am going to do an ESXi all-in-one so I can make some use of the server resources for VM purposes too. If that doesn't pan out, I will probably at least make it my primary DB server and run some other stuff from it so I didn't waste the $ building it to a VM server spec.
 
My Q6600/8G seemed to be getting a little long in the tooth as my primary workstation so I too am building a box off the Supermicro X8DTH-6F motherboard using these parts:

2x XEON [email protected]
6x 4G DDR3 ECC REG DIMM
1x 300GB SAS 15K
2x 1T 7200's 16M
1x 1.5T 7200 32M
1x EVGA GEFORCE 460 GTX 2GB
 
Just posting an update, I finally have all the hardware, and will be building this as I have time. So far I have most of it in the chassis, and so far even though I love the density I am not sure I love the chassis. It is significantly louder than my Dell R710 and 2950, and can easily be heard through 2 sets of walls/doors. The chassis is also a bit awkward to work in. The motherboard area slides back so I can access the rear backplane, but with everything installed it was very difficult to get it to slide forward again. With the motherboard and CPUs in, I couldn't get it to slide in without running into the power cables. I ended up cutting all the zip ties for the motherboard power leads and shoving the individual wires anywhere I could to get them out of the way. I will post pictures later on to better show what I mean. Supermicro could easily have left a bigger notch out of the sliding tray so the power could shove through it and the excess would fit under the tray where the internal drives go.

I love the IPMI on the motherboard, and have no regrets about the money it cost to get a board with it. I have already remotely installed ESX with the virtual CD, and can do everything from a safe distance from all the noise of the chassis.

I picked up a SSDNow 96GB from the Amazon deal, so that should make a suitable OS drive I hope. I am skipping RAID for the ESX/NAS datastore because the ESX install is pretty quick and the NAS I will keep a backup copy of for DR. I figure I can toss any SATA disk in and be back up in hours if I lose that SSD.
 
Good to know that you are getting more done.

I can imagine that it can get cramped with all of that in one case, but you properly don't have to do much in there when its first running. I'm putting mine in my basement with some isolation on top so i hope that will keep then noise in. But i haven't got my motherboard so i haven't checked out the noise yet.

How does that IPMI work?
 
Freak1 said:
Good to know that you are getting more done.

I can imagine that it can get cramped with all of that in one case, but you properly don't have to do much in there when its first running. I'm putting mine in my basement with some isolation on top so i hope that will keep then noise in. But i haven't got my motherboard so i haven't checked out the noise yet.

How does that IPMI work?
Click to expand...

Not sure if you mean technically or how well does it work, so I will answer both.

Technically, I plugged in the IPMI NIC and it DHCPed. Then I just had to open a browser to that IP, log in, and I get a bunch of options.
- I can see sensor readings, which shows temps for a few dozen things including each dimm and cpu, etc.
- I can set up email alerts for things, I assume including chassis intrusion and temp thresholds. I haven't tried yet.
- I have virtual power and reset buttons so I can do a reset, power on, hard power off, and soft power off.
- I can attach virtual media by pointing it to an ISO on an SMB share.
- I can launch the virtual console, where I see everything in a browser applet I would see from the actual console. This gives me full bios access, etc.
- I can update the firmware


If you meant how well does it work, so far it's great. The ability to mount ISOs is important to me when I am building a system with no optical drive. I hate having to rig up a usb or internal one just for the OS install or booting a livecd. The latter is a bit slow if you try it by the way. So far it has been a little flakey with some mouse stuff and keyboard mappings. Trying to hit alt-f1 in ESX doesn't work for me yet, and if the mouse gets out of sync I can't click things properly in a livecd boot. I usually use keyboard to spawn VNC if I am booting a linux cd and then do it that way. All in all, I wouldn't use it as a main remote access method, but if you need to power the server and mess around in the bios, or even just figure out why you can't reach it over the network, it is a huge benefit.
 
I might be missing something on that mobo, but 20 drives in raid 5 or even 10 drives in RAID5 seems a bit "tight"
 
Trepidati0n said:
I might be missing something on that mobo, but 20 drives in raid 5 or even 10 drives in RAID5 seems a bit "tight"
Click to expand...

I don't know if I understand what you mean. If you mean connections, I have the 2 m1015s plus the onboard 2008, for a total of 6*4 or 24 disks. Or am I misunderstanding what you are saying?
 
edlin,

your mouse issues in a linux live cd environment while using the IPMI KVM console may be because the console is using absolute mouse positioning. Patrick @ ServeTheHome did an article on this very topic just this past week.

http://www.servethehome.com/absolute-relative-mouse-ami-megarac-ipmi-20-based-kvmoverip/

i can';t say for sure that this is your problem, but my experience with the problem you describe was solved by changing the mouse mode a couple weeks ago
 
You must log in or register to reply here.
Back
Top