Looking for suggestions to upgrading my lab

T

thecoffeeguy

n00b
Joined
Jun 27, 2015
Messages
42
Hello friends.
As the title states, looking for some suggestions and feedback to upgrade and improve my home lab.

I currently have a shuttle box running a Intel quad core with 32gb memory. Internally, it has (2) 10,000 SCSI drives.

My lab is going to be mostly Linux (some solaris), and sprinkling of Windows machines here and there. It will primarily be for work as i am constantly testing new stuff as well as playing around with new technologies.

That said, I want to add one more ESXi server to my mix and maybe some storage (maybe FreeNAS or something). In addition, i also need a 24 port switch as well as I plan to have a lot of NICs in my setup.

For the new ESXi box, I was looking at this server from SuperMicro server:

SUPERMICRO SYS-5018A-FTN4 1U Rackmount Server Barebone FCBGA 1283 DDR3 1600/1333 - Newegg.com

Most of what I am testing is more memory and IO intensive. Not so much CPU.
Thoughts on this server? I am on a bit of a budget.
Alternatives? Was lookign to start with 32gigs and when the 16gb modules drop in price, up it to 64.

Also, looking for a good 24 port switch. With the server above (4 NIC's), my existing ESXi server, 3 NIC's, and looking to add a storage machine, I want to make sure I have plenty of space to add VLAN's, test networking etc.

Lastly, wall mount server. Would like something around a 6U size. Nothing to fancy. Clean wiring etc.

I am open to suggestions as well. Going to place this soon as i desperately need to get this up and running.

Thanks everyone!

TCG
 
Greetings! I would be happy to share some experiences as I just built out my home lab a bit.

About a month ago I built out a FreeNAS box, based on this ASRock miniITX board. The reason was that I had been running a single ESX host with local SSDs for datastores, but wanted the option of dabbling with clusters. I can't say enough good things about this board for a FreeNAS build. You get more than enough CPU resources, it has a ridiculous amount of SATA ports and it can be loaded with 32GB ECC RAM. I then picked up a second ESX host and serve the storage up as iSCSI LUNs.

The experience so far is really great. The FreeNAS box is presenting two iSCSI volumes; one is x3 2TB WDGreen drives with x2 180GB SSDs for L2ARC, and I then also threw together a JBOD of SSDs. I don't care so much about losing that datastore because I back up regularly using the free edition of Veeam. Most of the VMs are running on that WD Green iSCSI LUN and after giving the L2ARC cache some time to warm up, response times for that datastore are generally sub 10ms. Granted I am not running any crazy workloads.

So in a nutshell, I would say FreeNAS is a great way to go especially if you have some drives already laying around like I did. I think you would be hard pressed to find a better ESX host option, especially on a budget. If you go the Xeon route, you are looking at quite a bit of $$ and frankly for a home lab it would be overkill from a CPU resource perspective. I was even looking at Xeon-D based motherboards but even those are a bit pricey and you have to go with DDR4. It is just too easy to pick up DDR3 cheaply these days.

PS - I got burned when getting RAM for those Atom boards! I had some registered ecc dimms laying around and didn't realize you need unregistered dimms. I ended up picking up 4 of these modules and they work well:
http://www.amazon.com/gp/product/B008EMA5VU?psc=1&redirect=true&ref_=oh_aui_detailpage_o04_s00
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
thecoffeeguy said:
<snipped>

Most of what I am testing is more memory and IO intensive. Not so much CPU.
Thoughts on this server? I am on a bit of a budget.
Alternatives? Was lookign to start with 32gigs and when the 16gb modules drop in price, up it to 64.

</snipped>

TCG
Click to expand...

What does IO intensive mean? Storage? Network?
If you know for sure you will want more than 32GB RAM, look elsewhere.

While that board/cpu combo will do 64GB RAM, those 16GB modules have remained $300+
from the one vendor that sells them AFAIK. I can't stomach $1200 for memory when I can build a complete
system for same money - with 64GB of memory.

My lab has some similar components to what you seek:
Netwerkz101 Lab
 
I will just say, I am running one of Supermicro's dual proc boards and I am really pleased. You can find them on EBay for around $300. The nice thing is, you can run with 1 proc and 12 RAM slots or go 2 proc and access all 24 slots. 8GB ECC modules are fairly cheap so you have lots of room to grow. You can also find Xeons at very reasonable prices, especially the engineering samples that pop up from time to time.
 
Netwerkz101 said:
What does IO intensive mean? Storage? Network?
If you know for sure you will want more than 32GB RAM, look elsewhere.

While that board/cpu combo will do 64GB RAM, those 16GB modules have remained $300+
from the one vendor that sells them AFAIK. I can't stomach $1200 for memory when I can build a complete
system for same money - with 64GB of memory.

My lab has some similar components to what you seek:
Netwerkz101 Lab
Click to expand...

ya, i started to look at memory and that is expensive.
i think i should be good with 32gigs. If i can upgrade to 64, even better.

The big thing I keep coming back to is power draw. I may end up with a few of these at my house (3 tops), and i would like it to NOT raise my electricity bill.

I am also up for building my own if i can get what i am looking for.

Been quite awhile since i have built a server.
I looked at your lab and that is sweet.
do you know what the power draw on that is (per node)?

i like your compute nodes. What is the chassis on that if you dont mind me asking? Do you have a ballpark on what the cost would be?
That seems to be exactly what i am looking for.

REally appreciate it.

TCG
 
Hey all. Just quickly update here.
Turns out my lab is growing. Wife came told me today she wants a home media setup so we can store all or DVD's and Blu-rays and accessing them via something like Plex and make it easier for her and my kids to find movies (instead of digging through cases.)
Cool with me. More toys to play with

That said, still looking forward a power efficient setup.

From Netwerks101 setup above, that is pretty sweet setup.

From what, i know for a fact I will be needing:

Rackmount of some sort, something around 6U.
One ESXi host, for now, but the idea is to add another node for expansion. I want this to support at least 64gb memory. Even if i start with 32gigs (say 1 16gb dim to start with), i can grow into it.
NAS storage. Funny enough, a co-worker of mine gave me a case he has with a nice setup: supermicro board, good CPU and 32gigs of memory in it. It has a 5.25 to 2.5 caddy in it where I can put in 6 SSD drives for storage.

At this point, i am leaning towards building my own 1U server.
I am looking for suggestions on a good chassis to look out for (or specific brand name). The one in netwerkz101 picture looks pretty slick.

Outside of that, is there anything else I should consider moving forward and before I purchase?

There are some sales going on right now at newegg and hoping to make the plunge here soon.

Things just got a lot more interesting and fuN!

Cheers!

EDIT:
Thoughts?

Ok. So I think I found my build. Here is what I am thinking for my core ESXi host:

Mobo: Supermicro | Products | Motherboards | Xeon® Boards | X11SSi-LN4F

CPU: Intel Xeon E3-1225 v5 SkyLake 3.3 GHz 8MB L3 Cache LGA 1151 80W BX80662E31225V5 Server Processor - Newegg.com

Chassis:

Supermicro | Products | Chassis | 1U | SC512F-350B

The only thing I need to pick out is memory now and some SSD's.

Going to add a NAS setup and throw in some ICY docks so i can add extra hard drives to my NAS and mount that to my ESXi box.

Any thoughts on those components above?
 
Last edited:
thecoffeeguy said:
ya, i started to look at memory and that is expensive.
i think i should be good with 32gigs. If i can upgrade to 64, even better.

The big thing I keep coming back to is power draw. I may end up with a few of these at my house (3 tops), and i would like it to NOT raise my electricity bill.

I am also up for building my own if i can get what i am looking for.

Been quite awhile since i have built a server.
I looked at your lab and that is sweet.
do you know what the power draw on that is (per node)?

i like your compute nodes. What is the chassis on that if you dont mind me asking? Do you have a ballpark on what the cost would be?
That seems to be exactly what i am looking for.

REally appreciate it.

TCG
Click to expand...

The power draw at idle is 35W-40W per compute node (2)
... i can measure later with a kill-a-watt

The atom based node was less than 35W with low use
The storage node was about 40W low use ..SSD drives pretty much idle

From what I remember, the whole lab was about/under 300W - everything in the rack.

The cases used = Supermicro SC504-203B and each was $99 - it includes the 200W power supply.

You should know that my lab is a little deceptive because I wanted quiet and to be able to use most of the PCIe slots,
I doubled the cases...each compute node uses two of those cases.
 
Netwerkz101 said:
The power draw at idle is 35W-40W per compute node (2)
... i can measure later with a kill-a-watt

The atom based node was less than 35W with low use
The storage node was about 40W low use ..SSD drives pretty much idle

From what I remember, the whole lab was about/under 300W - everything in the rack.

The cases used = Supermicro SC504-203B and each was $99 - it includes the 200W power supply.

You should know that my lab is a little deceptive because I wanted quiet and to be able to use most of the PCIe slots,
I doubled the cases...each compute node uses two of those cases.
Click to expand...


Thanks Netwerkz101. Really appreciate it. Fantastic information.
Would a chassis like that work with something like this:

Mobo:
SUPERMICRO MBD-X11SSI-LN4F-O ATX Server Motherboard LGA 1151 Intel C236 - Newegg.com

CPU:
Intel Core i5-6500 6M Skylake Quad-Core 3.2 GHz LGA 1151 65W BX80662I56500 Desktop Processor Intel HD Graphics 530 - Newegg.com

Or, similar to your build:

Mobo: Same as your build:

CPU:
Intel Xeon E5-2603 v3 Haswell 1.6 GHz 6 x 256KB L2 Cache 15MB L3 Cache LGA 2011-3 85W BX80644E52603V3 Server Processor - Newegg.com


Never build a server before so this is a first. Just trying to get a nice combo down with good resources (I want 64gb RAM), 2 internal drive bays for SSD's and the ability to mount storage via NFS or iSCSI.

Some good deals at NewEgg at the moment.

Really appreciate you taking the time to answer my questions.

Thanks again!

TCG
 
thecoffeeguy said:
Thanks Netwerkz101. Really appreciate it. Fantastic information.
Would a chassis like that work with something like this:

Mobo:
SUPERMICRO MBD-X11SSI-LN4F-O ATX Server Motherboard LGA 1151 Intel C236 - Newegg.com

CPU:
Intel Core i5-6500 6M Skylake Quad-Core 3.2 GHz LGA 1151 65W BX80662I56500 Desktop Processor Intel HD Graphics 530 - Newegg.com

Or, similar to your build:

Mobo: Same as your build:

CPU:
Intel Xeon E5-2603 v3 Haswell 1.6 GHz 6 x 256KB L2 Cache 15MB L3 Cache LGA 2011-3 85W BX80644E52603V3 Server Processor - Newegg.com


Never build a server before so this is a first. Just trying to get a nice combo down with good resources (I want 64gb RAM), 2 internal drive bays for SSD's and the ability to mount storage via NFS or iSCSI.

Some good deals at NewEgg at the moment.

Really appreciate you taking the time to answer my questions.

Thanks again!

TCG
Click to expand...

As far as the Supermicro 504-203b case goes .. it is meant for Supermicro motherboards.
It was designed specifically for Mini-ITX and Micro-ATX offerings from them.

I can only tell you it _looks_like_it_SHOULD_fit ....but I can only base that on pictures of the I/O backplane
on the motherboard.
Mobos I have that fit the horizontal plane:
- A1SRi-2758F-O Mini-ITX
- X10SL7-F-O Micro-ATX
- X10SRi-F ATX

Case is 9.8" deep and largest ATX motherboard I have in one is 9.6" deep - the smallest ATX standard depth.

You have to keep cooling components in mind when you build with a 1U case ...
not hard to break the vertical limit of a 1U case with CPU + heatsink + fan.
This is why I went 2U based on two short depth 1U cases - that and to use HH/low profile PCIe cards.

The E5-2603v3 CPU was one I considered to keep my build cost down ...
but knowing the type of VMs and what those VMs were going to be doing,
I got the feeling that 1.6GHz wasn't going to cut it for my 30+ VMs on a single node
and that's why I opted for the E5-2620v3 @ 2.4GHz.
(your post does not quantify anything - so no real way to tell you one way or the other)

The good thing is you can upgrade it later if you need to.
Microcenter was the cheapest I could find my CPU ..by almost $100.
 
Well ... interesting result with the Kill-A-Watt ... it seems the lab idles at 150W.
It did spike ti ~200W during boot of the (1) storage node + (2) compute nodes.

Everything in the 12u rack is plugged into the UPS and the UPS into the Kill-a-watt.
 
Netwerkz101 said:
As far as the Supermicro 504-203b case goes .. it is meant for Supermicro motherboards.
It was designed specifically for Mini-ITX and Micro-ATX offerings from them.

I can only tell you it _looks_like_it_SHOULD_fit ....but I can only base that on pictures of the I/O backplane
on the motherboard.
Mobos I have that fit the horizontal plane:
- A1SRi-2758F-O Mini-ITX
- X10SL7-F-O Micro-ATX
- X10SRi-F ATX

Case is 9.8" deep and largest ATX motherboard I have in one is 9.6" deep - the smallest ATX standard depth.

You have to keep cooling components in mind when you build with a 1U case ...
not hard to break the vertical limit of a 1U case with CPU + heatsink + fan.
This is why I went 2U based on two short depth 1U cases - that and to use HH/low profile PCIe cards.

The E5-2603v3 CPU was one I considered to keep my build cost down ...
but knowing the type of VMs and what those VMs were going to be doing,
I got the feeling that 1.6GHz wasn't going to cut it for my 30+ VMs on a single node
and that's why I opted for the E5-2620v3 @ 2.4GHz.
(your post does not quantify anything - so no real way to tell you one way or the other)

The good thing is you can upgrade it later if you need to.
Microcenter was the cheapest I could find my CPU ..by almost $100.
Click to expand...

Thanks Netwerkz101. That is super helpful information and helps me plan out my setup

In regards to cooling, did you do additional cooling in your chassis?
makes me wonder if i should look for a 2U, just for extra space and cooling.

I liked the idea of doing a 1U in a rackmount for ease and it being clean. I was going to buy a 6U server rack, thinking that should be plenty.

Last thing I want to happen is get this setup and realize i am running into cooling problems.

As far as the CPU, what i will be running will be mostly memory and IO intensive. Not so much CPU. Figure this node would run between 8-12 VM's (mix of OS's) and part of an exisiting shuttle box I have. Add in some NAS storage and I think i should be set.

Just based on the build above, think i could run into cooling problems? Suggestions on how to combat that?

Really cant thank you enough for your time and help on this. Been incredibly helpful and gives me lots of cool things to think of.

Thank you again!

Cheers,

Coffeeguy
 
thecoffeeguy said:
Thanks Netwerkz101. That is super helpful information and helps me plan out my setup

In regards to cooling, did you do additional cooling in your chassis?
makes me wonder if i should look for a 2U, just for extra space and cooling.

I liked the idea of doing a 1U in a rackmount for ease and it being clean. I was going to buy a 6U server rack, thinking that should be plenty.

Last thing I want to happen is get this setup and realize i am running into cooling problems.

As far as the CPU, what i will be running will be mostly memory and IO intensive. Not so much CPU. Figure this node would run between 8-12 VM's (mix of OS's) and part of an exisiting shuttle box I have. Add in some NAS storage and I think i should be set.

Just based on the build above, think i could run into cooling problems? Suggestions on how to combat that?

Really cant thank you enough for your time and help on this. Been incredibly helpful and gives me lots of cool things to think of.

Thank you again!

Cheers,

Coffeeguy
Click to expand...

No additional cooling needed for my build as my lab sits in a basement that never goes above 85 degrees F in hottest months.
I simply have a 60mm Noctua fan for the cpu heatsink and of course the power supply fan is an exhaust fan.
Thats it for the compute nodes. Similar for my storage node..it just uses a stock Intel heatsink/fan combo that came with the CPU.

For the gateway node running the Atom C2758F, there is a 25mm Noctua in that 1U case blowing directly over the CPU heatsink.
It is mounted horizontal on top to blow or exhaust air to/from the CPU. It's enough for that low power Atom.
It will overheat with no cooling though.
If it was any hotter in my environment, I would add another 25mm and mount both vertically and point them to blow across the cpu/heatsink front to back.

I cannot tell you if you will have cooling problems without knowing the environment in which you will deploy your lab.

1U is not possible with my components..at least not without modding. CPU+Heat sink were probably 1-2mm above the 1U height limit.
I could have shave the fins and ran a high RPM fan or two...but again..didn't want noise.

Also ...your budget is limited as you say....but the Supermicro Motherboard/CPU combos based on D1540/1541 CPU are perfect for 1U beast compute nodes.
It would be similar to what you had in your original post but much more powerful/capable. They fit the SC504-203 cases too.

Keep an open mind ... you can build what you want ...it's easy.
The hard part...... knowing what it is you want.Most people fail here.

Based on 1U and 2U cases, 6U rack can support:
6 x 1U
3 x 2U
4 x 1U + 2U
2 x 2U + 2 x 1U

You can mount switches horizontal in the rack or vertical on the outside of the rack and save 1U.
 
Netwerkz101 said:
No additional cooling needed for my build as my lab sits in a basement that never goes above 85 degrees F in hottest months.
I simply have a 60mm Noctua fan for the cpu heatsink and of course the power supply fan is an exhaust fan.
Thats it for the compute nodes. Similar for my storage node..it just uses a stock Intel heatsink/fan combo that came with the CPU.

For the gateway node running the Atom C2758F, there is a 25mm Noctua in that 1U case blowing directly over the CPU heatsink.
It is mounted horizontal on top to blow or exhaust air to/from the CPU. It's enough for that low power Atom.
It will overheat with no cooling though.
If it was any hotter in my environment, I would add another 25mm and mount both vertically and point them to blow across the cpu/heatsink front to back.

I cannot tell you if you will have cooling problems without knowing the environment in which you will deploy your lab.

1U is not possible with my components..at least not without modding. CPU+Heat sink were probably 1-2mm above the 1U height limit.
I could have shave the fins and ran a high RPM fan or two...but again..didn't want noise.

Also ...your budget is limited as you say....but the Supermicro Motherboard/CPU combos based on D1540/1541 CPU are perfect for 1U beast compute nodes.
It would be similar to what you had in your original post but much more powerful/capable. They fit the SC504-203 cases too.

Keep an open mind ... you can build what you want ...it's easy.
The hard part...... knowing what it is you want.Most people fail here.

Based on 1U and 2U cases, 6U rack can support:
6 x 1U
3 x 2U
4 x 1U + 2U
2 x 2U + 2 x 1U

You can mount switches horizontal in the rack or vertical on the outside of the rack and save 1U.
Click to expand...

Thanks again Netwerkz101.
I thought about this and think i am going to take your advice and will update my CPU. i just thought it would be better to have the extra cores and power, just in case. It would be unfortunate to buy the CPU, only to realize rather quickly, you need a faster CPU. I'll opt for a xeon based CPU with a few more cores and power.

Makes sense on the cooling piece. Something I will consider as the room I will have this in will be my office. I have a portable AC in there, but it should not get above 85 degrees there in the summer.

For the chassis you listed from SuperMIcro, that is kind what i am learning towards because it is easy (rather inexpensive). Unless there is a better one i find.

That should get me started for now.

Now i just need to buy and build it. :)

Really appreciate your time an feedback.
Much obliged.

Cheers,

Coffee
 
Just thought I would update with some items since my last post.

I narrowed down a mobo/cpu combo for my new ESXi build

Here is what I found:

Setup #1: Mobo: (up to 64gb memory supported)

Supermicro | Products | Motherboards | Xeon® Boards | X11SSi-LN4F

CPU option 1: Intel Xeon E3-1220 v5 SkyLake 3.0 GHz 8MB L3 Cache LGA 1151 80W BX80662E31220V5 Server Processor - Newegg.com

Or...

mobo: LOTS of room for memory, 512gb

Supermicro | Products | Motherboards | Xeon® Boards | X10SRL-F

CPU:

Intel Xeon E5-2603 v3 Haswell 1.6 GHz 6 x 256KB L2 Cache 15MB L3 Cache LGA 2011-3 85W BX80644E52603V3 Server Processor - Newegg.com

Pulling the trigger today.

My hopes here is for friends to tell me the positives and negatives for the above. I am not sure which chipset is the better one, so looking for expertise here.

This will strictly be for server based stuff, running between (hopefully) 8+ VM's.

Still need to pick out memory, chassis and storage. Would like to add an extra quad NIC in as well for networking options.

Was going to look at this chassis:

SUPERMICRO SuperChassis CSE-512F-350B Black 1U Rackmount Server Chassis 350W - Newegg.com

But not sure if fits with the second mobo listed above.
I dont think i can go wrong with either one.
One gives me memory expansion capabilities.
Filling out 64gb memory (which is where i want to go) is going to be expensive. Kinda hoping memory drops in price here soon.

Anyone see any issues with the above? Any gotchas?

Thanks friends.

Coffee
 
You wrote that you will buy another quad nic to that mobo - maybe it'd better to buy mobo with more nic embedded ? Supermicro has many models with quad nic,, 10g nics.
If you care about power draw - the best would be Xeon-D.
 
You must log in or register to reply here.
Back
Top