New Build: Modest Home Lab

S

Sparky310

n00b
Joined
Dec 2, 2014
Messages
6
I am using the holidays (and the holiday deals) as an excuse to build a virtualization server for a home lab for myself. I have been out of touch with the hardware world for quite awhile, and a couple people pointed me in the direction of this forum for advice. I considered posting in one of the other similar threads, but I didn't want to hijack someone else's build thread, so I figured I would make my own; if this is the wrong forum or I should not have made a new thread I would appreciate a move!

A couple relevant points:

1) My familiarity with virtualization is at the conceptual / light usage level. At the urging of a number of people I am leaning heavily towards using ESXi.

2) I have built a system myself before but was hoping to go pre-built for the ease, but after initial research it seems parting and assembling the build is the way to go for value (happy for someone to prove me wrong, though).

3) I work in the infosec field, so the primary purpose of the lab will be to build an environment I can stand up large numbers of victim VMs (across OS and services spectrums) in and play with e.g. Kali, Snort, etc. Work has been generous enough to agree to cover some certs for me, so I'll also be rebuilding the lab envs I encounter for further study/play after I lose access to the cert env. I am also considering trying my hand at writing for some industry magazines / journals, so I might do some code development, larger scale setups ("mock enterprise"), etc. to support that, too.

4) My gaming rig is quite long in the tooth as well, but I think I am resigned to building the virtualization server now and just waiting to replace the gaming rig for a while.

5) Budget is fairly flexible, preferably <$1000 but can stretch to say $1500 for compelling reasons. I am building or buying from scratch, excepting a monitor I will re-use. Aside from the horsepower to support '3', the only other hardware requirement I have would be at least one preferably two additional monitors (not counting those in the 1-1.5k budget). Things like power usage, noise, heat are considerations, but they're hard seconds to horsepower and cost (power is more so than noise or heat, prefer not to drive my electric bill through the roof).

Any thoughts or suggestions you can offer would be sincerely appreciated. As I said, I have built before, but that was for a standard gaming rig, so this is definitely a new experience for me. That said, the parts I am considering after research:

CPU: Intel i7-5930k ~$250
Motherboard: ASRock X99 Extreme4 LGA 2011-v3 Intel X99 ~$240
Case: ?????
PSU: SeaSonic S12G 650W ATX12v ~$100
RAM: Crucial 4x8GB 288-pin DDR4 SDRAM DDR4 2133 ~$420 (cheapest 4x8GB I saw)
GPU: VGA GTX 750 1GB 128-Bit GDDR5 PCIe 3.0 FTW ~$100
Storage: 1or2x Samsung 840 EVO 2.5" 500 GB SATA3 ~$190ea
DVD-RW: 2x ASUS 24x DVD burner ~$20ea

Prices were pulled a day or two ago so aren't accurate right now.

The i7-5930k would be a special offer from a friend; it is significantly less than the MSRP, but due to forcing DDR4 and not supporting onboard video, it will end up resulting in a ~$300 price premium. For the performance bump that benchmark data (as relevant as it is to virtualization, anyway) seems to suggest I would get, I think I am willing to pay the premium. It also gives me an excuse to grab a discrete video card that I can either stick in this rig, or swap with my GTS 250 in my gaming rig for a small upgrade.

It's interesting that the ASRock board says it is dual gigabit lan as, unless I am blind or stupid, I definitely only see one RJ-45 port. Thinking the PSU is probably overboard as well, but with the max draw from the processor being 140 and needing a discreet graphics card, I figured I would give it some room (unsure if I'll steal this card for my gaming rig and put my old GTS 250 in this rig or put this card straight into my server). The card doesn't have HDCP, but from what I understand of it, that really won't matter unless I intend to use the system as a media center (I don't). The DVD burners are for flexibility; actually considered blu-ray burners, but the cheapest ones seem like they're 2.5x the cost and have crappy reviews.

I think I did all my homework and all the hardware itself is compatible, but ESXi is the real question mark to me. Unfortunately, Google search results aren't proving terribly friendly and most of the sites I see discussing building whiteboxes are covering early (e.g. Q1) 2014 hardware. VMWare's site seems atrocious, too.
 
Are you interested in learning ESXi or just want something easy to use so you can run a bunch of VMs?

Bear in mind as well that ESXi doesn't support or offer its own software RAID. If you get two 500GB SSDs, each will appear as their own datastore. You'll need a hardware RAID card if you want to mirror them or use RAID.

Also, the onboard NIC is a desktop chipset so VMware won't recognize it natively. You'll have to create a custom install ISO with the proper driver installed or the install will fail. I, too, see only one network port.

https://communities.vmware.com/thread/483693
 
My primary goal is to run the VMs to assist with my learning / professional development in my core competency area(s), but I am willing to make learning ESXi a strong secondary goal if the feedback is that there are compelling reasons to (and from what I have heard and read so far, it seems there are). It would probably be a good experience to struggle a little with it in general and learn from that; after all, that is in part the whole intent of the lab for me.

I think I will be sticking with 1x SSD to start. I was considering going with a normal spinning drive, but swapped to the Samsung after seeing some of the deals (unfortunately the best ones already ended it seemed) and the feedback I got about virtualization and IOPS. At some point in the future I may add another, at which point I'll tackle the question of going with a RAID setup or just individual stores.

I did note that about the NIC when I was googling. I forgot to ask if anyone ended up having success getting this particular one to work. The discussion in that link seemed to end inconclusively as to whether there was a final working solution the OP (and others) arrived at. I am willing to fight with the hassle of creating the custom ISO (again, learning experience), but given my experience level I would need a fairly solid guide to work from.
 
If you look at Asrock's website you'll see it only has Intel gigabit LAN.
 
I would not rule out VMware Workstation in your primary use case.
You can build the gaming machine and run VMware Workstation on it.


As far as ESXi and NICs go .. can always add in a supported NIC.
 
Quartz-1 said:
If you look at Asrock's website you'll see it only has Intel gigabit LAN.
Click to expand...

Huh, so it absolutely does. I guess I saw the start of the graphics below the initial spec text and it looked the same as on NewEgg, so I assumed it was (on NewEgg it says dual LAN). I suppose this validates the saying about what assuming things does!

I would not rule out VMware Workstation in your primary use case.
You can build the gaming machine and run VMware Workstation on it.
Click to expand...

I have certainly considered going the type 2 route, but I was under the impression VMware Workstation is a paid product. Am I mistaken in that? I had always thought ESXi was as well, prior to this project (probably an indication as to how much I have followed virtualization solutions ...). I guess I can't think of an immediate reason to go type 1 with ESXi over type 2 with Player/Workstation beyond "it's how businesses (mostly) do it".

What would you say some compelling reasons for going with ESXi might be?
 
Sparky310 said:
Huh, so it absolutely does. I guess I saw the start of the graphics below the initial spec text and it looked the same as on NewEgg, so I assumed it was (on NewEgg it says dual LAN). I suppose this validates the saying about what assuming things does!



I have certainly considered going the type 2 route, but I was under the impression VMware Workstation is a paid product. Am I mistaken in that? I had always thought ESXi was as well, prior to this project (probably an indication as to how much I have followed virtualization solutions ...). I guess I can't think of an immediate reason to go type 1 with ESXi over type 2 with Player/Workstation beyond "it's how businesses (mostly) do it".

What would you say some compelling reasons for going with ESXi might be?
Click to expand...

VMware Workstation is a paid product offering - you are correct.

A few quick reasons to use dedicated ESXi server over VMware Workstation in your case:
1. Performance - bare metal ... one less layer
2. OS independent - OS crash or reboot won't affect your VMs
3. Mock enterprise hardware setups.
4. Free version

If you decide to go the hardware route and want a dedicated ESXi server,
I would look at the server grade boards (ex: Supermicro X10SRi-F).

Of course there are always the entry level servers from Dell (T110-II) and Lenovo (TS140)
that pop up at discounted prices.

Again .. in your use case, you can run all your dev/test VMs + a nested ESXi lab for learning
VMware ESXi all on a "gaming" system via VMware Workstation.
 
Netwerkz101 said:
VMware Workstation is a paid product offering - you are correct.

A few quick reasons to use dedicated ESXi server over VMware Workstation in your case:
1. Performance - bare metal ... one less layer
2. OS independent - OS crash or reboot won't affect your VMs
3. Mock enterprise hardware setups.
4. Free version
Click to expand...

I'm really tempted to go with ESXi, if for no other reason than the learning factor. Not sure if that is being more hard-headed than anything else or what. I would prefer the victims and setups and I create be as close as reasonably possible to "real world," but in most cases I don't know that Workstation would be any less "real."

The cost of Workstation certainly doesn't help. From VMware's site it appears to be ~$300 for a year, and I really don't know that I want to create this lab with a recurring support cost from day one. I suppose I could buy a copy of Win8Pro and use Hyper-V, too.

If you decide to go the hardware route and want a dedicated ESXi server,
I would look at the server grade boards (ex: Supermicro X10SRi-F).

Of course there are always the entry level servers from Dell (T110-II) and Lenovo (TS140)
that pop up at discounted prices.
Click to expand...

Any particular reason on the server grade boards? The processor is already limited to 64GB RAM, which the Extreme4 can easily support with 8 slots, has plenty of PCI/USB slots, etc. Primarily a stability thing, or?

I looked through some of Dell and Lenovo's stuff during the Black Friday deals, and it just didn't seem like they were able to offer prices good enough to justify the pre-built route. It seemed like no matter what I looked it I was going to pay at least a couple hundred more for a similar or less powerful system.
 
Desktop parts will be fine, whether you go the VMware Workstation or ESXi route. I've run desktop parts in my VMware ESXi lab for years. It's more than good enough for learning, especially since you need to bake in some hardware drivers to get desktop NICs or what not to work. You can also buy dual and quad port Intel NICs off ebay for $40-$80 that don't require special drivers (avoid Broadcom).
 
Sparky310 said:
I'm really tempted to go with ESXi, if for no other reason than the learning factor. Not sure if that is being more hard-headed than anything else or what. I would prefer the victims and setups and I create be as close as reasonably possible to "real world," but in most cases I don't know that Workstation would be any less "real."

The cost of Workstation certainly doesn't help. From VMware's site it appears to be ~$300 for a year, and I really don't know that I want to create this lab with a recurring support cost from day one. I suppose I could buy a copy of Win8Pro and use Hyper-V, too.



Any particular reason on the server grade boards? The processor is already limited to 64GB RAM, which the Extreme4 can easily support with 8 slots, has plenty of PCI/USB slots, etc. Primarily a stability thing, or?

I looked through some of Dell and Lenovo's stuff during the Black Friday deals, and it just didn't seem like they were able to offer prices good enough to justify the pre-built route. It seemed like no matter what I looked it I was going to pay at least a couple hundred more for a similar or less powerful system.
Click to expand...

What is "real" in the world of virtualization? :D
On that same thought of wanting to be close to "real world" and the fact you mentioned
"mock enterprise" makes me think a server grade board is more ideal.

Desktop boards do work fine .... my own lab uses all the above that I mentioned:
Intel Desktop grade boards, Supermicro Server Boards, and Dell Poweredge T110.

If you got the example SuperMicro board I mentioned as an example:
- should not need to customize an ISO for ESXi and you start with 2 NICs vs 1 on the desktop board.
- Don't need to purchase a video card ...and IPMI allows for out of band management.
- Xeon Processor >64GB RAM capable/scalable to 512GB if needed (many more victims)

I can understand the cost of VMware Workstation being an inhibitor.
You are not being hard-headed by going the ESXi route .. it's your lab and choice in the end.
I run both ... Workstation on my laptop.

Forgot ... while I mentioned building your game machine and running VMware Workstation on it,
I failed to mentioned multibooting with ESXi and whatever OS(s) you choose...something else I do in my lab.
 
Last edited:
Netwerkz101 said:
What is "real" in the world of virtualization? :D
Click to expand...

Yea, that's what I mean about Workstation not being any less "real," and possibly being hard-headed about ESXi :p I think I'll start with the idea of using ESXi, and if that fails, I can always go the Workstation or Hyper-V routes.

On that same thought of wanting to be close to "real world" and the fact you mentioned
"mock enterprise" makes me think a server grade board is more ideal ...
Click to expand...

The dual LAN ports would be nice, but otherwise with my current goals and knowledge level I don't really see the value in the other server board features. IPMI seems like a neat feature, I just don't know what I would realistically use it for that would justify building around a server mobo.

The extra RAM capability is certainly nice as well, but I don't see myself going beyond 64GB in this build; if I were to ever exceed that, it would probably be in a complete rebuild. While I want this rig to last me as long as possible, I think I would be ok with lifespan of 2-3 years. I figure at that point, assuming this was successful, I can take my lessons learned and build another server from scratch. Hopefully by then I'll be in a significantly different place professionally as well, which might heavily influence the focus of the build (not to mention budget, or so I hope).

Forgot ... while I mentioned building your game machine and running VMware Workstation on it, I failed to mentioned multibooting with ESXi and whatever OS(s) you choose...something else I do in my lab.
Click to expand...

Certainly an option if that is something one can do alongside ESXi.


I sincerely appreciate the feedback from this thread. I think I am looking to pull the trigger on the parts either tonight or this weekend. Any other thoughts or recommendations, hardware or otherwise? What are the popular/quality case manufacturers these days? I'm also still on the fence with the PSU and whether it seems way overboard.
 
Power Supply:
- Pick a reputable brand
- Estimate expected power use for your build + little overhead
- Ensure connectors are what you need
The 450W version of the unit you are looking at would be fine for your build as listed.


Case:
- I tend to pick the cheapest metal case that can contain all components and allow
for adequate cooling of those components

My only recent experience in shopping for ATX cases with 8 x 3.5" internal bays led me to:
- NZXT Source 210 ($39)
- Fractal Design R4 Blackout ($60-$69 on sale)

I wish I had bought the NZXT Source 210 Elite for $10 more because it was tool-less + had an additional cooling fan included IIRC.
 
That's a third or fourth vote for Fractal Design across people I am talking with, so I will definitely check them out. Looks like they even take 2.5" drives (though not sure how unusual that is these days). Thanks for the suggestion!
 
I would personally go with a server board and Xeon. You might be able to find them cheaper, but here are three combo deals from NewEgg.

$450.98 - Supermicro X10SRL-F Server Board + Intel Xeon E5-2603 v3
$528.99 - Supermicro X10SRL-F Server Board + Intel Xeon E5-2609 v3
$653.98 - Supermicro X10SRL-F Server Board + Intel Xeon E5-2620 v3

You can get a lot more memory this way in the future and there is no need to buy a video card. My ESXi server is nearly four years old now and is running 2x 4-core AMD Opteron 4130 processors with 128 GB of memory. I bet all three of the Intel CPUs listed above outperform my AMD processors. I run a decent amount of VMs, so I am currently consuming almost 80 GB.
 
KapsZ28 said:
I would personally go with a server board and Xeon. You might be able to find them cheaper, but here are three combo deals from NewEgg.

$450.98 - Supermicro X10SRL-F Server Board + Intel Xeon E5-2603 v3
$528.99 - Supermicro X10SRL-F Server Board + Intel Xeon E5-2609 v3
$653.98 - Supermicro X10SRL-F Server Board + Intel Xeon E5-2620 v3

You can get a lot more memory this way in the future and there is no need to buy a video card. My ESXi server is nearly four years old now and is running 2x 4-core AMD Opteron 4130 processors with 128 GB of memory. I bet all three of the Intel CPUs listed above outperform my AMD processors. I run a decent amount of VMs, so I am currently consuming almost 80 GB.
Click to expand...

I guess you missed the OP? he's getting a 5930K for $250 from the intel retail edge program, CPU is WAY WAY overkill for his needs and he's paying a premium for the DDR4 ram but he's ok with that.

It's more powerful and cheaper than the Xeons you listed.

OP, as for the video card, ESXi literally has no benefit from it unless your passing through a GPU for gaming but you're not, I'd go with something cheap ($20) like the AMD 5450 unless you want to update your gaming machine with a 750 for $100 (but in that case, why not grab an AMD 280X for $100 which more than twice the power of that card?)
 
*The*Dark*Lord* said:
I guess you missed the OP? he's getting a 5930K for $250 from the intel retail edge program, CPU is WAY WAY overkill for his needs and he's paying a premium for the DDR4 ram but he's ok with that.

It's more powerful and cheaper than the Xeons you listed.

OP, as for the video card, ESXi literally has no benefit from it unless your passing through a GPU for gaming but you're not, I'd go with something cheap ($20) like the AMD 5450 unless you want to update your gaming machine with a 750 for $100 (but in that case, why not grab an AMD 280X for $100 which more than twice the power of that card?)
Click to expand...

I read what he posted, and like I stated, I recommend the server board. Especially for the amount of memory you can use.
 
Sparky310 said:
Aside from the horsepower to support '3', the only other hardware requirement I have would be at least one preferably two additional monitors (not counting those in the 1-1.5k budget).
Click to expand...

Just to be clear, ESXi is not meant for a system you plug monitors into outside of configuring the root password and network config. You'll need another machine to run vSphere on to manage the server. Your video card would be useless unless you pass it through to a VM, in which case I believe Radeon's have better support. Then you would only be able to use the GPU from within the VM, it wouldn't be part of managing the host and you'd still need another machine to manage the ESXi box to get this VM setup with the GPU passed through. Once it is passed through and you're using the VM directly, it could manage ESXi but if the VM went down so would your management of ESXi.

If you want to have a single system, you want VMWare Workstation.
 
ser_rhaegar said:
Just to be clear, ESXi is not meant for a system you plug monitors into outside of configuring the root password and network config. You'll need another machine to run vSphere on to manage the server. Your video card would be useless unless you pass it through to a VM, in which case I believe Radeon's have better support. Then you would only be able to use the GPU from within the VM, it wouldn't be part of managing the host and you'd still need another machine to manage the ESXi box to get this VM setup with the GPU passed through. Once it is passed through and you're using the VM directly, it could manage ESXi but if the VM went down so would your management of ESXi.

If you want to have a single system, you want VMWare Workstation.
Click to expand...

+1 to all of that

I'd also agree with others who recommended a server grade mobo. They just have much better ESXi support. I've read a bunch of posts from people with desktop motherboards which claim to support vt-d (pci passthrough) but don't fully support it in real life. Supermicro is everywhere for ESXi builds, so when your learning and run into an issue, chances are some googling will find someone with the same problem on the same hardware. Can't say the same for desktop/pro-sumer boards.
 
You must log in or register to reply here.
Back
Top