VM home lab suggestions

[dxun1]

Hello everyone, new poster here.

Would like your thoughts/recommendations on this home build I am currently working at:

CPU: Intel i5-4590 ($209 CAD)
CPU Cooler: CoolerMaster 212 Evo ($25 CAD)
Motherboard: Gigabyte GA-Z97-WIFI ($141 CAD)
Memory: G.Skill RipjawsX Series 16GB PC3-14900 (2x8 GB) ($166 CAD)
Storage: Samsung F3 1TB (already owned)
Case: Fractal Design Node 304 Mini ITX Tower Case ($90 CAD)
Power Supply: Antec High Current Gamer 400W (modular) ($44 CAD)
Network Card: Intel PRO/1000 PCI-e Gigabit Dual-Port LAN (EXPI9402PTBLK) ($62 CAD)

The main purpose of the machine is to run ESXi host with several VMs for my own purposes, most notably:
- firewall/router/proxy/DNS server/VPN (pfSense distro)
- web server (CentOS distro)
- database server (CentOS distro)
- build server (CentOS distro)

Let me explain my choice of components and reasoning:

ESXi is notoriously finnicky when it comes to hardware so the reason for taking a Z97 board despite having a non-K CPU is ESXi support - I am simply not sure how would it react to H97 board (which is only 10$ cheaper than Z97 anyhow). Same goes for the Intel's network card - it seems to be certified by VMWare and it's highly unlikely ESXi would play well with on-board Intel i217V (which is not certified by ESXi), let alone second Atheros port - so I'll probably just make them available as passthrough adapters to client operating systems (the same with WiFi).

Moreover, I could not find any decent, yet considerably cheaper board with a single LAN port (I'd even be willing to sacrifice WiFi for such a board) - any suggestions? I've been considering Supermicro boards but I am just not sure how well differents chipsets play with ESXi (in particular - is there any difference ESXi compatibility-wise, when it comes to Z97/H97 and Z87/H87/H81?)

I also had trouble finding relatively efficient, low-powered ATX PSUs - since the machine will be lightly stressed most of the time, I don't expect I would exceed 70-80 W on average and there is very little in the way of reliable PSUs with efficiency > 80% in that power range, especially given the price of Antec PSU being considered here ($45). Even if I wanted to consider them, SFX power supplies are even worse - cheapest Silverstone 450W SFX PSU is priced around $70 and I don't even need such vast power. Seasonic has a few SFX PSUs in 300W-power range but they are more expensive and less efficient than this Antec here. Again - any suggestions/thoughts here would be most welcome.

Storage is reused from a previous machine I owned - I will try to make ESXi work with NFS shares I have available on my NAS, so theoretically, I may not even need any storage, save for booting the ESXi host (which itself shouldn't consume more than a 1 or 2 GB of disk space). I am also considering buying a 16 GB reliable stick (such as Sandisk Extreme) and booting ESXi off of that.

Thanks for your suggestions in advance

 

[Zarathustra[H]]

The only comments I have at this point are:

1.). Do both your CPU and that motherboard have published VT-D support? This is necessary if you ever want to direct I/O forward anything to your guests. (Not everyone wants to do this, but for me its crucial. Again depends on what you are planning on doing with it.

This is one of the reasons many who build their own ESXi boxes with consumer hardware do so with AMD systems. While most non-K Intel CPU's support VT-d it is very difficult to find motherboards that support the feature. On the AMD side almost all of them do (but its called by its technical name, IOMMU, instead of Intel's brand name VT-d. They are essentially the same thing)

If you instead use official server hardware VT-d/IOMMU support is pretty much universal.

2.). How many ram slots do you have? With virtualization one of the rules is you will likely need more RAM than you think. Over provision up front and make sure you can expand later. Since 16gig UDIMMs are still as rare as unicorns, it limits your max amount of RAM. This is the main reason I recently switched away from my home built server, as the four RAM slots effectively limited me to 32 gigs.

3.) Consider using ECC RAM (and if you do, make sure your CPU/Motherboard support it). May not be strictly necessary for your application, but it is typically a good idea. For me it was as I have a ZFS based NAS as one of my guests, and ZFS has been known to implode on itself in an unrecoverable manner due to RAM faults without ECC.

Good Luck!

 

[Zarathustra[H]]

Also worth noting,

For what you are spending on consumer parts, you could pick up an off lease rack mount server on eBay with 30day warranty, and then you have real server parts, ECC ram, etc. etc.

I just ordered a 12 bay HP DL180 G6 with dual Xeon L5640's (12 cores + HT = 24 threads) and 64GB of registered ECC ram for $780 shipped (and judging from my research I probably overpaid)

I enjoy building my own stuff, but when it comes to servers, I've come to the conclusion lately that used data center servers are the better buy (just avoid the hard drives, as they likely have way too many hours on them)

 

[fantabulous]

Getting an off lease server is a pretty good suggestion if you need ESXI compatibility specifically, if you also have the room for rackmount hardware that's not in your living areas. Do you need ESXI specifically, though? I mention this because managing a single KVM node with virt-manager is pretty easy and you have broader hardware compatibility.

Zarathustra[H] brings up relevant points about VT-d compatibility as well. Even with a "supported" consumer chipset, it's still a bit of a crapshoot as you also need BIOS/UEFI support for the feature in addition to chipset and processor. Consider also the Q87 and B85 chipsets for VT-d support if you go the consumer hardware route.

Regarding the hardware specs given, that'll probably be fine. You're going to be quite limited on disk I/O with a single 1TB spinning drive, and you'll be limited to 16GB RAM at most with the mini-itx form factor and 1150 platform. One SSD can alleviate your IO issues at that scale if you end up needing it. 16GB RAM will probably be plenty for the given VMs, but keep in mind you'll either need to build larger or build another node if you run into that limit.

Also of note, I'm not sure that 212 Evo is going to have enough clearance in the Node 304. The 92mm Hyper TX-3 should fit if you put the fan on the "back" (closest to the exhaust fan). I have a relatively similar H77n-wifi in a Node 304 for my gaming box.

 

[TType85]

Take a look for

ASRock E3C226D2I Mini ITX Server Motherboard LGA 1150 Intel C226 DDR3 1600/1333
or
ASRock E3C224D2I Mini ITX Server Motherboard LGA 1150 Intel C224 DDR3 1600/1333

They might fit better for what you need. From what I can tell, the only issue with them is they are picky on RAM but you get 2x Intel LAN, on board USB Type-A port and IPMI which is nice.

You may need to inject the ESXi ISO with the network drivers to get it to install though. http://www.servethehome.com/install-vmware-esxi-5x-intel-i210-intel-i350-ethernet-adapters/

 

[dxun1]

Thanks for your answers, they were very helpful - here's what I have learned but before that, let me just say, the purpose of this build is a small-scale home lab machine for personal projects and web-hosting. As such, it will be installed in the living room, so silence, efficiency, affordability and low profile are key features - for the most part, the pfsense VM will be mostly hit with traffic from other computers in the house but I expect to be the sole inhabitant of the remainder of VMs, at least for the near future. That would also obviate the need to have any server-grade hardware (not to mention server-grade hardware tends to be on the noisy side as well).

1) As of time of this writing, it seems to me nothing but Q87 chipset natively supports VT-d capabilites. This came as a bit of a surprise to me, as I was certain the VT-d instructions on the CPU is the only requirement to make direct IO happen.

Therefore, I am now considering Gigabyte GA-Q87M-D2H - I have found here that this board seems to support VT-d instructions so could you please open the manual PDF behind the link, jump to page 25 and tell me if you think otherwise? It seems to be specifically supported in BIOS.

The board also supports 32 GB of RAM so I should be in pretty good shape expansion-wise.

As this is a microATX board, I'll be switching to a different case as well, so my prime choice is this case. Any thoughts on it?

2) CPU is an i5-4590 non-K version and Ark states it supports VT-d so I'd say that should cover my bases regarding direct IO feature?

3) Regarding ESXi compatibility, the card I mentioned previously is a used Intel card I had already ordered from ebay - I've verified that the backing controller is supported by ESXi, so my only (slight) worries are MBO/VT-d. Truth be told, VT-d's are a nice-to-have feature; I don't think it would be a noticeable hit on performance in my use case (correct me if I am wrong), so as long as ESXi installs, boots up and gives me access to my VMs, I will be ok.

4) Storage shouldn't be an issue - the VMs datastore is intended to be stored on a NAS (QNAP TS-469L) via NFS. The datastore will be backed by a RAID-5 array, the NAS unit supports LACP and I have an LACP-capable switch (Netgear GS108T) - do you think I am still running a risk on limited disk I/O?
In fact I am planning to chuck that 1TB out and use flash drive as bootstrap for ESXi (next question is on that).

5) I would like to put that flash drive inside the case - I've seen some motherboards sport an adapter that directly hooks to the USB 10-pin connector on motherboard but I've failed to find anything similar as a standalone product. How is that thing called and can you even buy it separately?

 

[Zarathustra[H]]

Therefore, I am now considering Gigabyte GA-Q87M-D2H - I have found here that this board seems to support VT-d instructions so could you please open the manual PDF behind the link, jump to page 25 and tell me if you think otherwise? It seems to be specifically supported in BIOS.

Judging from the manual that SEEMS to be the case, but I would be a lot more comfortable if I could find a specification line (like in intel CPU's) that said "VT-D support: Yes" or something like that.

There are plenty of stories of people struggling to get to work, or having working VT-D only to have a BIOS update and then it is gone.

The only reliable solution for VT-d/IOMMU in consumer hardware is - IMHO - AMD's 990FX chipset. All those boards seem to support it. Unfortunately I haven't really seen any micro-ATX or mini-ITX boards equipped with it though...

I mean, it sounds like you might not even need VT-d from what you are describing.

From personal experience though, these things never wind up being used the way they are intended for very long, and when you change things you are going to wish you had flexibility.

My server progression was as follows:

1.) Dell Zino HD mini cube (needed something small to serve and it was cheap)
2.) AMD E-350 Zacate build (needed a bigger case to fit drives and more expansion)
3.) AMD FX-8120 build. (needed more power and IOMMU)
4.) AMD Fx-8350 build. (thought I needed more power but really didn't)
5.) Dual Xeon HP server (in sig) (needed more than 32gigs of RAM.

If I had just bought the server I have now from the beginning, I would have saved myself a lot of time and money

I suggest leaving the door open for massive RAM upgrades and expansion space, or you'll kick yourself, like I have a number of times over the last few years.

 

[TType85]

1) As of time of this writing, it seems to me nothing but Q87 chipset natively supports VT-d capabilites. This came as a bit of a surprise to me, as I was certain the VT-d instructions on the CPU is the only requirement to make direct IO happen.

Therefore, I am now considering Gigabyte GA-Q87M-D2H - I have found here that this board seems to support VT-d instructions so could you please open the manual PDF behind the link, jump to page 25 and tell me if you think otherwise? It seems to be specifically supported in BIOS.

The board also supports 32 GB of RAM so I should be in pretty good shape expansion-wise.

As this is a microATX board, I'll be switching to a different case as well, so my prime choice is this case. Any thoughts on it?

2) CPU is an i5-4590 non-K version and Ark states it supports VT-d so I'd say that should cover my bases regarding direct IO feature?

3) Regarding ESXi compatibility, the card I mentioned previously is a used Intel card I had already ordered from ebay - I've verified that the backing controller is supported by ESXi, so my only (slight) worries are MBO/VT-d. Truth be told, VT-d's are a nice-to-have feature; I don't think it would be a noticeable hit on performance in my use case (correct me if I am wrong), so as long as ESXi installs, boots up and gives me access to my VMs, I will be ok.

4) Storage shouldn't be an issue - the VMs datastore is intended to be stored on a NAS (QNAP TS-469L) via NFS. The datastore will be backed by a RAID-5 array, the NAS unit supports LACP and I have an LACP-capable switch (Netgear GS108T) - do you think I am still running a risk on limited disk I/O?
In fact I am planning to chuck that 1TB out and use flash drive as bootstrap for ESXi (next question is on that).

5) I would like to put that flash drive inside the case - I've seen some motherboards sport an adapter that directly hooks to the USB 10-pin connector on motherboard but I've failed to find anything similar as a standalone product. How is that thing called and can you even buy it separately?

Search the boards I listed. ASrock C224/226 chipset boards should support VT-D, heck ASrock Z77's support VT-D. If you go mATX, go for a Supermicro board (something like a X10SLM+-F-O) and E3-1220v3 and ECC ram. Cost is not much more for a solid system. Server boards usually have a Type-A USB port on them. You can find adapters to take internal headers to USB Type-A

 

[_Gea]

If you go mATX, go for a Supermicro board (something like a X10SLM+-F-O) and E3-1220v3 and ECC ram. Cost is not much more for a solid system.

+1
SuperMicros are rock solid - no problem with ESXi.
Such a server class system offers all what you want: ECC, vt-d, enough fast pci-e slots, dual Intel Nics, and IPMI remote control.

If you think about vt-d (you mainly need this if you want to do storage virtualisation), you may think about a SuperMicro X10-SL7-F that adds a 8 channel high performance SAS/Sata controller (its like a LSI SAS 9207 that costs separatly nearly the price of the board)

 

[/usr/sbin]

edit: wrong thread, ignore this.

 

[dxun1]

OK, you've definitely swayed me towards Supermicro MBO, but since I've never dealt with server-grade hardware, you'll have to excuse my questions and ignorance.

Here's the current build I am considering:
Case: Fractal Design Core 1000 ($40 CAD)
Motherboard: SUPERMICRO MBD-X10SLM+-F-O uATX ($200 CAD)
CPU: Intel Xeon E3-1220v3 ($216 CAD)
CPU Cooler: CoolerMaster Hyper T4 ($20 CAD)
Memory: Crucial 16GB (2 x 8GB) ECC Unbuffered DDR3 ($194 CAD)
Storage: Sandisk Extreme USB 3.0 32 GB flash drive (for ESXi bootstrap) + NAS NFS-backed VM datastore
Power Supply: Antec High Current Gamer 400W (modular) ($44 CAD)
Network Card: Intel PRO/1000 PCI-e Gigabit Dual-Port LAN (EXPI9402PTBLK) ($62 CAD -- already bought over ebay, so can't return it)

Here is my current understanding, conclusions I have drawn from travelling around the net together with my questions - by all means, feel free to correct me at any time or point me to a better solution than I've chosen; just consider the goals of my home lab (silence, efficiency, affordability and low profile).

1) SuperMicro boards should be bullet-proof when it comes ESXi so I would expect zero problems whatsoever when installing the ESXi (VT-d and passthrough included). I also expect integrated LAN to work out of the box without any need of ESXi customization. Are there any gotchas here? I have found some people describing booting up ESXi from flash drive as....slightly problematic. What is your take on this? Can I use the SuperMicro USB Type-A receptacle for this purpose?

2) I've noticed there is a more powerful version of chipset available (C226 vs C224 which I've selected based on your recommendations). What is the material difference between the two and is it relevant for my use case? The best I could see on ARK is the C226 has more USB and PCI ports available.

3) These boards also have IPMI capability which I have never dealt with and this is the first time I've actually heard of it. As far as I can understand, the IPMI deals provides remote-management-over-LAN (my ad hoc coinage ) and there is a dedicated RJ-45 port on the MBO just for these purposes. I guess the board itself is running some form of web-server internally which enables you to access the remote management features from web browser. Given that, here are my questions:
3.1) How expansive are these remote management capabilities? What can you do with it and what do SuperMicro MBOs typically expose over this interface?
3.2) Since I do not plan on buying a dedicated graphics card, will I perhaps be able to install everyting from grounds up over LAN? The Xeon doesn't have integrated graphics but I can see the board has an onboard video (Aspeed AST2400) but only D-Sub output. If the system cannot be installed over LAN (which I would really like), does that mean I can only install it by hooking the box to a D-Sub-capable monitor?

4) I have opted for Cruicial ECC RAM but choice was really between these three modules. Since I won't be using huge RAM sizes ever (I don't consider 32 GB huge), I don't see the value of paying more for Buffered/Registered ECC RAM. Am I wrong in assuming that? Also, is there any difference in reputability between Crucial and Kingston (I personally regard the former as more reputable)?

5) Do you have any comments on my storage approach? Do you have any concerns about IO or configuration itself? FWIW, the NAS is backed by a collection of 3x4TB Seagate NAS drives arranged in RAID-5 array. My needs and expectations aren't too high but I generally like to extract every iota of performance from hardware so if there are any NFS-related configuration issues/experiences that you'd care to mention?

6) How does LACP configuration work in ESXi? From what I can gather, the configuration is done on vSphere client and hosted OSs are pretty much oblivious of link aggregation - they would just see a 2 GBit/s network link with a single IP.

7) Since I've already ordered and payed good money for the dedicated Intel LAN card, I would like to hear your suggestions how to put it to good use in my current setup. Mind you, one of the primary missions of this build is to function as a router/firewall, so extra LAN ports won't go to waste - I would just like them to use as best I can (e.g. to increase performance, efficiency or scalability) so if you have any suggestions, fire them this way.

 

[TType85]

Storage: Sandisk Extreme USB 3.0 32 GB flash drive (for ESXi bootstrap) + NAS NFS-backed VM datastore
Network Card: Intel PRO/1000 PCI-e Gigabit Dual-Port LAN (EXPI9402PTBLK) ($62 CAD -- already bought over ebay, so can't return it)

Go with a smaller USB stick (4-8GB) you will not be able to have a VM Datastore on it. It is just to boot/run ESXi
More network ports is better. You could pass the dual port through to your PFsense to really isolate it.

1) SuperMicro boards should be bullet-proof when it comes ESXi so I would expect zero problems whatsoever when installing the ESXi (VT-d and passthrough included). I also expect integrated LAN to work out of the box without any need of ESXi customization. Are there any gotchas here? I have found some people describing booting up ESXi from flash drive as....slightly problematic. What is your take on this? Can I use the SuperMicro USB Type-A receptacle for this purpose?

ESXi is slow to add network drivers. You are on newer hadware than when even 5.5 came out. See below, just make the custom ISO to install.
http://www.servethehome.com/install-vmware-esxi-5x-intel-i210-intel-i350-ethernet-adapters/

I have multiple Supermicro production servers that boot ESXi off USB and have had no issue.

One thing to note, server boards are much slower to post and have a longer boot sequence before getting to loading the OS.

3) These boards also have IPMI capability which I have never dealt with and this is the first time I've actually heard of it. As far as I can understand, the IPMI deals provides remote-management-over-LAN (my ad hoc coinage ) and there is a dedicated RJ-45 port on the MBO just for these purposes. I guess the board itself is running some form of web-server internally which enables you to access the remote management features from web browser. Given that, here are my questions:
3.1) How expansive are these remote management capabilities? What can you do with it and what do SuperMicro MBOs typically expose over this interface?
3.2) Since I do not plan on buying a dedicated graphics card, will I perhaps be able to install everyting from grounds up over LAN? The Xeon doesn't have integrated graphics but I can see the board has an onboard video (Aspeed AST2400) but only D-Sub output. If the system cannot be installed over LAN (which I would really like), does that mean I can only install it by hooking the box to a D-Sub-capable monitor?

IPMI can be used over the dedicated port or shared on the first port. You can do everything from remote (power up, power down, mount virtual iso's for install, etc) very powerful. Sever boards usually have built in graphics. It's not going to be something you are going to want to use for gaming but it displays text/gui. Once you use IPMI and get used to it you will love it. I have a SM machine in our disaster recover datacenter that I have full control of as long as it has power/network plugged in. ASRock uses the Aspeed which is graphics + IPMI I think.

4) I have opted for Cruicial ECC RAM but choice was really between these three modules. Since I won't be using huge RAM sizes ever (I don't consider 32 GB huge), I don't see the value of paying more for Buffered/Registered ECC RAM. Am I wrong in assuming that? Also, is there any difference in reputability between Crucial and Kingston (I personally regard the former as more reputable)?

I have Crucial in my boxes and it has been fine. E3 Xeons only support unbuffered ECC ram. 32GB max RAM (4x8GB). Thats where the older server come in. My T5500 (in the FS section) has 9 slots for ram (72GB using 8GB sticks). My Z800 has 12 slots and will take the 16GB modules.

7) Since I've already ordered and payed good money for the dedicated Intel LAN card, I would like to hear your suggestions how to put it to good use in my current setup. Mind you, one of the primary missions of this build is to function as a router/firewall, so extra LAN ports won't go to waste - I would just like them to use as best I can (e.g. to increase performance, efficiency or scalability) so if you have any suggestions, fire them this way.

The more ports the better. You could pass the NIC through to pfSense and isolate it from the rest just as if it was a physical box.

You are on the right track. You need to remember dealing with server hardware is a bit different than normal consumer grade stuff. There may be a few gotchas (some times ram computability is a pain) but once you get it up and running it's solid. I have machines that have been up over a year now without a reboot, running production systems.

 

[Hagrid]

Judging from the manual that SEEMS to be the case, but I would be a lot more comfortable if I could find a specification line (like in intel CPU's) that said "VT-D support: Yes" or something like that.

There are plenty of stories of people struggling to get to work, or having working VT-D only to have a BIOS update and then it is gone.

The only reliable solution for VT-d/IOMMU in consumer hardware is - IMHO - AMD's 990FX chipset. All those boards seem to support it. Unfortunately I haven't really seen any micro-ATX or mini-ITX boards equipped with it though...

I mean, it sounds like you might not even need VT-d from what you are describing.

From personal experience though, these things never wind up being used the way they are intended for very long, and when you change things you are going to wish you had flexibility.

My server progression was as follows:

1.) Dell Zino HD mini cube (needed something small to serve and it was cheap)
2.) AMD E-350 Zacate build (needed a bigger case to fit drives and more expansion)
3.) AMD FX-8120 build. (needed more power and IOMMU)
4.) AMD Fx-8350 build. (thought I needed more power but really didn't)
5.) Dual Xeon HP server (in sig) (needed more than 32gigs of RAM.

If I had just bought the server I have now from the beginning, I would have saved myself a lot of time and money

I suggest leaving the door open for massive RAM upgrades and expansion space, or you'll kick yourself, like I have a number of times over the last few years.

I believe most if not all of the 970's also have it. Both of my 970 boards do.

 

[dxun1]

Go with a smaller USB stick (4-8GB) you will not be able to have a VM Datastore on it. It is just to boot/run ESXi

Unfortunately, I could not find a smaller stick at my local dealer (Memory Express) nor at the newegg.ca. However, for 29.99, I think it's a good deal, even though most of the space will alas go to waste. The drive popped up as the recommended one at a good number of reviews I've seen (e.g. here) but I guess I could also buy a Patriot Supersonic Boost XT 8 GB drive for boot purposes. I am basically at a loss how to figure out what kind of flash is the manufacturer using so I am looking at the advertised/tested speeds plus subjective reputability factor to get an idea how much of a trash the NAND under the bonnet is.
I admit my method of estimation is unscientific to say the least but I don't know of any better. However, I would be quite...displeased to find the stick just failing to boot after being plugged-in for X months and having virtually no write wear at all.

Speaking of which, that brings me to disaster recovery.

So on the VM datastore side, I find things should be fairly straight forward - as long as I perform periodic VM snapshots (so that I can recover back from misconfiguration or similar misshaps) and take care of my RAID array, I should be good.
However, should the boot drive fail, is it enough simply to toss the failed USB drive, find a new one, install ESXi on it, restore configuration and start VMs? How do you backup your vSphere configuration - what are best practices?

Also, is upgrade to new ESXi version as painless as I've described it here?

More network ports is better. You could pass the dual port through to your PFsense to really isolate it.

How would I go about doing that - are there any tutorials/info you can point me to? I surmise this is what VT-d is for and I'll be effectively removing the support for the given ports in the ESXi host but if I pass through a couple of ports, won't they show up at all the host OSs as unknown network ports? How do you "focus" your passthrough to a singular VM (namely, pfSense)?

ESXi is slow to add network drivers. You are on newer hadware than when even 5.5 came out. See below, just make the custom ISO to install.
http://www.servethehome.com/install-vmware-esxi-5x-intel-i210-intel-i350-ethernet-adapters/

Understood. I'll customize an ESXi ISO then - it doesn't appear to be particularly troublesome.

IPMI can be used over the dedicated port or shared on the first port. You can do everything from remote (power up, power down, mount virtual iso's for install, etc) very powerful. Sever boards usually have built in graphics. It's not going to be something you are going to want to use for gaming but it displays text/gui. Once you use IPMI and get used to it you will love it. I have a SM machine in our disaster recover datacenter that I have full control of as long as it has power/network plugged in.

So, the way I see it this - after finishing the build, I'll hook up the dedicated IPMI port directly to switch. The IPMI will be accessible via static IP and as long as there is power to the motherboard. From there I can power off/on the machine, mount ISOs, and install stuff. Correct?

I have Crucial in my boxes and it has been fine. E3 Xeons only support unbuffered ECC ram. 32GB max RAM (4x8GB). Thats where the older server come in. My T5500 (in the FS section) has 9 slots for ram (72GB using 8GB sticks). My Z800 has 12 slots and will take the 16GB modules.

So for my particular SM motherboard, I have found the following modules are explicitly compatible but I cannot see Crucial among them. Will this be an issue?

Also, I am correct in assuming this is DDR3L memory, since it's operating on 1.35V (even though the page doesn't explicitly name it as such)?

 

[Tony Ou]

I also had trouble finding relatively efficient, low-powered ATX PSUs - since the machine will be lightly stressed most of the time, I don't expect I would exceed 70-80 W on average and there is very little in the way of reliable PSUs with efficiency > 80% in that power range, especially given the price of Antec PSU being considered here ($45). Even if I wanted to consider them, SFX power supplies are even worse - cheapest Silverstone 450W SFX PSU is priced around $70 and I don't even need such vast power. Seasonic has a few SFX PSUs in 300W-power range but they are more expensive and less efficient than this Antec here. Again - any suggestions/thoughts here would be most welcome.

We also have a 300W SFX PSU that is best in class and cost around $50. For your setup, it'll likely be running in fanless mode most of the time if the ambient temperature is low enough.

 

[dxun1]

We also have a 300W SFX PSU that is best in class and cost around $50. For your setup, it'll likely be running in fanless mode most of the time if the ambient temperature is low enough.

Thanks for pointing that out, however, at the time it was very hard to get hold of this PSU and the Antec HCG-400 was a better deal (price was lower to begin with, it was available immediately, there were no shipping costs associated and I honestly don't think any efficiency differential would make up for the increased costs and the hassle of going through the hoops).

Is there a reason there is no high-efficiency ~200W SFX PSUs available? I would imagine that kind of PSU would be driving a lot of small-scale rigs like mine.

I am actually surprised SFX PSUs are so hard to come by, especially given the rise of HTPC and NUCs.

 

[Zarathustra[H]]

If you need a low power super efficient small PSU, check out the picoPSU from Mini-Box.

 

[dxun1]

If you need a low power super efficient small PSU, check out the picoPSU from Mini-Box.

Thanks, yes, I was also considering pico but what ruled it out for me is the following:
1) their top units were underpowered (150 W - I'd need at least 50-100 W more to have some healthy headroom)
2) I didn't want external bricks, especially with noisy fans (such was the 120W unit, at least the one tested by SPCR)
3) I was uncomfortable with bare electronics and wires in the case
4) power efficiency is not that superior - Antec HCG-400 was measured with ~82% eff. at 40 W, whereas picoPSU was measured by SPCR at ~85% at same power. Considering the former is a German review, it was probably tested at 220 V, so my assumption is Antec's efficiency at 40 W/110 V is around 78-79%. That is a ~3 W difference in favour of picoPSU and frankly that much difference even for 24/7 operation seemed like too small a gain to fiddle with it. I didn't run the numbers but that would seem like a difference of ~1$/month for 24/7 operation.
4) to top it all, picoPSU is more expensive than Antec, consider this - so by taking rough estimate from above it would've taken at least 1.5 years of continuous operation just to break even and for what? For virtually no power headroom (Xeon is 80 W, MBO 20 W, RAM 10-20 W, plus ~10 W for fans and other), operating at sub-optimal power range (50% is usually the most efficient), having cables exposed, an extra brick, a non-standard solution and small difference in noise (Antec should be very quiet at 40 W).

I concluded picoPSU just offered too little for my use case so I removed it from my considerations.

Now, had the picoPSU had +95% efficiency at all power loads and had it cost 50% of Antec, I would have seriously considered it.....but you can't beat physics; I am a software, not an electrical engineer but it is obviously very hard to produce PSUs with such high efficiency at low loads and with low prices which is (answering my own question from previous post) probably the reason why we don't see 200-250 W SFX high efficiency PSUs being offered.

 

[TType85]

Unfortunately, I could not find a smaller stick at my local dealer (Memory Express) nor at the newegg.ca. However, for 29.99, I think it's a good deal, even though most of the space will alas go to waste. The drive popped up as the recommended one at a good number of reviews I've seen (e.g. here) but I guess I could also buy a Patriot Supersonic Boost XT 8 GB drive for boot purposes. I am basically at a loss how to figure out what kind of flash is the manufacturer using so I am looking at the advertised/tested speeds plus subjective reputability factor to get an idea how much of a trash the NAND under the bonnet is.
I admit my method of estimation is unscientific to say the least but I don't know of any better. However, I would be quite...displeased to find the stick just failing to boot after being plugged-in for X months and having virtually no write wear at all.

Speaking of which, that brings me to disaster recovery.

So on the VM datastore side, I find things should be fairly straight forward - as long as I perform periodic VM snapshots (so that I can recover back from misconfiguration or similar misshaps) and take care of my RAID array, I should be good.
However, should the boot drive fail, is it enough simply to toss the failed USB drive, find a new one, install ESXi on it, restore configuration and start VMs? How do you backup your vSphere configuration - what are best practices?

Also, is upgrade to new ESXi version as painless as I've described it here?

The USB drive for ESXi I use are usually just cheap ones, I have yet to have one fail on me. I cloned an image of one to have just in case but so far have not had to use it. Speed is not that important as the USB stick is JUST used to load ESXi (which stays in memory, as far as I know the stick rarely gets hit after loading)

How would I go about doing that - are there any tutorials/info you can point me to? I surmise this is what VT-d is for and I'll be effectively removing the support for the given ports in the ESXi host but if I pass through a couple of ports, won't they show up at all the host OSs as unknown network ports? How do you "focus" your passthrough to a singular VM (namely, pfSense)?

VT-D will allow you to pass the whole card through to a VM so it appears ONLY to that VM and it has direct access to it, ESXi can no longer see/use it. In the "Advanced" area of the configuration on ESXi you can set up passthrough, when you create/edit the VM you can then add a PCI device and your hardware you have marked as passthrough will be listed and be able to be select. pfSense will be pretty much plug and play with it.

So, the way I see it this - after finishing the build, I'll hook up the dedicated IPMI port directly to switch. The IPMI will be accessible via static IP and as long as there is power to the motherboard. From there I can power off/on the machine, mount ISOs, and install stuff. Correct?

Thats pretty much how it works. IPMI will pick up a DHCP address by default and the board will come with a program to find the address and connect to it. You can also connect to it in the web browser and work on it.

So for my particular SM motherboard, I have found the following modules are explicitly compatible but I cannot see Crucial among them. Will this be an issue?

Also, I am correct in assuming this is DDR3L memory, since it's operating on 1.35V (even though the page doesn't explicitly name it as such)?

Safest bet is to stick to the modules SM says will work, or if you can find reviews that says a certain modules works. Server parts are usually a bit particular about RAM. It looks like the newer SM boards support the 1.35v RAM, which is DDR3L