New Home Lab

[luckylinux]

I'd like to create a virtualisation server at home (ESXi if possible). Xenserver would also be an option (though non-supported guests are known to be VERY slow). I'm a bit confused though: ESXi is free but the Vcenter control management software is only o 60 days trial? Would Xenserver be better at that point? I'd like to hear opinions on this if possible.

First I though it might be fun to use a dual socket motherboard plus 2 CPUs, but since its quite costly and might well do the same with 2 builds (which will perform better as well, since higher CPU frequencies) in the end (after seing that most of your virtualisation labs are quite "modest") I though of these possibilities:

Configuration 1

• 1 x ASUS M5A99X EVO ~ 130$
• 1 x AMD FX-8120 ~ 210$ (I think of overclocking it back to the FX-8150 level, not much further - if any)
• 4 x 4GB Kingston KVR1333D3E9S/4GEC ECC-unbuffered memory (already tested it works with this motherboard) ~ 4 x 40$

Configuration 2

• 1 x Intel DQ67OW ~ 120$
• 1 x Intel I7-2600 (non-K because of its lack of VT-D support)
• 4 x 4 GB patriot/corsair budget memory of 4 x 8 GB corsair high-end memory (like Vengeance for instance) ~ 280$

(Common to both)
Power supply to be discussed (I own a Corsair AX750 which is overkill for another server I described here) or maybe a Bequiet efficient 400-500W PSU.
I don't plan on using HDD since storage will be done in the network (Solaris 11 supports SAN / VMWARE Esxi supports SAN ?). Just an USB stick should do the job (basically just hold the bootloader and a few hundreds MB). I don't know if I should use SSD caching (60GB is not too expensive for me), though it may wear the SSD overtime. In this case I wouldn't need the USB stick.
A few Intel NICs anyway are required by both configurations.

Now I'm sure everyone will say the FX-... is not so good.
If I sum it up these are pros of one versus the other
Configuration 1

• ECC support (unbuffered) for "free" (compared to Intel's Xeon E3 series where this costs A LOT)
• 8 core support
• Overclock (though not too strong)
• 6 PCI/PCIe slots
• Power consumption high at LOAD (only)
• Integrated non-INTEL NIC (just trash IMHO)

Configuration 2

• Power consumption very limited (both IDLE & LOAD)
• Integrated INTEL NIC
• 4 cores / 8 threads CPU
• Overclock: this CPU may not be "recycled" or resold since its marked its pretty small I think (all enthusiast buy the I7-2600K)
• No ECC support
• 4 PCI/PCIe slots (which means I'd have to buy the INTEL dual port NIC which is very expensive ~ 160$ compared to ~ 30$ single NIC)

Both platforms may upgrade to 32GB RAM (though not ECC, since 8gb unbuffered ECC sticks are VERY expensive).
What will this computer be used for you may ask ? I'd say quite a few

• Gentoo compilation This is definitively my favorite
• Standard testing platform
• Virtualized media server
• Folding @ Home

I may in the end buy 2 of those. I think I have to make an inventory of my computers otherwise I'm gonna get lost here

Do you have any special reccomendation / platform change ? I know the case is very big therefore it acheives very low density but I'm not so happy to buy small sized ATX cases to pile one over the other (like this for instance), since they're pretty expensive for what they offer IMHO.

Thank you very much in advance for your help.

 

[pitne]

how is ecc support work with the configuration 1, like why do you say it's free? Unbuffered ECC costs the most doesnt it?

 

[Tiporaro]

I think he means that he doesn't need a server grade mb/CPU to enable ecc support. That being said, being able to get 32gb on 4 ecc dimm slots is definitely a premium, so it's a double edged sword and should be considered as to whether your application is worth the ecc costs.

 

[Child of Wonder]

Don't worry about ECC, just get the Intel board, 32GB of desktop RAM ($90 for 2x8GB of GSkill at Newegg right now), and either an i5-2400 or i7-2600 (depending on whether HT is important to you or not). Then get yourself a nice Intel dual or quad port NIC off ebay and slap it in there.

Cost per host:

Intel mobo - $120
i7-2600 - $275
32GB RAM - $180
Intel quad port NIC off ebay - $125

Total $700 per host

If you don't care about HT, just get the i5-2400. Microcenter has it for $150 right now. You'll run out of RAM before you run out of CPU power.

 

[Netwerkz101]

<snipped>
I'd like to create a virtualisation server at home (ESXi if possible).
Xenserver would also be an option (though non-supported guests are known to be VERY slow).
I'm a bit confused though: ESXi is free but the Vcenter control management software is only o 60 days trial?
Would Xenserver be better at that point? I'd like to hear opinions on this if possible.

<snipped>

XenServer vs. ESXi ?

If you already know your primary guest will have performance issues with XenServer = vote for ESXi.

A couple of benefits to XenServer:
- 82579LM NIC works out the box for the Intel board you are looking at.
- XenMotion is included in the "Free" version - but you need 2+ hosts

With a single host only, you only need the included management client (XenCenter or vSphere Client).

Compare the DQ67SW to the DQ67OW

You may want USB3 and Firewire ports.

 

[luckylinux]

Thanks a bunch for the replies.

how is ecc support work with the configuration 1, like why do you say it's free? Unbuffered ECC costs the most doesnt it?

Nope. Asus AM3(+) motherboards support unbuffered ECC almost "for free" and unbuffered ECC (around 40$/each) 4gb sticks comes almost for the same price as non-ECC (around 35$/each). Registered ECC memory doesn't cost really that much, but it require a MB & CPU that put together cost around 2 times MORE (3 times in total) the unbuffered ECC configuration. That's what I meant.

I think he means that he doesn't need a server grade mb/CPU to enable ecc support. That being said, being able to get 32gb on 4 ecc dimm slots is definitely a premium, so it's a double edged sword and should be considered as to whether your application is worth the ecc costs.

Correct. 8gb ECC unbuffered sticks are around 200$ instead of 80$ for the ECC registered ones. Not sure if ECC is worth "the trouble" in fact. Surely I want it for my storage server where I want to avoid data corruption. However on the internet you always find that although it's not "mandatory", it's better to have it for the day where actually something goes wrong. 32gb unbuffered ECC is not really an option. However 32gb non-ECC cost around 280$, so you're right: if I run out of memory with 16GB unbuffered ECC, will have to switch to 32GB non-ECC.

Don't worry about ECC, just get the Intel board, 32GB of desktop RAM ($90 for 2x8GB of GSkill at Newegg right now), and either an i5-2400 or i7-2600 (depending on whether HT is important to you or not). Then get yourself a nice Intel dual or quad port NIC off ebay and slap it in there.

Cost per host:

Intel mobo - $120
i7-2600 - $275
32GB RAM - $180
Intel quad port NIC off ebay - $125

Total $700 per host

If you don't care about HT, just get the i5-2400. Microcenter has it for $150 right now. You'll run out of RAM before you run out of CPU power.

You mean that for this particular application I suppose. Not for EVERY application, do you?
Thanks for the reply. Sorry but newegg doesn't ship overseas so it's not really an option.
Will look for an online retailer then. I must say hovewer I always had some bad RAM even at my local retailer, so maybe it's better to go with them since I'd not pay for shipping back to RMA.

Not so sure about the ebay buying procedure. From Germany plus shipping I'd save 40-50% of the price here. How would the RMA procedure work then? Should I directly ship to Intel?

Regarding your suggestion: RAM is a little more expensive here and if I'd gone the Configuration's 1 route I could've use standard 30$ NICs (from INTEL).

XenServer vs. ESXi ?

If you already know your primary guest will have performance issues with XenServer = vote for ESXi.

A couple of benefits to XenServer:
- 82579LM NIC works out the box for the Intel board you are looking at.
- XenMotion is included in the "Free" version - but you need 2+ hosts

With a single host only, you only need the included management client (XenCenter or vSphere Client).

Compare the DQ67SW to the DQ67OW

You may want USB3 and Firewire ports.

For instance Gentoo is NOT supported. The Intel Board's NIC will not work with Esxi? Or you said that because you're using it under Xenserver?
About the need for firewire I'd say no (always had the firewire ports but never used them since I have no device using that protocol).
About USB 3 you're right. I'll definitively take the DQ67SW in consideration as USB 3 might be nice (around the same price).

Still it is unclear why some users in the forum I linked used a trial version of the management application. ESXi is free to use but it's management client is not?
Don't worry, I may have a 2nd virtualisation host sooner than you think , therefore is there a management client with >= 2 host support that can be used for free?

 

[bmh.01]

Still it is unclear why some users in the forum I linked used a trial version of the management application. ESXi is free to use but it's management client is not?
Don't worry, I may have a 2nd virtualisation host sooner than you think , therefore is there a management client with >= 2 host support that can be used for free?

No the vsphere management client is free, vcenter is a seperate product that is payed license only. Although no vmotion etc without it, for a home lab you can find licenses easily enough if you know where to look. I'll say no more.

 

[luckylinux]

No the vsphere management client is free, vcenter is a seperate product that is payed license only. Although no vmotion etc without it, for a home lab you can find licenses easily enough if you know where to look. I'll say no more.

No need to. You explained quite well I think

Just wonder if the vmware vsphere client support multiples machines or I should connect each time to a different host, etc

@bmh.01 what do you think about Xenserver and/or Xen + Convirture open source? Have any experiences with them? Or simply Vmware Esxi was so good you don't need to look further around?

From what I understood, Vmware Esxi doesn't allow programmers to use their API and/or create programs that use their CLI. Is that correct? Xen might be better at this, though Xen only runs good with supported guests, but this page seems to tell otherwise (now).

 

[Vader]

Just wonder if the vmware vsphere client support multiples machines or I should connect each time to a different host, etc

Without vCenter you'll have to connect to your hosts separately using the vSphere client.

 

[luckylinux]

Without vCenter you'll have to connect to your hosts separately using the vSphere client.

What about APIs? Can they be used for free? Can I use the CLI directy if I ever would like to write an application which interacts with Esxi?

Btw how would PWM fans behave with Esxi (home build lab, not Dell's or IBM's which allow IPMI sensors)? Any experience? Thank you.

 

[Vader]

The free version, ESXi Hypervisor, does not allow write access using the API's via CLI, PowerCLI commands..etc. It only allows read/write access via the vsphere client.

If you want access to the API's for "write" commands such as powering on a VM etc, you'll have to change your licensing.

 

[-Dragon-]

If your budget is... lets say "flexible" and you're not in a super hurry to get this up and running you may want to hold off and check out some of the Xeon E5 16xx amd 26xx series chips based on the SB-E socket. While quad channel RAM is a bit overkill in the gaming arena, it allows down right monsterous RAM configurations in servers, and has the added benefit of using the much cheaper RDIMMs instead of the UDIMMs if you decide you really do want ECC, so the extra you spend in the MB/CPU would most likely be saved on RAM and then some.

The SB-E Xeon parts seem to be in tight supply right now (MBs and such at least) but supporting 128-384GB RAM per CPU, and 4x8GB RDIMMs being practically cheap would make for an amazing and long lasting lab.

For the record I have something similar to what you're considering with 2 Xeon E3 boxes with 16GB each and they're frankly feeling cramped. When the SB-E E5's stock and prices stabilize in a few weeks to a month I can see myself easily pulling the trigger on a dual E5-2620 box with 64GB to actually give me room to "play".

I want to see if it can play Crysis...

Remotely...

With multiple sessions.

 

[Child of Wonder]

You mean that for this particular application I suppose. Not for EVERY application, do you?

Yes. ECC is not necessary for a home lab.

 

[Tiporaro]

If your budget is... lets say "flexible" and you're not in a super hurry to get this up and running you may want to hold off and check out some of the Xeon E5 16xx amd 26xx series chips based on the SB-E socket. While quad channel RAM is a bit overkill in the gaming arena, it allows down right monsterous RAM configurations in servers, and has the added benefit of using the much cheaper RDIMMs instead of the UDIMMs if you decide you really do want ECC, so the extra you spend in the MB/CPU would most likely be saved on RAM and then some.

The SB-E Xeon parts seem to be in tight supply right now (MBs and such at least) but supporting 128-384GB RAM per CPU, and 4x8GB RDIMMs being practically cheap would make for an amazing and long lasting lab.

For the record I have something similar to what you're considering with 2 Xeon E3 boxes with 16GB each and they're frankly feeling cramped. When the SB-E E5's stock and prices stabilize in a few weeks to a month I can see myself easily pulling the trigger on a dual E5-2620 box with 64GB to actually give me room to "play".

I want to see if it can play Crysis...

Remotely...

With multiple sessions.

hahah, the E5's look very tempting. I similarly felt a little constrained by a E3 build I had - not that it was bad, but the 1155 platform is somewhat restrictive in memory options and # of pci-e lanes.

If the 1620 had been out at launch I would have strongly considered it, but didn't have time to wait out for it. That being said, going with the E5's invokes the corresponding server boards that are going to run a premium, especially depending on what type of applications you're running for (more robust pci-e slot setups are not going to be cheap). For a home lab I'm experimenting with the X79 desktop platform and couldn't be happier so far - cheaper mb (than the E5 server boards, not than E3/1155), 8 dimm's non-ecc, and easy vmdirectpath passthrough. VGA passthrough is a new addiction =)

 

[CEpeep]

What about APIs? Can they be used for free? Can I use the CLI directy if I ever would like to write an application which interacts with Esxi?

No API access, but you can use the shell commands over SSH. Not what you would want to do in a production environment, but for a lab it should be ok.

 

[GigaPie]

If your budget is... lets say "flexible" and you're not in a super hurry to get this up and running you may want to hold off and check out some of the Xeon E5 16xx amd 26xx series chips based on the SB-E socket. While quad channel RAM is a bit overkill in the gaming arena, it allows down right monsterous RAM configurations in servers, and has the added benefit of using the much cheaper RDIMMs instead of the UDIMMs if you decide you really do want ECC, so the extra you spend in the MB/CPU would most likely be saved on RAM and then some.

The SB-E Xeon parts seem to be in tight supply right now (MBs and such at least) but supporting 128-384GB RAM per CPU, and 4x8GB RDIMMs being practically cheap would make for an amazing and long lasting lab.

For the record I have something similar to what you're considering with 2 Xeon E3 boxes with 16GB each and they're frankly feeling cramped. When the SB-E E5's stock and prices stabilize in a few weeks to a month I can see myself easily pulling the trigger on a dual E5-2620 box with 64GB to actually give me room to "play".

I want to see if it can play Crysis...

Remotely...

With multiple sessions.

Maybe if the prices came down on that, it would be a great bet for myself too. The CPU/MB will cost you almost 1k alone.
Looks like all LGA2011 boards are dual socket as well. Would be easy to upgrade the CPU in the future, though 6 cores/12 threads is quite a bit. The temptation of having more then 32GB of RAM is there though.

 

[luckylinux]

hahah, the E5's look very tempting. I similarly felt a little constrained by a E3 build I had - not that it was bad, but the 1155 platform is somewhat restrictive in memory options and # of pci-e lanes.

If the 1620 had been out at launch I would have strongly considered it, but didn't have time to wait out for it. That being said, going with the E5's invokes the corresponding server boards that are going to run a premium, especially depending on what type of applications you're running for (more robust pci-e slot setups are not going to be cheap). For a home lab I'm experimenting with the X79 desktop platform and couldn't be happier so far - cheaper mb (than the E5 server boards, not than E3/1155), 8 dimm's non-ecc, and easy vmdirectpath passthrough. VGA passthrough is a new addiction =)

There's a bug concerning the 2011 socket for desktop CPUs: VT-D doesn't work correctly. That's why Intel released a new stepping revision.

If your budget is... lets say "flexible" and you're not in a super hurry to get this up and running you may want to hold off and check out some of the Xeon E5 16xx amd 26xx series chips based on the SB-E socket. While quad channel RAM is a bit overkill in the gaming arena, it allows down right monsterous RAM configurations in servers, and has the added benefit of using the much cheaper RDIMMs instead of the UDIMMs if you decide you really do want ECC, so the extra you spend in the MB/CPU would most likely be saved on RAM and then some.

The SB-E Xeon parts seem to be in tight supply right now (MBs and such at least) but supporting 128-384GB RAM per CPU, and 4x8GB RDIMMs being practically cheap would make for an amazing and long lasting lab.

For the record I have something similar to what you're considering with 2 Xeon E3 boxes with 16GB each and they're frankly feeling cramped. When the SB-E E5's stock and prices stabilize in a few weeks to a month I can see myself easily pulling the trigger on a dual E5-2620 box with 64GB to actually give me room to "play".

I want to see if it can play Crysis...

Remotely...

With multiple sessions.

No game server in my plans ... however I'm more tempted in a 2 x single socket configuration than 1 x dual socket, because CPUs running in dual socket configuration cost about the same but have a lower frequency (to compare: 2.0GHz vs 3.1GHz), as we've seen for the 1366 socket.

I'm not in a hurry since I burned my budget for this month on some books and a few SSDs.
However the E5-16xx hasn't even shown up on my local retailer's site and the E5-26xx have too low frequency for what I'd take as acceptable. Furthermore 2 x MB means some kind of "redundancy": if one VM host breaks I can transfer its VMs on another one. Costly-wise it's about the same (except for the case).
For a single motherboard configuration I could go with a not-so-costly-efficient Silverstone GD01 case, since they can be put one over the other. That allows ATX motherboards (and microATX) as well as up to 135mm CPU cooler heigth. This for ~ 120$ for each case.
Plus RAM (~ 4x80$), CPU (~ 600$), MB (~ 300$), PSU (~ 70$), graphic card (? ~ 30$), NICs (several $$) for each build. Not so bad. A bit of a shame that this time 6x2.0GHz dual socket compatible are for ~ 480$ while 6 x 3.1GHz single socket are for ~ 600$ though. With lga1366 it was not like this (at least not right now). Surely I can not fire up to 24 threads for each VM but I don't think I'll need to.
What would you do? 2 x single socket or 1 x dual socket? Thanks

 

[Tiporaro]

There's a bug concerning the 2011 socket for desktop CPUs: VT-D doesn't work correctly. That's why Intel released a new stepping revision.

Yes, the c1 revisions of the 3930/60 didn't have vt-d. C2 revisions and the 3820 have vt-d working just fine, assuming the mb has it implemented correctly. I can personally vouch for this on at least Asus' X79 line.

 

[-Dragon-]

No game server in my plans ... however I'm more tempted in a 2 x single socket configuration than 1 x dual socket, because CPUs running in dual socket configuration cost about the same but have a lower frequency (to compare: 2.0GHz vs 3.1GHz), as we've seen for the 1366 socket.

I'm not in a hurry since I burned my budget for this month on some books and a few SSDs.
However the E5-16xx hasn't even shown up on my local retailer's site and the E5-26xx have too low frequency for what I'd take as acceptable. Furthermore 2 x MB means some kind of "redundancy": if one VM host breaks I can transfer its VMs on another one. Costly-wise it's about the same (except for the case).
For a single motherboard configuration I could go with a not-so-costly-efficient Silverstone GD01 case, since they can be put one over the other. That allows ATX motherboards (and microATX) as well as up to 135mm CPU cooler heigth. This for ~ 120$ for each case.
Plus RAM (~ 4x80$), CPU (~ 600$), MB (~ 300$), PSU (~ 70$), graphic card (? ~ 30$), NICs (several $$) for each build. Not so bad. A bit of a shame that this time 6x2.0GHz dual socket compatible are for ~ 480$ while 6 x 3.1GHz single socket are for ~ 600$ though. With lga1366 it was not like this (at least not right now). Surely I can not fire up to 24 threads for each VM but I don't think I'll need to.
What would you do? 2 x single socket or 1 x dual socket? Thanks

I don't really want to make mine a game server either I just want to see if it's beastly enough to stream multiple instances of a game to multiple RDP sessions.

As far as what kind of configuration to go with it highly depends on what you intend to do with your lab, and how much you're willing to spend to do it. For me I wanted to run a domain on my home network and play around with some of the MS servers and network management apps, basically download everything I can off technet and play around with it, including all the forefront and system center apps. MS stuff isn't exactly lax on memory demands either.

I opted for 2x single sockets to start with to give myself both hardware and software redundancy, and after a few of the problems I've encountered in the past I totally see why they recommend having an odd number of redundant servers, but like I said I'm feeling the memory pinch now @ 16GB per server and the cost to upgrade to even 24GB each is several hundred dollars.

So to (try) to answer your question you need to answer a few first:

How important will your lab be to your day to day life? Is it going to be something you fire up when you're bored and shut down when you're done or are you going to do some real world testing and use it as a file server, have it manage your network, redirect your mail from your current host to a mail server on premises, or install some sort of firewall/routing OS to a VM and actually run your internet connection through it? If the server being down could actually negatively impact your day to day then you probably want hardware redundancy, if you're not gonna go quite that crazy, then I'd probably say go with one big one, you can still do software redundancy in a single piece of hardware, but it's a lot easier when you have one big memory pool than trying to juggle VMs around trying to turn that last 1GB you have in each system into 0GB in one and 2GB in the other to make room for that test client VM you'd like to setup.

If you need more than one set of hardware then you need to ask yourself how much you're willing to spend on all of this. Right now you're asking if you should do 2x medium sized servers or 1x mega server but maybe the right answer is one or two small servers along side a mega server. Lets say you intend to do a setup similar to mine where you have reundant domain controllers (2x), mail servers (2x), and a firewall VM (1x). Of all the VMs I run those 5, or at least 3 functioning ones, are the most important and if I had to I could probably cram them all into an 8GB system (would be tight cause exchange and tmg are mem hogs but servicable in a pinch, although with the price of non ECC RAM not really much reason NOT to go 16GB). These are all fairly light on CPU demand too so you could probably go with a cheaper processor as well and save a few hundred bucks, and VT-d support would be optional since they don't really have high IO demands. The right answer may be to have one smaller system like a cheap i5-2300 to run your redundant VMs and either an E5 system if you really want to go crazy or a more reasonable i7-3820 system with 64GB RAM.

As far as what I would do if I had it to do all over again, knowing what I know now and with the current products on the market, I'd definitely go one E3-1230 w/ 16GB and a dual E5-2620 w/ 64GB and probably a GTX560Ti for remotefx, possibly a second E3, but if not then I'd definitely still want it (tho less than I'm wanting the dualy E5 currently). I would NOT go with consumer chips/boards and non ECC RAM, because to me it just doesn't feel right and the price difference between going cheap and "doing it right" is fairly small relative to the full system price either way. Also I LOVE the IPMI's on my SM boards since the servers are stashed away without any monitors or KBs.

Slightly OT but another thing to consider if you haven't is you likely want a decent L2 or L3 switch to connect all of this to, you seem to think you need a lot of LAN connections, which if you have a switch that supports VLANs you can eliminate/combine a lot of networks that might otherwise actually require different connections. For my home lab I personally regret the extra network cards I got and would not buy them again if I had the choice. With a good switch and some VLAN planning you can easily get by with one NIC, 2 is a luxury (again for home use)

 

[luckylinux]

Dragon, thanks for the (very long) reply. And I though I was the only one writing LONG posts and texts all the time

How important will your lab be to your day to day life? Is it going to be something you fire up when you're bored and shut down when you're done or are you going to do some real world testing and use it as a file server, have it manage your network, redirect your mail from your current host to a mail server on premises, or install some sort of firewall/routing OS to a VM and actually run your internet connection through it? If the server being down could actually negatively impact your day to day then you probably want hardware redundancy, if you're not gonna go quite that crazy, then I'd probably say go with one big one, you can still do software redundancy in a single piece of hardware, but it's a lot easier when you have one big memory pool than trying to juggle VMs around trying to turn that last 1GB you have in each system into 0GB in one and 2GB in the other to make room for that test client VM you'd like to setup.

Let's say it's not really mission critical. I have a "dedicated" server with Intel I5-2500K (no ECC, I know ) which will NOT be virtualized, which will do most operations (like hosting a SVN server, WEB server, ...). Maybe I can set up some kind of synchronisation between them or something (between this and one VM) for redundancy. Not sure how to do this with GNU/Linux (actually, not sure how to do it with any OS ).
To sum up: if the VM server is down a day I can live with it. Might be interesting to know if I can daily backup its configuration (not data, since that is centralized) so that if it fails I can access the VMs from ther other server running Esxi.

Let's say I tend to play a LOT with Linux distributions and might begin with FreeBSD as well. Gentoo distributed compiling would be fine as well I know, many people say Gentoo is "bad" for its compile times and I don't really look for optimisations, but I really like its versioning features (like choising one version, compile, test, install and see if it works) and in the worst case scenario I can always choose another version of the packages to install. That way I update my VM first, compile everything, see if it works and only then install packages on the client PCs (which are at least quad core, but why loose time and stability when you can test in a VM hosted in another server ?).

I think the Supermicro X9SRA is quite good as a motherboard (~ 360$ which is a bit more than others, but provides more expandibility: 2xPCIe 3.0 x16, 2 x PCI-E 3.0 x4 (in x8 slot) , 1 x PCI-32). Other motherboards provide IPMI which may be nice, though they provide less expansion slots (only 3, see the Supermicro X9SRi-F for ~ 410$).
Have to decide if it would be better to have IPMI or more expansion slots. The mixed alternative with PCI-X slots is not available here ATM (though I'm not really sure you can fit a PCI-32 NIC inside a PCI-X slot ).

If you need more than one set of hardware then you need to ask yourself how much you're willing to spend on all of this. Right now you're asking if you should do 2x medium sized servers or 1x mega server but maybe the right answer is one or two small servers along side a mega server. Lets say you intend to do a setup similar to mine where you have reundant domain controllers (2x), mail servers (2x), and a firewall VM (1x). Of all the VMs I run those 5, or at least 3 functioning ones, are the most important and if I had to I could probably cram them all into an 8GB system (would be tight cause exchange and tmg are mem hogs but servicable in a pinch, although with the price of non ECC RAM not really much reason NOT to go 16GB). These are all fairly light on CPU demand too so you could probably go with a cheaper processor as well and save a few hundred bucks, and VT-d support would be optional since they don't really have high IO demands. The right answer may be to have one smaller system like a cheap i5-2300 to run your redundant VMs and either an E5 system if you really want to go crazy or a more reasonable i7-3820 system with 64GB RAM.

Yeah: the I5-2xxx series is "relatively" cheap (even the i7-2600) compared to the Xeon E5-1650 (6C/12T). Right now I already plan to virtualize an AMD Athlon X4 server with 16GB of RAM I already own - converted from a media center which never started, therefore I think I can do well mythtv on Esxi with no much problems (definitively not much powerful, but for basics should be enough - for now ). As of right now, since I already own a CPU in the same class of the I5-2300 you suggest, I think I'll go with a single socket Xeon E5-1650 which has ECC. Going all out: 8x8GB ECC REGISTERED = 64GB Registered. 8GB ECC is not so much expensive (~ 80$ each). 16GB modules are harder to and cost ~ 210$ each. Not worth it for now IMHO.
Not so sure if I can put a single socket 130W beast in such a small case as the Grandia 01 from Silverstone though. 135mm max cooler heigth might be a little too restrictive.

As far as what I would do if I had it to do all over again, knowing what I know now and with the current products on the market, I'd definitely go one E3-1230 w/ 16GB and a dual E5-2620 w/ 64GB and probably a GTX560Ti for remotefx, possibly a second E3, but if not then I'd definitely still want it (tho less than I'm wanting the dualy E5 currently). I would NOT go with consumer chips/boards and non ECC RAM, because to me it just doesn't feel right and the price difference between going cheap and "doing it right" is fairly small relative to the full system price either way. Also I LOVE the IPMI's on my SM boards since the servers are stashed away without any monitors or KBs.

About the E3's I think not too: the only MBs supporting them cost around 300$ here and the CPU is around 200$. For that I can do anoth FX-8120 server for 200$+120$+ECC RAM (unbuffered).

In my case I could always connect them via DVI or HDMI using a hardware KVM (or just the Dell's many ports input) since my desktop will be around 1 meter from the VM host (or near the TV, haven't decided yet). You got the point though (and near the TV I always have a thin client where I could use hardware KVM as well).

I feel the same about ECC: already bough (I think) 4 sets of 16GB of NON-ECC RAM and after reading pro vs cons ECC I felt bad since I was trowing my money away. As many say it's better to have it and (hypothetically) not use it, than not having it and something goes terribly wrong. 8GB Ram sticks ECC Registered aren't much more expensive than consumer non-ECC (at least, here).

Slightly OT but another thing to consider if you haven't is you likely want a decent L2 or L3 switch to connect all of this to, you seem to think you need a lot of LAN connections, which if you have a switch that supports VLANs you can eliminate/combine a lot of networks that might otherwise actually require different connections. For my home lab I personally regret the extra network cards I got and would not buy them again if I had the choice. With a good switch and some VLAN planning you can easily get by with one NIC, 2 is a luxury (again for home use)

I decided to build a WAN/LAN router a month ago with Pfsense. I had (and still have) 3 consumer grade switches. Since I wanted link aggregation (not yet implemented though) I bough 1 x HP Procurve 1810g-8x and 1 x HP Procurve 1810g-24x. Always see good reviews on them. They support VLANs.
About the NICs I'm not so sure: not to say the more the better, but hosting VMs on SSD and if each VM has its dedicated link to the storage pool might increase performance a lot. Though at the start I agree, definitively I NIC / VM. I'm a bit at unease at buying dual port INTEL NICs on ebay since to get them I'd have to pay 50% more for shipping as well as taxes,... Here they are around 160$ (depends on where you buy it) but that's a bit too much (at least for now). I can always upgrade later imho.

Some other questions

• Does EsXI support what is commonly know as "overselling" (I know it's at home, but here what I mean is: I have for instance 6 cores / 12 threads, can I allocate 6T to a machine and 10T to another = 16T in total > 12T available, and if they're running at the same time they'll coordinate themselves ?). Doing this when the sum of dedicated threads of ACTIVE machine is < 12T (i.e. two machines with 6T each active plus one machine with 6C turned off) ?
• What about PWM fans? Can they be controlled by the guest? Is there some specific configuration for this in Esxi? Or I should just do this in BIOS or by using a Hardware FAN controller.
• A bit OT (maybe you know of another discussion): how can you phisically isolate your VMs from your LAN but not LAN (I may want to do some folding but for security reasons I think it's better to only leave it access internet). Or should I just setup deny entries in all of my boxes for that IP address (which may be a pain IMHO)

Edit: I just saw this on ebay. The price is somewhat wrong ? How can a new brand dual port (even though PCI-X) card cost less than a single port card ? Is there something going on ? There is lots of positive feedback though. 3 of those and will do a 2 integrated + 6 using expander port Something's off or can it be since PCI-X is a bit old ?

 

[CEpeep]

[*]Does EsXI support what is commonly know as "overselling" (I know it's at home, but here what I mean is: I have for instance 6 cores / 12 threads, can I allocate 6T to a machine and 10T to another = 16T in total > 12T available, and if they're running at the same time they'll coordinate themselves ?). Doing this when the sum of dedicated threads of ACTIVE machine is < 12T (i.e. two machines with 6T each active plus one machine with 6C turned off) ?

Yup. VMWare's implementation of this is arguably the best. It makes very efficient use of shared resources. It works best with RAM, but you can over-allocate vCPU cores as well.

[*]A bit OT (maybe you know of another discussion): how can you phisically isolate your VMs from your LAN but not LAN (I may want to do some folding but for security reasons I think it's better to only leave it access internet). Or should I just setup deny entries in all of my boxes for that IP address (which may be a pain IMHO)

We run a VM with pfSense in our labs that acts as the router for all traffic coming into and leaving the VM space. ESXi 5 has some limited routing/firewalling controls but pfSense is great for this. You can also use Untangle, Vyatta or anything else that you are familiar with.

 

[luckylinux]

We run a VM with pfSense in our labs that acts as the router for all traffic coming into and leaving the VM space. ESXi 5 has some limited routing/firewalling controls but pfSense is great for this. You can also use Untangle, Vyatta or anything else that you are familiar with.

Thank you. I though something else could be done (using differents subnets ?) which does not involve a whole new router configuration. Began to use Pfsense a month ago. A bit tricky to get started and haven't done much yet. Not sure about the other distros, though GNU/Linux might be easier for me to understand. Even tried clearos but got no luck with it.

Edit: Since it would only be firewalling any distros with IPTables might as well do the job (using shell scripting or Webmin). Just wanted to know if there existed other solutions. VLAN may also be an option since HP Procurves 1810g-8x or 24x support this.

 

[CEpeep]

Thank you. I though something else could be done (using differents subnets ?) which does not involve a whole new router configuration. Began to use Pfsense a month ago. A bit tricky to get started and haven't done much yet. Not sure about the other distros, though GNU/Linux might be easier for me to understand. Even tried clearos but got no luck with it.

Edit: Since it would only be firewalling any distros with IPTables might as well do the job (using shell scripting or Webmin). Just wanted to know if there existed other solutions. VLAN may also be an option since HP Procurves 1810g-8x or 24x support this.

iptables works fine too, if that's what you are comfortable with. ESXi fully supports VLANs as well, and if you already have a managed router that might be the way to go. Basically, don't worry about it; whatever networking infrastructure you possess/are used to should work fine with ESXi.

 

[-Dragon-]

About the NICs I'm not so sure: not to say the more the better, but hosting VMs on SSD and if each VM has its dedicated link to the storage pool might increase performance a lot.

This will matter all of 3-5 seconds a day when two or more VMs happen to be using more than ~100MB/sec disk access at the same time.

• A bit OT (maybe you know of another discussion): how can you phisically isolate your VMs from your LAN but not LAN (I may want to do some folding but for security reasons I think it's better to only leave it access internet). Or should I just setup deny entries in all of my boxes for that IP address (which may be a pain IMHO)

I assume you meant LAN but not WAN and yeah I have 2 networks, a 10. LAN with full internet access and a 192.168 isolated network for troubleshooting potentially infected machines (for friends and such) that only has port 80 and 443 access to the internet and DNS access to my local DNS servers (but not the internet) and gets DHCP assigned from my DHCP servers via a DHCP relay on my firewall. Just have to setup a routing path between both networks and enable appropriate firewall rules to allow and/or deny the desired traffic. I use 3 VLANs for my TMG box over one physical cable, untagged for the 10., VLAN2 for public internet, and VLAN5 for the 192.168.

 

[luckylinux]

Thank for you reply -Dragon-. Yeah I meant LAN NOT WAN.
Is it then possibly to shedule to backup Esxi's configuration daily (for instance) so that I can always import VMs to another host if that one fails?
About the single socket I don't really think it'd be a problem anymore since 64GB of ECC registered RAM should be plenty (for now), don't you think?

 

[-Dragon-]

I don't use ESXi personally, but if you're using a NAS for your VM VHDs, you should only have to back up the ESXi config whenever you change it, past that it shouldn't really change, I mean all the config is should be things like "this VM has this boot device, X CPU cores, Y GB of RAM, 2 NIC addapters, first assigned to VLAN A with MAC AAAA.BBBB.CCCC, and second untagged with MAC DDDD.EEEE.FFFF" etc... that should only change when you modify your VM for whatever reason.

 

[luckylinux]

I don't use ESXi personally, but if you're using a NAS for your VM VHDs, you should only have to back up the ESXi config whenever you change it, past that it shouldn't really change, I mean all the config is should be things like "this VM has this boot device, X CPU cores, Y GB of RAM, 2 NIC addapters, first assigned to VLAN A with MAC AAAA.BBBB.CCCC, and second untagged with MAC DDDD.EEEE.FFFF" etc... that should only change when you modify your VM for whatever reason.

Yeah, that's true. But I'd rather not do it myself (if it's possible). I don't know if you can create a sheduled task to do exactly that (backing up configuration).


PS: what are you using? Xenserver? Xen free?

 

[RussC]

I have not "played" with XenServer at all. If you do go with ESXi you could always look at Kaviza's virtual appliance which is now owned by Citrix. I'm not sure of the cost, but once you add multiple hosts into Kaviza's grid, it controls the load balancing without the need of vCenter.

Just a thought.

 

[luckylinux]

Yeah, but I think that's just too much of a budget at home for that appliance. I don't even think I need load balancing.
I still don't understand if Xen's performance hit is because of the non-supported GNU/Linux distributions or the fact they don't have PV support in their kernel (which is not the same thing, since I can always recompile the kernel in the "worst" case).

 

[-Dragon-]

I just use server 2008 R2 with a hyper-v role since everything is windows based, also I've got a ceton infinitv in one of them that requires a full windows install for the drivers to work in bridged mode (can't use hyper-v server even because you had to change a setting in device manager). I realize hyper-v isn't a particularly popular choice in these parts but after hearing all the crap that came with esxi 5, esp memory limits (esp now that 64-128GB on a dualy SB-E E5 board is reasonably affordable), manual updates, etc... and the fact I don't use any *nix VMs makes it a pretty solid choice.

 

[luckylinux]

Furthermore: I just checked prices here and single socket 2011 @ 410$ and dual socket 2011 @ 520$. At this point better with dual socket: 100$ more for MB, 2x450$ instead of hypotethycally 1 x 620$. What do you think ? The dual socket Supermicro X9DR3-F doesn't seem that bad. That way I can fit all PCIe slots with single (and cheap) port INTEL NICs. Sorry if I changed my mind, but it all depends on price:

- 1 x dual socket (Supermicro X9DR3-F) ~ 520$ + 2x450$ (Intel Xeon E5-2620, 6C/12T, lower frequency) ~ 1420$
- 2 x single socket (SUPERMICRO X9SRi-F) ~ 2x410$ + 2x620$ (Intel Xeon E5-1650, 6C/12T) ~ 2060$
- 2 x single socket (SUPERMICRO X9SRi-F) ~ 2x410$ + 2x310$ (Intel Xeon E5-2609, 4C/4T) ~ 1440$

What do you think ? Seeing the updated prices seem like the dual socket became the better alternative. IPMI included with all of them. 7PCIe slots with the dual socket and 16 DIMM slots x 8GB = 128GB RAM. The single socket board only has 3PCIe slots. Saving a few $$ I could get the SUPERMICRO X9SRA (~ 380$ each) but you got the point IMHO.
Plus double cases, PSUs, ...

OK. Right now I think dual socket it's better: same price as 2x4C/4T (but lower frequency) but I get 12C/24T. I won't buy all RAM DIMMs at first since 16x80$ ~ 1280 (other) $$
Maybe start with "just" 32GB or at max 64GB.

I repeat, sorry for saying I was going to build 2 x single socket configurations. Seeing the actual prices, they're not worth it (IMHO) except for redundancy - and if a board is bugged maybe both will die anyway.

 

[-Dragon-]

check out some of the Xeon E5 16xx amd 26xx series chips based on the SB-E socket.

This is exactly why I said the above in my first post

 

[luckylinux]

This is exactly why I said the above in my first post

I know you said that. Wasn't really sure if you were actually suggesting going 2 x single socket route (as you did with the E3) or the dual socket (which you said you'll probably do later).
Sorry for the misunderstanding

 

[-Dragon-]

I didn't mean to come across as a dick I'm was just saying the E5's were fairly new and chances are you hadn't given them much of a look to see what a ridiculous value they were. But now obviously you have and have seen first hand why it's such a ridiculously good deal.

 

[luckylinux]

I didn't mean to come across as a dick I'm was just saying the E5's were fairly new and chances are you hadn't given them much of a look to see what a ridiculous value they were. But now obviously you have and have seen first hand why it's such a ridiculously good deal.

No worries. In fact I was expecting (much) worse of a price.
And now it's cores&threads vs frequency.
I suppose for a second CPU I (only) need a second power plug with (4) pins [sorry, actually it's 8 and seems to be called EPS connector]. Since chances are that it'll probably be ON most of the time I though of the Corsair AX750 (80+ Gold, 84 months warranty). I already own one and I saw two 4 pin power plugs (and not for PCIe) so I suppose that should do the job. That PSI is around 190$ at my preferred retailer. Do you feel like suggesting another one (of another brand too)? I always went with Corsair for their warranty.

Edit: the AX850 isn't much more expensive (~ 210$) and the Corsair HX1050 for 230$ seems a very good deal (though that is "only" 80+ Silver - though that has 2 EPS connectors, so suitable for dual socket). Right now I own a HX750 and I'm not sure it has a 2nd 12V CPU plug (probably not [Edit: in fact it only has an EPS connector]). With 1050W I could even attach a pair of graphic cards and play as you suggested

 

[-Dragon-]

Bear in mind that the dualy boards are generally E-ATX not ATX and require a case that can support such a huge board. You would probably want something along the lines of this at least, which comes with its own PSU suitable for DP boards. Alternately you could go with something like this which has redundant PSUs and includes heat sinks that work with the internal cooling duct for the CPUs and RAM. But shit if you're gonna spend that much you might as well go with THIS for an extra hundred that supports 768GB max (24 total DIMMs slots)!

Edit: Oops here's an actual product link for the last one forgot the sales site was sparse on details

Edit2: Wrong link above, that was for MB only not barebones server that the sales link is for

Also for people click shy let me share some of the details of that last one:

Dual Socket R (LGA 2011)
Supports up to 768 GB DDR3 ECC Registered memory (RDIMM) in 24 DIMM sockets
Intel® i350 Quad Port Gigabit Ethernet
2x SATA 3.0 ports (6Gbps)
4x SATA 2.0 ports (3Gbps)
8x SAS ports from C606
4 (x16) PCI-E 3.0 slots
1 (x8) PCI-E 3.0 slot
1 (x4) PCI-E 3.0 (in x8) slot
8x Hot-swap 3.5" SAS/SATA Drive Trays
920W high-efficiency (94%+) AC-DC Redundant power supplies with PMBus and I2C

BRB fappin

 

[luckylinux]

Bear in mind that the dualy boards are generally E-ATX not ATX and require a case that can support such a huge board. You would probably want something along the lines of this at least, which comes with its own PSU suitable for DP boards. Alternately you could go with something like this which has redundant PSUs and includes heat sinks that work with the internal cooling duct for the CPUs and RAM. But shit if you're gonna spend that much you might as well go with THIS for an extra hundred that supports 768GB max (24 total DIMMs slots)!

Edit: Oops here's an actual product link for the last one forgot the sales site was sparse on details

Thank you very much for your reply.
However I have some points to point out:
a) The Fractal Design Define XL support E-ATX (if the Supermicro are standard E-ATX !) boards as wll (~ 140$). Plus a good PSU. Though that won't be redundant.
b) About the Supermicro cases I think they are too costly. I particularly like the Fractal's because they're pretty quiet (as long as the CPU cooler is). Basically they're like ANTEC's cases but a lower price. Been happy with them. Supermicro might be a bit loud, and costly.In the end I may only spare 50$ (on the first board you suggested), but still.
c) About these 1200$-1300$ cases I'd say no. I think it's pretty useless to me IMHO. Furthermore, I already bought 2 x Supermicro SC847E16-R1400UB with 1400W dual-redundant PSU. These are though only to be storage bays. The shit with them is that I have to use UIO cards to put in PCIe slots. I wanted virtualisation servers separated from Storage servers as well. But that way redundancy and 2x1400W PSU is "thrown away"

The Supermicro X9DR3-LN4F+ is not available here: we have the X9DRi-LN4F+ which seem to be similar. However you need an EEATX (double E: Enhanced Extended ATX, about 4cm wider/larger than normal). These cases are only made from Supermicro and not available here. However I think 128GB of RAM should do it's job well (and I might even go with 512GB once 16GB and 32GB sticks become affordable).

Edit: about the Supermicro SYS-7047R-TRF, you have it on newegg for ~ 1300$, but here it is for ~ $1550 which IMHO is a bit too much. The other expensive case is only available as a case for 850$. No MB included. Furthermore it's only E-ATX compatible, not EEATX so I don't think the 24 DIMM MB will fit.

IMHO the less costly solution would be a build from "scratch": 520$ dual socket MB, 140$ case, say even 240$ PSU << 1550$. Sorry but it's a fact that here some things are much more expensive, when others are around the same price (or cheaper: the 2 cases I bough from Supermicro with 2x1400W PSU were 700$ each, normally they are for 1600$ each on newegg). The only advantage I find from the cases you suggested it's the 80+ platinum certification: not so sure there is much difference with 80+ gold or silver, expecially under "light" (<= 30%) load of around 300W which is 2xTDP plus something.

 

[luckylinux]

There exist less expensive cases even from Supermicro than the one suggested here (50% less expensive). I don't understand however why there may be compatibility issues with a tower case. E-ATX is standard (with EE-ATX I will have definitively have a problem) and the only problem I can think of is ventilation.
Here is what they replies me: link.

Anybody having direct experiences using an unsupported (as far as Supermicro says anyway)? Thank you.