Building a hypervisor+Internet gateway rig

W

Wolvenmoon

Weaksauce
Joined
Apr 6, 2009
Messages
67
Hello all!

I've recently upgraded my workstation off of an old Q9450 w/ 8 gigs of RAM and a P5Q pro. Usually I retire my workstations to serve as Internet gateways and servers. I started to do that with my old workstation only to find I needed to go back five years and kick myself in the butt for not buying a board with VT-d.

So I'm in a bit of a bind. I'm looking for a system that can host a number of Xen VMs, including one that'll act as my home network's gateway/NAS server (Running ClearOS). I do some 3D modeling and rendering that makes GPGPU very important to me, and I'd like to go grab a few last-gen low power GPUs that have a grotesque amount of RAM strapped to them, and toss them into this system so that it can chew on rendering tasks.

I am completely out of the loop when it comes to AMD's CPU offerings and by extension I haven't the first clue as to their motherboards. I'm looking for something with as many cores as possible that'll run on as little power as possible to run a bunch of virtual machines while being able to support two or three Nvidia GPUs.

What should I look at getting?
 
FX-83xx 8-core CPU paired with a 990FX chipset motherboard work great for IOMMU (AMD's equivalent to VT-d). Many 990FX boards have enough PCI-E slots and lanes to support as many as 4 GPUs.
 
Yeah low power, fast, and "AMD" don't mix too well together in this case.

Anyway, pretty much what rtanwai said: an FX 8320 or FX 8350 CPU with a 990FX motherboard. However if you want us to setup a complete build for you or better critique any build you come up with, please answer the stickied "ANSWER THESE QUESTIONS FIRST!". We can help you better that way.
 
I read the sticky before posting. I personally think I'm good to go, but here are my overall plans:

CPU: Unknown.
HSF: Stock or something inexpensive
Mobo: Unknown
RAM: 32 gigs of cheap DDR3-1600 or DDR3-1866
PSU: An aging Seasonic 550W
Case: Corsair Carbide 200r
Storage: A ton of old, unwanted HDDs with a possibility of a small SSD for a boot drive
GPU(s): Whatever old Nvidia cards from Fermi onwards that I can find

I started looking at the 8350 vs the 8320. It looks like the 8350 is 11% faster overall? http://www.cpubenchmark.net/high_end_cpus.html

What's the power consumption like on these? Am I asking for trouble if I'm considering overclocking the 8320 up to the 8350's stock speed, and leaving it there?

If Intel's a better choice here I'm not averse to another Intel system, I just sort of wanted to support AMD.
 
Wolvenmoon said:
I read the sticky before posting. I personally think I'm good to go, but here are my overall plans:
Click to expand...
Your post still doesn't clearly answer questions 2 to 5 and 8 to 9. So please answer those questions from the stickied "ANSWER THESE QUESTIONS FIRST!"
Wolvenmoon said:
What's the power consumption like on these?
Click to expand...
Relatively high once fully loaded. Roughly 100W more power usage than a similarly priced Intel CPU.
Wolvenmoon said:
Am I asking for trouble if I'm considering overclocking the 8320 up to the 8350's stock speed, and leaving it there?
Click to expand...
I would not risk overclocking considering that this is a virtualization server.
Wolvenmoon said:
If Intel's a better choice here I'm not averse to another Intel system, I just sort of wanted to support AMD.
Click to expand...
When it comes to virtualization, AMD is the better choice from a bang for the buck perspective. However, AMD's CPU power consumption is significantly higher than that of Intel. In other words, what matters more for you? Low power or good bang for the buck value? For a virtualization setup like the one you're proposing, you won't be able to get both.
 
Going down the sticky:

My budget ceiling with 32 gigs of RAM is ~$550.

I'm in Oklahoma in the U.S., city should be inapplicable unless someone wants to give me free parts in which case I'll oblige. ;)

I've been clear that I'm only looking at CPU+Mobo+HSF.

My power supply will be bought or re-used depending on the system's need. At this point I have not decided. More on that momentarily. I will not be buying new HDDs nor new GPUs.

I'm considering the 8350 and underclocking+undervolting at the moment.

I plan on building this system this June if possible.

The motherboard needs to have at least two PCI-E x16 ports and then enough to add in an IDE controller card, two gigabit LAN cards, and a TV tuner card. I also want at least 6 SATA ports. Generation of SATA doesn't particularly matter to me.


Thinking about this during normal daylight hours when my brain is fully on has made me realize another issue. I keep this system attached to a battery backup - an inexpensive one. This seriously throws a wrench into AMD being a good value for me, because buying a new PSU (currently have a 350W or 430W passive PFC PSU, I forget which) and/or a new UPS that does a decent sine wave compatible with active PFC PSUs potentially sets the system's initial cost up to be more than it would be if I built around a Xeon E3-1230 (I'm picking these CPUs out of http://www.cpubenchmark.net/high_end_cpus.html )


Additionally, I already have a Core i7 3770K with a pair of GTX 480s and a core i7 920 with no GPU, both overclocked, sitting on this same 120V 15A circuit. I'm getting close to hitting a hard limit on the amount of power I can draw. So while bang for the buck is incredibly important to me, I'm going to hit a point where I need an electrician or efficient systems. Also, it's going to be 115 F out this summer. I'll be backing off my overclocks because all of these systems are in the same room! :D
 
Wolvenmoon said:
I've been clear that I'm only looking at CPU+Mobo+HSF.
Click to expand...
Unfortunately you weren't that clear: I've seen people post a list of parts exactly the way you did in your second post and then tell us later on that those are ALL the parts they need to buy, not the parts they have. The only clear indication that some of the parts were being reused were the PSU and hard drive. The rest you didn't make clear that you were going to reuse.

When we do builds here, we take a look at the whole system. No point in recommending a system if you're gonna pair it with a crappy PSU, bad case, incompatible RAM, etc. So we try to make sure that the system you have is good enough for the parts we recommend.

Wolvenmoon said:
I'm considering the 8350 and underclocking+undervolting at the moment.
Click to expand...
Well that's kinda of counter-intuitive: If you're just going to underclock the CPU anyway, why not just get the lower clocked FX-8320 to begin with? It's already clocked 500Mhz lower than the FX-8350. Besides the lower clock, the two CPUs are virtually the same.

Wolvenmoon said:
because buying a new PSU (currently have a 350W or 430W passive PFC PSU, I forget which)
Click to expand...
Well no matter what you're looking at a new PSU as generally PSUs with no APFC are generally either of low quality, old age, or both. So is a new PSU part of that $550 budget or a seperate budget?

Also this doesn't jive with what you said in an earlier post:
Wolvenmoon said:
PSU: An aging Seasonic 550W
Click to expand...

So do you or do you not have a Seasonic PSU? If it's a Seasonic, unless that Seasonic was from like 10+ years ago, it should have APFC.

Wolvenmoon said:
(I'm picking these CPUs out of http://www.cpubenchmark.net/high_end_cpus.html )
Click to expand...
Ignore that website. It's a synthetic benchmark website. As such, it has little to no bearing on the indication of real world performance.

Wolvenmoon said:
So while bang for the buck is incredibly important to me, I'm going to hit a point where I need an electrician or efficient systems. Also, it's going to be 115 F out this summer. I'll be backing off my overclocks because all of these systems are in the same room! :D
Click to expand...
So just to be clear: Do you want Intel or AMD? Do note that you will probably have to increase your budget for the Intel setup due to your 32GB of RAM requirement.
 
In my first post I explained in general terms what I was doing and what I was looking for, then asked what I was looking for. I can empathize with you - you deal with newbies who gripe about not having their hands held. However, the bullet-by-bullet argument style employed on political debate forums combined with some of your phrasing is seriously off-putting.

I understand I am an irregular on this forum, however I do know what I'm doing. I've done this type of thing before.

Anyway. We're looking at mobo+CPU with the assumption that I eventually want to run 32 gigs of RAM. RAM recommendations are not unwelcome.

Underclocking/undervolting a binned processor will let me run the 8350 at the same clock speed as its slower sibling at a lower voltage than said sibling. The hope would be that the CPU would run cool enough not to noticably heat my room up more than the Athlon 64 3200+ Venice it would be replacing. I'd be paying a price premium for the 8350 for the same reason I'd pay the premium for an Intel CPU of similar caliber. I'm really not too happy with the thought of having to do that out of the box so that I don't roast this summer.

I'm looking at Mobo+CPU+RAM in a $500 budget ceiling with an additional $250 to be invested in the system over the next year or two. The end result will have 32 gigs of RAM, a small SSD for the hypervisor and ClearOS, several SATA drives, a TV tuner card, and a few old GPUs. All I need to get the system to turn on right now, so that means not doing too much, too fast.

I do not want to replace my PSU or my UPS right now, so this means I'm going to have to estimate the wattage of any proposed system and weigh it against what all I have on my UPS.

It appears I'm looking at the 8350 vs the Xeon E3-1230. I haven't ever shopped motherboards with the intent on cutting corners, so I need a bit of assistance here. The motherboard for either needs to support virtualizing everything, have two PCI-e x16 slots running at x8 or better, then one running at x4. There are no special requirements on the RAM. I have access to the add in cards to run the hypervisors I'm considering.

So what would you suggest for an E3-1230? What about an 8350?
 
Wolvenmoon said:
In my first post I explained in general terms what I was doing and what I was looking for, then asked what I was looking for. I can empathize with you - you deal with newbies who gripe about not having their hands held. However, the bullet-by-bullet argument style employed on political debate forums combined with some of your phrasing is seriously off-putting.
Click to expand...
From my experiences here on the forum, the bullet-by-bullet style works well in making sure that whoever I'm helping understands exactly what I'm trying to refute, point out, assess, comment on, etc and to make sure that nothing I say is taken out of context. Not to mention, it helps me nitpick people's build a bit better. I'm sorry that you were offput by that but unfortuantely I dispise quoting long posts and trying to respond to that as is.


Wolvenmoon said:
Underclocking/undervolting a binned processor will let me run the 8350 at the same clock speed as its slower sibling at a lower voltage than said sibling. .
Click to expand...
I see.

Wolvenmoon said:
I do not want to replace my PSU or my UPS right now, so this means I'm going to have to estimate the wattage of any proposed system and weigh it against what all I have on my UPS.
Click to expand...
As I noted earlier, you've mentioned two PSUs: a 350W/430W PSU that has no APFC and a Seasonic 550W PSU that does have APFC. So which PSU do you have? What's the specific model numbers? What UPS do you have?

Wolvenmoon said:
It appears I'm looking at the 8350 vs the Xeon E3-1230. I haven't ever shopped motherboards with the intent on cutting corners, so I need a bit of assistance here. The motherboard for either needs to support virtualizing everything, have two PCI-e x16 slots running at x8 or better, then one running at x4. There are no special requirements on the RAM. I have access to the add in cards to run the hypervisors I'm considering.

So what would you suggest for an E3-1230? What about an 8350?
Click to expand...
For the AMD setup, I recommend looking at the Gigabyte GA-9990FXA-UD3 AMD 990FX ATX Motherboard. Quite a few people here in this forum uses that mobo for ESXi just fine and IOMMU supposedly works great. As such, for your planned Zen setup with GPGPU computing, that mobo should do the trick. Another option would be the Asus M5A99FX Pro.

For the Intel, the normal general recommendation here on the forum for a Xeon CPU is a server-class mobo. Unfortunately, there aren't any reliable server class mobos that would fit your requirements for three PCI-E x16 slots. The closest would be this Supermicro mATX mobo whcih would require you to use single slot GPUs in order to use the rest of the PCI-E ports. Also note that the Supermicro mobo will require ECC unbuffered RAM. That will further increase the price as ECC RAM is often 1.5 to 2 times the price of regular RAM:
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182253

If you absolutely want to be able to use dual-slot GPUs and want lower power with Intel, a consumer grade Intel mobo might work. However I shall have to get back to you on that as I'm still looking for a consumer grade Intel mobo that's been confirmed to have VT-D/IOMMU/direct hardware passthrough working. So far, I'm only seeing mATX Intel mobos having that compatibility. But at least a consumer grade Intel mobo would let you use regular RAM instead of ECC RAM.
 
There's a lot of confusion out there regarding VT-d/IOMMU and the z77. Last time I looked closely at it, the unlocked K series processors did not support VT-d, but the chipset did. Looking closer at it, Google's reporting a mess. I did pull http://www.overclock.net/t/1338063/vt-d-compatible-motherboards out of it.

On one hand, there's this: http://communities.intel.com/thread/38604 showing it in the data sheet.

On the other: http://ark.intel.com/products/64024/

I just finished reading through this thread: http://communities.vmware.com/thread/401594?start=0&tstart=0

This is a *bleep* headache.
 
The Z77 chipset is capable of supporting VT-d, but it is up to the motherboard manufacturers to enable the support in the BIOS. ASUS doesn't enable it on any of their Z77 boards, ASROCK does, so do several other manufacturers.

With regards to the CPU's, none of the K-series support VT-d but several other LGA1155 CPUs do. For example, a 3770K does not support VT-d, but the 3770 does.

I did the same homework as everyone else building a whitebox bare-metal VM host for a home lab and I came to the same conclusion as many: go AMD. Price-performance ratio is much better and there is a lot less confusion because a far greater range of CPU/motherboard combinations support IOMMU. I just built a FX-8350/Gigabyte GA-990FXA-UD5 and it works beautifully, ESXi 5.1 sees all the hardware without any intervention from me.

In your case you have the extra problem that you want lots of PCI-E lanes, which the Z77 simply doesn't have without a PLX chip. The 990FX has lanes to burn, you can go 3 or 4 GPUs and still have lanes left over for NICs, RAID controllers, etc.
 
Last edited:
I'm sold on AMD. I can't find a single good reason to go for Intel if I'm trying to keep all of my bodily organs. Average power consumption is 25%-33% more, but the PITA-factor is 25%-33% less. I'll likely order this next month/as soon as I can sell my Q9450.

Also, it looks like this CPU undervolts decently, this dude got a 14% reduction in power consumption with a .1V drop. http://www.hardcoreware.net/amd-piledriver-fx-review-vishera-8350/11/ . I'll probably inch it down until it crashes at an apocalyptic load, then bring it back up. An 11-19% difference in power consumption is fine with me.

My worry with power consumption is more a concern with heat dissipation while trapped in a room in an Oklahoman 115 degree Fahrenheit Summer with a [email protected] (from 3.5) + 2xGTX 480s (running at 1.0V from 1.075 ), an i7-920 running at 3.2 GHZ from 2.66 with no GPU (currently), and then this system. What I really need is 600W worth of solar panels and a window AC unit! :D
 
I have multiple PSUs including the Seasonic 550W with active PFC.(SS-550HT) I believe I'm using a Thermaltake 430W with passive PFC in the system, but checking would require me to shut the system down and pull it out of its corner.

I use a cheap UPS to keep this system from being dumped when the power flashes - ClearOS is really horrible at dealing with being dumped, it often corrupts a certain user database. I forget which database because it's been years since I've had to look at it.

Cheap UPSes output stepped waves that cause active PFC power supplies to shut off instantly.

My plan is to eventually switch to an 80+ silver active PFC PSU and a UPS that outputs a sine wave, but I'll only do that if/when the system is making me money as a tool or if the system is running at load frequently.

Edit: Also, a window AC would start throwing my breaker. 15A*120V=1800W maximum load. I know with these three systems I'll max out at around 1300-1400W while rendering. A window AC is asking for trouble.
 
Wolvenmoon said:
I have multiple PSUs including the Seasonic 550W with active PFC.(SS-550HT) I believe I'm using a Thermaltake 430W with passive PFC in the system, but checking would require me to shut the system down and pull it out of its corner.
Click to expand...

Use the Seasonic PSU for the new server. That Thermaltake PSU is probably the Thermaltake TR2 430W. The TR2 is well known among PSU enthusiasts for being a very crappy PSU that's only capable of half its rated wattage under the most ideal of conditions. In other words, you must have a lot of luck if that PSU hasn't blown up and taken out the PC.
 
There's an equal chance that I'm using an Antec 300W PSU from God-knows-when, in this system. I do know that both that Thermaltake PSU and my ancient Antec PSU are missing from the shelf that hosts my power supply Krakken.

I'd guesstimate this system's running at about 150W under maximum load, but it's idle 90% of the time. Hmmm. I suppose I'll have to pop it open and look once the other Internet addicts in the house have gone to bed.
 
Either way, use that Seasonic. Some of the older Antecs were crap as well.
 
So we'll need to find a UPS that'll not shut off when it detects 500-550W of load (I can't reasonably foresee the system ever using more than that even with a pair of mid-range GPUs in it) to fit into my budget.

I can take a wild guess, but my wild guess is going to be for 'inexpensive'.
 
That doesn't output a sine wave, I actually started doing some heavy reading on this over here: http://www.jonnyguru.com/forums/showthread.php?t=3964&page=4

It's fairly interesting and makes me eager to get into my EE coursework this Fall.

I can't order until June 1st though. I just got told I needed to get into 5 credit hours over the summer and my wallet lost so much weight so quickly one would think it was working out.

...I'm on page 7 of that thread and have not gotten a reason as to why my active PFC PSUs turn off on my UPS (barring changeover time, which wouldn't be the problem, as the Seasonic has about a 333-500MS holdup time before it powers off when the power is cut).

The TL;DR is that I should have a sine wave line interactive UPS and not a stepped or approximated one in my usage, because the system will be on 24/7.
 
You must log in or register to reply here.
Back
Top