Cheap ESXi 5.1 IOMMU/Vt-d build?

E

evlo

n00b
Joined
Aug 18, 2012
Messages
24
Hello,

i looked at "Your home ESX server lab hardware specs?" and got even more confused at it. I'm building my home server, one part of it would be "NAS" which would run unRaid so i figured out i would need to passtrough HBA to it. There will be other machines running like ruTorrent and my custom downloaded shows downloader (it would handle seeding and savings downloaded shows to correct places in library as well as replacing HDTV with web-dl releases). Mind that downloading tv shows and movies of the internet is actually NOT against the law in my country. Last thing that i would like to run on it is PLEX.

I already bought some used case for 16HDDs and i'm now waiting for it to arrive (which should take about a week) so it is ideal time to plan the hardware that would go inside it. It should be very low power consuming as there will be not much need for actual computing power - for the time being it should save as file storage described above and not as the windows development lab (tough it is possible that i would want run windows servers on in the future :) ).


So i got the feeling that "IBM Express ServeRAID M1015" should be IOMMU compatible and probably will work with case internal sas expander.
But for the rest of hardware i'm totally confused and more i read more confused i get. I would probably like to go with AMD A6 as these have low TDP and are cheap, but i can't find out board that would be granted to support IOMMU and allow SAS controller in PCIe slot.

My idea was (after i found out that only amd FX procesors on and 9xx and 8xx chipsets supports iommu):
IBM Express ServeRAID M1015
16GB DDR3 RAM
GIGABYTE GA-970A-DS3
AMD FX-4100

But i'm unsure about IOMMU support passtrough of M1015? And ideally even the onboard SATA controller? In ESXi 5.0 or 5.1?

Can you make me somewhat less confused and help me finally pick something? :)
 
Last edited:
evlo said:
Hello,

i looked at "Your home ESX server lab hardware specs?" and got even more confused at it. I'm building my home server, one part of it would be "NAS" which would run unRaid so i figured out i would need to passtrough HBA to it. There will be other machines running like ruTorrent and my custom downloaded shows downloader (it would handle seeding and savings downloaded shows to correct places in library as well as replacing HDTV with web-dl releases). Mind that downloading tv shows and movies of the internet is actually NOT against the law in my country. Last thing that i would like to run on it is PLEX.

I already bought some used case for 16HDDs and i'm now waiting for it to arrive (which should take about a week) so it is ideal time to plan the hardware that would go inside it. It should be very low power consuming as there will be not much need for actual computing power - for the time being it should save as file storage described above and not as the windows development lab (tough it is possible that i would want run windows servers on in the future :) ).


So i got the feeling that "IBM Express ServeRAID M1015" should be IOMMU compatible and probably will work with case internal sas expander.
But for the rest of hardware i'm totally confused and more i read more confused i get. I would probably like to go with AMD A6 as these have low TDP and are cheap, but i can't find out board that would be granted to support IOMMU and allow SAS controller in PCIe slot.

My idea was (after i found out that only amd FX procesors on and 9xx and 8xx chipsets supports iommu):
IBM Express ServeRAID M1015
16GB DDR3 RAM
GIGABYTE GA-970A-DS3
AMD FX-4100

But i'm unsure about IOMMU support passtrough of M1015? And ideally even the onboard SATA controller? In ESXi 5.0 or 5.1?

Can you make me somewhat less confused and help me finally pick something? :)
Click to expand...

I believe that the 970 and up have IOMMU, like my 970 extreme 3 with a 1045 has it.
If you dont need the horsepower you can always downvolt/downclock the CPU.
 
I'm in the process of planning a home lab/server to learn and play with. I'm going to use the FX-8120 and an ASRock 970 EXTREME4 as it completely supports IOMMU. If you have a Microcenter near by this combo can be very cheap.
 
ziggo0 said:
I'm in the process of planning a home lab/server to learn and play with. I'm going to use the FX-8120 and an ASRock 970 EXTREME4 as it completely supports IOMMU. If you have a Microcenter near by this combo can be very cheap.
Click to expand...

The 8 core would be good if you plan a AIO server(or a few VM's), expecially since most of the boards support 32gb's of memory.
 
Yep, I plan to pass through a video card for HTPC, move my NAS over to a VM as well and a few other things. I figure the FX-8120 is so cheap may as well...it's what, $139 at MC? They usually have some AMD CPU/Board/RAM combo to get 50 ~ 70$ off anyways.
 
As i like gigabyte better i will try the GIGABYTE GA-970A-DS3 and if it does not work i would go for the ASRock 970 Pro3 you recommend. Thank you.

Is it possible to run AMD FX-8120 and underclock it so i would get lower power consumption and then when i would need more power for windows virtualization to clock it up again? Tough it is quite a shame that you can't use 4x 16GB rdimm modules with the AMD 970. 32GB limit is not much when i would like to start playing with windows development, but i can work with it.
Or better question would there be any noticable powersaving between underclocking AMD FX-4100 and AMD FX-8120 as much as possible?

It seams (based on some article on anandtech) that build would draw about 100W idle with AMD FX and ibout 80W witch CORE i5 2400S. I guess there is not any more power effecient solution?
BTW 20W diffence would made the TCO same for both AMD and Intel builds in 2 years in my country. :)

EDIT: now i found another article on xbitlabs that claims 16w for sytem idle for core i5 2400S, that is 40w lower then anandtech. So much choices and no first hand info.


In this article http://www.xbitlabs.com/articles/cpu/display/core-i5-2400s_6.html#sect0 they say 2400 has 16W idle for whole system and here http://www.xbitlabs.com/articles/cpu/display/core-i5-ivy-bridge_9.html they say 63w. Eh?
 
Last edited:
evlo said:
As i like gigabyte better i will try the GIGABYTE GA-970A-DS3 and if it does not work i would go for the ASRock 970 Pro3 you recommend. Thank you.

Is it possible to run AMD FX-8120 and underclock it so i would get lower power consumption and then when i would need more power for windows virtualization to clock it up again? Tough it is quite a shame that you can't use 4x 16GB rdimm modules with the AMD 970. 32GB limit is not much when i would like to start playing with windows development, but i can work with it.
Or better question would there be any noticable powersaving between underclocking AMD FX-4100 and AMD FX-8120 as much as possible?

It seams (based on some article on anandtech) that build would draw about 100W idle with AMD FX and ibout 80W witch CORE i5 2400S. I guess there is not any more power effecient solution?
BTW 20W diffence would made the TCO same for both AMD and Intel builds in 2 years in my country. :)

EDIT: now i found another article on xbitlabs that claims 16w for sytem idle for core i5 2400S, that is 40w lower then anandtech. So much choices and no first hand info.


In this article http://www.xbitlabs.com/articles/cpu/display/core-i5-2400s_6.html#sect0 they say 2400 has 16W idle for whole system and here http://www.xbitlabs.com/articles/cpu/display/core-i5-ivy-bridge_9.html they say 63w. Eh?
Click to expand...

GIGABYTE GA-990FXA-UD3 is also an option.

http://en.wikipedia.org/wiki/IOMMU_hardware_list

Hope that helps.
 
evlo said:
As i like gigabyte better i will try the GIGABYTE GA-970A-DS3 and if it does not work i would go for the ASRock 970 Pro3 you recommend. Thank you.

Is it possible to run AMD FX-8120 and underclock it so i would get lower power consumption and then when i would need more power for windows virtualization to clock it up again? Tough it is quite a shame that you can't use 4x 16GB rdimm modules with the AMD 970. 32GB limit is not much when i would like to start playing with windows development, but i can work with it.
Or better question would there be any noticable powersaving between underclocking AMD FX-4100 and AMD FX-8120 as much as possible?

It seams (based on some article on anandtech) that build would draw about 100W idle with AMD FX and ibout 80W witch CORE i5 2400S. I guess there is not any more power effecient solution?
BTW 20W diffence would made the TCO same for both AMD and Intel builds in 2 years in my country. :)

EDIT: now i found another article on xbitlabs that claims 16w for sytem idle for core i5 2400S, that is 40w lower then anandtech. So much choices and no first hand info.


In this article http://www.xbitlabs.com/articles/cpu/display/core-i5-2400s_6.html#sect0 they say 2400 has 16W idle for whole system and here http://www.xbitlabs.com/articles/cpu/display/core-i5-ivy-bridge_9.html they say 63w. Eh?
Click to expand...

If electricity is that expensive, it might be a logical step to look into a workstation class laptop. a W530 idle's around 10 watts and the CPU has a TDP of 45....and the machine supports 32gb of ram.
 
Hagrid said:
GIGABYTE GA-990FXA-UD3 is also an option.

http://en.wikipedia.org/wiki/IOMMU_hardware_list

Hope that helps.
Click to expand...

I own 990FXA-UD5 V1.X with FX-4100. unbuffer ECC memory, IOMMU works, and power management are working without a hitch on esxi 5.0 including PCI express passthrough.
this is my testing esxi machine.

I stay away from esxi 5.1 at the moment since vmware does not release patch that fixes pink screen of death when passing-through PCI (or could be PCI express, see on vmware forum to look in detail for certain motherboards)

power consumption is pretty good around ~100W with 2 IBM M1015 and one adaptec 3805 while serving/running Virtual machines, the consumption will shoot to the roof when one or more Virtual machines utilize CPU intensive application/program.

I would say, intell has better power consumption than AMD, but... total cost with AMD is a way low than Intel.

Everything goes to everyone where choosing the hardwares and spending cost.

oops. one thing is bothering me. I have to disable on-board AMD Sata to make M1015 and 3805 to work properly.

the onboard E-sata and internal Sata (marvel SATA3 ) works perfectly. I think can be passthrough in esxi ( as I remember)

just my though.
 
Are you planning to run PLEX media server on Linux or Mac?

The mac version of PMS is still much better than the Linux one (I used the Linux one for 6mo before switching back to Mac), however you will need an Intel CPU to unlock ESXi for Mac support. Just something to think about.
 
omega-x: electricity is relatively expensive, 40wats difference would make up for the price difference between core i5 2400 and AMD FX-4100 in one year. Laptop is not an option as it should be primary NAS. I'm trying to figure out the best build for me and i'm ok in paying more per month if there is some significant advantage.

reebzor: Good point about mac osx version of plex. Thank you! I I currently run windows version and it is still usable, but i know that macOSX is better.

Now i'm almost decided to to buy core i5-2400. Still have some time so i will sleep on before ordering :)
Probably with Intel Elkhorn Creek BLKDB75EN as it is cheapest with VT-d and i really don't need anything extra.
 
But is it possible to boot up without graphics without any issues? Seams to add quite a hustle as the board has only one big PCIe slot, tough obviously xenon has way more cool factor :) ... Intel Core i5-3470 / Intel Xeon E3-1220V2



I measured my q6600 at idle with one disk spinning and it is pulling 100W, would Core i5-3470 or Xeon E3-1220V2 system really pull less then half of it?
 
Last edited:
evlo said:
But is it possible to boot up without graphics without any issues? Seams to add quite a hustle as the board has only one big PCIe slot, tough obviously xenon has way more cool factor :) ... Intel Core i5-3470 / Intel Xeon E3-1220V2



I measured my q6600 at idle with one disk spinning and it is pulling 100W, would Core i5-3470 or Xeon E3-1220V2 system really pull less then half of it?
Click to expand...

E3 ( 2nd and 3rd) will have many less than 100W when not in idle( light cycle) :D
I5... should be less than 100W, can not confirm since not having I5 2nd generation

E3 with entry level server motherboard where has already on-board VGA (most in this forum suggesting SM, I would say Tyan/SM/Asus should be ok).
if you need spend a bit of cost, find motherboard with IPMI.

I have 1U SM with dual core 2 duo. the system average is 80W during light cycle, will jump to 100W-120W during cpu intensive. I would say Intel is good on trimming power consumption starting with core duo an up...
 
One more thing came across my mind:
Could there be issue that DESKTOP motherboard with only one pciE 16x slot would not work correctly with SAS controller in it as it is usually used for graphics card?



Then i also wonder if plex would be able to detect changes on unRaid storage, but it is quite offtopic here :)
 
Last edited:
evlo said:
One more thing came across my mind:
Could there be issue that DESKTOP motherboard with only one pciE 16x slot would not work correctly with SAS controller in it as it is usually used for graphics card?

)
Click to expand...

Yes and No...
you have to try the motherboard first :p...

Like mine for testing machine at the moment, Gigabyte 990-FXA UD5 can detect add-ons PCI Express SAS/SATA card when AMD onboard SATA is disable :p.

you can create a new thread asking DESKTOP mobo that work perfectly with SAS/SATA card :p.

plex is Out of Topic hhehehe..
 
May i ask here too, since my main question also about IOMMU?
I need to bypass PCI-X card to ESXi VM. Will X8SAX and Xeon, well, L5520 do the trick?
 
Rivera said:
May i ask here too, since my main question also about IOMMU?
I need to bypass PCI-X card to ESXi VM. Will X8SAX and Xeon, well, L5520 do the trick?
Click to expand...

check your bios , search on vt-d.
if your bios has selection to enable vt-d, you are good to go with passthrough
 
You must log in or register to reply here.
Back
Top