New build - ESXi Home Server Lab

Discussion in 'Virtualized Computing' started by rufik, Sep 16, 2013.

  1. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    Hello


    Im preparing to build a new home vmware lab.

    I was thinking about desptop part (i7) to build that lab, but i think better hoice will be
    on intel xeon processor and board also from intel (supermicro)

    For that project i have at this moment:

    1. Computer case from Fractal Design DEFINE XL R2 TITANIUM GREY
    http://www.fractal-design.com/?view=product&category=2&prod=112

    2. Hardware RAid controller from LSI
    LSI MegaRAID SAS 9271-8i

    3. Six hard drive's from WD (6 x 3 TB)
    WD Red WD30EFRX (3TB, 64MB, SATA/600)

    4. RAM 16 GB / soon upgrade max to 32 GB
    G.SKILL DDR3 2x8GB 1600MHz CL10 RipjawsX

    And right know i need

    1. CPU
    2. BOARD

    -> For Board from INTEL <-

    http://ark.intel.com/compare/66137,71384,71385,61042

    Which one will be best choice ?

    -> For processor (CPU) <-

    http://ark.intel.com/compare/75054,75465,75057,75464,75056

    Which one will be best choice ?

    Most important difference is SOCKET for LGA1150 (Socket B2 LGA-1356 vs LGA1150) and size of board (SSI ATX 12" X 9.6" vs uATX)

    Socket B2 LGA-1356 vs LGA1150 ?
    SSI ATX 12" X 9.6" vs uATX ?

    ps.

    What are most difference's beetwen processor Intel Xeon E3 and E5 ?
     
  2. peanuthead

    peanuthead [H]ardness Supreme

    Messages:
    4,593
    Joined:
    Feb 1, 2006
    Before we go anywhere, what are you using the system for? What kind of load on the vms will you place on them?
     
  3. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013


    Hey

    Sure :)

    1. Active Directory - Windows Server 2012
    2. Linux VATTA - Internet connection
    3. Linux for Cisco, Juniper lab (ios, jun os, asa)
    4. Free NAS
    5. Linux for DNS
    6. Windows 2003 srv to connection from VPN (terminal)
    7. Linux web server with mysql
    8. Linux services monitoring (centreon, cacti)
    9. ESXi 60 day evo... lab
     
  4. sybreeder

    sybreeder Limp Gawd

    Messages:
    193
    Joined:
    Oct 24, 2010
    Are you planning to use FreeNAS As your main storage - All in one solution ? If so you should go with HBA not hardware Raid card.
    Rest - like E5 vs E3 is related to fact if you want for example more than 32gb ram in the future or how many pcie lanes do you want.
     
  5. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    Hello

    FreeNAS only for shared disk allocation for some VM's

    Main storage will be provided by Active Directory (into domain for home users)

    I want use max 32gb memory.

    Also i have question

    http://ark.intel.com/compare/75054,75465,75057,75464,75056

    Can i use 32 gb memory in my board if on specification i can see

    Max Memory Bandwidth - 25.6 GB/s

    What it mean's ? Because i don't understand that

    I will have 32 GB and Max Memory Bandwidth 25.6 GB/s.

    Do there will be system slow perfomance or what ?
     
  6. peanuthead

    peanuthead [H]ardness Supreme

    Messages:
    4,593
    Joined:
    Feb 1, 2006
    Don't get hung up on the max. memory bandwidth. For where you are at, just ignore it.
     
  7. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    Regarding you previous question about pciex

    1. controller raid
    2. i need one for adidtional NIC

    and i thik thats all

    I have here integrated GPU so i dont need externel GPU.

    Do i need more pciex ?

    ps. Could you explain for newbie what is going with that memory bandwith, what is a reason of this value ? etc... ?
     
  8. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
  9. GaryS

    GaryS [H]Lite

    Messages:
    94
    Joined:
    Dec 25, 2012
    Supermicro (for Xeon) vs other

    I've had very good luck with SuperMicro.
    I have two computers with Xeon E3 CPUs. One currently replacing a troubled workstation. The other dedicated for ESXi. Would have preferred the E5... but they're expensive and I've noticed the CPU generally underutilized on my 2xCPU workstation. So, couldn't justify the extra budget for underutilized horsepower.

    The new E3 (haswell?) also runs cool and quite. Doesn't heat up the office.

    Some things to think about:
    - RAM gets used up very quickly... 32GB, in my opinion, is bare minimum.
    - Get lots of efficient storage. I spin up several VMs for quick testing, analysis, and then move them to cold storage. I don't have the time to move them off to external storage. It's way easier to simply move them internally, within the box, or just leave them be (don't forget to take notes on which-did-what). When the disks fill up? Buy more. Disks are cheep.
    - Make sure you have enough PCI-E slots for extra NICs (intel 2xport). The additional ports are a nice feature if you like routing traffic through load-balancers (e.g. F5), etc.
    - IPMI is a "must have". Couldn't live without it.
    - LSI Raid. Though some here disagree... I like the caching controller card. It's damn fast with the 512MB (or more). I didn't get the on-card battery backup. Instead I have my machine plugged into a big UPS.
    - Expensive SSD? I personally consider SSD overkill for a home lab "server" (ESXi). Many here disagree.
     
  10. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    What is IPMI ?
    Were i Can use it ?
     
  11. GaryS

    GaryS [H]Lite

    Messages:
    94
    Joined:
    Dec 25, 2012
    IPMI is like an onboard KVM (keyboard Video Mouse).

    IPMI server runs on the motherboard, uses its own network/lan port, and connects to a computer running the Java IPMI client.

    For example, my SuperMicro motherboard has 3 network ports:
    1 and 2 are basic on-board NiCs
    The 3rd is the IPMI port.

    I can then connect to the server via my desktop running the IPMI client software.
    The client presents a view just like a display plugged directly into the computer. You even get to see boot-up and can interrupt for BIOS configs, etc. (however... be careful about flashing the BIOS or IPMI software over IPMI).

    Some features:
    • start/stop
    • temp monitors
    • fan speeds
    • remote media

    Link to SuperMicro IPMI page
     
  12. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    Nice. I checked it i want it ! :)
    I can see that on intel boards all of them have ipmi.

    I am almost decided for E3 Intel XEON board for chipset C226

    In my opinion C226 is better. Am i wrong ?
    http://ark.intel.com/compare/63984,36785,64015,75522,75516

    But there is a problem regarding Supermicro board, only one is supporting ipmi :((((

    http://www.supermicro.com.tw/products/motherboard/Xeon/C220/X10SLH-F.cfm
    Big minus this is mini ATX. Better for me will be ATX.

    Full ATX has more pciex, mini not ...
    Supermicro has two full ATX boards
    http://www.supermicro.com.tw/products/motherboard/Xeon/C220/X10SAE.cfm
    http://www.supermicro.com.tw/products/motherboard/Xeon/C220/X10SAT.cfm



    Im confused and i need your advise

    Main questions

    1. uATX vs ATX ?
    2. Board with IPMI or SuperDoctor® III (for Supermicro) ?
     
  13. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    no recommendations?
     
  14. zb2good

    zb2good n00b

    Messages:
    4
    Joined:
    Sep 11, 2013
    x1220 v3 and x1225 v3 do not offer hyperthreading.
    source
     
  15. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
  16. GaryS

    GaryS [H]Lite

    Messages:
    94
    Joined:
    Dec 25, 2012
    Recommend reading this blog:
    Rootwyrm's Corner
    and
    BabyDragon II Hardware Compatibility List


    re:
    uATX vs ATX: I would go with function and features first. Then, begin looking at size issues.

    How I did this:
    - watch for the "extended" description in mobo size - example: "e-atx". The "extended" dimensions aren't all the same. Absolutely not an issue as long as you have a big case and are not shy of slight modifications. Most people blog about modifying cases to fit the larger SuperMicro boards.

    I would avoid the "proprietary" sizes as these are specific to SuperMicro cases. If you're going proprietary then you're better off buying the box as a unit (don't mess with installation).

    My last two SuperMicro boards were uATX dimensions (NOTE: and what is the "u" in uATX? Honestly do not know. They did fit my LianLi ATX mid-tower case).

    IPMI vs Doctor

    Haven't used "SuperDoctor"...
    I do use IPMI - it works well from a win7 Desktop.
    NOTE: I haven't been able to find a VMWare vSphere Client for Linux (CentOS). Consequently, I'm currently running a Win7 guest on my CentOS desktop.

    Xeon CPU:
    If I had more money and time... i'd go with the E5.
    However, the E3 is a very good workhorse. No issues. Very popular and well supported.
    Make sure you get an E3 with hyper-threading! (review the E3 models carefully)
    Also, unless you need the embedded video, don't pay any extra. Just happened that I purchased my workstation E3 CPU on special so the extra Graphics GPU (embedded) was a "freebee". However, couldn't get the Xeon graphics working for a 3x display desktop and eventually switched over to FirePro (I have a CentOS workstation).

    My ESXi server has the E3-1270 V2 3.5GHz model (purchased this prior to V3).
    My Workstation has the newer V3 addition... but, "meh..." no real empirical differences given my applications.

    Here's my ESXi server specs (please note the similarities to rootwyrm specifications)

    Haven't had a free minute to write-up my new E3 workstation build...

    Motherboard:

    Carefully review support and compatibility for RAM (ECC), CPU, multi-threading, and passthrough. SuperMicro does an outstanding job of listing what their boards can and cannot do. They also list tested OS (ESXi, etc.)

    Don't think you're going to get IPMI support on the "cross-over" models for workstation (extra PCI-E 3.0 x16). If your server is doubling as a workstation... then you really don't need IPMI (you'll have a physical display connected anyway). But, if you're building an ESXi server... then you don't need the extra x16 PCI-E slots unless you're adding some damn-expensive RAID cards... etc. This then brings up a point on exactly what sort of adapters/cards you intend to add to your server.

    Make sure you have enough slots on your motherboard. PCI-E will spread (the x16 becomes an x8, etc) with additional cards. Read the motherboards documentation and understand EXACTLY what each slot supports based on the usage and number of slots consumed.
     
  17. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    I need pcies because i have RAID LSI 9271 8i :)


    What about power suply for E3 or E5 ?

    600 W will be ok ? or more ?
     
  18. peanuthead

    peanuthead [H]ardness Supreme

    Messages:
    4,593
    Joined:
    Feb 1, 2006
    It should be fine. 350w is more than enough. Now buy the parts! :)
     
  19. Treyshadow

    Treyshadow [H]ard|Gawd

    Messages:
    1,392
    Joined:
    Sep 7, 2003
    It sounds fun and great to do this, but I am going to suggest something that makes way too much sense and just works well.

    Get 2 HP Gen 8 MicroServers. Replace their processors with Xeon E3-1220V2 processors and put in 16GB of ECC ram. Then you have 2 really small very powerful LAB devices. Storage is of your choosing, but I prefer something like a Synology for ease of use. This lab may not be fun for those that want to just build, but it works well if not better for LAB purposes. Also it will give you the opportunity to understand why large companies stick with true production servers for their setups. They just work.

    Also if you want the ILO to be available, you can do that with the purchase of an advanced license key from ebay.
     
  20. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013

    Could you show me some URLs from shop ? ebay ?

    I need to be sure, that we understand :)

    I will install

    6 x WD RED 3 TB
    1 x PCIEX LSI harware raid
    2 x 8 GB ECC memory

    E3 or E5 procesor :)
    I need to compare all of super micro for LGA2011 and signle CPU/ATX only mainboard

    350W it is ok ? Im nor sure :p

    So I need advice in that case
     
  21. jabbernotty

    jabbernotty n00b

    Messages:
    43
    Joined:
    Sep 2, 2013
    You could start by adding up the total draw yourself.
     
  22. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    I dont understand :p
    Could you explain ?
     
  23. Appleby

    Appleby n00b

    Messages:
    7
    Joined:
    Apr 28, 2013
    u should actually be the greek letter mu (&#956;) as in &#956;ATX. Its the scientific standard prefix for micro, hence micro atx.

    To do this start looking at the datasheets/web pages of the components you want to use. They should all state maximum power draw. Add it all up and you'll see how big a power supply you will need. You will be surprised how frugal you can make a virtualisation server!

    :)
     
  24. rufik

    rufik Limp Gawd

    Messages:
    151
    Joined:
    Sep 16, 2013
    I dont believe. Men it a Xeon TDP likr in i7. RAM, Six (6) hard drisks from WD (RED)

    Wrr i need calculate :p