chune's house build

Discussion in 'Multiprocessing Systems' started by chune, Mar 24, 2016.

  1. chune

    chune n00bie

    Messages:
    57
    Joined:
    Nov 2, 2013
    Long time lurker, potential first time thread starter. Here is my build. I am running esxi 5.5 on a ga-7pesh3 with donk's unlocker which allows for mac guests to be run on PC hardware (with GPU passthrough!!). This is all backed by a ZFS-powered NFS datastore (omniOS VM + nappit + onboard SAS2008 IT mode pci-passthrough). This started as just a windows + mac gpu passthrough rig and has slowly sprawled throughout my entire house. I am currently running a dual monitor win 8.1 workstation VM, Mavericks OSX dev VM, SteamOS arcade VM, and a win10 HTPC VM (Plex HT) all with individual GPUs passed through. I just realized how much shit i chained off one mini-DP port and had to document it. Yes, the dual apple cinema displays all work great from 100 feet away running through all the adapters with no relics or signal degradation. The DVI KVM Matrix allows me to mix and match any input to any output on my cinema displays. It comes with a cat-5 remote (not pictured) that allows me to select a source per monitor. The SnapX allows me to change my secondary monitor over to a local macbook laptop if needed. _Gea should get a real kick out of this one =)

    [​IMG]
     
    Last edited: Mar 24, 2016
    {NG}Fidel likes this.
  2. KazeoHin

    KazeoHin [H]ardness Supreme

    Messages:
    6,501
    Joined:
    Sep 7, 2011
    So, How does this work in terms of running multiple users simultaneously, without little niggling problems like end users accidentally closing their VM, or dragging their VM window to another monitor, recovering from crashes... or even something as basic as rebooting the whole system without having to re-set-up all the screens and hardware.

    I'm not being snarky, this is a SERIOUS interest of mine to have the entire house essentially run off of one supercomputer and simply have everyone using monitors, keyboards, controllers, etc. connected via wireless/wall plugs. Everyone I've asked said it was impossible, citing the issues above among others...
     
  3. chune

    chune n00bie

    Messages:
    57
    Joined:
    Nov 2, 2013
    So it sounds like you are talking about a ghetto setup that has everything running off one machine that uses eyefinity and some third party software to enable concurrent sessions and mouse/keyboard mapping.

    This setup uses an enterprise grade hypervisor and each VM is completely isolated from the other. Each VM has its own dedicated video card and memory. The way I overcome the USB issue is using a four port, four controller USB 3.0 card that allows me to pass through individual USB ports to each VM. The 65 foot active USB repeater cables then carry the USB port to wherever the endpoint resides.

    This setup is extremely stable and works great for gaming. A user cannot accidentally "close" their VM, but they can shut it down. This can be disabled by local security policy if it is an issue. There is also esxi cron scripts that check for non-running VMs every so often and turn them on. When performing a host reboot, you can have the VMs auto power on/off using guest shutdown in an orderly fashion. Everything persists across host reboots and all VMs will remember their monitor layout. The monitor(s) at each workstation plug directly into the GPU dedicated to that VM so it is physically impossible to "drag your VM to another monitor". The worst thing a user could do is accidentally enable the console monitor in which case they will notice really poor graphics performance but it won't lock them out or anything.
     
    Last edited: May 16, 2016
  4. Track Drew

    Track Drew Limp Gawd

    Messages:
    487
    Joined:
    Dec 6, 2007
    This is very cool, something I've toyed with but never made the jump to purchase all the enterprise hardware. You have flexibility to move displays around, but I'm thinking if things would always be static you could cut out the KVM entirely.

    How are you doing audio? I see the DVI-KVM switches audio, but nothing is mapped on your diagram. USB hub at each endpoint w/ mouse, keyboard, and USB audio adapter?

    I'd be more worried about signal issues with the VGA to the 36" CRT, but I'm guessing "basement rack" and "basement arcade" are close enough to negate this issue.

    What's your power draw at both idle and peak of the server?
     
  5. chune

    chune n00bie

    Messages:
    57
    Joined:
    Nov 2, 2013
    Right, this is a super over-complicated setup. The KVM is by no means required, it was just a nice alternative to running additional DVI-D cables upstairs through 1" smurf tube (not fun) for input 2 or 3 on my monitors. If you didn't use apple cinema displays on windows VMs, you could further eliminate both c24dls, both monoprice adapters and the snapX if you used video cards with dvi-d ports.

    The audio is handled by an internal usb sound card on the ACDs feeding the built-in speakers. That was one of the main things I liked about these monitors: 27", 2560x1440, mic, speakers, webcam and usb hub all on one (good looking) package. The Ultimate Endpoint

    I have not checked any power draw specs
     
    Last edited: Aug 30, 2017
  6. NobleX13

    NobleX13 2[H]4U

    Messages:
    2,799
    Joined:
    Jun 15, 2010
    I am loving that Gateway destination... I have not seen one of those in a few years.
     
  7. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    chune how did you get the Startech STCPEXUSB3S44V quad usb controller working on OS X ?

    I've been down this rabbit hole before when I did my workstation and tried a few options but in the end settled on a Synergy solution - which worked great in a single user environment - However, for this new build I need a similar setup to yours of having totally separate "terminals" that need to behave separately; I have all the ESXi setup done and have OSX installed and working great with an nvidia GT710 passed through -- I just can't control it! I'd be happy using bluetooth peripherals so long as I can pass the bluetooth controller through directly (without passing through the whole usb controller)

    After reading your post I ended up getting the same Supermicro case so I could vertically mount it in a cupboard while having relatively easy access to the IO panel and PCIe cards; Landed on a Xeon 2683 (14 core, 28 thread) monster on eBay for a relatively cheap price to avoid going down the double socket route (at the expense of having less pcie lanes)

    Also found 3x startech PS2/USB KVMs for £9.99 a piece; They only use VGA which sucks and PS2 which sucks even more but just means running a single 20m cat6 cable to the end point instead; I might run further cables in the future but for now it's not needed;

    I'll try and write up what I did eventually; really appreciate your original post - gave me the confidence to do this twice!

    Thanks,
    Chris

     
  8. chune

    chune n00bie

    Messages:
    57
    Joined:
    Nov 2, 2013
    I actually gave up trying to get USB 3.0 working in OSX. I ended up passing through an onboard usb 2.0 hub and it worked fine. In a previous iteration of this build I did not have any usb 2.0 hubs on the motherboard that worked for passthrough so i ended up just buying the cheapest low profile usb 2.0 pci add in card and using that for mac. It was a startech PEXUSB4DP
     
  9. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    Thanks for replying so quick! This post hasn't been updated in over a year so I didn't know if you'd come back and reply!

    Cool; I have an ASRock Extreme 4; which appears to have 2 on board controllers -- however passing them through yields nothing, so not sure which ports they control -- likely the two headers on the motherboard that don't have ports attached;

    I bought a few PEXUSB3S23 (Etron chipset) that work great for Windows 10 and Linux but so far not had any luck with OSX; Plus it's only a single controller so wastes an entire pci lane slot for 1 machine;

    Thanks for pointing out the PEXUSB4DP; just ordered one on Amazon so hoping that pushes the OSX build along; Next hurdle is whether the SV565UTPGB works on mac, I have a few bluetooth dongles and peripherals as backup though. Still can't believe I picked up 3 of those new for £9.99 when they sell for £150+; to be fair they're PS2 and VGA that nobody likely wants so suprised they are so expensive still.

    Thanks for pointing out that low profile usb 2.0 card; might go with that if I run out of ideas; Planning on eventually running 4 separate "terminals", 1x OSX and 3x windows 10 boxes so will need another motherboard at some point with more pci slots;

    Have looked into pci switches (mining seems to have brought the price of these down) but just can't figure out a way to mount them sensibly in a case; one for another day;

    Thanks,
    Chris
     
    Last edited: Aug 30, 2017
  10. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    While I'm here; DVI-D cable ends are just a tad bigger than 1"; I guess you could maybe pull them at an angle but still I can't imagine trying to do this over a long distance without causing damage to the connector! Did you really use 1" tube? I also can't imagine terminating a DVI cable...

    I was planning on going down the HDMI route eventually since they can run 20m+ without extenders and have a fairly small connector at the end; I'm using 32mm waste pipes rather than conduit since they're cheaper and a lot more rigid; Since they're super smooth for water flow they pull cables effortlessly;
     
  11. chune

    chune n00bie

    Messages:
    57
    Joined:
    Nov 2, 2013
    http://www.avsforum.com/forum/19-de...ution-2x-dual-link-dvi-through-1-conduit.html
     
  12. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    Were there any particular steps you took to get this working under Windows?

    I bought one from eBay a few months back for half price and never got it to work - assuming it was faulty I bought a brand new one from Amazon that arrived today that behaves exactly the same. With the Microsoft drivers I get code 10 on the Root hub; With the startech controllers I get a code 10 on the controller itself; The card works during boot and is usable for ESXi console;

    * Tried disabling power savings on the controller/hub;
    * Tried various BIOS options; noticed disabling USB3 on the mobo disables all USB so wondering if there's a "handover" issue with this motherboard; noticed some configurations result in "overlapping memory region" errors from ESXi which prevent it booting the guest.
    * Tried all but one of the PCI-e x16 slots; From the manual of the Asrock Extreme 4 it suggests putting x4 cards into the second slot but I've tried the first and fifth (not tried the 3rd yet) - read something about ICH6 (this board has ICH10) having issues with this particular card; figured it might be a bug on both chip sets
    * Tried molex or sata connectors onto the card itself


    I might try the card in my z170 workstation and see if its the same there; also need to try testing it on bare metal windows 10 rather than via ESXi; slightly annoyed I paid so much for this card and it not been plug and play (also the official drivers haven't been updated since 2012 wtf startech!?)

    The PEXUSB4DP arrived this morning -- works great in Windows; I didn't hook up the power connector yet since I don't have a molex floppy connector (who has these!?!); Under El Capitan it complains of excessive power draw and disables all the ports (even when nothing is plugged in) so didn't get that working yet either;

    Amazed I managed to get GPU passthrough to work so effortlessly but having so much trouble getting basic USB to work :D;

    The wife started complaining about the fan noise already so looking at some replacement 80mm fans for the rear; I turned them around already so the airflow goes up through the case since its going to be mounted vertically and that seems to have made them a bit quieter but the Asrock fan controller doesn't seem to let them spin down any slower than 2000rpm (which is slow compared to the 5-6000 rpm they can spin up to; Might have to play around a bit more with the fan controller settings;

    When it works; its great but getting these builds going eats so much time; Especially restarting the host ESXi between passthrough changes; meh again...
     
    Last edited: Sep 1, 2017
  13. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    So at the weekend I set up a bare metal Windows 10 install and got both the new and original ebay STCPEXUSB3S44V controller working; upgraded both of their firmware to the latest; upgraded the bios on the Extreme4 and still neither of them would play nice under a guest Windows 10 via ESXi (but work fine in a normal non-virtualised environment)

    I tried every combination of USB bios option; as well as every slot on the motherboard; tried various esxi options including disabling MSI; still nothing; So much wasted time -- especially since each change requires rebooting the host...

    There's so many people out there claiming this controller works fine I decided to buy another motherboard in the hope it works with that; went for an Asus x99-e WS; which has 7x 16x pcie slots in case that doesn't work and I have to revert to a controller card per terminal (4 gpu, 3 pcie usb and hopefully an on board controller to pass through); The extreme4 was only a stand in for the time being anyway since it was cheap on ebay and apparently had been used for esxi previously;

    I've read so many motherboard manuals over the last few days; surprised how little of them have this many slots; or do but have limitations on what you can do with them;
     
    Last edited: Sep 5, 2017
  14. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    The Asus x99-e WS turned up yesterday; awesome board! (So so heavy!!) Still no luck with the PEXUSB3S44V; I feel like it's a little closer this time -- sometimes when the guest boots there aren't any errors; and I can see via a USB optical mouse that keeps starting; then going dim; it cycles this for about a minute (didn't time it) before a code 43 appears instead; I haven't exhausted all USB options on this board since there are so many more! I tried a few combinations as well as some of the obvious things I'd tried on the Extreme4;

    To be fair after researching this even more; there's only really 2 people: chune and a guy on one of the overclocker forums who I've seen this working on! everyone else seems to be having issues and gives up; I haven't tried passing the on-board controllers yet; only looks like there's 2 (not sure if more appear if you plug into the headers?) seems strange having so many usb ports on this board been driven by so few controllers.

    I'm going to try some other things like going back to ESXi 5.5 (and try 6.0); In 6.5 it looks like they changed something in the USB layer so maybe something won't let go properly. I haven't tried QEMU+KVM yet on this build to see if I get any more info there;

    I reviewed the kernel.log from the web interface and I can see some errors for the card about vectors and pin assignments so I feel it is something to do with IRQs or interrupts; The other error that was odd was an issue taking control of the parent pci device for the controller; which I can't pass through.

    Tried disabling interrupt (re?)routing and also tried disabling the acs check options in ESXi;

    Main thing I don't understand is why disabling USB on the motherboard prevents it working entirely from within a guest; surely its just a pci device...
     
  15. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    Gave up in the end; requested a refund from Amazon to send the most recent PEXUSB3S44V back; Can't waste any more time on this card for the sake of future expansion :D; Tried everything I'd tried before but back on ESXi 5.5 (so strange going back to the desktop client); The on board USB controllers pass through fine; as does the PEXUSB4DP and another Startech Etron pci USB controller I had lying around; it's enough to set up the 3x terminals I wanted for now; I've read about software solutions that share USB devices that I might try at some later date!

    I hate failure and giving up; but it sure is a relief to stop spending so much time hitting my head against a brick wall; I can't be bothered buying the other manufacturer versions of this card/chipset and wasting any more time either; Have read about the Sonnet Pro and a few other non name brands but at this point I'd rather stick with what works;
     
  16. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    Managed to get at least 2 of the on-board controllers passed through; the vm needs to boot with the devices attached or else the port fails with various errors but its unlikely we'll need to do any hot swapping; I've yet to try if it works via a powered hub.

    For some reason the PEXUSB4DP only works in OSX via a USB 2.0 hub; Apart from that it works "out of the box". Still waiting on the floppy adaptor cable to power this one properly so that might be the cause; OSX continuously reports "over current" errors; however the hub is bus powered so I don't understand how its any different;
     
  17. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    Turns out I'd not fully seated the USB controller connector on the back panel PCB so the two usb 2.0 controllers on the board worked fine! So I have 2x USB 2.0 and 2x USB 3.0 controllers just from the motherboard itself; the on board ones don't need a hub to work with OS X unlike the PEXUSB4DP; So I should be fine now running at least 4-5 terminals each with their own controller;

    Started looking into the noise from the rear fans; chune I assume you just mounted it in the basement and forgot about it? Was thinking about swapping the fans for quieter ones, just needs careful consideration to make sure I don't overheat anything.
     
    Last edited: Sep 11, 2017
  18. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    Ordered a 60mm and 80mm 4-pin PWM noctua fans; one for the CPU and one for the rear of the case to test; I have a feeling they won't be able to push enough air through to keep it at a low enough temp but its worth a shot; if it doesn't work I can always send them back; Amazon is so great with returns! :D
     
  19. lightfoot256

    lightfoot256 n00bie

    Messages:
    11
    Joined:
    Aug 29, 2017
    The noctua fans are awesome! Incredibly quiet; however, I'm slightly concerned about their ability to move the same amount of air :D;

    After running the machine with these quiet fans I've now realised the majority of the noise comes from the 1U PSU (DUH!) which switches to full fan as soon as the mobo powers on; I think resolving this will have a better effect than quieting the cpu and system fans so looking into alternatives; I love this case so much for the front io and the fact I can get the air to raise from the back to the front I'm considering taking out the PSU fixtures and eitehr using a 2u PSU or mounting a super quiet ATX psu somewhere else since I already have the ATX extension cable :D;

    Looks like seasonic do a few quieter 1u PSUs; Mine seems to be about 75-80db in the "on" state; I'd be happy with something in the 40-50 range instead;

    If the loft/attic space wasn't so hot I'd just chuck it up there -- also slightly worrying having something so hot left up there too; Really wish we'd bought a house with a cellar/basement...