[Build Log] Excelsior! Threadripper KVM Build with GPU Passthrough and ZFS

Discussion in 'Worklogs' started by TheHobbyist, Dec 6, 2018.

  1. TheHobbyist

    TheHobbyist Hugs Hard Johnnies [H]ard

    Messages:
    456
    Joined:
    Apr 8, 2008
    Greetings True Believers!

    Lets go on a journey together. Join me as I plan and execute an amazing computer for myself and my family that will go by the name of “Excelsior!” in honor of the great Stan Lee.

    To live up to its name, it must be something truly exceptional. My vision for Excelsior! is that it will have a host such as Proxmox that supports GPU Passthrough as well as ZFS and it will serve as 2 Windows 10 Gaming Workstations, a NAS, and whatever else we can dream and implement!

    The Hardware:

    • Processor: AMD Threadripper 2950X
    • CPU Cooler: Cooler Master Wraith Ripper
    • Motherboard: ASRock X399 Taichi
    • Memory: 128GB DDR4 ECC
      • (8x) Crucial 16GB DDR4-2666 ECC UDIMM
        • Part #: CT16G4WFD8266
        • Crucial Part #: CT12146164
    • Storage:
      • (3x) Samsung 970 Evo 500GB M.2 NVME
      • (8x) Seagate 12TB IronWolf ST12000VN0007
    • Video Cards:
      • GPU (Host)
      • GPU (Windows 10 Guest)
      • GPU (Windows 10 Guest)
    • Case: Fractal Design Define R6 Blackout FD-CA-DEF-R6C-BKO
    • Power Supply: Seasonic PRIME 1300 Platimun SSR-1300PD
    Excelsior! PC Part Picker

    The Software:

    • Host:
      • Proxmox
      • ?Other Linux KVM?
    • Guest:
      • Windows 10 with GPU passthrough
        • Main Workstation
      • Windows 10 with GPU passthrough
        • NVIDIA GameStream VM to NVIDIA Shield TV in living room
        • Parsec Server
    Research, Design, Planning:

    • Phase 1: Research and Design
    • Phase 2: Acquire Hardware
    • Phase 3: Windows 10 Bare Metal
      • Hardware Stage 1:
        • Windows 10 Bare Metal Install
          • OEM Cooler Master Wraith Ripper Thermal Interface Material
          • Single GPU
          • Single NVME
          • No 3.5" HDDs
        • Overclocking and Performance Benchmarking
      • Hardware Stage 2:
        • Windows 10 Bare Metal Install
          • Replace OEM Cooler Master Wraith Ripper Thermal Interface Material with Prolimatech Pk-3 Using [H] Method
        • Overclocking and Performance Benchmarking
    • Phase 4: Linux
      • Hardware Stage 3:
        • Dual Boot Windows 10 Bare Metal and Linux
          • Single GPU
          • (2x) Samsung 970 Evo 500GB M.2 NVME
          • (8x) Seagate 12TB IronWolf SATA HDD
        • Learn Linux, Debian, ProxMox, Arch Linux, ZFS
        • Decide on hypervisor and final hardware configuration
    • Phase 5: Virtualized
      • Hardware Stage 4:
        • Single Boot Linux Hypervisor
          • Full Hardware Build
            • Boot from ZFS Mirrorred USB 3.1 Drives
            • Multi-GPU
            • (3x) Samsung 970 Evo 500GB M.2 NVME
          • Windows 10 Guest 1
            • 8 cores, 16 threads
            • 64GB RAM
            • (1x) Samsung 970 Evo 500GB M.2 NVME
            • Single GPU Passthrough
          • Windows 10 Guest 2
            • 4 cores, 8 threads
            • 8GB RAM
            • (1x) Samsung 970 Evo 500GB M.2 NVME
            • Single GPU Passthrough
        • Performance Benchmarking and Comparison vs Bare Metal
    • Phase 6: ...

    The Build:


    g9gLcgo.jpg

    It all starts with a blank canvas.

    fvNIuMq.jpg

    Hardware acquired!

    aOgNiP6.jpg

    Don't forget the case that comes in an absolutely massive (but protective!) box.

    AQpAxnm.jpg

    Lets invite some old friends to the party.

    ijz2pg7.jpg

    2 SATA cables. Hmm... that's not gonna cut it!

    2aQ0W9y.jpg

    A beautiful motherboard, but it looks lonely. I think a team up is in order!

    IXyQPlu.jpg

    Tai Chi, meet Threadripper. This unboxing was very satisfying, not gonna lie. An included torque wrench? Gigity!

    kM1hQrm.jpg

    Alright, now they need to figure out how to work together.

    nmTQ6W4.jpg

    This is my first go at a Threadripper build, lets see if I can NOT BREAK IT.

    5Xb2W5b.jpg

    Oh lawd... so many fragile pins exposed...

    13qzGXY.jpg

    It's a fascinating install process. It feels very substantial and well engineered.

    8j3Ajw6.jpg

    Threadripper in place. Almost there!

    ohoUbWn.jpg

    Phew. Threadripper safe and sound in its socket.

    bH9knBG.jpg

    Watch out Tai Chi! Ninjas! Oh, wait, you guys wanna team up too?

    HNNJfyd.jpg

    Lets see what you've got Ninjas.

    0a9P2CU.jpg

    Triple flying ninja kick!

    lpvKFyg.jpg

    Man, I can't believe you guys landed that.

    9UcsOvX.jpg

    Our team is growing in infamy. Mighty champions flock to us to join our righteous cause.

    K9dZxkl.jpg

    These emerald warriors fit right in. The time before they were part of the team feels like a distant memory.

    Z6mt0Ab.jpg

    Watch out! A Wraith Ripper! Be you friend or foe?

    mZYCFR0.jpg

    Wraith Ripper, with the help of your friend T.I.M., I think you and Threadripper will work well together.

    fumQiy4.jpg

    What an overwhelming presence!

    NPpGDkK.jpg

    The protective films are off. This is getting serious! Wraith Ripper, such a towering behemoth!

    I will return soon to further flesh out this first post, but for now I eagerly await your feedback, whether it be on hardware choices, system design, software advice, or otherwise!

    Excelsior!
     
    Last edited: Dec 11, 2018
  2. BoiseTech

    BoiseTech Limp Gawd

    Messages:
    308
    Joined:
    Mar 15, 2018
    My only feedback is. DAAAYUMN!
     
    Adidas4275, N4CR and TheHobbyist like this.
  3. N4CR

    N4CR 2[H]4U

    Messages:
    3,622
    Joined:
    Oct 17, 2011
    Pretty much my dream build... can you be my dad?
     
    TheHobbyist likes this.
  4. Fritzz

    Fritzz Gawd

    Messages:
    683
    Joined:
    Feb 11, 2008
    Sub'd for epicness.....
     
    TheHobbyist likes this.
  5. Engr62

    Engr62 Limp Gawd

    Messages:
    467
    Joined:
    Mar 24, 2015
    Stan the Man!
     
    TheHobbyist likes this.
  6. DogsofJune

    DogsofJune 2[H]4U

    Messages:
    2,335
    Joined:
    Nov 7, 2008
    This should be cool. Looking forward to updates
     
    TheHobbyist likes this.
  7. fullvietFX

    fullvietFX [H]ard|Gawd

    Messages:
    1,777
    Joined:
    Sep 1, 2004
    Wow this is epic. That's a lot of storage.
     
    TheHobbyist likes this.
  8. TheHobbyist

    TheHobbyist Hugs Hard Johnnies [H]ard

    Messages:
    456
    Joined:
    Apr 8, 2008
    Johnny, do you play baseball? lol!

     
    pendragon1, N4CR and Engr62 like this.
  9. N4CR

    N4CR 2[H]4U

    Messages:
    3,622
    Joined:
    Oct 17, 2011
    LOL perfect, a fellow fenslerfilm fan to boot - if you're interested I have an ISO of that original DVD from before it got lawyered and I don't know of anyone with the full DVD rips uploaded. Can slowly upload it to mega for you if you wish. Just post lots of build porn later lol
     
    TheHobbyist likes this.
  10. TheHobbyist

    TheHobbyist Hugs Hard Johnnies [H]ard

    Messages:
    456
    Joined:
    Apr 8, 2008
    I had some time to work on the build and added some content to the OP. Enjoy!
     
  11. TheHobbyist

    TheHobbyist Hugs Hard Johnnies [H]ard

    Messages:
    456
    Joined:
    Apr 8, 2008
    I'll PM you about that DVD! Which one is your favorite?
     
  12. fullvietFX

    fullvietFX [H]ard|Gawd

    Messages:
    1,777
    Joined:
    Sep 1, 2004
    I never realized how big that cooler is holy crap.
     
    TheHobbyist likes this.
  13. DogsofJune

    DogsofJune 2[H]4U

    Messages:
    2,335
    Joined:
    Nov 7, 2008
    No doubt. Glad I went with a waterblock.

    Nice progress so far
     
    Adidas4275 and TheHobbyist like this.
  14. N4CR

    N4CR 2[H]4U

    Messages:
    3,622
    Joined:
    Oct 17, 2011
    Gosh that's a tricky question.. computer one is always a classic, the australian one is absurd, medicine one and the indian camping one come to mind but really.. they're all hilarious in their own way for different settings.
     
    TheHobbyist likes this.
  15. TheHobbyist

    TheHobbyist Hugs Hard Johnnies [H]ard

    Messages:
    456
    Joined:
    Apr 8, 2008
    I’m trying to decide on a HOST os. Originally, it seemed like ProxMox VE would be a good choice as it has tons of the feature I need. However, ProxMox VE 5.3 is based on Debian Stretch 9.61, kernel 4.15.18, QEMU 2.12.1, and ZFS 0.7.12 . I have read that for threadripper, I really want a kernel that is 4.18 or later and a recent version of QEMU (3.0 or later, preferably with the PCI-E link speed negotiation patch). From what I have read, I will not be able to easily update the Kernel in ProxMox VE 5.3 (at least with my skill level) but I’m not sure if I could update QEMU. Going down this rabbit hole, it seems like ArchLinux would be my best bet for a Host OS that is properly updated. What do you think?
     
  16. IdiotInCharge

    IdiotInCharge [H]ardForum Junkie

    Messages:
    9,643
    Joined:
    Jun 13, 2003
    Arch if you're into cults- Manjaro is Arch-based and lets you swap kernels at will.