Rate my setup, server style

[grasshoppa]

I'm moving from esxi to proxmox, and I've been playing with different setups. I've been somewhat isolated in a small business of late, so I'd like some outside perspective on the best way to proceed.

Servers
- Integrated storage, Threadripper for each of the two servers. ZFS SSD striped mirrors, with a LOG device, as well as an HDD pool for VM logs ( to keep writes to an SSD to a minimum ). No shared storage

Servers 2
- Truenas ( MiniX+ with a total of 64gb of memory ) backend with 20tb of rust, ssd log via nfs. Same server hardware from the first option ( ie: two threadrippers ), replacing the storage options with just enough to boot to proxmox ( ssd zfs raid 1 ).

Servers 3
- 2x truenas from option 2 replicating their data ( for extra redundancy ), and instead of a threadripper VM host, using 4 minisforum UM790 ( 64gb, 1tb zfs raid 1 ) as the VM hosts.

Options 2 & 3 include a 10g switch and associated hardware.

I'm leaning towards #3 for the HA options, despite the non-server level hardware ( no ecc ), but I've been out of the game for a while, so...what say you, [H]?

 

[ChRoNo16]

Pictures or GTFO

 

[grasshoppa]

Pictures or GTFO

Haven't bought anything on the lists yet.

Well, that's not entirely true; I did buy the UM790 for my home lab

Tiny but mighty.

 

[Scoobydo2]

Have you considered hyper converged?

I just built a hyper converged (3 hosts) in proxmox and spent roughly $6000-$8000 (dont remember exactly) on it about 6 months ago.

Essentially 3 - AMD Ryzen 7900's 12 core-24 thread, 128 gigs of ECC RAM, 64 gig boot/logging SSD, 2 TB fast "host" ssd, 14 gig TB slow "storage" hdd, 10 gig ethernet for ceph backend between all 3. This allows HA. All the hardware uses enterprise level components (the drives, ram). The only thing that isn't are the motherboards, while they support ECC, they are not server boards. I didn't need the server remote control chip (BMC).

Host 1 contains:
-LDAP server
-Mail gateway (proxmox)
-Modoboa email server
-OpenMediaVault fileserver (truenas equiv I think) for windows shares.

Host 2 contains:
-Linux Mint client
-PLEX media server
(extra room still)

Host 3 contains:
-Postgres SQL
-Postgres pgbouncer db pool
-Application server
-Apache Website (external reverse proxy)
-Apache rest API website (internal website)

I also had an old server to put on PfSense for the firewall/router.
I also had another old fileserver I used as an offline backup storage for the 3 hosts using proxmox backup server.

 

[grasshoppa]

Have you considered hyper converged?

I just built a hyper converged (3 hosts) in proxmox and spent roughly $6000-$8000 (dont remember exactly) on it about 6 months ago.

Essentially 3 - AMD Ryzen 7900's 12 core-24 thread, 128 gigs of ECC RAM, 64 gig boot/logging SSD, 2 TB fast "host" ssd, 14 gig TB slow "storage" hdd, 10 gig ethernet for ceph backend between all 3. This allows HA. All the hardware uses enterprise level components (the drives, ram). The only thing that isn't are the motherboards, while they support ECC, they are not server boards. I didn't need the server remote control chip (BMC).

Host 1 contains:
-LDAP server
-Mail gateway (proxmox)
-Modoboa email server
-OpenMediaVault fileserver (truenas equiv I think) for windows shares.

Host 2 contains:
-Linux Mint client
-PLEX media server
(extra room still)

Host 3 contains:
-Postgres SQL
-Postgres pgbouncer db pool
-Application server
-Apache Website (external reverse proxy)
-Apache rest API website (internal website)

I also had an old server to put on PfSense for the firewall/router.
I also had another old fileserver I used as an offline backup storage for the 3 hosts using proxmox backup server.

Bumping this 2 years later for an update.

I did end up going with 4 Minisforum UM790s ( 2x2tb storage, 96gb ram ) in a proxmox cluster. Initially I did the hyperconverged thing with ceph, but as these units only have 1x2.5g network connection it wasn't all that exciting. I ended up throwing together an SSD nas sitting on a 10g connection for the shared storage. No shade with ceph, but a more traditional NAS is easier to admin ( and train a replacement in supporting ), plus because of the limited connectivity of the devices ceph really didn't get to stretch its legs.

Anyway, two years in and this setup is awesome. I think I spent under 6g for the entire set up ( including the nas ), and for the speed/capacity I got it was pretty damn incredible. Yes, I'm missing ECC, but that hasn't presented as an issue so far ( and I don't anticipate it BEING an issue going forward ). I did have a node fail, but it was as simply as buying a replacement node, moving the ram/storage over and booting it up; fixed.

All in all, extremely pleased with this setup. Oh, I would have loved to have played with the threadrippers + have ECC, but as I'm finding here; it would have dramatically added to the cost with no real return on investment ( excepting my own amusement ).

 

[littlegreenmenfromhome]

Proxmox have come a long way.

After having looked at Nutanix(Expensive), XCP-ng (a bit to locked, although not quite immutable), Unbuntu Microcloud (Too easy to crash under heavy workloads), Proxmox and Dell Aether (Not out in public yet) we choose Promox for internal VM's (and Nutanix for customer UC platforms) due to stability, costs and freddom of tweaking).