Are EPYC chips difficult to find?

Zarathustra[H]

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
38,822
So,

I was recently forced to replace the Supermicro motherboard on my server (see sig) and even with the one I ebayed, I am now seeing intermittent hard freezes. (once every ~14 days or so)

I was holding on to the current server for a long time due to the fact that I already have my 256GB of registered ECC RAM, and I wasn't looking forward to rebuying it all in DDR4, but maybe it is time after all.

Anyway, I started playing around with the idea of moving from my old dual socket Intel server with two 8C/16T chips, to a single socket 16C/32T EPYC.

Then I started looking around for parts. Supermicro motherboards seem to exist at somewhat reasonable prices, but EPYC CPU's seem to be going for more than list price everywhere I can find them.

Is there a secret to finding them at the right prices, or is there a shortage, and this is just the way it is?

Much obliged
 
Normally I'd swing by the ewaste piles on each floor of our data centers but the guys that work there have been stripping them out.

Those guys have a pretty good side hustle reselling gear they find thrown onto a pallet with AMD parts right now. I mean most of what they do at our colos is secure our cages. Branded polo shirt Dell/HP/Cisco/Juniper/etc bodies do the actual work. Colo staff are basically janitor/garbage men.

I'd go figure out how to get on a want list with one of them.

Be warned, most of those guys are not people I'd want to be friends with, just friendly.
 
Limited supplies, lots of demand (especially on the enterprise side right now), hard to find, etc.

Unless you want the mega 64C beast. I can get those for MSRP... but that's one hell of an MSRP.
 
Zarathustra[H] said:
So,

I was recently forced to replace the Supermicro motherboard on my server (see sig) and even with the one I ebayed, I am now seeing intermittent hard freezes. (once every ~14 days or so)

I was holding on to the current server for a long time due to the fact that I already have my 256GB of registered ECC RAM, and I wasn't looking forward to rebuying it all in DDR4, but maybe it is time after all.

Anyway, I started playing around with the idea of moving from my old dual socket Intel server with two 8C/16T chips, to a single socket 16C/32T EPYC.

Then I started looking around for parts. Supermicro motherboards seem to exist at somewhat reasonable prices, but EPYC CPU's seem to be going for more than list price everywhere I can find them.

Is there a secret to finding them at the right prices, or is there a shortage, and this is just the way it is?

Much obliged
Click to expand...

What are you doing with that thing? I have 2x L5640's (6/12 each) with 96GB of registered ECC and it barely ever breaks a sweat :). Was thinking about setting it up as a software rendering farm for Blender, but honestly I'd be better off buying a few GPU's and offloading. I'm probably going to end up replacing it with a high end zen or low end threadripper, I just think EPYC is overkill for my needs, but always curious what people are able to put together. A single Ryzen 1600 gets about the same multi-core score as my dual xeons. I just bought my son a 3700x for his computer which would walk all over my current server. Every time I think about upgrading it, I remember... it just works and I really don't have a compelling reason. Anyways, just making a post and listening in for some suggestions because you never know ;). That link to servethehome has a few lower priced deals. Finding 'upgrades' isn't a half bad idea, they are the same damn chip, lol, they just call it an upgrade for one of their systems.

ps. Not sure how much you're looking at spending, but:
https://www.dell.com/en-us/work/sho...aBPVsQDM4LUxOQp45KgaApO9EALw_wcB&gclsrc=aw.ds

Can go to a few places like dell, hp, etc and put some together to get an idea on prices.
 
Ready4Dis said:
What are you doing with that thing? I have 2x L5640's (6/12 each) with 96GB of registered ECC and it barely ever breaks a sweat :). Was thinking about setting it up as a software rendering farm for Blender, but honestly I'd be better off buying a few GPU's and offloading. I'm probably going to end up replacing it with a high end zen or low end threadripper, I just think EPYC is overkill for my needs, but always curious what people are able to put together. A single Ryzen 1600 gets about the same multi-core score as my dual xeons. I just bought my son a 3700x for his computer which would walk all over my current server. Every time I think about upgrading it, I remember... it just works and I really don't have a compelling reason. Anyways, just making a post and listening in for some suggestions because you never know ;). That link to servethehome has a few lower priced deals. Finding 'upgrades' isn't a half bad idea, they are the same damn chip, lol, they just call it an upgrade for one of their systems.

ps. Not sure how much you're looking at spending, but:
https://www.dell.com/en-us/work/sho...aBPVsQDM4LUxOQp45KgaApO9EALw_wcB&gclsrc=aw.ds

Can go to a few places like dell, hp, etc and put some together to get an idea on prices.
Click to expand...

I had a few VM farms at home that I used to use all the time; less so now (less need, more power at work to use, etc).
 
Ready4Dis said:
What are you doing with that thing? I have 2x L5640's (6/12 each) with 96GB of registered ECC and it barely ever breaks a sweat :). Was thinking about setting it up as a software rendering farm for Blender, but honestly I'd be better off buying a few GPU's and offloading. I'm probably going to end up replacing it with a high end zen or low end threadripper, I just think EPYC is overkill for my needs, but always curious what people are able to put together. A single Ryzen 1600 gets about the same multi-core score as my dual xeons. I just bought my son a 3700x for his computer which would walk all over my current server. Every time I think about upgrading it, I remember... it just works and I really don't have a compelling reason. Anyways, just making a post and listening in for some suggestions because you never know ;). That link to servethehome has a few lower priced deals. Finding 'upgrades' isn't a half bad idea, they are the same damn chip, lol, they just call it an upgrade for one of their systems.
Click to expand...

I find the biggest motivation to replace older servers is the power consumption. I was surprised how much my old Bloomfield era Xeons were costing me every month in power.

My server is somewhat of a "home production" server. It runs a bunch of VM and containers for my home use, some of them which are media based, and require real time responsiveness in order to not cause problems, so I have been very careful to not oversubscribe the CPU cores. No more than one virtual core per physical core has been my philosophy.

I'm less concerned abput having screaming fast cores, just want to make sure that the VM's arent stealing CPU time from eachother should they kick in at the same time.

The server is running Proxmox so that I can easily run both KVM and LXC with a nice web interface. Things I have running on it:
  • House NAS via native ZFS on Linux. The NAS stores all house files (very little is stored on local machines) as well as the house media library (mostly ripped blurays in native disc format)).
  • Dedicated rsyslogd container for offloading all the firewall logging from my standalone pfSense box (Ubuntu Server 18.04 LTS)
  • Dedicated LXC container for SMB/CIFS shares from NAS (Ubuntu Server 16.04 LTS)
  • Dedicated container for a MythTV backend which I use for PVR/DVR (Ubuntu Server 18.04 LTS with LXDE installed, accessed via X2Go)
  • Dedicated container for a small DHCP server to hand the HD Homerun tuners their IP addresses on a separate subnet (*1) (Ubuntu Server 16.04 LTS)
  • General Purpose Ubuntu server container I use from console for things I want to run offline when I am using my desktop for other things (mainly long downloads, but other stuff as well) (Ubuntu server 16.04 LTS)
  • Dedicated (and fully Chrooted) SFTP Server container I use for sharing files with my friends (Ubuntu Server 16.04 LTS)
  • Dedicated container for my Unifi Wireless Access point software. (Ubuntu Server 16.04 LTS)
  • Dedicated container for the server for my Unifi-Video Security Cameras. (Ubuntu Server 16.04 LTS)
  • Dedicated container for my TeamSpeak server (Ubuntu Server 16.04 LTS)
  • Dedicated VM for Wireguard VPN server (Ubuntu Server 18.04 LTS)
  • Dedicated VM for a friends remote backups (*2) (UBuntu Server 16.04 LTS)


(*1) The HDHomerun boxes have a nasty habit (which cannot be disabled) of announcing themselves to everything (windows boxes, smart TV's etc) via multicast, and if any user uses these devices to request a tuner, they immediately stop what they are doing and respond. MythTV expects dedicated tuners and does not play well with others, so I needed to isolate them. Turns out they lack the ability to set static IP's so I needed a DCHP server on that subnet

(*2) A few years back I was getting frustrated at how expensive colocation was. I had this extra server and I wanted to stash it somewhere to run remote backups to it via ZFS built in incremental block based Send/Recv command. Turns out a friend of mine was looking for something similar. I gave him an encrypted TB of storage on a VM for him to back up to and he let me stash my extra server in his basement. Nice arrangement.

When I first started doing this, I was using ESXi and everything was a VM. After I got fed up with VMWare, and migrated to Proxmox and KVM in 2016, I initially ran everything in VM's as well, but then I gradually migrated most of my stuff to LXC containers as they are much more lightweight. I only kept the VM's where I needed them either for more secure isolation or different operating systems. Now there are only 2 of them left!

I have to admit, the server does sit with pretty low load most of the time. This is pretty typical:

1594257095612.png


This is the last years worth of system averages. (There was some downtime in May/June due to my move)

Networking data is confusing as it is in Bytes, not in Bits as expected.

ZFS annd VM's love RAM though!

Ready4Dis said:
ps. Not sure how much you're looking at spending, but:
https://www.dell.com/en-us/work/sho...aBPVsQDM4LUxOQp45KgaApO9EALw_wcB&gclsrc=aw.ds

Can go to a few places like dell, hp, etc and put some together to get an idea on prices.
Click to expand...

Yeah, I'll be building my own, probably using a Supermicro (or maybe Tyan) board. I already have a 4U Supermicro CSE 846 which I really like. I spent some time modding it to put in a custom 3x120mm fan wall (instead of the awful high speed noisy 80mm fans it came with) and ordered their specialty quiet dual 920W power supplies.

I was completely tuned off from pre-built servers years ago when I bought a HP DL 180 G6 barebones machine on ebay. The thing was loud even under ideal circumstances. It had 8x redundant 80mm 18krpm fans. The fun part was that HP put thermal sensors on all of their supported PCIe cards which dialed home to the HP ILO to tell it everything was A-OK temperature wise. I needed an HBA for ZFS though, not the hardware RAID card it shipped with, and at the time there werent any, so I installed a 3rd party PCIe SAS HBA. It didn't have the temp sensor the HP ILO expected, so the system freaked out, assuming it was in an overheated state, and maxed out the 8x 18krpm fans. The thing literally sounded like a taxiing jumbojet.

Apparently others agreed, as I have seen it referred to as the HP DL180 G6 Dreamliner.

Listen to this video what it sounds like at post. Then picture it revving up maybe 25% more than that and staying there if an unsupported PCIe card was inserted....



I had it in my basement, but I could still hear it two stories up in my second floor bedroom, even with all the doors shut in between. it was crazy.

These HP/Dell/Etc servers are intended for datacenter use, where they are great, but I don't ever want to deal with a noisy server like that again. I tried modding it (fan volt mods etc.) but could never get it to behave quietly enough to where I could use it at home. Years later I was able to get a compatible HP SAS HBA, which quieted it down and used it for a little bit as my remote server at my friends house, until I upgraded my main server (In a Norco 4U case) to my current Supermicro SC846 system, and replaced the remote one wit the Norco.

Anyway, since I finally have a case and fan setup I really like, my next upgrade/replacement will simply be a standard form factor server board from SuperMicro or Tyan which will be dropped in this existing case.
 
Zarathustra[H] said:
I find the biggest motivation to replace older servers is the power consumption.
Click to expand...
Yeah, my server averages 140-150w. So lets say 150w * 24 = 3.6kw per day. I pay 11 cents per KW/h (less at night, but it's the average) so 365*3.6 = 1,314kw/h * .11 = $144.54 per year (~$12 per month)
Lets say a "new" server will average 1/2 the power (not sure on a real #, I really should look up power figures for newer stuff but it's hard to find especially when the load changes). So I can cut my bill by about 1/2 so now my running cost is $75 per year a savings of about $75. If I can build a new server for $1,000 (lol, yeah right) it'd only take me 13 years to recoup my money....

Ok, lets pretend my new server would use zero power (which it won't)... and the new one will cost me $1k. It would still take 6.9 years to pay off the new server. I just don't see how power savings add up enough for this to make sense. Everytime I look to upgrade I'm reminded it's not really netting me much. So basically until my work load goes so high that I am at 300+ watts and I can pick up a server that can do that work @ 50w for under $1k, I don't see how it would make any sense. I've done the math, I understand that large companies that have servers churning 24/7 and pay high costs for power it makes sense. For my server that does a little bit here and there, ramps up when i need it and goes back to idle. I just can't make the #'s work to justify it. That said, I will eventually get a new server when I feel this one is getting to old to support my needs or a new need arrives, but every year or two I check and see what's available and realize it really won't net me much just cost me $.

Zarathustra[H] said:
Yeah, I'll be building my own, probably using a Supermicro (or maybe Tyan) board
Click to expand...
I figured as much, was just saying you can get an idea what they're going for so when you're building you know if you're doing ok.

MY Dell R710 was loud too... some reason the fans didn't spin down as far as they should at lower speeds. Common complaint. I wrote a C++ program that checks the CPU core temperatures every ~10 seconds and adjust the fans through the IPMI interface, since Dell decided the fans should not be correlated to temperature at all I decided to fix that :). Now my server is very quiet when at lower powers or short spikes but will ramp up based on my custom fan curve in my app. My server sits in my dining room. My kitchen is between my dining and living rooms and there are no doors between any of them. I thought about going another route (new fans or custom fan controller with one my microcontrollers) but was able to get the data from the MB for CPU temps and manually control fans through the IPMI interface/tools and it no longer bothers me at all. The fans used to sit at 3,000 RPM at idle (they run over 12k at full speed). Now they are around 1500 and very rarely break 2000. When they do it's because I have a sustained load and even then I rarely see it break 3k, which was idle speeds prior.


My power monitor: Average 147w over the last week, which is typical.
xvWuuPn.jpg
 
Last edited:
It's so sad what businesses do with their old hardware. I work for a giant financial processor and we literally decommission perfectly capable and maintained hardware just because of age and that's it. These are 10/20/30k+ server racks that just get piled up and we pay a company to come and dispose of it. Sadly we aren't allowed to pick through any of it. These servers aren't even close to being worn out in reality, they lived their life in temp and humidity controlled rooms with the cleanest conditioned power sources, and for the most part sitting idle. When it comes to processing most of the bottleneck is storage and network throughput anyhows so these systems never get taxed.
 
I built a 7702p workstation about 2-3 months back for work. The only luck I had getting near MSRP was buying a motherboard+CPU combo.

This gets you pseudo-close to MRSP...maybe a $50 premium depending on what you value the motherboard at. Keep in mind that, in the unlikely event you need RMA service, you would have to RMA (the CPU) through Supermicro, since it's a combo - and you 'only' get a 1-year warranty. You'll also want the latest BIOS update for that motherboard because it fixes quite a few nuances.

All of that said, the 7702p workstation I built (512GB of 2666 MHz registered ECC in 8-channel config in addition to the 64-core CPU) is rock solid and is a real workhorse.

Best of luck.
 
Why not go x399 TR2? The expense of Epyc seems wasted on a casual home server.
 
You must log in or register to reply here.
Back
Top