FS: Cheyenne Computer

erek

erek

[H]F Junkie
Joined
Dec 19, 2005
Messages
10,987
1714529646431.png

https://gsaauctions.gov/auctions/preview/282996
 
I hope whoever gets a bargain on a former top-20 system redeploys it at scale and it doesn't end up parted out. Specs-wise it's still got some gas in it, aside from bad (by 2024 standards) perf/watt and needing a cooling loop rebuild.

This was one of the last (maybe the last?) of the large-scale SGI clusters before they got swallowed by HPE. Technically SGI-branded Rackable Systems but with interconnect architecture inherited from old SGI.
 
Fine fine fine.. but will it allow me to have a thousand chrome tabs open and not experience slow down or artifacts?

The 5.34-petaflops (Linpack) system was the 20th most powerful supercomputer in the world when it launched, but now would be considered slow and inefficient when compared to cutting edge systems.
The water-cooled supercomputer features SGI ICE XA modules, 28 racks, and 8,064 Intel E5-2697v4 CPUs (for a total of 145,152 cores). It has DDR4-2400 ECC single-rank, 64 GB per node, with 3 High Memory E-Cells having 128GB per node, totaling 313,344GB.

Two air-cooled management racks consist of 26 1U Servers (20 with 128 GB RAM, 6 with 256 GB RAM), 10 Extreme Switches, and 2 Extreme Switch power units. Each rack weighs 2500 lbs.

Fiber and CAT5/6 cabling are not included.
Click to expand...

Oooooooooh so there's the issue. Always a catch. :)
"The system is currently experiencing maintenance limitations due to faulty quick disconnects causing water spray," the auction listing notes.
Click to expand...

https://www.datacenterdynamics.com/...etired-cheyenne-supercomputer-up-for-auction/
 
That's one heck of a homelab setup. :ROFLMAO: :ROFLMAO: :ROFLMAO:

Honestly, I would take this and put it in Northern Canada where cooling is cheap, and then build out a solar array grid to power it. If the solar grid and the auction is cheap enough, may be the biggest bang for buck ever considering there would be nearly zero opEx to keep it running.
 
SamirD said:
That's one heck of a homelab setup. :ROFLMAO: :ROFLMAO: :ROFLMAO:

Honestly, I would take this and put it in Northern Canada where cooling is cheap, and then build out a solar array grid to power it. If the solar grid and the auction is cheap enough, may be the biggest bang for buck ever considering there would be nearly zero opEx to keep it running.
Click to expand...

You're gonna need to find a good auction source for your solar array... 1.7 MW is a lot of power.
 
toast0 said:
You're gonna need to find a good auction source for your solar array... 1.7 MW is a lot of power.
Click to expand...
Oh yeah, and I would need to find a sunny and cold part of Canada, lol.

I still don't know why Canada isn't the data center capital of N. America--so much free cooling!
 
Free cooling is nice, but extra latency to where the people are is kind of meh.

And access to major existing networking. (Which is self reinforcing, but oh well)
 
toast0 said:
Free cooling is nice, but extra latency to where the people are is kind of meh.

And access to major existing networking. (Which is self reinforcing, but oh well)
Click to expand...
True, but how much extra you think it would be? If cloud computing vendors can do it, why not a datacenter? Even if you have to build it out I would think the roi would be there.
 
You're looking at probably 20-30 ms to go to Canada from the southern parts of the US. If it matters a lot or not, I dunno. Cloud vendors have stuff in Canada, but they've also got stuff all over the place. No big deal to have a couple datacenters in Canada when you have like 10 in the US. GCP has one in Las Vegas which tends to have worse latency to everywhere else, even people in Las Vegas are likely to get lower latency to Los Angeles or Salt Lake City; I've seen data!
 
toast0 said:
You're looking at probably 20-30 ms to go to Canada from the southern parts of the US. If it matters a lot or not, I dunno. Cloud vendors have stuff in Canada, but they've also got stuff all over the place. No big deal to have a couple datacenters in Canada when you have like 10 in the US. GCP has one in Las Vegas which tends to have worse latency to everywhere else, even people in Las Vegas are likely to get lower latency to Los Angeles or Salt Lake City; I've seen data!
Click to expand...
That would be great! I see 80ms to go from CA to AL. :ROFLMAO: Honestly the latency doesn't matter until it gets to over 90--then I've got some minor issues.
 
UCAR says that Cheynne was originally slated to be replaced after five years, but the COVID-19 pandemic severely disrupted supply chains, and it clocked in two extra years in its tour of duty. The auction page says that Cheyenne recently experienced maintenance limitations due to faulty quick disconnects in its cooling system. As a result, approximately 1 percent of the compute nodes have failed, primarily due to ECC errors in the DIMMs. Given the expense and downtime associated with repairs, the decision was made to auction off the components.
Click to expand...

I hope they tell the winner which ones those were, because that sounds like an absolute bloody nightmare to try to find that in this cluster.

As far as the price, 480k... well I looked up these chips on ebay, it looks like they're selling for about $40-50 a piece? They just got 8000 of them. 1% might be broken (?) so ~7983 chips operational. 7983*50=399150. Then there's all of the cabinets and motherboards, and ECC RAM, so... overall, about a decent price. Crazy how much tech depreciates over time. I wonder how this thing would hold up to a cluster of Nvidia AI chips (with some random modern CPU) that cost the same amount in contemporary times.

The problem is that even if they intend to just flip it, how the hell do you get rid of 8k chips on Ebay without the price for them just deflating to almost nothing? Hell, I'd be on board for grabbing some Xeons and ECC ram on the (really) cheap if they do piece it out lol.

Absalom said:
Walked away? More like trucked away. I can only imagine the cost to transport all of that. I am sure it will resemble something like what Elon did with Twitter's data center in 2022.
Click to expand...
The transportation costs are going to be absurdly high. I've had display cabinets from auctions that were priced for transport for higher than the cost of the cabinet. This is specialty tech gear.
 
StoleMyOwnCar said:
I hope they tell the winner which ones those were, because that sounds like an absolute bloody nightmare to try to find that in this cluster.

As far as the price, 480k... well I looked up these chips on ebay, it looks like they're selling for about $40-50 a piece? They just got 8000 of them. 1% might be broken (?) so ~7983 chips operational. 7983*50=399150. Then there's all of the cabinets and motherboards, and ECC RAM, so... overall, about a decent price. Crazy how much tech depreciates over time. I wonder how this thing would hold up to a cluster of Nvidia AI chips (with some random modern CPU) that cost the same amount in contemporary times.

The problem is that even if they intend to just flip it, how the hell do you get rid of 8k chips on Ebay without the price for them just deflating to almost nothing? Hell, I'd be on board for grabbing some Xeons and ECC ram on the (really) cheap if they do piece it out lol.


The transportation costs are going to be absurdly high. I've had display cabinets from auctions that were priced for transport for higher than the cost of the cabinet. This is specialty tech gear.
Click to expand...
It'll tell you which nodes - if you bother to hook it all back up again.

And no, the AI chips would run rings around it.
 
You must log in or register to reply here.
Back
Top