Advice requested on large-capacity RAM build - 512gb

P

PiSquared

Limp Gawd
Joined
Oct 15, 2002
Messages
269
Hi folks,

I'm trying to put together a cost-effective build that needs 512gb RAM. Yes, that's RAM, not storage. I know it won't be cheap, per se, but I want to at least be efficient. There's no absolute budget limit but as I said I want to be efficient with the spend.

Purpose: I'm trying to help a friend with a long-term project. He doesn't want me to be specific about it, but imagine programming something that has to analyze a crapload of things like a chess engine. He wants a huge pile of RAM to work with and will be using Windows 10 Pro. As far as I can tell he's serious and plans to use this for at least a few years so I want to make sure he doesn't get hosed or buy something dumb. Used parts are A-OK if you have suggestions. The RAM capacity is the key, does not have to be the latest technology. It wouldn't hurt to get at least 8 cores/16 threads with a decent boost clock but that's secondary. Spending time on PC Part Picker was a bit fruitless so I thought I would check with the experts here as this is out of my lane (I'm a gamer and builder of gaming PCs). It's also possible I just didn't know the right things to look for.

Any advice is greatly welcome! Thanks for reading this far.

Cheers, and have a great weekend.
 
X9DRi-LN4F+ (dual lga 2011) dual e5 2687w v2 (or other cpus) and 24 32gb sticks of ddr3 ecc ram. (~1k for 768gb of ram)

I'm not convinced there exists a workload that only is dependent on ram capacity with disregard to ram speed, or other processing capabilities.

Buy a high end pcie or nvme ssd?
 
cdabc123 said:
I'm not convinced there exists a workload that only is dependent on ram capacity with disregard to ram speed, or other processing capabilities.
Click to expand...
To be honest I'm not fully either. My understanding (which is not necessarily accurate) is that he wants to do a lot of modifying data in RAM which on a SSD would be writes on a program that is designed to run all the time. Think of chia mining/farming but probably not quite that intense on the writes. Do I know that this is the best way to do it? Nope! I'm trying to fulfil the brief of 512 gb RAM without getting hosed.

How much slower is DDR3 than DDR4 in terms of latency and transfers anyway? I'm guessing it is still noticeably faster than a SSD so long as the tasks stay within RAM
 
PiSquared said:
To be honest I'm not fully either. My understanding (which is not necessarily accurate) is that he wants to do a lot of modifying data in RAM which on a SSD would be writes on a program that is designed to run all the time. Think of chia mining/farming but probably not quite that intense on the writes. Do I know that this is the best way to do it? Nope! I'm trying to fulfil the brief of 512 gb RAM without getting hosed.

How much slower is DDR3 than DDR4 in terms of latency and transfers anyway? I'm guessing it is still noticeably faster than a SSD so long as the tasks stay within RAM
Click to expand...
Its slower especially the ecc stuff but 512gb of ddr4 would be extremely expensive.

You do gain the advantage of write endurance with ram vs sdds. Something like mining chia could be done plotting with just ram.

It is also worth noteing there are ssds that can withstand pretty decent amounts of write cycles. So I guess the question of could you use a ssd would depend on exactly how much endurance you would need and if you could make software utilize a ssd instead.

As far as cost goes I don't see any practical way to make this happen without using ddr3 ecc sticks
 
I'm going to pile onto the used server plan. Used ddr3 registered ecc sometimes goes for super cheap, because the secondary market in server gear is small. If you're running your machine hard, power efficiency increases mean you don't want to run old servers, and you can't run registered memory in desktop boards so...

I've run Xeon 2690 v1-4 with up to 768G. We were using database software where everything had to fit in ram, so there you go. We ran data services on the same servers that held the data, many of the databases either weren't that big, or needed more cpu to process them, but one of our databases was real big and didn't use much cpu, so I think we had about 256 machines with all the ram we could stuff in them. Supermicro boards. Check the board revision notes, I think for some of the machines where we didin't order with gobs of ram, we had to swap boards to get full capacity.
 
Thanks to you both for the suggestions! Any ideas on where to get a used server for (relatively) cheap? If we're going to go this route I'd much rather find something used if possible and I have no idea where is good to look.
 
I haven't bought from them, but maybe take a look at Unix Surplus (Santa Clara, CA), something like https://unixsurplus.com/supermicro-1u-x9dri-ln4f-1-1-4bay-3-5-tq-w-cpu-and-memory/

Their server configurator doesn't go above 128 Gb ram (that I saw), but the servers can take it, so it might be worth email/call to see if they can stick more ram in it; or maybe recommend something. I'd check on ebay too, just to make sure prices are reasonable.
 
toast0 said:
I haven't bought from them, but maybe take a look at Unix Surplus (Santa Clara, CA), something like https://unixsurplus.com/supermicro-1u-x9dri-ln4f-1-1-4bay-3-5-tq-w-cpu-and-memory/

Their server configurator doesn't go above 128 Gb ram (that I saw), but the servers can take it, so it might be worth email/call to see if they can stick more ram in it; or maybe recommend something. I'd check on ebay too, just to make sure prices are reasonable.
Click to expand...
not a bad price for the whole server

supermicro psu PWS-563-1H is one of my favorite power supplies :D
 
cdabc123 said:
X9DRi-LN4F+ (dual lga 2011) dual e5 2687w v2 (or other cpus) and 24 32gb sticks of ddr3 ecc ram. (~1k for 768gb of ram)

I'm not convinced there exists a workload that only is dependent on ram capacity with disregard to ram speed, or other processing capabilities.

Buy a high end pcie or nvme ssd?
Click to expand...

If you start swapping your performance tanks to basically zero unless your code was explicitly written to swap in and out of disk, even Optane is orders of magnitude slower than RAM in most metrics.
I'd caution buying old hardware with tons of DDR3 for compute though, I have a similar system and it's been useful in the past, but only for very specific applications. Ivy Bridge also doesn't support AVX2 which can be a problem for HPC codes that require it (the code might not be faster without AVX2, but it won't run regardless). The old DDR3 stuff is good for VMs, if you are running some test servers or virtual office machines performance isn't important and the cheap RAM helps reduce the amount of juggling you have to do to keep the VMs all in RAM.
 
noko said:
https://www.newegg.com/supermicro-m...pro-3000wx-series-processor/p/N82E16813183727

Something way more potent and modern. Since workload is unclear.
Click to expand...

Depends how sure the OPs friend is that it won't be a massively CPU bottlenecked workload/and or need run often enough for it to matter. ie for a one off run, 5 vs 15 days might not really matter.

24 x32GB of DDR4 RDIMMs will run ~$1000-1500 more than DDR3 (I couldn't find any deals as good as cdabc123 gave on ebay; but even new it's ~$70 vs $110 for cheapest sticks on Newegg).

OTOH at a certain point if the runtime is short enough getting a monster VM on AWS/etc could become cheaper. An x2gd.8xlarge (Linux, Ohio data center) with 32 cores and 512Gb of ram is only $2.672/hour ($1923/month).

Building a old tech server to do it locally will probably run ~$2500-3000; for new tech roughly double that.

https://aws.amazon.com/ec2/pricing/on-demand/
 
You must log in or register to reply here.
Back
Top