Second one isn't available yet, that's what I've been waiting for. I'll check out the ASRock thanks (though I still need to rewire my brain from ASRock as the shit budget vendor they used to be)
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Anyone order Threadripper 2 yet? User feedback is highly valuable.
- Thread starter tangoseal
- Start date
Second one isn't available yet, that's what I've been waiting for. I'll check out the ASRock thanks (though I still need to rewire my brain from ASRock as the shit budget vendor they used to be)
Its gonna be a monster!
In terms of what I’ll use it for. It’s just going to be a general purpose ‘god’ box for shared compute without going to the cloud.
I’ve got a startup and we do some stuff that requires intermittent *very* heavy processing. Plus because the devs use macs it’s handy to have a central docker host for local development (where you need access to a lot of memory) or where there is integration.
Unfortunately one of the applications that does the heavy processing is windows only which makes it harder but the setup we’ll do is to put kvm on it. That would run
Spec I went with is 2990wx, got 128gb of ecc 2666mhz ram on the way plus an optane 480gb 905p (2.5” version) to run everything on.
I’ll update with experiences but as I say, waiting on a motherboard and memory. Experience won’t be that relevant to people here but it might be interesting.
I’ve got a startup and we do some stuff that requires intermittent *very* heavy processing. Plus because the devs use macs it’s handy to have a central docker host for local development (where you need access to a lot of memory) or where there is integration.
Unfortunately one of the applications that does the heavy processing is windows only which makes it harder but the setup we’ll do is to put kvm on it. That would run
- windows VM that can start up with GPU passthrough
- Other VMs to run K8s (virtualisation on top of virtualisation basically)
- Look into whether I can make the GPU passthrough changeable to move from the windows vms to Linux one so docker can use it. That might have to be kvm on kvm, not really thought it through and it’s melting my brain to think about it.
Spec I went with is 2990wx, got 128gb of ecc 2666mhz ram on the way plus an optane 480gb 905p (2.5” version) to run everything on.
I’ll update with experiences but as I say, waiting on a motherboard and memory. Experience won’t be that relevant to people here but it might be interesting.
I hope you didn't get the crucial 2666 ecc kits, the samsung 2400 or even 2133 ecc will work better on Zen cores. None of that xeon bullshit here, the memory clocks are wide open for you to push to the limit. All my samsung unbuffered ecc runs at least 2933 stable on multiple TR boxes, thats DR with 8 dimms, the SR 4 dimms all do 3200.
The only other 2666 16GB sticks I know are registered samsung, which while good for Epyc will not run on TR.
I put my 2990WX in its new home, one of the asrock X399 fatalities. Grabbed an X399M because leaving a 1950x naked would be a shame.
For now it is still in the same case and enermax cooler, it works but definitely a lot warmer lol. Will take awhile to move it into the caselabs (now "collectible") and put those shiny heatkiller blocks in.
Threadripper 2nd gen is ECC certified officially unlike first gen. Shouldn't be an issue as long as the board maker qvl checks out.
Oh right. Registered is definitely no go. I didnt read that in the post. Selective attention I am subject to it seems ha
Balls... luckily they’re on order so I can change it.
Noobie mistake. Cheers for the advice on the slower Samsung dimms. Saves me $120 as well which is nice.
I should have known better, I even went through the crucial matcher, but then when I went to order I was like oooo Samsung have a 2666 let’s do that. You can tell I’m used toan account exec catching the stupid mistakes on an order, now I’m paying for stuff myself if I make mistakes as the vendor tax isn’t worth buying off the shelf yet.
With so many PCI-E lanes, why wait for integrated?
To get to use it immediately instead of later?
I'm sure integrated cost substantially less than an addin card.
I don't think several dozen bucks matter much for a 2990WX 128 GB ECC RAM build.
It's hundreds for 10gbase-t unless I get an Asus card. Which is still over a hundred and I wont buy the Asus, I've been burned many many times over the years with add in cards that I only get a few oems stuff, Intel in this case.
I've had $50,000 worth of 10gbe sitting in boxes around my desk having had to strip them from a whole hosting environment because they didn't work with more than 32gb of ram, which even the vendor didn't know (that was 2008 in fairness). Don't get me started on infiniband dramas unless it said mellanox.
If everything arrives and I still haven't got one (the ASRock mobo seems to have had problems going by reviews) then I'll look again but right now I'd rather save a few hundred bucks. It's a sale I don't have to make
Has been a bit of an adjustment going from a program with a $1.4m a week burn rate just for staff, to worrying about $400
I would rather get a dual port X540 for $225 and save some difference on a 10GB-less motherboard, for verified VM support and general Intel network chips reliability.It's hundreds for 10gbase-t unless I get an Asus card. Which is still over a hundred and I wont buy the Asus, I've been burned many many times over the years with add in cards that I only get a few oems stuff, Intel in this case.
I've had $50,000 worth of 10gbe sitting in boxes around my desk having had to strip them from a whole hosting environment because they didn't work with more than 32gb of ram, which even the vendor didn't know (that was 2008 in fairness). Don't get me started on infiniband dramas unless it said mellanox.
If everything arrives and I still haven't got one (the ASRock mobo seems to have had problems going by reviews) then I'll look again but right now I'd rather save a few hundred bucks. It's a sale I don't have to make
Has been a bit of an adjustment going from a program with a $1.4m a week burn rate just for staff, to worrying about $400
Last edited:
Don't use VMware so support is less important. As long as it runs on ubuntu I'm golden.
Take your point though, I hadnt seen them that cheap (I'm in australia atm) guess there is a side bar of trying to save slots but really if it has that many GPUs added it means I'm busy enough that I can buy 5 machines like this anyway.
Take your point though, I hadnt seen them that cheap (I'm in australia atm) guess there is a side bar of trying to save slots but really if it has that many GPUs added it means I'm busy enough that I can buy 5 machines like this anyway.
It's hundreds for 10gbase-t unless I get an Asus card. Which is still over a hundred and I wont buy the Asus, I've been burned many many times over the years with add in cards that I only get a few oems stuff, Intel in this case.
I've had $50,000 worth of 10gbe sitting in boxes around my desk having had to strip them from a whole hosting environment because they didn't work with more than 32gb of ram, which even the vendor didn't know (that was 2008 in fairness). Don't get me started on infiniband dramas unless it said mellanox.
If everything arrives and I still haven't got one (the ASRock mobo seems to have had problems going by reviews) then I'll look again but right now I'd rather save a few hundred bucks. It's a sale I don't have to make
Has been a bit of an adjustment going from a program with a $1.4m a week burn rate just for staff, to worrying about $400
I only use Intel XF SR 10gb Fiber Optic Cards. They have worked for me in build that had 512 GB of Reg ECC in the data center. I have 4 of them at home right now and two of them in use, one between my 7820x and my FreeNAS appliance I built. My nVME Samsung 960 Pro can max the 10G link between my desktop and my FreeNAS ZFS ARC with ease. Even my 256GB L2ARC gets pegged at 5.5 Gbps when it needs to be used.
I would not use any other off brand adapter outside of Intel. However, I am really toying with the idea of getting a Ubiquiti 10gb 16 port switch and mixing SFPs between fiber and copper in my home and just making everything 10gb that can do it.
1gbps is truly getting slow for today's Massive home file servers.
Ubiquiti stuff is pretty good. I'm happy with it, though it would be nice if it had a better API, have a shell script to start up the VPN to AWS every day which is a bit clunky (It saves a grand total of $6 a week )
Use the netgear (XS708E) for a 10GbE switch though, just because of 10-BaseT. It's only to go between a shared storage device and everyone else (this would be the first workstation corrected directly). I'd definitely get the Ubiquiti US-16-XG if I needed fibre though. At the moment it just sits in the corner of the office on a shelf so not much need.
Anyway, ways off topic, if I can save $150 dollars on a NIC I'll do it but do take the point, we'll see how long gigabyte take, if the reordered memory arrives and I still don't have a motherboard then I'll just get an external NIC but it's $450 Australian.. $6000 workstation aside (sans graphics cards) I'm pretty tight. as you can see, I already force people to save $0.80 cents a day on VPN (but conversely is why the vpn is from the router not from the workstations when they're in the office)
I save the money for things like salaries
)Use the netgear (XS708E) for a 10GbE switch though, just because of 10-BaseT. It's only to go between a shared storage device and everyone else (this would be the first workstation corrected directly). I'd definitely get the Ubiquiti US-16-XG if I needed fibre though. At the moment it just sits in the corner of the office on a shelf so not much need.
Anyway, ways off topic, if I can save $150 dollars on a NIC I'll do it but do take the point, we'll see how long gigabyte take, if the reordered memory arrives and I still don't have a motherboard then I'll just get an external NIC but it's $450 Australian.. $6000 workstation aside (sans graphics cards) I'm pretty tight. as you can see, I already force people to save $0.80 cents a day on VPN (but conversely is why the vpn is from the router not from the workstations when they're in the office)
I save the money for things like salaries
Ubiquiti stuff is pretty good. I'm happy with it, though it would be nice if it had a better API, have a shell script to start up the VPN to AWS every day which is a bit clunky (It saves a grand total of $6 a week
Use the netgear (XS708E) for a 10GbE switch though, just because of 10-BaseT. It's only to go between a shared storage device and everyone else (this would be the first workstation corrected directly). I'd definitely get the Ubiquiti US-16-XG if I needed fibre though. At the moment it just sits in the corner of the office on a shelf so not much need.
Anyway, ways off topic, if I can save $150 dollars on a NIC I'll do it but do take the point, we'll see how long gigabyte take, if the reordered memory arrives and I still don't have a motherboard then I'll just get an external NIC but it's $450 Australian.. $6000 workstation aside (sans graphics cards) I'm pretty tight. as you can see, I already force people to save $0.80 cents a day on VPN (but conversely is why the vpn is from the router not from the workstations when they're in the office)
I save the money for things like salaries
The SFPs for the UBNT 16xg are like $25.00 its so cheap its laughable how they are able to deliver such a low cost for such a high performing switch.
That's the goal of the thread, hopefully!
Both the ASRock and the Gigabyte are already in stock at e-tailers across Europe. So it probably won't be long. I'd prefer the Gigabyte mobo too.
Threadripper has officially supported ECC from day one. Nothing has changed there between first and second gen.
Aorus Extreme is available now.
Cooking with gas , weeee
https://www.newegg.com/Product/Prod...e=gigabyte_x399_xtreme-_-13-145-084-_-Product
Cooking with gas , weeee
https://www.newegg.com/Product/Prod...e=gigabyte_x399_xtreme-_-13-145-084-_-Product
I cant quite afford 32 core yet but I almost bought that board to get it out of the way. But decided to wait till microcenter has it so I can get replacement warranty.
Gideon
2[H]4U
- Joined
- Apr 13, 2006
- Messages
- 3,556
In my experience "2133" or "2400" Samsung unbuffered ecc b-die (M391A2K43BB1 around $200/16GB buy once cry once) should be able to run at 8x16 2933 16-16-16 on TR, better timings/speeds based on luck and/or using less ranks/DimmsPerChannel. Sample size now 5, all asrock.
As for real, validated feedback: its 32 cores, if your code wants threads and doesn't hate numa it'll love it. You trade 8 channel for quad channel with higher boost and faster fabric based on ram speeds, ~3.2Ghz/2666- on Epyc vs ~4Ghz/2933+ on TR. Registered adds a 1 ram tick clock penalty and server ram default timings are very conservative.
It likes good cooling and idles a good bit warmer on a stock 280 aio that tamed 1950x well, no surprise there. Custom is only real option if you care about overclocking. I kinda don't with this but will do it anyways because fun.
PS, fuck intel.
As for real, validated feedback: its 32 cores, if your code wants threads and doesn't hate numa it'll love it. You trade 8 channel for quad channel with higher boost and faster fabric based on ram speeds, ~3.2Ghz/2666- on Epyc vs ~4Ghz/2933+ on TR. Registered adds a 1 ram tick clock penalty and server ram default timings are very conservative.
It likes good cooling and idles a good bit warmer on a stock 280 aio that tamed 1950x well, no surprise there. Custom is only real option if you care about overclocking. I kinda don't with this but will do it anyways because fun.
PS, fuck intel.
cyberguyz
Gawd
- Joined
- Aug 28, 2014
- Messages
- 710
fixed it for ya
cyberguyz
Gawd
- Joined
- Aug 28, 2014
- Messages
- 710
Yup. numa means each package is associated only with its bank of 2 memory channels. uma adds latency as cores try to access non-local memory by talking to the memory controller on the far package. A lot of applications hate that - mostly in the game world so you have to set numa mode with one of the dies disabled to make those work right. I know of at least one game that simply will not run at all in uma (creator) mode.
IdiotInCharge
NVIDIA SHILL
- Joined
- Jun 13, 2003
- Messages
- 14,675
It's $80, shipped, from the people that make the chipset that goes on every 10Gbase-T board: Aquantia AQN-107
For fiber, sure, but these switches have been finicky both on the RJ-45 ports and with RJ-45 SFP+ transceivers.
I'm... not going to argue with this. There are some complaints with respect to reliability with the Aquantia NICs. I'll say that I haven't seen any, but I only have a sample of two, one built-in and one AIC.
The 1Gbps SFP RJ-45 transceivers are pretty cheap as are the 10Gbps fiber SFP+ transceivers- but the 10Gbps RJ-45 transceivers?
That's where 10Gbit still hurts, and generally speaking, RJ-45 with CAT-6a is where 'multi-gig' is going at 2.5Gbps, 5Gbps, and 10Gbps; 10Gbps fiber is passe, they've moved on to 40-100Gbps+. So cheap to build 10Gbps fiber today with used/discounted parts, but going forward you'll want the twisted-pair stuff.
As an Amazon Associate, HardForum may earn from qualifying purchases.
/dev/null
[H]F Junkie
- Joined
- Mar 31, 2001
- Messages
- 15,182
My mellanox connectx-2s were $15 ea., My connectx-3s were I think $22.50 shipped. SFP+ gbics were $4 ea. in bulk.
/dev/null
[H]F Junkie
- Joined
- Mar 31, 2001
- Messages
- 15,182
I love when fast enterprise stuff gets cheap. Then it comes into my home lab
Working on upgrading from connectx2 to connectx3 sfps at the moment. Switch is a Mikrotik CRS317
As an Amazon Associate, HardForum may earn from qualifying purchases.
I love when fast enterprise stuff gets cheap. Then it comes into my home lab
Working on upgrading from connectx2 to connectx3 sfps at the moment. Switch is a Mikrotik CRS317
Yeah, I ran myricom cx4 10g cards for years off ebay they were super cheap. Only downside was the cx4 max cable length.
prne10
Limp Gawd
- Joined
- Oct 26, 2005
- Messages
- 217
Started testing my OC this weekend on my water cooled 2990wx. Stable so far at 3.6ghz. Going to push higher as I have some thermal overhead.
Just remember that you wont get high single core clocks overclocking manually. If that matters to your usage scenario.
I know this 100% a productivity chip. But load some games up and see how it runs for you while managing your affinities locking the game down to the ccx with direct mem access.
cyberguyz
Gawd
- Joined
- Aug 28, 2014
- Messages
- 710
Yeah that's goodness. Keep in mind if you want to go much higher you have to start adding voltage. An we all know more voltage = more heat.
prne10
Limp Gawd
- Joined
- Oct 26, 2005
- Messages
- 217
yeah der8auer's youtube video on overclocking the 2990wx seemed to suggest a 3.7-3.8ghz across all cores is the sweet spot for stability. Also from Kyle's review, power draw becomes insane when you push towards 4ghz. If I can keep it under 80c (ideally under 75c) at full prime95 load I think I'd be happy with that.
Neapolitan6th
[H]ard|Gawd
- Joined
- Nov 18, 2016
- Messages
- 1,182
A humble/slightly pointless request for 2990WX owners with too much time on their hands:
I'd really be interested to see how far the best 1,2,4,6,8,12,16, etc.. cores could be overclocked.
It would require manually testing each core at a time and documenting each's required vcore at different frequencies. (Requiring disabling cores in bios)
Out of all 32 cores I am extremely curious if a user could find at least 1 core that would run at 4.4, 4.5, or perhaps even 4.6ghz under a reasonable ~1.4 vcore
Its mainly a curiosity of mine. My experience has been the silicon lottery can vary significantly even on 1 CPU. Especially with the 2990WX since it has 4 seperate dies.
It would be awesome to find 4 hyperthreaded cores that would run say 4.5, 4.6 ghz. If only for that rainy day when clockspeed is of greater importance.
I get its kind of pointless, but I am more curious of what is possible as an indicator of the performance we could see in a future Ryzen chip.
I'd really be interested to see how far the best 1,2,4,6,8,12,16, etc.. cores could be overclocked.
It would require manually testing each core at a time and documenting each's required vcore at different frequencies. (Requiring disabling cores in bios)
Out of all 32 cores I am extremely curious if a user could find at least 1 core that would run at 4.4, 4.5, or perhaps even 4.6ghz under a reasonable ~1.4 vcore
Its mainly a curiosity of mine. My experience has been the silicon lottery can vary significantly even on 1 CPU. Especially with the 2990WX since it has 4 seperate dies.
It would be awesome to find 4 hyperthreaded cores that would run say 4.5, 4.6 ghz. If only for that rainy day when clockspeed is of greater importance.
I get its kind of pointless, but I am more curious of what is possible as an indicator of the performance we could see in a future Ryzen chip.