Air cooled 36 cores / 72 HT

Venturi

Venturi

Limp Gawd
Joined
Nov 23, 2004
Messages
264
Et Ali

I rebuilt the prior box and this is its new incarnation
Yes, dropped the water cooling



I tried to make an ultimate pc. My personal rules is that it had to be low maintenance, stable, very powerful and do anything I threw at it.

I run at 4096x2160 true 4k and I like my games to be full eye candy and I like to run a lot of VM for the various things I do.

While at first glance one would not notice, but there where a lot of details worked out in this build.


Well here are my specs and some pictures, please enjoy

thx

V




Asus Z10PE-D8 WS dual mobo (version 1.04, bios 1202)
(2) E5-2699 v3 (36cores/72HT) (O.C. 2.6/4.0 Turbo) (sequential sr numbers)
256GB LR Samsung ram DDR4 ecc reg
(QUAD SLI) 4 Titan X SC
Corsair Aix1500 DIGITAL PSU
1 Samsung NVMe 950Pro pcie 3.0 (os drive)
8 Samsung 850 Pro 1TB SSD RAID 0 (apps drive)
9 noctua fans (almost silent)
Corsair Aluminum Red LED Keyboard, red level 10 mouse
LG 31MU97z 4096x2160 true 4K at 60hz dp
CaseLabs Mercury
2 modified Scythe ninja 3 rev b heatsinks
Thermal Grizzly Kryonaut TIM
Windows server 2012 R2, Ubuntu 15, LinuxMint
1lb. of flesh and a pint of blood

OC rock stable up to 109mhz using bios profiles, etc etc



































medium to high load temps when room is 75F




My prior über workstation was the Z9PED8 WS and unfortunately it can't do NVMe or m.2 with 4x lanes,


for compatibility and speed the xp941 is very fast and as is the sm951

in order of speed (high to higher) it's xp941, sm951 ahci, sm951 nvme, 950pro nvme.

in order of heat generated (warm to warmer) it's xp941, 950pro nvme, sm951 name, sm951 ahci

I describe the heat from the drives as warm. Under load all chips are less than 49c on all models.
Having 4 titan x sc right above them may increase the environment heat, but its still with normal parameters.

If you look closely at the picture you'll see that I have heatsinks on many motherboard components as well.

There was a company this summer that did a review of the z10 and took a thermal image of the board, so when I saw the review I used the picture as guide to get rid of hot spots

in addition the 950pro only has chips on the top surface which aids in cooling and allows for all those little aluminum heatsinks.


The hottest thing on the whole board now is the chipset controller the c612, and that runs about 104F to about 109F and that sits directly under 2.5 out of 4 video cards.

I can't imagine the drives on motherboard being a heat issue, maybe, but I don't know it might be a heat issue in a tight enclosed environment like a surface or tablet laptop


All temps are so farrrr below spec I'm not concerned ;)



 
Last edited:
damn... gaming benchmark please.

I really want to see what a 20.000$+ machine can do for gaming..
 
You can't possibly just use that for gaming. There is not way to justify that.

I assume render rig?
 
The only issue I have is the quad SLI, what does 4 Titans provide that you can't do with 2?

Is there some kind of GPU based rendering that this helps with?
 
It does double duty as gaming rig and as an fusion software workstation, a medical imaging (pacs) station, a cad station etc. I get use out of it ;)
 
Last edited:
post-17505-Anchorman-by-the-beard-of-Zeus-lpvm.gif
 
Synthetic BENCHMARKS (starting to)

I've run a few benchmarks that make more sense to me but I'll run some of the more generic ones so people can have relevant comparisons

example here is Cinebench 15, however(1) it should be noted that cinebench may see all the cores but only utilize them 1-12% so its not a relevant example other than a synthetic benchmark. (2) cinebench does not invoke the turbo mode out of the intel xeons, so the score is running at the base 2.3ghz

Cinebench cpu scores on windows 764 were:

4788
4801
4721
4696

so it also lacks consistency, but here is a pick if the last one from the prior 764 build:





On win srv 2012 r2 and the scores are actually higher, so I don't really see the synthetic benches as accurate or consistent

Win2012 r2 cinebench 15

4861
4804
4799
4830

Hence my lack of faith in Cinebench 15. It is however handy for some quick checks


I'm currently benchmarking the system using AIDA64 and I'll post those scores as I wrap up.


I'll run some gaming benchmarks in srv 2012 r2, however those also are not as relevant. "Heaven" for example runs a high score, median score, low low high. Every time, its a reproducible pattern, so I'm not sure why the software mimics that pattern but it does, and its not load or temperature dependent.

The problem with video benchmarks is that not everyone benchmarks the same parameters for an apples to apples comparisons (exception is locked parameters such as cinebench)

Example, I run at real 4096x2160 with max eye candy, occlusion, parallax, ssao, AF16. FSAA etc. The gaming benchmarks will be skewed at the very least by the fact its at real 4K, but I'll run those shortly. I also edit and modify games to show all it can and I modify the cfg and ini etc to use more threads in both physics and cpu. I also script it to run cpu affinity for real cores (not HT) and all cores (real cores and the HT threads)

every says they have 4k, but what they are referring to is 3840x2160, my monitor and set up is 4096x2160.

The benchmarks I like are the built in Metro Last Light, the Painkiller Black edition built in bench, the doom 3 built in bench, the FEAR built in bench etc


SLI profiles are custom with max visuals and tweaked, in most cases I run AFR2 4GPU SLI mod profiles with the multi threaded optimization



Game BENCHMARKS, all at 4096x2160 maxed visuals, maxed threads, maxed physics (again, parameters are well beyond the game settings menu)

Doom 3, modified, cfg maxed, using full suite of higher textures, higher lighting, rag doll, liquids, ssao, occlusion, adv physics mods: 97FPS



Painkiller Black cfg maxed 424FPS

Cryostasis, full physics, threading, full eye candy, textures, etc 291FPS

Wolfenstein 2009, cfg maxed, havok physics threading, game threading, max eye candy, veil fully rendered: 147FPS

FEAR cfg maxed, ssao, physics, occlusion, true 4k, liquids, lighting, destructive physics, particle motion etc etc Min 146FPS, Max 281FPS, Av 188FPS




I'm currently tweaking "Outlast", and I'm tweaking the Unreal 4 engine "Kholat" game. The new engine can really take advantage of the hardware if configured correctly.


At the end of the benchmarking the video card temp high was 69-71C (158F) the cpus highest was 41C (105F) ram temp was 44C (111F)

there are other types of benchmarks, that are more important to me

DATA read write Throughput -->

example, using MACRIUM Reflect Server version, I can compress and back up 1.9TB full system and data in 6 min (6:18) to a full self restoring image.



and I can restore full windows, apps, and mail data blank blank to fully loaded in about 1 1/2 minutes (1:23)






In addition I have moved to Win SRV 2012 R2 from Win 764 ultimate.

Took a few registry hacks to get the classic desktop and
about the same privacy as 7-64
I used various built in tools it's a workstation and so far I'm content
I've got a few days of tweaking ahead of me.

All games installed ok, a few fps of improvement across the board
Also was able to do GPT and M.2 boot without workarounds

Verified file transfer speed, sustained, 2.3 GB/Sec on the 2012R2
Back up transfer speed says 13.5 Gb/Sec but I'll have to test, not sure if it can be trusted yet.

Compared to 764 ein 2012r2 offered a few improvements:
a. I have some hardware in my build that benefitted from the 2012r2, mainly the m.2 drives, and the new sata express. I was using several macrium tricks to re-install from a back up to do the gpt and m.2 as the C drive for boot as well.

b. There are a few settings in 2012r2 that benefitted from the e5-2699 v3 and the z10 when using "cache on Die" in the memory handler. Do not use early or late snoop.

c. and there are some benefits to using Hyper V in a very controlled fashion.


Do these benefits rank high enough to outweigh the win 764 build? - probably not...

For now I'm tweaking and exploring the 2012r2 and tweaking Ubuntu 15, and my known addiction to DRM free shooters in high rez to incorporate into the fun


I'm getting two additional perks now that I've had it running for a few weeks:

1. Specific games that have multicore and multi gpu malleability benefit from running on 2012r2. This might be how 2012r2 makes the environment available to an app. DX11 games in combination with NVIDIA drivers really saw a bump. Example Metro Last Light Redux built in benchmark had a performance increase in Max frames by 34 f/s, increase of 17 frames in Av f/s and raised the min f/s by 28.
Open GL games perked evenly by several f/s.

2. Load and precaching time on game levels has dropped noticeably. I have not benched this aspect since its hard to quantify with consistent results.

3. Compared to 8.1, the tile streaming, and other user burdens, were automatically not installed. In server manager you add and subtract features very easily, much easier than win 8.1 and more control than 8.1. In fact, having server 2012 r2 as a workstation was cake.

4. Boot and shutdown time - this is relative since I have a longer power to loader moment due to the nature of the power cycle of the z10ped8ws. However, once the video card displays the first part of the bios/uefi splash, the os boot time is about 7 seconds to login. That is considerably faster than 764 which was more than 30 seconds. The shutdown process is about 1-3 seconds to all power off.

I started easily with a site to guide my initial efforts. Thank you to several forum member for suggesting http://www.win2012workstation.com as it did make it much easier.

Pain in the arse issues that have been solved:
1
Office protection platform service kept running after you open outlook. Fixed by registry entry
Key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\OfficeSoftwa reProtectionPlatform
ValueName: InactivityShutdownDelay
ValueType: DWORD
ValueData: The number in seconds, that the Office Software Protection Platform will stay running for, after the most recent activity.

I set mine to 30sec

2 Windows update service keeps running unless I set it to disable. usually I have it set to manual and when I manually go and do updates and select then it will start, after a reboot its gone no prob. However, now with 2012r2, I have update set to never and service set to manual (so I can do it on my terms) but the update service on reboot is running. In task scheduler I removed the triggers but they magically reappear in task scheduler after a reboot. So now I have the service disabled until I figure out how to keep it on manual and keep the pesky triggers out of task scheduler.


I hope that helps on initial benches and perception, and forgive my long winded explanation, the devil is in the details.


Ciao


V
 
Last edited:
How is the SLI scaling working? When Linus tested 3x and 4x SLI on the 980 he found that it didn't work so well in many games and introduced stuttering.
 
It's so beautiful!!!!!!!!

my core2quad would just stall out and die sitting next to that lol.
 
Can we see cinebench or handbrake benches?

thats a crazy system.


Are the xeons able to be overclocked in any way? strap(fsb) overclock?, or even multiplier?
 
Quartz-1 said:
How is the SLI scaling working? When Linus tested 3x and 4x SLI on the 980 he found that it didn't work so well in many games and introduced stuttering.
Click to expand...

I have not had any issues with Sli. 4 cards is noticeably and measurably faster than 3 cards, scaling is progressively noticeable. Rock solid :D



The reason why a lot of folks have a problem is because when using 4 cards on the bus:

1 must "enable above 4G decoding"

2. must disable power management in windows power options on the pcie link under advanced power options.

3. Must have all the cards in exactly the same kind lane and widths on the bus. Make sure its not just enough sockets but that each one can command the needed bandwidth.



Do ALL this or all sorts of stuttering and mishaps occur :eek:

I hope that helps.

V
 
Last edited:
Loop242 said:
How long to handbrake Blu-ray of The Two Towers
Click to expand...

I have not done that yet, but when I do I'll post it ;)



Unoid said:
Can we see cinebench or handbrake benches?

thats a crazy system.


Are the xeons able to be overclocked in any way? strap(fsb) overclock?, or even multiplier?
Click to expand...

The machine overclocks rock stable to about 109mhz. Anything above that has little niggly issues.

I run at 105mhz all the time, well within the my "mental" safety zone.

The Asus Z10PED8 has lovely editable overclocking profiles right in the bios.


Cinebench 15 highest scores for this rig range from 4696 to 4861, a picture of the lowest score is in the header post for reference

I also have a small rant in the post about how the score is very synthetic. :)





V
 
Last edited:
In comparison to the Asus Z10PED8 WS, there have been significant changes in motherboards, but not many if you consider the amount of time that has passed since I first got into this around the mid 90's.

I've been building my own personal workstation since 1998. In 1999 and 2000 was water-cooling a dual cpu rig with raid. I even water-cooled the video card back then. I had to use dual power supplies at that time. I used to run an obsidian x-24 card then moved up to 3d wildcat dual GPU/CPU video card etc. I did one of the first sli rigs on a test mule than mobo. Then moved to an 8 core dual 2 motherboard rig from tyan. Then more tyan boards and the first quad sli build that I know of. Lots of different builds to end up to the one today.

This just happens to be the 7th incarnation of the original build. Great for games and when I have to do work from home.


Anyhow, here are some of my first PCs from 15+ years ago



















My "Blue Period" and first quad SLI

(NVidia had not even made the quad Sli sound yet, but beta driver support was there in developer drivers)
and always multi cpu and raid ;)








"Cable Zen" was a pain then too






earlier this year water-cooling





The last one was my water-cooling attempts on the Z10PED8 WS build. While it worked well, the results were not any better than air cooling and not with the maintenance and complexity of the liquid cooling. So not the box is completely air cooled now.

Bringing us back to "today" :





 
Last edited:
venturi said:
It does double duty as gaming rig and as an isotope software workstation, a medical imaging (pacs) station, a cad station etc. I get use out of it ;)
Click to expand...

Ummm

If you are hooking this to a MALDI TOF Mass spec (or any mass spec) to do chromato peak analysis, you aren't using this for gaming.
 
I long time member asked me about some of the more relevant bios settings for stability. The bios has many screens but I'm posting some of the more important settings


make sure you have the latest bios 1202



pic the default auto unless you want overclocking profiles, but first make sure your system is stable



main index


enable Above 4G decoding or you WILL NOT realize the benefit of the 4th card and you will otherwise have stability issues and STUTTERING


These are your best cpu parametrs for speed and stability


if your cpu configuration supports it pick these specific setting in the qpi
if possible select "cluster on die" in snoop


enable numa
enable acpi
enable default mem unless you are overclocking


in CSM enable UEFI when possible, here is a mixed config just as an example


In boot make sure you enable UEFI and Legacy


also if you use UEFI, those peski separate raid config splashes are bypassed in the initial boot sequence, saving time.

I hope that helps

V
 
Question, where'd you source those cool SATA plugs?
 
Mithan said:
And chances are it will run ARK at 30 fps with Quad SLI.

Waste of money.
Click to expand...

Apparently you didn't even bother to read the thread and see what he actually uses it for. If you haven't figured it out just by looking at the workstation grade hardware, its a... w o r k s t a t i o n.
 
CaptNumbNutz said:
Apparently you didn't even bother to read the thread and see what he actually uses it for. If you haven't figured it out just by looking at the workstation grade hardware, its a... w o r k s t a t i o n.
Click to expand...

Forgive me for being out of the loop, is ARK supposed to be Arkum City or Asylum?

I don't have the latest version, because its slightly problematic on pc, but I do have the first two. I think one is the GOTY version

Both of those play at over 100fps based on the game internal benchmark (with all candy on)
had to tweak the Sli profile. But I don't have them installed anymore because it was originally steam, and I don't do steam anymore. --> I'll spare you the lecture on my feelings about drm games, now, I just use GOG or find way with the games so I don't have to use a client like steam, EA, ubiplay, blizzard etc. I like to click on the shortcut and be in the game in the next frame. I buy all my games, always have. I just feel they should be able to be enjoyed without all the layers one goes through just to play them.


Somewhere I might have the benchmark screenshot and I'll dig through and see if I spot it.


Otherwise, I have no idea what ARK is.


Also, really, its a workstation first ;) If it was supposed to be games only, there would not be a need for all the extra grunt it has
 
What does 8 ssds look like when you run as-ssd? I'm guessing after 3 they don't scale.
 
venturi said:
Forgive me for being out of the loop, is ARK supposed to be Arkum City or Asylum?

I don't have the latest version, because its slightly problematic on pc, but I do have the first two. I think one is the GOTY version

Both of those play at over 100fps based on the game internal benchmark (with all candy on)
had to tweak the Sli profile. But I don't have them installed anymore because it was originally steam, and I don't do steam anymore. --> I'll spare you the lecture on my feelings about drm games, now, I just use GOG or find way with the games so I don't have to use a client like steam, EA, ubiplay, blizzard etc. I like to click on the shortcut and be in the game in the next frame. I buy all my games, always have. I just feel they should be able to be enjoyed without all the layers one goes through just to play them.


Somewhere I might have the benchmark screenshot and I'll dig through and see if I spot it.


Otherwise, I have no idea what ARK is.


Also, really, its a workstation first ;) If it was supposed to be games only, there would not be a need for all the extra grunt it has
Click to expand...

He's talking about this game Venturi. ARK: Survival Evolved.

Oh and awesome PC man! ;)
 
You must log in or register to reply here.
Back
Top