Testing Methodology Thread

[Erasmus354]

Ok since it seems to be the hot topic around here (and getting a number of threads off topic). I thought it might be pertinent to try and move the testing methodology discussion to its own thread.

I have added a poll to this thread where you can vote on what you believe is the best testing methodology. To get it started where we left off I believe the discussion was centered around die sim vs. real world testing.

Which do you prefer and why...why do you think either or both of the methods are invalid for testing waterblocks?

 

[Cathar]

Hmmm, where's the 75% die simulator, 25% overclock (on bare-die CPU's only) poll option?

 

[Erasmus354]

Hmmm, where's the 80% simulator, 20% overclock poll option?

both of couse! (it doesn't specify how much)

I think I agree with you Cathar. Both options should be included in a comprehensive review. The meat and potatoes however should be the simulator testing. Because that is the only repeatable and statistically accurate method of testing we have atm.

I know others think differently.

 

[R1ckCa1n]

Hmmm, where's the 75% die simulator, 25% overclock (on bare-die CPU's only) poll option?

True but I would like it the other way round

Cathar - you used to quantify your new block on how well it overclocked the same system of yours (if I remember it was an AMD XP CPU). These results are somewhere deep in the overclockers.com.au's forums. Those posts where the reason I bought my first WW block way back!

 

[plywood99]

The meat and potatoes however should be the simulator testing. Because that is the only repeatable and statistically accurate method of testing we have atm.

Yup...

 

[menlatin]

well you can calculate heat output of a cpu though cna;t you? The cou and the same speed and same voltage should be the same heat output right? as long as it using the smae program to heat it up

 

[plywood99]

well you can calculate heat output of a cpu though cna;t you? The cou and the same speed and same voltage should be the same heat output right? as long as it using the smae program to heat it up

Yes, but it is much more complicated than it appears...

 

[Cathar]

True but I would like it the other way round

Cathar - you used to quantify your new block on how well it overclocked the same system of yours (if I remember it was an AMD XP CPU). These results are somewhere deep in the overclockers.com.au's forums. Those posts where the reason I bought my first WW block way back!

I still do this. It is still a primary part of my testing methodology, and I have continued to report the overclocking gains that I get in by block development threads.

First and foremost though for me is the temperature gains achieved. I tend to find that overclock gains go pretty much hand-in-hand with temperature gains, although I have also focused on what can be loosely described "smoothing" of CPU hotspots to help for overclocking, the effects of which can't quite be seen in pure temperature gains, and so overclocking has always remained an important part of testing, but that tends to happen later on once I can verify that a design is at least providing good temperatures too.

 

[DFI Daishi]

in reviews i want thermal die testing.

i hear plenty of first hand real world experience from acutal users. i look to reviews for what user commentary can't give me: which hardware comes out ahead with two products that compete pretty closely.

we measure to the milisecond in races. a runner who is competing in an international event is so much faster than a normal person that there is no real need to time a race between the two. when a number of contenders face off, it's another matter: we use high speed cameras, precision stop watches and precise definitions of what part of the body must cross the finish line.

i feel that a similar situation exists with high-end waterblocks, and that testing must be to a similar standard. i'm not interested in what a domestic high flow block can do as comparted to stock cooling. i'm not interested in what german gear can do against stock cooling. i'm interseted in what german watercooling parts do compated to domestic watercooling parts, using the same test setup to measure both.

i am willing to give the german gear evey advantage, including flow rates, and therefore cooling performance, well beyond what it will give in a typical low-flow loop. if it posts numbers similar to those of high-end high-flowing equipment, i will just shut up about it being underperforming. i will still probably comment to the effiect that a stronger pump could get more out of the blocks, but i will lay off on that particular point.

if the blocks don't post decent numbers even with good flow, i think that those claiming that the german engineers can do something that domestics can't with regard to getting good performance with low flow should really reconsider their choosen arguments.

 

[Top Nurse]

...although I have also focused on what can be loosely described "smoothing" of CPU hotspots to help for overclocking, the effects of which can't quite be seen in pure temperature gains...

Is this done by making sure you have one or more of the jet impingement sockets in the same area as the higher temperature areas of the die? I saw a thermal scan of a CPU somewhere and it was interesting because the heat wasn't concentric from the CPU mid point.

 

[Cathar]

Basically done by attempting to ensure that there aren't any poor heat-paths that would tend to "trap" heat, or not spread heat evenly. All waterblocks don't have a perfectly even rate of thermal dissipation at different points on the base-plate, and this unevenness also depends on what the design is doing internally. It also involves carefully balancing base-plate thickness, sometimes not merely for best possible temperatures, but to ensure that localised heat will spread quickly and evenly to the primary areas of highest convectional transfer, and ensuring that those primary convectional areas are spaced accordingly. So even if some hot spot occurs directly underneath the "worst" point-spot in terms of the block's internal geometry to deal with that localised heat load, we are balancing off the heat spreading ability of the copper vs the local primary "heat sink" locations within the block's design. As stated, this may involve choosing a base-plate thickness that does not necessarily achieve the lowest possible average die temperature across the entire die's surface (such as when purely tuning for a linear die-sim), but rather attempts to address the localised hot-spots surrounded by cool-spots that occur on real CPU's.

Skeptics may reckon the above sounds like trash. Doesn't bother me. I reckon it sounds a bit like black voodoo too. I just go by results.

 

[Top Nurse]

Skeptics may reckon the above sounds like trash. Doesn't bother me. I reckon it sounds a bit like black voodoo too. I just go by results.

Actually it makes more sense than this nonsense about testing CPU's with a heat simulator as the simulator doesn't take into account any of this real life thermal characteristics. I like your designs because you go the extra mile. Now if only you had a G5 or G7 with a clear top.

 

[mohammedtaha]

Actually it makes more sense than this nonsense about testing CPU's with a heat simulator as the simulator doesn't take into account any of this real life thermal characteristics. I like your designs because you go the extra mile. Now if only you had a G5 or G7 with a clear top.

Testing with a heat simulator PROVES that water block X can remove heat better than Water block Y. Isn't that the WHOLE point of innovation?

TESTING WITH A REAL CPU WITH AN IHS WILL NEVER GIVE YOU THE RESULTS YOU WANT TO SEE !!!

IF a CPU had a perfectly flat block of copper that perfectly sits on the cpu shielding it and providing 100% contact all around the die then I would go with real world performance and die simulator.

This isn't happening today so I'm going with die-simulator. You CAN NOT compare real-world performance when the test results of cooling a cpu using a Maze 3, a STORM and a Apogee are almost the same.

 

[Erasmus354]

Actually it makes more sense than this nonsense about testing CPU's with a heat simulator as the simulator doesn't take into account any of this real life thermal characteristics. I like your designs because you go the extra mile. Now if only you had a G5 or G7 with a clear top.

but wait...wasn't it you (or was it r1ck) that is professing that you cant buy a proc without an IHS, and therefore you need to take the IHS into consideration. The last time I checked you cant by a Tbred (unless from some weird online store or used somewhere). So doesn't that mean by your standards Cathar's testing on an old A-XP is moot as well? Flippity Flop Flippity Flop

EDIT : wow that sounds really mean after reading it...I guess it shows how much of a d1ck I can be when I have to wake up at 6:30am on saturday to take a final

 

[mohammedtaha]

EDIT : wow that sounds really mean after reading it...I guess it shows how much of a d1ck I can be when I have to wake up at 6:30am on saturday to take a final

wrote mine last night at 7:00pm .. Software Engineering ... I hate UML's

 

[R1ckCa1n]

but wait...wasn't it you (or was it r1ck) that is professing that you cant buy a proc without an IHS, and therefore you need to take the IHS into consideration. The last time I checked you cant by a Tbred (unless from some weird online store or used somewhere). So doesn't that mean by your standards Cathar's testing on an old A-XP is moot as well? Flippity Flop Flippity Flop

EDIT : wow that sounds really mean after reading it...I guess it shows how much of a d1ck I can be when I have to wake up at 6:30am on saturday to take a final

I think the 6:30 wake up caused you to not read that conversation correctly.

TN was saying she respects the work Cathar puts into his blocks. Not only does he try to acheive the best possible temps, he is interested in how well the block cools the "hot spots". This is way I always have and will respected the way Cathar tests and designs. If it can't get his personal rig to certain temps and/or overclocks it sounds like he goes back to the drawing boards. Yes he does it on an AMD bare CPU, but he is consistant, which matters more.

 

[Erasmus354]

I think the 6:30 wake up caused you to not read that conversation correctly.

TN was saying she respects the work Cathar puts into his blocks. Not only does he try to acheive the best possible temps, he is interested in how well the block cools the "hot spots". This is way I always have and will respected the way Cathar tests and designs. If it can't get his personal rig to certain temps and/or overclocks it sounds like he goes back to the drawing boards. Yes he does it on an AMD bare CPU, but he is consistant, which matters more.

Once again you failed to understand what I am saying. I did not misunderstand anything. In the other thread (which this one is based off of) where discussion about testing methodology was going on, TN argued that because you couldn't buy a processor without an IHS, that you should only test with an IHS. So once again you try to take pot shots at me without understanding fully what is going on...it is getting quite tiresome to say the least.

 

[Top Nurse]

I think it is you who got a problem as you can't seem to get it straight who is who around here. Also if you are going to quote from different threads then please have the decency to post the quote. Perhaps you should put down the pipe for a while...

 

[Ice Czar]

enough

discuss the topic and leave the interpersonal accusations out of it

 

[R1ckCa1n]

enough

discuss the topic and leave the interpersonal accusations out of it

QFT

ontopic - so far we have some interesting results. Seems pretty neutral.

 

[DFI Daishi]

@Ice Czar: do yo have anything that you would like to share regarding how the new cooling review testbed is shaping up?

how do you sit, with regard to the present discusstion?

 

[Ice Czar]

I was afraid someone might ask that

Cathar, Joe, pHaestus, BillA, Les, ect ect ect
have set the bar quite high when it comes to quantifiable testing of individual components
this gives an affecionado definative figures to mix and match an ensemble of components with a reasonable expectation of the performance they would get. They have even made great strides in interactive component combination tools with that data.

However Kyle also has a point, which is that real world examples of component configurations are what new entrants to the field really require, the above sophistication being part of thier learning curve yet to be aquired.

Kyle knows his customer, and Id have to agree.
Thus the question is really one of who is the information targeted at and how understandable is it. The obvious answer is that both methodologies have thier place but that its the conclusions of the reviewer that are relied on, and that expressing the caveates involved is quite critical.

The trajectory Ive set for the lab has a dual purpose, as it will be addressing energy as a whole, meaning that like Lee "Robotech" Garbutt, I'll be looking at both power requirements, supplys and cooling solutions and trying to accomplish a more holistic approach, to that end Ive been working on building and privately financing an environmental test chamber with humidity and temperature control. And a fully automated data aquisition system.
(Its hard to ask Kyle for $10,000 with nothing to back it up but a plan )
Its a steep learning curve.

To Date here is an aquired equipment list

LABview 7.1
National Instruments PCI-GPIB\LP NI-488.2
National Instruments PCI-DIO-96
National Instruments SCB-100 breakout box and 2m 100 pin cable
pCon Beta PC based multiloop PID controller
Keithley 2700 Digital Multimeter & Data Aquisition System
Keithley 7708 40 channel Diff Mux Module w/ Automatic Cold Junction Compensation
Keithley 7701 32 channel Differential Multiplexer Module
Some 20 J Type Thermocouples with shielded extention grade wire
Various 2 wire and 4 wire RTDs
HP 6050A 1800 Watt dc Electronic Load Mainframe
2x HP 60501B 150 Watt dc Electronic Load Modules
Bruel & Kjaer 2203 Sound Level Meter with 1613 Octave Filter Set
Bruel & Kjaer 4131 condenser microphone
Omega CN77342-C2 Autotune PID Temperature Controller
Rotronic HC321A Humidity & Temperature dewpoint Computer w\ Hygrometer
Aquafine SP-1 UV Water Treatment Unit
Endress+ Hauser M-Point Massflow Meter
3 Air conditioners being retrofit to various duties (chillers ect)
various blowers, pumps, insulation, sheet metal, hardware, plumbing
2 computers for DAQ and Control
2 more computers for testbeds (Prescott & FX55)


Items still on my shopping list:
Rosemount 1151 Differential Pressure Transmitter
2 x HP 50602B 300 Watt dc Electronic Load Modules
various GPIB DC Power Supplys
a GPIB Variable AC Power Supply
construct a Stainless Steel Mesh Heater
Vaccum Pump & Bell Jar w\ various wire feedthroughs
Watlow Cartridge Heaters (both die simulation and component simulation is cases)
F.W. Bell 640 Gaussmeter w\ probes
more RTD's

Im interested in some questions Im not sure have been fully explored in the cooling arena and where I feel I might be able to make a new contribution, namely the operating environment variable and to what extent that will degrade a given solution under adverse conditions. As well as airflow questions inside a case, overall resistance, case volume, layout, additional heat sources (say what happens with 10 HDD) ect. Thus being able to replicate an environemnt (temperature\humidity\pressure) from one test to another is critical.

 

[DFI Daishi]

I was afraid someone might ask that

you should have known that it would happen if you posted in this thread.

however, that equipment list makes me halfway wish that i had not asked in the first place. even if i don't know the specifics of most of the hardware, it is quite impressive that you are putting together such a "substantial" battery of equipment for evaluating cooling equipement.

now, given your noise measuring hardware, i have to ask: have you sound proofed the room itself and what is the localty of the test facility?

when you're listening to the hard rock that i typically enjoy, minor noise from the outside environment hardly makes a difference. if you're trying to pick up the faintest sound, it's not unusual for the floor of the recording studio of an orchestral performance to be mounted on rubber blocks, and the building to be located out inthe country.

if you are going to be evaluating to a lower than 30 dB sound level, it would be a good investment to pad the walls of the room with egg carton style matress material. it's cheaper than real sound proofing, and it does the job VERY well.

 

[BillA]

hot shit !
a guy with a budget (I was wondering who bought the mass-flow unit, lol)

both a die and CPUs ?
(please do groove a 775 CPU and start collecting some IHS temps, all will become clear)

 

[Ice Czar]

you should have known that it would happen if you posted in this thread.

was why I held off
well until I was forced to post inorder to keep the natives inline

yes there will be a sound test chamber though likely not as advanced as a real sound lab
as with most measurements it will be reasonably reliable to compare data from test to test, but no it wont be NIST or ISO traceable or certified, or even comparible as definitive values to another site. (SPCR)

hot shit !
a guy with a budget (I was wondering who bought the mass-flow unit, lol)

both a die and CPUs ?
(please do groove a 775 CPU and start collecting some IHS temps, all will become clear)

now I had to buy that massflow meter, the titanium alone was worth more than its auction price
trying to get documentation has been a bitch, its currently being ignored

is there a thread on the grooving proceedure and thermocouple\RTD placement others are employing?

 

[plywood99]

(please do groove a 775 CPU and start collecting some IHS temps, all will become clear)

How so? If that is all that is needed, why the muddled mess we have today???

 

[BillA]

How so? If that is all that is needed, why the muddled mess we have today???

the suggestion was to gather the data, implicit was that its relevance would be apparent
the IHS temp is the 'die temp' plus that due to the the resistance of Tim1 and the IHS, minus the effect of the sink
- it is necessary to address the variability of Tim1 as an issue apart
then the muddle dissipates somewhat

IC
hellish impressed with the equip
several topics of interest with you and normally I'd jack this thread into testing
here, new testing thread, or the other forum ?

 

[Erasmus354]

Where is a drooling smily when you need one? Very impressive IceCzar, do you have any sort of time table when you expect your rig to be done? Summer 06 maybe?

 

[Ice Czar]

IC
hellish impressed with the equip
several topics of interest with you and normally I'd jack this thread into testing
here, new testing thread, or the other forum ?

Id say a different thread or over at procooling
today however Im doing the frontpage with a little light admin duties and actually trying to work on the lab.
and the immediate concern, getting my girls a good run so they will settle down
(Border Collie, a BC mix and a Karelian Bear Dog)

when you expect your rig to be done? Summer 06 maybe?

Kyle would insert a metal rod up my butt and have a long pig barbeque if it took that long

at least for the cooling reviews, it might take that long before I can fully finance the PSU test setup,
depends on what falls out of the trees on ebay, 60502B moduals run $1000 per used, good GPIB Variable AC Power Supply over $1K used

on the plus side I have two pals that are helping me as time allows, one is an ISO certification consulatnt that has built several environmental test chambers, the other a automated robotics veteran (packaging)