• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Improving 3DMark

B

BoogerBomb

Supreme [H]ardness
Joined
Jan 10, 2003
Messages
6,470
It is beyond universal that 3DMark doesn't due what gamers want it to. So instead of threatening to delete the free version that hurts FM is no way or continuing to post that 3DMark sucks, how about we start a thread about what is expected and how you think it should be done.

Most complaints are that they use and engine that no one else uses. Which engine should it use then? There are a myriad of choices that current games use and it can't test them all in one benchmark. Some engines dont make use of any of the advanced features of a card so what could FM do to test those features in a way that is of importance to gamers? Should it use DX or OGL? Once again it can't do both at the same time.

I only mention a few aspects that should be considered so feel free to add any other aspects you feel.

Hopefully this will give them some direction in what we like to see in a useful benchmark.

This is an attempt to solve a problem and please reply with only solutions rather than flames.
 
The fundamental problem is that you are wanting one application that can tell you gaming performance.

That just doesn’t work though.

To test gaming performance you need to test games.

That’s why its called gaming performance.
 
I think you should also post this at the FM forums, as that way the makers can take note.
 
The solution would be to use the games you want to play to benchmark with. The rebuttal to that is that all games you want to play dont have benchmarking utilities built into them. My response to that is to ask the developers to implement it, which is a start and pretty much all we can do.

As for how to fix 3dmark05, you cant. An engine is everything but the game code, it handles the graphics, sound, networking, etc. An engine cannot simply be added to 3dmark05. 3dmark05 is its own engine that attempts to recreate what the game of tommorow will implement.

And the problem with 3dmark05 is that it doesnt do this to the extent that most would like. (you cant make every game out of one benchmarking demo) So it is only useful to test certain aspects of your video card, (fill rate etc) and to test video dependant settings. Which are better handled by a real game benchmark, one game may be very cpu/memory dependant or very video card dependant etc.
 
What could be done besides testing every game out there? Most games aren't comparable and so don't tell you how it will perform in other games. How does running a doom 3 benchmark tell me how the new lord of the rings rts game will play? Or how does running a hl2 benchmark tell me how a Rome Total War will play, or a racing game or a sports game for that matter? The above poster is right in that you should benchmark games you want to play, but I think one problem with most sites is that they all use too many first person shooters for benchmarks. Not everyone plays first person shooters, but those dominate the tests most times.
 
Why can't Futuremark get licenses to the primary popular engines of today? For example, they could utlitze the CRYEngine, the Doom3 Engine, and the Source engine. Throw in a few RTS and RPG engines, and you may have a benchmark that actually reflects game performance a bit better.
 
Yes there is. However, you need to own almost all the titles in question. This isn't practical for most people.
 
Use features of today's graphics cards extensively to demonstrate performance differences and image quality differences between this generation's video cards. 3dmark05 is no more than an intensive SM2.0 test. There is no extensive use of SM2.0B or SM3.0 (despite basic support of those modes) i.e. no geometry instancing, no dynamic branching, no longer shaders. There is no OpenEXR HDR support, which is the future of HDR lighting. No 3DC, so we can't see how that helps performance wise or IQ wise. Virtually none of the features introduced into this gen's GPUs were utilized significantly in 3dmark05 beyond a marketing standpoint. If Futuremark could not find a use for these things in its demos, I suggest they create different and more diverse demos, because some of the most popular real life games are using or will be using these features this year, and they have shown significant gains even from moderate use of them. I'd also recommend picking up OpenGL support if they truly want to be the "gamers" benchmark as Doom3 engine games all use OpenGL.

3dmark05 as a benchmark test best measures performance potential of the FX 5950 Ultra versus the 9800XT on games from 2003. Instead, it should be measuring the performance potential and image quality potential of the 6800 Ultra versus the X800XT on games from 2004 and beyond. This is where 3dmark05 fails, and this is where the improvement needs to be made. Even staunch supporters of synthetic benchmarks could not make a good argument that 3dmark05 is an accurate predictor for performance and IQ of today's cards, because it fails to significantly utilize most of the new technology today's cards brought to the table to enhance performance and image quality.
 
I think stop bashing the "benchmark" and accept it for what it is would be a great step. I look at it like a radio station or website; if you don't like it, don't tune in or browse their website. It does, however, give people the ablitlity to run the same "benchmark" across multiple platforms for comparision. All review sites give is how well games will run on one configuration for each video card (typically).

If this "benchmark" was so "useless" and "sucked", ATI and Nvidia wouldn't spend more time coding drivers to Futurmark than actual games.

Just my 2 cents
 
The reason why ATI and Nvidia spend so much time trying to win the 3dmark benchmark with their flagships is because the general public views it as the definitive 3d performance test for a graphics card.
 
Well I think what we have here is a turning point in 3D Mark.
I do think there is a use for 3D Mark. What we all need is to define exactly what purpose or use that would be.
I see it as a useful tool for comparing hardware such as when you swap out parts and such. I don't see it as a tool for defining good or bad hardware. Definitely not a game gauge.
 
I think if they changed the name of the benchmark to 'The 3D Tweaker's Benchmark' compared to 'The Gamer's Benchmark', all would be better.
 
I agree with tranCendenZ, 3Dmark is nothing more then a generic DX9 benchmark but if it used the most "cutting edge" features then it would really be worth something as a way to tell video card performance.

What FutureMark should do is work alittle less with the video card makers and a little more with the game makers. It can keep the crappy benchmark stuff that it has currently (well it is pretty to look at) but it should work with game makers to enable "hooks" within games to create a benchmark app for them. Think about it, loading up a level in Doom 3 or Farcry and having the option of letting 3Dmark run a benchamrk for that game that actually test game performance.

Either video card makers can continue to optimize for a synthetic or they could optimize for a game engine. That's what I'd want to see from FutureMark. :rolleyes:
 
reposting from other thread hope this is ok


I haven't posted much less read hard|forum for quite a long time... Since I got banned quite a while ago for something that was a bit dumb, but no hard feelings.


Anyways I suggest to Futuremark that to improve the "usefulness" of their benchmark, they need to diversify the 3dmark scores into a few different tests.

One 3dmark number just isn't going to do it.

I suggest that when you start up 3dmark you pick the tests you want to run, or run all of them, but at the end you get a few different 3dmark points.

Just off the top of my head:

Futuristic Points:
Current Points:
Old School Points:

Just divide up the tests so that Futuristic is your best guess at what tech games in the future will use and how well cards do at that. Current is taliored to features and tech current game engines are using, and Old School is for year- 2 year old engines.

Then we can all use "current" as the most important 3dmark, and note "futuristic" as important, but not as important as you can't actually buy a game that would relate to.

Thanks for the time.

 
tranCendenZ said:
Use features of today's graphics cards extensively to demonstrate performance differences and image quality differences between this generation's video cards. 3dmark05 is no more than an intensive SM2.0 test. There is no extensive use of SM2.0B or SM3.0 (despite basic support of those modes) i.e. no geometry instancing, no dynamic branching, no longer shaders. There is no OpenEXR HDR support, which is the future of HDR lighting. No 3DC, so we can't see how that helps performance wise or IQ wise. Virtually none of the features introduced into this gen's GPUs were utilized significantly in 3dmark05 beyond a marketing standpoint. If Futuremark could not find a use for these things in its demos, I suggest they create different and more diverse demos, because some of the most popular real life games are using or will be using these features this year, and they have shown significant gains even from moderate use of them. I'd also recommend picking up OpenGL support if they truly want to be the "gamers" benchmark as Doom3 engine games all use OpenGL.

3dmark05 as a benchmark test best measures performance potential of the FX 5950 Ultra versus the 9800XT on games from 2003. Instead, it should be measuring the performance potential and image quality potential of the 6800 Ultra versus the X800XT on games from 2004 and beyond. This is where 3dmark05 fails, and this is where the improvement needs to be made. Even staunch supporters of synthetic benchmarks could not make a good argument that 3dmark05 is an accurate predictor for performance and IQ of today's cards, because it fails to significantly utilize most of the new technology today's cards brought to the table to enhance performance and image quality.
Click to expand...

I usually don't even like to read your posts because of the negativity, though you do have truth to what you say. This is well said, and I completely agree. You saved me the time of posting the exact same thing :). While it took quite some work to make 3dmark i'm assuming,, they could have have improved significantly on it. I'll leave it at that :). Nice job tranCendenZ.
 
R1ckCa1n said:
I think stop bashing the "benchmark" and accept it for what it is would be a great step. I look at it like a radio station or website; if you don't like it, don't tune in or browse their website. It does, however, give people the ablitlity to run the same "benchmark" across multiple platforms for comparision. All review sites give is how well games will run on one configuration for each video card (typically).

If this "benchmark" was so "useless" and "sucked", ATI and Nvidia wouldn't spend more time coding drivers to Futurmark than actual games.

Just my 2 cents
Click to expand...

It is somewhat useless and it does kind of suck. ATI and Nvidia knew it drew a large fan club and they are out to make money. Getting people higher scores and wanting to buy better video cards for it is what makes them happy. We throw our money away for that stupid benchmark. I'm just not going to run the benchmark anymore, not because i'm trying to make a statement, because hardocp helped me realize that it was pretty worthless to use that benchmark to evaluate my video game performance. They helped me make my decision for my x800 pro. MIght not have been completely smart to rush out and buy, but you get the point now. Now I see your jumping on the other bandwagon, the "anti anti 3dmark" bandwagon. Whatever, i've had enough of the stupid bandwagon claiming crap. No more for me. I've been drinking too. Time to go pass out on the lawn.
 
Lord of Shadows said:
How much does 3dmark05 tell you how well game X is going to run. =)
Click to expand...

I scored ~4,100 with a 6800 Ultra. With some new drivers (4.10, I think) some guy with a 9800 Pro scored ~5,300.

I would like to know what game that represents.
 
OriginalReaper said:
I scored ~4,100 with a 6800 Ultra. With some new drivers (4.10, I think) some guy with a 9800 Pro scored ~5,300.

I would like to know what game that represents.
Click to expand...

Probably b/c your core system sucks... and he's running an FX 939 with everything (especially videocard) overclocked.

Anyway, here's my idea for improving 3Mark......

Instead of just using reference images.... use an automated comparison test. Show the 2 scenes (reference image & currently rendered image) merged with the differences highlighted in white....

you know the black screen with white dots where the differences are....... welll add up the white dots and subtract (use some calculation) from the final score.

So if you're using optimizations to increase your score the "Quality Reference Comparison Test" will subtract roughly the amount of points off that you gained.

This will counter act cheating. If the differences are minor then only a minor subtraction, but if the scene is rendered very poorly b/c of all kinds of optimizations then subtract more points.

I'm not sure if it would catch artifacting from overclocking too far, but if it did that would be great too. B/c those arent valid scores if its so overclocked that you're getting all kinds of artifacts!!

Ex:
all optimizations off: 5000 points + no quality difference = 5000 points
turn optimizations on: 5500 points - 400 quality points for deviations from reference image = 5100 points

- OK, I THINK MY IDEA IS PURE GENIUS!! - (patting myself on the back)
 
chrisf6969 said:
Probably b/c your core system sucks... and he's running an FX 939 with everything (especially videocard) overclocked.

Anyway, here's my idea for improving 3Mark......

Instead of just using reference images.... use an automated comparison test. Show the 2 scenes (reference image & currently rendered image) merged with the differences highlighted in white....

you know the black screen with white dots where the differences are....... welll add up the white dots and subtract (use some calculation) from the final score.

So if you're using optimizations to increase your score the "Quality Reference Comparison Test" will subtract roughly the amount of points off that you gained.

This will counter act cheating. If the differences are minor then only a minor subtraction, but if the scene is rendered very poorly b/c of all kinds of optimizations then subtract more points.

I'm not sure if it would catch artifacting from overclocking too far, but if it did that would be great too. B/c those arent valid scores if its so overclocked that you're getting all kinds of artifacts!!

Ex:
all optimizations off: 5000 points + no quality difference = 5000 points
turn optimizations on: 5500 points - 400 quality points for deviations from reference image = 5100 points

- OK, I THINK MY IDEA IS PURE GENIUS!! - (patting myself on the back)
Click to expand...
Too bad that means I can null pixel shaders and get a really good score then...
 
It's really pretty simple. If Futuremark wants to be relevant there is no way it can do so unless it is running engines of todays and future games. I propose this: Our friends at Futuremark need to make nice with the developers of upcoming games. They need access to beta quality builds of the engines to use. They would of course use their own talented artists to come up with their own content but they would use real engines. The benchmark would be modular and updates to engines such as all the different variations of the Quake 3 engine would be added as they neared completion. This would allow gamers to get a real taste of upcoming technology. Not only that but they could actually then have premium content such as short playable versions of the demos for registered users instead of just the non interactive stuff. This would allow their coders to do their magic as well and give folks more incentive to purchase a subscription.

Futuremark is not a shop that builds game engines. They doubtfully ever will. They need to get with folks like the guys building stalker and sign up some NDA's and get some code and release the STALKER 3DMark Module and let us have at it. Wanna see if the new NFS:U2 engine will chug on your older box? Download that module and try it out. Modularity and real engines if not real content. That is the key to the future of benchmarking. If Futuremark is working on 3DMark XE or whatever, this is what they should be working on. Relationships with gamers and relationships with devs. Going the lone wolf route is just alienating them from the community.
 
kcthebrewer said:
Too bad that means I can null pixel shaders and get a really good score then...
Click to expand...

No, you couldn't. You would get a much different result, at least when compared to a pixel overlay. And the variation subtraction points would offset the score to a similar or lower score.

Another thing why have the 3dMark 2001, 3D Accelerator Comparison & Processor Comparison things been broken for so long? And why don't you have those 2 options for 3dMark2003 & 3dMark2005. At least for the pay customers. That was one of the coolest features. To be able to click the link... and see if you're in the right place for your processor or GPU. I was always in the top percentile :)
 
I think the answer is where does Futuremark's money come from? ATI and nVidia pay them money so that their cards look better. If futuremark were to drop them (which they would have to do to be relevant) how would they make money? By paying to license engines? Not bloody likely. By charging gamers to use their product? Not gonna happen. I'd never pay for that. Their current business model is the only financially viable one, so that is the one to which they will stick.

The solution is open-source. Let some coder crank out a bench that stresses the cards. Who cares if it looks pretty or not, it's the numbers that matter.
 
This is a repost from the other 3Dmark thread:

Here is my problem: With 3dMark05, there are deliberate scenes in there that run in the "Frames-per-minute" arena, intentionally so. Many of their tests and demo scenes are now in the pay category. Why is this? If there are ads and sponsors, wanting to get us to upgrade our hardware, why do the Average Joe’s have to pay anything?

If I would already be done if I were using a Falcon, would I even need the benchmark to tell me so?

I look at these benchmarks more as a tech demo, and lately, that demo is loosing its luster. How cool was the "Neo" demo, or, if I remember correctly, the precursor to Max Payne (or was that another benchmark?)?

Now, on an Athlon XP 2600+ with a 5800U, I am hitting the minimum baseline for the bench, and it is barely watchable (haven't had time to test it on my 3500 with X800XTPE). I can run Far Cry on my 1800+ with an old GeForce 3:Ti with much eye candy.

I guess my point here, is that this engine that they have built isn't scaleable. Wouldn't it be cool to have a benchmark that told you how much of your video card you could use and maintain a 45 FPS, or 60 FPS rate? A really powerful system might be able to run with full shadows at 1280x1024 with 16xAA and 8xAF, where a lesser system my need to run at 1024x768 with no AA and 2xAF and low detail shadows.

Base each factor with a point scale, but make it so that a lower resolution with higher AA and AF would be near a higher res with lower AA and AF, since visually they may be the same. Also, you could possibly put in flexible polygon LOD, reducing until you hit the FPS level. The higher the polygons, the higher your score. And the great thing, it would be possible to get this thing to scale nicely to new generations of video cards. Just make incredibly highly detailed base objects that probably wouldn't be fully renderable for many years.

The problem with that from a company’s perspective is that it shortens the revenue stream. Well, then they would just need to come out with new demos that plug into their engine. Load them up with adverts like they do now. Leave in the website type benchmark stuff for the pro version, but change the financial model to a yearly based one, and maybe halve the cost, or leave it at 2 years for the same price. Having plug in demos would also reduce the risk of cheating, since they could write demos that should get nearly the same score, but using a different type of scene.

Also, letting me store a few bits of data on server really doesn't cost a whole bunch and should be imbedded in the cost of advertising. It promotes usage of the software by users and that in turn promotes the usage of the benchmark by companies trying to sell their hardware. Limiting me to 5 benchmarks is quite restrictive.

Keith
 
"Probably b/c your core system sucks... and he's running an FX 939 with everything (especially videocard) overclocked."

Not really. I know the guy, and his video card is not overclocked. His CPU is 2.8 @ 3.0, and a gig of RAM. I have a 2400+Barton at 2.4GHz and a gig of RAM.

So no, it's a poor benchmark.
 
FM should use all that slush money they get from the hardware vendors and actually buy a licence to all the games they say that the benchmark is measuring. The reason the [H] way of benchmarking works is that it takes the item to be measured (say a video card) controls the testing environment (uses the same Mobo, RAM, CPU, etc...) and runs it through a repeatable set of parameters (games in this instance). Throw a different vid card on the same testbed and run the same tests, and you have an actual comparison.

Maybe that's the solution... what if they publish base line scores for certain configurations. It would piss off the losing vendors, but it would help validate it as a benchmark.
 
The mail I sent to Tero: :cool:

Dear Tero.

"If you make a mistake, and don't do anything to set it straight, you make a second mistake!" -Old Chinese saying

I read on HardOCP's pages that you have withdrawn your threats of legislasive action. Good to see that you are a man who can admit your mistakes, and do something about it. I can, however, understand your frustration of the massive, and often unfair critisism that 3D Mark is being exposed to. For years, 3D Mark has been the coolest benchmark around, and when '03 finally came out, i stayed overtime at work to wait for the download to finish, so I could burn it and take it home. I couldn't wait another day. At the time I had a brand new computer, and it is so cool to se what your hardware can do, and nice to have it maxed out for once, when you know your PC kicks ass...

I very much welcome your invitation to constructive critisism, and will delightfully contribute to (possibly) improve my favourite and most often used benchmark. I think one of the reasons for all the hammering on 3D Mark is that reviewers are provoked by the slogan "The Gamer's Benchmark". In some ways I agree to this critic - many of the tests in a 3D Mark run do not really resemble a gaming experience.

Some of the game tests are just "fancy eye-candy" that show us what can be done using the latest shaders and techniqes etc. And this is what makes 3D Mark fun! I already know how the existing titles perform on my PC, and I don't give a penny for knowing how many hundreds of frames per second I can get if I play Quake 3 at 640*480. It is simply not interesting. (My 2 cents to Tom's.)

What I would know from a 3D benchmark is how well my PC will be able to run future games ("Futuremark - get it?" ;-) as well as being able to compare it to other systems. (Read: Ditch my friends..;-)

What I do not like about 3D Mark:

1. It doesn't show how CPU power affects games. The CPU tests consist of turning off the built in hardware-accelleration of the graphics card, and making the CPU calculate the shaders etc. NO GAME DOES THIS! Even though my PC has (OK, used to have...) a cutting edge graphics card and a fairly good processor too, there are still games that makes it "lag" - because of the CPU-load. What a CPU does during an intensive game is AI and physics calculation. You were on the right track in 3D Mark 03 with the "ragtroll"-sequence. Disappointingly, there was nothing that involved the Havoc 2-engine in 3D Mark 05.

My advise: Scrap the current CPU-tests, and give us a couple of reeeeally heavy physics-calculations, preferrably involving water, explosions and the reaction of different surfaces. This should be CPU-test 1. Then you could give us a test from for instance an RTS, with hundreds of enemies on each side charging each other. (With intelligent AI programming behind it.) This could be CPU test 2. Finally, when the CPU tests are valid, they should count more on the overall 3D Mark score.

2. It doesn't really take advantage of the latest advancements in graphics technology. Althoug PS 3.0 is supported in 3D Mark 05, it doesn't utilize its potential. A friend of mine who has a 6800GT and plays FarCry a lot, said he got a 10 - 15% improvement when he installed DX 9.0C and the PS 3.0 patch. Nor does 3D mark support ATI's latest texture-compression technology. These technologies will (i predict) be crucial in upcoming games.

3. It doesn't test OpenGL performance. The difference between 3D Mark results and Quake 3 Benchmarks - no need to say more.

Maybe the solution for 3D Mark is to license some technologies /engines from popular game makers. Then 3D Mark will for sure return as being "The Gamer's Benchmark".

I hope you will read this and think about my suggestions, as they are well intended.

Yours sincerly
Click to expand...
 
I think the problem is that some believe 3DMark should be a one stop shop for all their game testing needs when it isnt and I've never really seen Futuremark try to put it out as being such. I think the big thing that people are missing is that the benchmark is broken up into different parts that tests different things. It's not all the same thing and I think this confusion is only perpetuated by the single score that the benchmark gives. They should scrap the single score and just give the 4 different game test fps results. They should also plainly explain what the specific game tests are doing. For what 3dmark does it is aan ok representation of specific not overall gaming performance when the games use similar techniques. So my suggestion for futuremark is to hammer that into the users thoughts.
 
Aren't the games of the future have 64bit ports? So, shouldn't 3dMark05 have a AMD64 port?
 
You must log in or register to reply here.
Back
Top