John Gruber: DxO Ratings Are Horsesh*t

[Megalith]

The author of Daring Fireball has declared that “no one should give a sh*t” about DxO’s camera ratings: the image processing software company assigns precise numbers to gauge photo and video quality, but Gruber insists that these give a false illusion of scientific rigor, noting that the overall quality of a camera is “utterly subjective.”

Particularly with their “overall” score, DXO is pretending to assign an objective scientific-looking measurement to something that is inherently subjective. It’s horsesh*t, but everyone in the media falls for it. I said it was horsesh*t last year when they named a Pixel their “highest rated ever”, and I say it’s bullsh*t now when they said that about an iPhone.

 

[gigaxtreme1]

Need another asterisk or two. Although, there is nothing wrong with a little horseshit on a Saturday morning.

 

[JosiahBradley]

Camera quality is completely objective. WTF is this guy talking about? If a camera reproduces a color accurately at a measurable sharpness, you can assign numbers to it. Nothing about photography technically is subjective besides the subject matter being photographed. It's literally an instrument.

 

[steakman1971]

I read his article and agree with his point of view. In my job, I have to estimate software projects. I break it down in a spreadsheet and have the major parts of the project broken down (at least what I know of). I include a range of time for each part - it looks great and does look scientific. However, I can assure you my estimates can be (and often are) bullshit. I tell clients this when I go over my estimates with them.
It is impossible for me to break down a component accurately. For one, do I know the software developer working on the project? It's not me. So maybe the guy is good or he turns out to be an idiot. I have no clue. What if that one feature sounds easy on paper but turns out to be a complicated mess? I know estimating software is different than camera quality - but the arguments he makes are the same.
Hell, just look at a lot of benchmarks out there. Does a certain 3D Mark cover the gamut of your needs? No. It just gives you an idea about something that might be similar to it.

 

[Deleted member 245375]

I actually would agree with this because to my obviously untrained eye the LG G4 still produces better looking visual data from the lens and sensor data than any other camera assembly or sensor in any smartphone that I've been able to test out to this day. I was just playing with a Note8 yesterday and an iPhone 8+ as well and took some sample shots at defaults aka "Auto" mode which is how DxOMark says they test devices and the G4 still just smokes 'em IMO.

I really don't know what Dark Magic went into that camera assembly/lens/sensor combination (it's a modified Sony Exmor sensor) in the LG G4 but, damned if it still doesn't produce images with better visual quality to my eye than anything else. Can't wait to see what the V30 is truly capable of but something tells me the G4 will still end up being at the top of my own "best camera" list.

 

[oROEchimaru]

I have had a pixel xl photo look as good as wifes iphone 6 plus

 

[Spidey329]

It's subjective, to a point.

If you're taking the same photo of the same subject (e.g. test cards) and lighting with different cameras, you can compare for sharpness, focus (speed and accuracy), color accuracy, and other metrics. You can do this with software (which is what I assume DxO has developed). One way is to compare the images against a known benchmark (e.g. the best DSLR). That's what benchmarks are. They aren't a definitive "this device is the best, no exceptions," it's a "in this controlled setup, this device outperformed other devices in the same setup."

The overall score may be deceiving as it's possible for a camera to be tremendous in daylight, but suck everywhere else - maybe the lowlight isn't good, or the autofocus can be slow. Whereas a camera that is just OK in all categories may score higher overall. That's why users should read what makes up the overall score.

Steve's Digicams used to compare point and shoot cameras by taking them to the same location (or same test charts) and time of day, and snapping photos. Users could view the gallery and make up their own mind. Were the bricks on the building sharper with one? The only thing DxO has altered from this is added values, which I assume are from some form of CV algorithm.

TLdr; Proper control of the environment can lead to a decent benchmark for quality which can be assigned a score. I assume DxO has developed an algorithm to compare systematically without bias. Truthfully, they could just be using interns who may or may not be drunk.

I actually would agree with this because to my obviously untrained eye the LG G4 still produces better looking visual data from the lens and sensor data than any other camera assembly or sensor in any smartphone that I've been able to test out to this day.

I have had a pixel xl photo look as good as wifes iphone 6 plus

Makes sense. Most phones use a form of the Sony EXMOR sensor. It's one market that Sony kills it in. Heck, GoPro was using versions of the EXMOR line for a long time (not sure if they switched once Sony competed with them). Top-end dashcams and security cams use it.

On a side note, it's one reason SONY (Samsung has this problem too) has a branding issue. They're so big, their R&D can innovate widgets for other manufacturers, but other branches of their company decide to compete in the same markets - competitors don't like buying internal components from someone who's going to also compete on the completed product (See Samsung/Apple chip fallout, GoPro/Sony camera fallout).

Once you have the sensor, the only changes to alter quality will be the focus system (laser? Ir?), tweaks to exterior optic packaging, and software (this is a major one, poor software and post processing can make a great lens look shit).

 

[Dekoth-E-]

All cellphone cameras are shit anyhow, who cares? If I just need a quick picture of something any cell works fine. If I need to take an actual photograph any cell looks like garbage against my Nikon.

 

[Spidey329]

All cellphone cameras are shit anyhow, who cares? If I just need a quick picture of something any cell works fine. If I need to take an actual photograph any cell looks like garbage against my Nikon.

I think that's the point though. Every evolution of camera has gotten to this stage. Your DSLR went through the same cycle.

The first innovation is about doing something different (e.g. digital over film, or shrinking the camera). The next innovations are improvements of the quality. Further innovations add features. Finally, they get to a point where the majority of users won't notice and most features have reached parity, so they start again (shrink it more). With cellphones it was quality improvements, speed of autofocus, speed of first shot, and lowlight.

It's just like the DSLR market. Compare your Nikon with the top of the market now. Can you really tell a difference?

 

[WorldExclusive]

The word "sharpness" is killing photography. Fewer people know what image rendering and character is.
That's why DxO mark isn not what anyone should go by when choosing a camera or lens.

 

[M76]

There are many uses for a camera, but if you're comparing camera phones scores are perfectly objective way to measure quality. if you're comparing apples to apples you'll be fine. And I shall hope noone willl be stupid enough to buy a iphone8 over any DSLR for professional work based on a score.

So I don't see what's the problem here.

 

[Ebernanut]

I think that's the point though. Every evolution of camera has gotten to this stage. Your DSLR went through the same cycle.

The first innovation is about doing something different (e.g. digital over film, or shrinking the camera). The next innovations are improvements of the quality. Further innovations add features. Finally, they get to a point where the majority of users won't notice and most features have reached parity, so they start again (shrink it more). With cellphones it was quality improvements, speed of autofocus, speed of first shot, and lowlight.

It's just like the DSLR market. Compare your Nikon with the top of the market now. Can you really tell a difference?

The problem with cellphone cameras is that they'll never have a real lens much less the multiple lenses you need for different situations when you're doing serious photography. The software and sensors are getting better and I know some phones are using an extra sensor to give the software another data set to work with but it's still just an attempt to fake what a real lens does and I'm not sure they can ever completely overcome that limitation.

 

[Dekoth-E-]

I think that's the point though. Every evolution of camera has gotten to this stage. Your DSLR went through the same cycle.

The first innovation is about doing something different (e.g. digital over film, or shrinking the camera). The next innovations are improvements of the quality. Further innovations add features. Finally, they get to a point where the majority of users won't notice and most features have reached parity, so they start again (shrink it more). With cellphones it was quality improvements, speed of autofocus, speed of first shot, and lowlight.

It's just like the DSLR market. Compare your Nikon with the top of the market now. Can you really tell a difference?

While what you say does apply to some features, it fails to address the primary difference between DSLRs or even regular SLRs. That feature is the very physical difference of lenses. Auto zoom and digital zoom will absolutely never overcome the physical limitations of not having enough glass and the correct amount of space between them to focus the light. That is why any decent SLR can still take overwhelmingly better photographs than any cellphone on the market. The digital features only enhance what exists.

 

[Stimpy88]

I'm assuming some butthurt Samsung lover didn't like the iPhone 8+ going to the top of the list? Or did Samsung pay him to call out DxO?

All the photo comparisons posted so far taken with the latest phones clearly are won by the iPhone 8+.

 

[dethklokworkorange]

I'm assuming some butthurt Samsung lover didn't like the iPhone 8+ going to the top of the list? Or did Samsung pay him to call out DxO?

All the photo comparisons posted so far taken with the latest phones clearly are won by the iPhone 8+.

It seems not, from the quote of the article.

I said it was horsesh*t last year when they named a Pixel their “highest rated ever”, and I say it’s bullsh*t now when they said that about an iPhone.

 

[Etherton]

The problem with cellphone cameras is that they'll never have a real lens much less the multiple lenses you need for different situations when you're doing serious photography.

That's not a problem. That is different tools for different jobs. Cellphones were never designed to be sports/wildlife cameras.

 

[Stimpy88]

It seems not, from the quote of the article.

You made my point.

 

[Ebernanut]

That's not a problem. That is different tools for different jobs. Cellphones were never designed to be sports/wildlife cameras.

Every type of photography benefits greatly from using a quality lens that fits the situation not just sports and wildlife. Phone cameras aren't useless(I use my phone camera more simply because I always have it on me) but the lack of proper lenses is a drawback that will be extremely difficult or impossible to overcome and have them catch up like the person I responded to suggested.

 

[darrpara]

Sigh I read this is as DX rating and I was like yeah, no companies care about government rated programs these days. Even amphenol doesn't give a damn.

 

[raz-0]

DxOs tests are relevant for those doing the digital darkroom thing working from RAW files. It measures how much the cameras are living up to the paper specs of the sensors. That's about it. Given their protocol, brands will differ based on the design of their camera as a whole. Exposure algorithm, signal path noise, sensor heat management, etc. Basically system noise and everything on the gazinta to the image processor both software and hardware. If you shoot to JPG, or let the factory bundled software do the RAW processing, you are now comparing each companies subjective opinions on photography, and there is no absolute in that regard. That is all choices on how to implement the image processing in camera, and not measures in their DSLR tests.

Case in point, I shoot pentax. When buying into pentax I also considered nikon. At the time I could get a lot more bang for my buck with pentax, but on top of that, while nikon theoretically beat it on low light noise, nikow was willing to blur the shit out of details to remove noise. Pentax chose maxmium information retention, or at least closer ot it. They use the same sensors, both are good at keeping the signal path pretty clean, but nikon is cool with removing detail for low light noise reduction, and for a long time was ahead of pentax in dead pixel mitigation and heat management. nikon would also go to more saturated and slightly overexpose photos if you let it process them in camera. Pentax would err towards mild underexposure to preserve detail. Nikon's choices reflect their history. Lots of flash photography and punchy colors and they do both of those things better in camera. Pentax shows a bit of their history, natural light photography, landscapes, and more mucking about in the darkroom. Their choices represent that background more. I happen to like that more.

And that's their good test.

The mobile test is measuring the whole stack, and while you can make some objective statements about autofocus, lens aperture, etc. A shit ton of the equation is going to come down to the software stack behind the camera, and the vast majority of that is going to be subjective with phones using near identical sensors with very similar lenses. You will like what you like.