Data Mining for Dummies

By Ctein

When you're shopping for a camera, you want to know how that particular model performs. Trend lines and averages are useful for winning bar bets or figuring out where the industry might be going. They're a very bad way to buy a camera. People here are smart enough not to buy a camera based solely on technical paper specs, right? Well, deciding that a camera is or isn't for you based entirely on a trend line or the average performance of cameras in its class is even worse. You're not even basing your decision on the paper specs for that specific camera but how some hypothetical camera of that general class might perform, if it performed like the average of all the cameras that came before it. Does that make sense?!

What you should be looking at is not averages and trends, but scatter. What you're really interested in is how individual models may differ from that average. If the scatter is very small, than the average or trend is a good indicator. If the scatter is large, it isn't.

Back when cameras moved from six megapixels to eight megapixels (and from 8 MP to ten) I warned people that the performance of cameras varied enough from model to model that the best of the six megapixel cameras were going to resolve comparably to the worst of the eight megapixel cameras. Sure, on average eight resolved better than six, but the average was of no help in deciding whether to buy a particular model.

I still read  frequent comments in these pages from readers who pronounce that a camera with some spec  (megapixels, sensor size, pixel count, bit depth, you name it) is ipso facto inferior to a camera with a different spec solely because of some average performance indicator. That's the same mistake. As I said, it may win bar bets, but it's not a way of ensuring that you're buying the camera that's best for you.

Let's take a look at a couple of the graphs on DxOmark to further illustrate what I'm talking about:

http://www.dxomark.com/itext/insights_paradoxical/image001.gif

http://www.dxomark.com/itext/insights_paradoxical/image002.gif

Both of these graphs show trends in noise and pixel count. Note how shallow the trends are compared to the scatter of the individual points (camera models). Even within a single manufacturing year, the variations in camera specs are greater than the trends over four or five years. This year's camera may not only be no better than last year's, it may be worse than one released several years earlier.

Since both plots used introduction date, you can match up pixel counts with noise between them. Sometimes there's a good correlation; sometimes it's really lousy. For example, in the first half of 2007, there is a very nice match between more pixels and worse noise. On the other hand, three cameras introduced in the first half of 2004, with a threefold difference in pixel count, all have essentially the same noise level.

http://www.dxomark.com/itext/insights_paradoxical/image003.gif

The third graph normalizes everything to pixel pitch, which is the new fave metric. And, yes, the data clusters more tightly, but there are still variations of several dB from the average. In fact, if one ignores the two very poorly performing cameras in the early years, there's no obvious trend at all; it's pretty much a flat line with any given model varying by a couple of dB. While useful for analyzing trends, it won't tell you which particular camera model to buy.

As the folks at DxO said, "Another thing to notice is that for a given release year, the dispersion across cameras is of approximately the same magnitude as the improvement over the past 5 years. This means that ...the differences among current DLSRs can be larger than the overall technology gain over the past five years."

They also note, later on, that Canon, on average, has the best pixel performance, but the single best performing full frame DSLR model happens to be a Nikon.

That's it for today's sermon. Go and spend your money wisely!

__________________

Ctein

Comments

[Henk Coetzee]

The words "not field relevant" in photozone.de's lens tests say a huge amount that we miss out on when looking at reviews.

[KenL]

Ctein,

I don't find your article clearly explained. It reminds me of the lack of clarity on the www.dxomark.com pages. I like to see graphs fully explained with meaningful labels on the axis and a meaningful title.

[Greg Smith]

Tendering all respect due, Ctein, I humbly posit the theory that you think too much...

best wishes

[WeeDram]

Does no one today just take a camera for test drives and simply take some pictures?

[Ctein]

Dear Ken,

Not all my columns, and not all web sites, are intended for novices. If you don't understand the dxomark's web pages, you shouldn't be reading them. Seriously. Making constructive use of that information requires a certain minimum amount of technical savvy. If you can't suss out what they're saying, you are better off skipping it.

pax / Ctein

[Ctein]

Dear Greg,

Au contraire, mon ami; others think far too little.

--------

Dear Wee,

You've hit upon the vexing part of buying a camera today. With the demise of so many local camera stores, it has become very difficult for most folks to find a physical camera to play with before buying. Even folks like Mike and I often cannot get loaned cameras we would like to test by the manufacturers. More and more, the old (and still wise) advice of "drive before you buy" isn't a real possibility. Potential buyers are forced to turn to third-party test data to figure out what camera might suit them best.

It is not a satisfactory state of affairs. I don't see any way to fix it, unfortunately.

pax / Ctein

[Bahi P]

WeeDram asked, "Does no one today just take a camera for test drives and simply take some pictures?"

The tests are still useful in helping set our expectations and removing the surprises that might crop up in situations in which we didn't get a chance to test.

I chose a camera that can be a bit noisy in low light even though I do much of my shooting in exactly those conditions. Thanks to lots of published tests, I made the choice knowing what to expect. The balance of things offered by the camera is the right one - the bright viewfinder, fast access to the controls like metering mode and focus mode, the general image quality and colour response, the price and the way it feels to use.

Published tests are useful (even really, really geeky tests). We're at risk of letting them becoming the be-all and end-all of the camera-buying process but that's being addressed very well in this article and others published here.

(I'm slowly losing my fear of digital noise, perhaps just through familiarity. It's only a matter of time before bold, unashamed and artful use of digital noise becomes mainstream, just the way similar use of film grain eventually did. Mr Johnston can then quite rightfully claim to have been singing its praises for years…)

[Jarrett]

Yes, agreed Greg.

When I browse DPR forums looking for NICE PHOTOS from say the G10, every post tends to begin with "here are a few test shots," "small comparison," and along those lines. See I care very little about people's personal fiddling with their new toy and I wish the 'newness' (if that is a word) would wear off sooner so I could start looking at some real photography coincidentally with a camera I am interested in.

[John Camp]

I would have expected some serious scatter over the past five years as that is the period in which some makers (Nikon, Canon, and to some extent, Kodak) were producing cameras of serious sophistication while others were just getting involved in the technology. [See the whole Leica M8 debate.] As the others catch up, I would expect dispersion to narrow, as, in fact, these graphs seem to demonstrate. I would be very interested to see a trend line on the pixel-pitch graph which eliminates all models but the high-end machines from Nikon, Canon and Sony (Minolta). I wouldn't be surprised if that were very narrow indeed -- or perhaps a new line that correlates closely with the maker's intention, high ISO or high-res. If you had two trend lines, one showing high ISO/noise effectiveness, and one showing high-resolution/noise effectiveness, I suspect that would show what we all believe, that is, that bigger pixels (D3) have less noise at higher ISOs, and smaller pixels (1DsIII, A900) have more noise and better resolution, and that everything is getting somewhat better over time. Which is to say that DxO has proven what any high school kid who just signed on to Digital Photography Review could have told you.

JC

[Dave Kee]

"If you can't suss out what they're saying, you are better off skipping it." OUCH!

Anyway, to the extent I understand the dxomark graphs I am amazed how well my little el cheapo, lightweight, high synch speed, 6MP Nikon D40 stands up compared to the grownup cameras.

[Yanchik]

"Data Mining for Dummies" but not novices ?

Y

[tim-j]

Thanks for nailing down some interesting ideas. I appreciate your clear thinking!

[John Roberts]

I can remember when cameras were for making photographs, not charts and graphs.

[Tim Darach]

Hi Ctein
An interesting article.
My initial thought were its a long time since I did any statistics, but I wonder how they did the regression analysis (drew the line) on the camera date vs SNR graph; without the D3 (2007 39) the line gets a lot steeper, I'd say closer to 2db - but that's based on me holding a biro to the screen and eyeballing it...

But then they do draw this out later by plotting different sensors sizes in different colours:
http://www.dxomark.com/index.php/eng/Insights/SNR-evolution-over-time/SNR-and-image-quality-evolution
which is more informative, and probably a better way of displaying the data.

Regarding SNR and how software has helped to reduce this: Where you have some control over the parameters, as with cameras that shoot RAW, this allows for a choice- one mans noise is another's Pointillism - then this is good.
But its a shame that all cameras - even the little point and shoots - don't offer RAW. For example there are hacks for Canon's Ixus (Powershot in USA) that allow you to record the RAW (CRW) file. I was amazed at the amount of processing - even at ISO 80 - and how weird the processed image looked compared to the RAW image. They hit the image with what you could mimic in Photoshop by going crazy with Dust and Scratches and then Fade to Darken.

Regards, Tim

[Imants]

ahaa......... I have more fun with this......
http://www.artouko.com/bum/

[Gingerbaker]

I have a couple of problems with the DxO data.

The D3 highlights the first of what I think is a problem with the DxO results. DxO reports on RAW output. And the D3 outputs very clean RAW files.

In fact, they are too clean, I think. How can a sensor output a true RAW file that has virtually no chroma noise, but plenty of luma noise? The answer - I don't think it can.

Nikon, I think - with the emphasis on "think" - is processing away its sensor chroma noise before its final RAW output.It isn't possible to have a CMOS chip that naturally outputs no chroma noise, is it?

So, I am not sure there is a level playing field for the Dxo data - pre-RAW output processing will give much higher marks. Yet, the D3 output at high ISO's may not deliver as good an image at higher ISO's than a camera whose output is processed in an outboard program.

My second problem with the DxO data is that I don't think - and I really could be off-base here - they have specified how their weightings and algorithm are calculated. Too much black box feeling for me, at least for now.

Thanks for another provocative article, CTEIN> :)

[Dennis Allshouse]

Gingerbaker said, "My second problem with the DxO data is that I don't think - and I really could be off-base here - they have specified how their weightings and algorithm are calculated. Too much black box feeling for me, at least for now."

I have to agree. Particularly regarding the 'sensor number'. I can't find a formula for this

[Christopher Perez]

Fascinating. Utterly fascinating.

Its like reading Road and Track where test drivers earnestly compare various automobiles. Its like reading Cycle magazine where they flog various motorcycles around a track and time the laps.

Friends and I being high tech engineers by trade created a similar situation a few years back. We performed a ton of camera/lens tests for 120 and 4x5/8x10inch formats. Learned a lot, actually. I measured and calculated and considered and published... and... and... even with all that concrete measurable information and knowledge, my photographs failed to improve.

It was only after I put aside the pursuit of the unimportant to concentrate on learning _how_ to make images that galleries showed in shows, that awards were awarded, and publications started publishing my new work.

The biggest lesson I learned is this; if one's goal is to create wonderful images, testing and measuring simply does not help.

For some people, measuring and testing becomes a convenient distraction.

[Ctein]

Dear Yanchik,

"DMfD" = "TiC"

("TiC" = "tongue-in-cheek")

Data mining is never for dummies; dummies who go data mining come back with fools gold every time.

---------------------

Dear John,

Interesting ideas, but unfortunately there are few enough data points that parsing them the way you suggest won't allow for an analysis. You could well turn out to be right, but for now it remains in the category of a bar bet rather than solid purchase information.

I'm not convinced that the technology has stabilized. In fact, that's going to be the topic of my already-written Solstice column. You have anticipated me!

Some characteristics have: in the early part of the decade you could find cameras of comparable pixel counts that had a factor of two or more difference in the number of spatially-resolved pixels in the image they produced. Nowadays performance is pretty consistent; if you assume that your real spatial resolution is about 40% of the pixel count, you're going to be close. Other qualities haven't converged, and I don't know when they will.

More bar bets I think.

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 

[Ctein]

Dear Tim,

I also have intuitive doubts about the robustness of some of the trendlines, but that would be a topic for another column, and not important to the topic I was writing about, so I didn't much go into it. Statistics of small numbers is such a dicey area...

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Ctein]

Dear Ginger,

If you dig through the technical papers on their website, you'll find mentioned the important fact that what is being measured are not RAW files but RGB files created from RAW. As they note, in some characteristics improvements in RAW conversion algorithms over the past decade have swamped improvements in the sensors themselves.

Also, RAW data is not the photoelectric signal collected by the sensor; it is highly massaged output data. You never get to see what the sensor collects, and you wouldn't want to. There needs to be lots of internal signal processing going on.

If you go back and reread my review of the Fuji S100fs, you'll see some speculation in there by me and DDB about the differences in internal signal processing between our Fuji and Nikon cameras.

Not that this really matters; you get what you get. You're buying a computer system, as it were, not individual components. Remember that these tests are supposed to be useful, not merely technical. Measuring a pre-processor sensor characteristic that the photographer can never, ever get access to is only interesting to designers.

All benchmarks are very black box; you just probably never noticed it before. Quick, tell me exactly how Kodak computes graininess these days, or Pop Photo computes SQF.

Or for that matter exactly how Modern Photography ran their lens tests back in the '80s, which were a heck of a lot more transparent! Which makes a point of importance: I would always get resolution figures substantially higher than MP's--like 30 to 50% higher! Way way beyond what should be normal experimental differences. Peter Moore and I talked about this; we never could figure out what we were doing differently. We also didn't think it really mattered; the important thing about a benchmark is that it produces consistent results across time and camera models, not that it be directly comparable to anyone else's benchmark.

So even if you knew what was under the hood, it wouldn't do you any good. The real thing you have to decide is whether the results are useful to you. If they are, it doesn't matter. If they're not…well, it also doesn't matter what's under the hood!

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Stephen S.]

Gingerbaker said: "It isn't possible to have a CMOS chip that naturally outputs no chroma noise, is it?"

Actually, that's the very nature of it.

With CCD sensors, noise reduction is performed by a separate processor, and most cameras provide raw images after the CCD readout, but before the noise reduction processing.

With CMOS sensors, noise reduction circuitry is built into the sensor itself, so the output from the sensor is post-noise reduction, and that's what the raw image stores, with no way of skipping that step.

[Gingerbaker]

Steven S

I am on thin ice here, but I think you may be mistaking electronic noise subtraction at the sensor level in CMOS chips for actual noise reduction done downstream.

I have never seen photographs from a CMOS sensor (or, for that matter a CCD sensor) which did not display BOTH chroma and luma noise. That is, until the D3 came out, and lo and behold, most if not all of the chroma noise is gone and only luma noise remained.

Nikon does a great job of this, however, The processing is super fast, and they have done a very good job of maintaining detail and color accuracy.

But, it is also, inescapably (I think), post processing and not an inherent characteristic of the sensor itself.

Until the D3, I think it is a fair statement to say that post processing done on board the camera has always been inferior to the result one can produce by outboard programs and a big CPU.

I have read some posts, but have NO first-hand knowledge, that this trend continues, despite the fine results from the D3. That is, one may get better final results from a different approach.

And so I question the DxO results, which claim to offer us information on the physical qualities of the sensors themselves.

As Ctein points out, there really is no thing as a pure sensor output, we can only read the output from what actually is a small computer system in each camera.

I am still uncomfortable about the objectivity of the DxO results for these reasons, and, combined with the proprietary nature of the evaluations, find no satisfaction from their availability. Especially since some users have pointed out seemingly nonsensical results in the data.

I also think that the DxO data is almost impossible, at least for me, to use as a basis for actually making a buying decision.

I am thinking about getting a small camera to carry with me, instead of my forty pound bag of DSLR's and lenses and kitchen sink. I have been following the on-line discussions and viewing samples of photographs from Canon G10 and Panasonic LX3 users. Michael Reichmann even had a serious post on the relative capabilities of the G10 vs a medium format digital back, and found nearly identical results under certain conditions when prints of a certain size were viewed. Amazing progress in small cams and the output from their sensors!

But, go to DxO and compare the results of a G10 to any DSLR. Would anyone in their right mind come away from the DxO graphic comparison of a G10 and a DSLR and come to Michael Reichmann's conclusion? I submit the answer would be a resounding "no". And hence, my reluctance to view the DxO results as useful to me.