Product Variability, Part I

By Ctein

No two pieces of the same photographic equipment behave the same way. Lenses, camera bodies, memory cards, computers, printers, they all vary from unit to unit.

Some of the ways they vary are unimportant to us. Generally we don't care if the clock speed of our computer or the read/write speed of our flash RAM card is some percent higher or lower than someone else's. Sometimes it can be very important; as Lloyd Chambers has written about at length, focus accuracy in cameras is problematical at best and can vary considerably from unit to unit. Sometimes it only matters to a subset of users; most of us don't care that different units of the same camera model will have slightly different low-light noise characteristics, but if you're an available-darkness photographer who's always living on the Stygian ragged edge, that's another matter.

The problem is, any review you read is about one particular unit. You, and the reviewer, hope it's a typical unit. With time and experience, reviewers develop instincts for when something is horribly wrong; normally, though, it's impossible for us to tell if a unit is merely better or worse than average. Obviously one can't learn anything about product variability from testing one unit, which is all there is usually the time or money to do. Unfortunately, the peculiar statistics of small numbers mean that you have to test many units or you will frequently be wrong.

Let me give you a numerical example. Suppose a company introduces a new piece of photographic equipment that is so poorly made that 20% of them perform horribly. I'd certainly not want to be recommending that to my readers. How many units would I have to test to discover that? If I tested 10, there would still be a 10% chance of me not getting a single lousy unit. If I reviewed two dozen products a year and I tested 10 of everything I reviewed, two or three times a year I'd be assuring my readers that some product was well made when it wasn't.

This would not endear me to my readers or my editors, with good reason.

Not that it's feasible for reviewers to test even ten units. We just can't collect enough information on product variability to provide reliable advice. We don't even try; if we did, readers would be thoroughly ticked at having to wait an extra six months for their reviews.

There are rare exceptions. Back in the early 1980s, when I was writing for Darkroom Photography magazine, the editor Ken Werner and I were very interested in knowing what the absolute best enlarging lenses were. (The results of this investigation are in my free downloadable book, Post Exposure.) There were lots of old shutterbugs' tales out there, but precious little hard information. The magazine paid me a then-considerable sum to test every single one of the top-tier enlarging lenses made, up to and including the fabled apo El-Nikkor (the only true apochromatic enlarging lens*). I think there were 60 or 70 lenses involved in all formats from 35mm through 4x5.

I wasn't too far into my testing before I got to the Schneider Componon 50mm ƒ/2.8, a lens that was widely regarded as excellent. Well, my sample wasn't. It was so badly decentered that there was smearing of the image on-axis! At the edges of the field, around the circumference, resolution varied by considerably more than a factor of two.

Ken contacted the people at Schneider, who were appropriately appalled. A second sample of that lens behaved normally. That got me looking at quality control. I tested all the lenses for decentering. In many cases, I had the manufacturer sent a second sample; there was a lot of random sample variation. When all was said and done, I'd looked at well over 100 lenses, and maybe five were "perfectly" centered. The rest all showed a visible degree of deviation from the ideal. Nearly a third were sufficiently decentered that I wouldn't consider them acceptable.

That's when I started firmly recommending that people should never buy a lens without return privileges, and that they test it thoroughly as soon as they got it. And we're talking about enlarging lenses here, the mechanically simplest of optics. If so many of them are bad, imagine what can happen with regular camera lenses.

That's in the past. Now we come to the present. We all know that computer printers and displays vary in their color rendition from unit to unit. That's why color management and custom profiles are an essential part of any serious worker's toolkit. It's a kind of product variability that we accept as common, and that we can fix with the proper software. What, though, about the variations that we can't fix? Specifically, do two different units of the same model printer produce equally "fine-grained" prints (i.e., with uniform ink droplet patterns)? If not, are the differences between units large enough to matter?

Recently I had the reason and the opportunity to investigate that question in Epson 3880 printers, and got some surprising results. Tune in in two weeks for the full story (as TOP will be closed for maintenance next week).

Ctein

[*I believe the Carl Zeiss S-Orthoplanar was also a true apochromat, although it was never readily available.  —Ed.]

Ctein's weekly column on TOP appears on Wednesdays, with only occasional sample variation.

Original contents copyright 2012 by Michael C. Johnston and/or the bylined author. All Rights Reserved.

Featured Comment by John Wilson: "Roger Cicala of Lens Rentals has done some interesting investigations on variability in camera equipment. He has the great advantage of having access to reasonable numbers of the same item (because he stocks them for rental) and has access to good measuring equipment. His first blog post on it is here. If you don't already read his blog I'd warmly recommend it."

Featured Comment by Allan Ostling: "Your Darkroom Photography test was a classic in its day. I pored over that article, then bought the Computar 65mm, one of the enlarging lenses to which you gave the top rating. I lost all respect for Schneider lenses, and this was reinforced when I got a de-centered Schneider zoom on a Samsung film P&S in 1996. I had a stack of those magazines from Issue #1. In 1990 I donated them to the library at the Center for Creative Photography, on the U of A campus in Tucson."

Featured Comment by James B: "There's also another variable for most modern equipment: local climatic conditions. Not extreme, but average climate from winter to summer. The manufacturers of consumer equipment all have a little paragraph tucked away somewhere in the user instructions advising of temperature limits (and it is normally pretty broad) of something like –20 to +50 Celsius (sorry my American friends, I only understand decimal). I've tested for military-grade use various thermal imaging units. These things are literally hand assembled and all are individually tested at the factory before being passed for service.

"Most are useless in a Norwegian winter or a Arabian summer. OK, those are the two extremes we test against. Most are still exhibiting significant variations in a European winter or summer. Some can exhibit variations on an April day in the U.K. in between pre-dawn and early afternoon, with a temperature difference of only about 15 degrees Celsius. Tiny variations in plastics quality and metal/plastic surfaces can twist the optics by that tiny amount as temperature changes. The principle applies just as much to commercial camera lenses as it does to military thermal imagers.

"Don't start me on humidity—that's an even worse variable.

"And worst of all is shock. You can mount an optic and calibrate it perfectly, but a big jolt and it's knocked off-axis. You can try to get it back in calibration, but what's actually the issue is that one of the 10 or so internal lenses has changed alignment, so it's a back to the factory job.

"I only have consumer lenses, less my completely beloved Nikkor 105mm ƒ/2 Defocus Control which I assume sits somewhere between consumer and pro. There's just too many variables for me to think of wasting my money on a pro lens, when I know that despite everything, so much depends on factors beyond my control."

Comments

[Richard]

"That's when I started firmly recommending that people should never buy a lens without return privileges, and that they test it thoroughly as soon as they got it. "

Certainly good advice.

I had to return a Canon EF-S 15-85mm lens when, on the initial tests, it showed terrible decentering. Many on the forums confirmed a serious product variability.

More recent are the Panasonic m4/3 14-42x and 45-175x lenses which have shown much variability, as confirmed by *many* on the forums.

Richard

[Bill OBrien]

"Recently I had the reason and the opportunity to investigate that question in Epson 3880 printers, and got some surprising results."

Is this where we find out what you did with all of the sample prints of the dockside cranes and water picture.

[David Dyer-Bennet]

Consumer Reports is the only consumer testing organization I know of that actually makes an effort at learning something about quality control (partly through multiple samples, partly through a large yearly questionnaire to their readers). They also make a point of getting their test samples through normal retail distribution, rather than in letting the manufacturers fine-tune them for optimal performance.

And they do sometimes test photo equipment, though not at this kind of level of detail. Their ratings are intended for normal consumers, not for enthusiasts or professionals.

[David Saxe]

Very interesting about the Schneider Componon 50mm ƒ/2.8, a lens. I used one in the 70's and thought it was very mediocre in spite of what other photographers told me. It was probably for the reasons you pointed out in your article. Eventually switched to an El-Nikkor and was more than pleased.

[Mark]

So far I've tested five samples of the infamous Panasonic PZ 14-42mm X lens (pancake zoom) on my G3. EVERY sample gave me blurred pictures at shutter speeds between 1/80 and 1/200, and almost every picture was more or less affected. (Btw. stabilizer on/off and with/without tripod doesn't matter, problem is always there.)
What does these statistics tell us about Panasonic...?
The main problems are that
- the lens behaves different on different m4/3 camera models (it's a problem related to the motion of the shutter),
- some people don't know how to test a lens for problems (and/or critical sharpness).
This makes it easier for Panasonic to deny this issue.
The problem is well documented (even dpreview has it in a review about the GX1), but Panasonic does nothing and the service says it's a problem of a single lens or two. (But they can't repair the defect or they send the lens back with the statement that it's ok.)
That's my statement about quality control and sample variations in modern camera gear.

[Kenneth Tanaka]

I must have led a charmed life. After having purchased and used quite a variety of photo equipment over the years I can say that I've never suffered the kind of manufacturing faults that many people report. Offhand I can only recall two such instances, and neither was a lens.

I'm either very lucky or a poor photo gear tester. Probably both, as I don't tend to spend much time analyzing products. But folks in photo forums seem to spend the majority of their free time "testing" stuff rather than using stuff. Perhaps that's a blessing in many cases.

Nevertheless, I cringe a the thought of getting a crappy 100lb printer. I've had excellent luck with my Epson 3800 and loathe the idea of having to replace it.

[David Bostedo]

I don't know if you've seen Roger Cicala's articles and analysis of lens copy variation. As the owner of lens rentals, he DOES have access to a bunch of copies of certain lenses and has a lot of data.

http://www.lensrentals.com/blog/2010/03/this-lens-is-soft-and-other-facts

http://www.lensrentals.com/blog/2011/10/notes-on-lens-and-camera-variation

[Tyler]

I'm glad to hear that there follow-up to the 3880 testing is on the way. I've been curious about the outcome since I sent my sample prints in to be a part of the evaluation.

[Emmanuel Huybrechts]

Ctein, you just opened a big can of worm!

[John Robison]

When I started this hobby (shortly after Pangaea started to break up) I was just happy to have more than one lens for my SLR. Any lens! Lens quality or sample variation never entered my mind.

I was happy.

Now I know too much to take good pictures.
(That was a bit tongue in cheek)

But, I think, in my old age I am coming full circle. If it'll fit my camera and not fall off I'll shoot with it.

Ah yes, now I'm feeling better.

[Richard Alan Fox]

I took part in this experiment and hope to hear that my printer is above average.

[Mathew Hargreaves]

I am looking to buy an Epson 3880 in order to print on metal and glass. Sadly, at the recent photo show in Seattle, the demo model's horizontal feeder was broken. So a test for feed skewering could not be done. Your review of the printer quality is of great interest.

CHEERS...Mathew

[ctein]

Dear Mike,

You could be right about the S-Orthoplanars. Arthur Kramer had strongly recommended I include them in my tests; he raved about them. Problem was that not only could I not get my hands on them, I couldn't even get full technical information about them. So I don't have any first-hand knowledge as to whether they are genuine apochromats or just incredibly good lenses.

Probably this is the place to stick in a note to the effect that all those enlarging lenses with APO at the beginning of their names? Those are not apochromats. It's deceptive marketing. It's a term the lens makers decided to agree upon to indicate a lens that was better than their usual achromats. But the lenses most decidedly do not meet the definition of apochromatic. Not even close.

~~~~

Dear Bill,

Yup.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[ctein]

Dear DDB,

One of the things Mike and I both agree on is that photographic manufacturers don't cherry pick their products for our reviews. We both have ample evidence that this is not the case, not with the numbers of defective products we get, and sometimes even the replacements for the defective products.

I suppose it's possible that if you're the 500 pound gorilla on the block like Popular Photography, there is actually someone at some of the companies who pays more attention to making sure you get a sweet unit (I doubt it, mind you, but I can't say it's impossible). But at the levels that folks like Mike and I get to play, it doesn't happen.

At least that's one less thing for us, as reviewers, to have to be worrying about. We're just not important enough to be “gamed.”

~~~~

Dear David,

Oh thanks for that! I'd forgotten all about Roger's articles when I was writing this column. Yeah, he's got some good information.

A bit of perspective-putting, though. Ignoring the couple of obviously defective samples he reports on, the resolution spreads he shows are less than + or - 10% around the average. That's measurable but essentially unobservable in real-world photography. So, yes, there's a difference that he can see in his laboratory, but a real-world user, even a fussy one, would not likely notice the variance unless they happened to get both near-best and near-worst lenses to compare.

~~~~

Dear Emmanuel,

OMG, I've never done anything like that before! I am in a panic. Whatever shall I do?

[TWFIC]

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Mathias Vejerslev]

Funny, I'm sitting here with a Canon EF 70-200 2.8 IS L II, bought new in march, couldnt focus consistently with the thing on my 5D II (measured it to +17 in MF adjustment!), returned to shop, they sent to canon repair, they found no flaw and have returned it to me, and here 3 months later I still cant get consistent AF on the (very expensive) thing... Next stop Canon repair with lens AND camera...

[Roger Moore]

A bit of perspective-putting, though. Ignoring the couple of obviously defective samples he reports on, the resolution spreads he shows are less than + or - 10% around the average. That's measurable but essentially unobservable in real-world photography.

I think that's about what he says. Most of his lenses cluster pretty tightly, with the defective ones sticking out like sore thumbs. I seem to recall that he also found some variability with lens and body matching- lens A looks great on body A but ordinary on body B- and even from test to test with exactly the same setup. So the best lenses in his tests may not be appreciably better than average, just have gotten a favorable test.

[davide]

Quick question about the math from your example, Ctein, as it's been a couple years since I took statistics:

Did you take the chance that a single unit would not be lousy (80%) and multiply that by itself 10 times to get the ~10% chance of getting 10 good units in a row?

[Jeff Markus]

I remember having to do that Schneider shuffle in the early 80's when I needed a 150 enlarging lens. Tested 6 or 8 to find a good one. A couple were horrendously bad, a couple were very good, and the remainder fair to middling. My 80mm and 100mm lenses were Componons, but both were excellent ones on the first try. For each of those I chose the best of three, but in both cases there was not much variation in quality that I could detect.

Would have considered a Nikkor 150, but really wanted that nifty stop-down lever that only the Componons had.

[Steve Jacob]

We are probably close to the limits on what we can expect regarding mechanical tolerances in the SLR world for the price we are willing to pay.

I have had issues with a large proportion of "consumer level" SLR gear and far less with professional grade stuff, although there is some variation there too.

The other answer of course is to design systems that are self-calibrating. Without a doubt the biggest advantage of mirrorless cameras is that the AF is taken directly off the sensor. Focus accuracy on my Xpro1 is considerably better than I could achieve on a D7000, especially at wide apertures.

Now all we need is a new way to build flexible lenses whose focal length, shape and RI can be controlled electronically...

Of course electronics has tolerances, a design life and a reject rate too, but to a certain extent correction can be built in to the system (eg. pixel mapping) and testing can be automated to a greater extent.

[Geoff Belfer]

Reminds me that a London UK camera store (i think it was RG Lewis) in the early 70s use to advertise that they tested before selling, all the lenses they stocked to guarantee you did not buy a 'lemon'. It was their way of being able to sell cameras/lenses at a higher price than the emerging camera discount stores at that time.

[Ctein]

Dear James,

This was a point I made, also, in my columns talking about why camera bodies mattered and were more than just boxes that held lenses. One of the differences that mattered to pros was how well a camera body held its settings in use ( shutter speeds, focus accuracy, lens alignment ). All things that would get shifted slightly when a camera was bounced about.

You could find amateur models that would perform just as well when tuned up (e.g., my Pentax ME Super), but they wouldn't stay that way very long. Whenever I needed to do a lens or film test, I always retuned my ME Super, even if it had only been a week since the last test. I always needed a bit of a new tweak to bring it back to "perfection."

pax / Ctein

[Ctein]

Dear Davide,

Yup. I hope I remembered my statistics right. (If I didn't I *know* someone will let me know.)

pax / Ctein

[Crabby Umbo]

FYI, when I was a wee lad in the studio business in the early 70's, the Schneiders were truly falling off the cart as far as lens quality was concerned...we used to test all lenses from our local supplier before we bought any, and I can tell you we went through more than a few of each focal length before we picked one to keep...I tell people all the time, the success of Nikon as a view camera lens company had nothing to do with price and a whole lot to do with quality, or maybe just an average higher quality per batch...I've seen some truly appallingly unsharp Schneiders in my lifetime, and a lot of studio people I know wouldn't have touched the Rodenstocks with a ten foot pole at all for any reason...

BTW, I've rarely found, or used a "panda" Schneider (silver front, black back) from the 60's, that wasn't just fine...go figger...

[Kenneth Tanaka]

Now that my eyes are temporarily sharpened to this topic I noticed that that lens rental business fellow has just "tested" his first batch of Canon 40mm f2.8 pancakes. It looks good in performance and consistency!

Mmmm...I like consistent batches of pancakes.....mmmm!