Experiments are Hard

By Ctein

My column last week about mysterious out-of-focus areas in a series of my photographs brought forth many useful and interesting suggestions from readers for experiments I might do to further pin down the cause of the problem.

I encountered only one small difficulty with these: the problem wasn't repeatable! I had 10 photographs with profoundly defocused areas, and that was it. Nothing before and nothing since. Makes it more than a little difficult to investigate further.

Mind you, I and those readers could (and did) do an awful lot of data-mining from the existing photographs. We could eliminate many possible sources for the problem from the available evidence and narrow it down to a few plausible ones. That, though, was as far as I could take it. I may never know the answer...unless the problem resurfaces in a way that lets me either test it further or get equipment repaired. If repairs are what is needed; even that isn't obvious.

This is one of the (many) reasons why doing useful photographic research is hard. Often the things you want to investigate, especially the problem areas, don't cooperate by appearing at your beck and call. That doesn't necessarily mean they are random or inexplicable occurrences, just that you don't know how to control the situation well enough to force them to appear reliably (or, more usefully, make them never appear).

This afflicts even the best researchers with the most well-equipped labs. Most of the major research I've undertaken over my photographic career has proven somewhat controversial for precisely that reason. Other people may not see the same results or have the same experience as I do. That doesn't make my research invalid, but it can make it very difficult to validate.

Likely the most famous example of that is my study into the cause of bronzing and silvering-out in RC black-and-white photographic papers, to which I devote considerable pages in my book Post Exposure, available free online. I'm not going to detail the research here; you can read my deathless prose in the downloadable book. The essence of it is that those papers are susceptible to damage because they generate their own reactive compounds under exposure to light. The damage can be entirely prevented (well, so far; it's only been 16 years) by very, very light selenium toning or treatment with a preservative solution that used to be sold by Agfa called Sistan.

That's it in a nutshell. Should be pretty easy to check out, right? And, indeed, when I published my results, Agfa, Kodak, and Ilford all paid them considerable attention (I do have some credibility). Agfa readily confirmed the self-oxidation problem and publicly acknowledged it, even going so far as to recall and suspend production of certain papers while they worked on reformulating them.

Kodak was less forthcoming about the matter. In private conversations with me they were happy to say that they agreed with my results. Publicly, they decided the less said the better.

Ilford was the major holdout. They persisted in the belief that there were no inherent problems with these papers and that any bronzing or silvering out could be attributed to poor processing or environmental damage. These were known to be problems; when Gary Mortensen was still running the photo department at the Minneapolis Institute of Arts, he showed me Ilford Multigrade IV RC prints that had clearly suffered damage from chemical compounds in the air. Some sort of "sick building syndrome" for photographs was at work. They had hundreds of these and it was most clear that RC black-and-white prints were not stable in the MIA's environment.

Ilford, though, would not admit that there was anything innately problematical about the materials. It was all bad processing or bad environment. They had a good reason for doing so. They had tried to replicate my results. They couldn't get the prints to deteriorate under exposure to light. It just wouldn't happen for them at all. What else were they going to do? It's not just that it's difficult to acknowledge a problem you can't make appear, though that's true enough. Had the roles been reversed, I'd have been skeptical. It's that if you do acknowledge it there's still absolutely nothing you can do about it! How do you eliminate a problem when you can't make it show up in the first place?

Inferentially, there must be other factors involved besides simple light exposure. That's not only supported by my and Ilford's results, but by photographers' real world experiences. Many photographers have experienced the silvering-out problem; many others haven't. Very few experienced it as reliably as I did. Without that reliability, testing different conditions to isolate the causative factors becomes nearly impossible. I happened to be the right person in the right place. Lucky me.

Ilford and I never did figure out why I was so "lucky" and they so extremely "unlucky." Like my fuzzy photographs last week, we'll probably never will figure out just why some properly-processed RC prints are subject to self-deterioration and others aren't.

Photography, like all of life, is filled with many mysteries. One must get used to living with a certain amount of uncertainty.

Ctein

Ctein's weekly column on TOP appears with adequate but not absolute reliability on Wednesdays.

Original contents copyright 2013 by Michael C. Johnston and/or the bylined author. All Rights Reserved. Links in this post may be to our affiliates; sales through affiliate links may benefit this site.

(To see all the comments, click on the "Comments" link below.)
Featured Comments from:

John Robinson: "I'm a software developer. A lot of software developers hate 'debugging'...but I love it. It is more science, than mathematics, because it is all about experiments. If you can't reproduce the problem, then its harder, and I like it more...that's just me.

"The most difficult problems are the ones for which the detection is farther away from the cause in time...in fact, the farther away the manifestation is from the discovery, the more difficult the problem is. Debugging is about guessing, and then building tests that will prove, conclusively, that the guess is wrong. You can never really prove it is right...but you can, in learning to reproduce the problem, prove definitely that your hypothesis was wrong. If you can not reproduce the problem, then you have to debug it 'in the wild.' Those are my favorites...also the most difficult.

"I have no idea what caused the problem you observed...my best guess would be some atmospheric (non-camera) phenomena...I would suggest returning to the same place on the same day of the week, or same day of the month, to see if it re-appears. Not easy, I know.

"And, as with all debugging, sometimes, you just want to know it doesn't happen all the time...then you'll keep an eye on it...wait for it to happen again...if it starts to happen a lot, then the priority goes up...otherwise its just one of those mysterious things....

"Life is full of mysteries...one of my favorites is 'why are there any rules at all?' Said another way, we humans tend to view order as a sign of intelligence. I'm not a big believer in God, or 'Intelligent Design,' but the fact that there is a mathematical equation that can reproduce the shape of a fern is, to me, incredibly profound...I don't know why this is, but accident is not the item at the top of my list.

"One of my favorite stories, from a friend, is of a woman who complained that her word processor (yes, this is going back a bit) was inserting long strings of random spaces in her documents. Several technician visits failed to uncover the malfunction. But the next time the problem was reported, another technician was dispatched. Rather than intervene, he just observed. Basically, he said, 'show me this problem.' The demonstration led to the observation that from time to time the woman's ample breasts were depressing the space bar, unbeknownst to her.

"Sometimes, very simple, obvious problems, take a long time to solve...because they're only simple, and obvious, once you have all the information."

J: "When I used to work at Lick Observatory on Mt. Hamilton (45 years ago), we used to sit on the wall overlooking the Bay Area enjoying the view of the lights along the Bay as the sun set (or rose). I remember the twinkling of the lights in the unstable evening air and the wisps of fog floating along the shore. We would wait hours for the air to stabilize before observations would begin. I guess you see where I'm heading with this...."

Ctein replies: This was covered in the previous column, but since you posted the comments here…. Atmospheric distortion does not jump around in such a way that when you're doing a panoramic sequence it only shows up in the right 25% of each frame and does so in each and every frame consistently. Well, not unless the gods are really out to get you! If it's a divine malevolence, not much I'm going to be able to do about it.

Bruce: "I repaired Olympus cameras for 20 years. I can...er, could...pull an OM-1 apart in my sleep. My biggest problem by far was when a camera came in with a fault, supported by negatives, but it was not reproducible. Drove us stark raving mad. Ctein has beautifully summarised the dilemmas we had in the repair industry. (I'm now out, selling cameras retail, and having a ball trying out all the goodies which come into the shop.)

"I have my own theory as to why the image is unsharp. Many of the posted comments are worthy, but not supported by facts. Without being at the place of taking, knowing the camera settings, it's very difficult to diagnose the problem. It really is a "glitch"  until it occurs again. (See real meaning of 'glitch' here—I heard it first from the Apollo space program in the '60s.

"My Canon 7D with 24–105mm gave intermittently unsharp pix in my aerial photos for a month, then miraculously cured itself. Still dunno why. Has never happened again. Ahh, the mysteries of the Universe." 

psu: "I also work in software, which means that most of what I do is dealing with what I have done wrong six months ago. On the topic of debugging: one of the best discussions of debugging ever put to paper is in the classic Tracy Kidder book The Soul Of A New Machine about a team of engineers designing and building a new computer in the late '70s. The chapter called 'The Case of the Missing Nand Gate' is a veritable reference manual on what you do to find those awful intermittent bugs.

"Oh. And even though the book is about a 30-year-old computer that you've never heard of, it will teach you more about computers than any other lay-person-readable text in the world. I'll pay you back if it doesn't."

Mike adds: I agree—one of the great books of the present era and a must-read for everyone, young or old, smart or slow, computer geek or normal civilian.

Comments

[Bob Rosinsky]

Every tool has a quirk or two--no matter how well designed. Embrace the oof and work with it. Oh, how Zen--an easy state to reach in the Bay area.

[Simon]

Not sure whether this has already been mentioned, but could it be a temperature-related problem? Is it possible for example that it was a cold night, and this caused contraction of the base to which the sensor was attached at a different rate than the sensor, introducing curvature in the sensor? Or just the two sides of the sensor being made of different materials, again introducing curvature. I would guess even a tiny amount of curvature might be enough to throw part of the focal plane out. I have no idea whether or not this is an issue with sensors (and whether for example it depends on how the sensor is attached to the camera or the materials that go into making the sensor), but it's worth a thought, especially if it was a cold night.

[rnewman]

Doing definitive bronzing/silvering studies on different papers probably will require access to spectrographic and/or chromatographic equipment and the money and time to do a full experimental design. Resources I am sure you don't have (me ether!). Once, Kodak might have been a resource, but now, outside a few government or university labs, or some otherwise oriented industrial labs, I have no idea where it could be done. Maybe Wilhelm labs could do this. If there was sufficient interest in the question. Given the great reduction in the use of silver based prints, I don't think the interest is there. Maybe a Phd thesis topic for some student looking for an 'arcane' topic?

[Richard]

I am glad at least someone was working on this B+W paper problem. I remember having so many fights with our lab on this and the only solution that was presented to us was the "Kodak promise of excellence" program. Basically Kodak would reprint your photo for free, however only if you also used Kodak film, which we hadn't done in the past. I believe they didn't cover the retouching either (understandable quite difficult to do) so the photographer was still out of pocket. It really was a frustrating time for B+W and all though we had a few prints returned to us I am sure there were quite a few clients who just scrapped the prints and thought less of us. We even had a print returned to us last month, but guess where Kodaks lifetime guarantee is now. We did have a printer we occasionally used for special prints who worked by hand and used Fuji paper, of which we never had a problem with.

[Lukasz Kubica]

Such cases are quite common in complex computer systems also. There are bugs which are not really repeatable or even worse, they happen quite often, but are not traceable because of heisenberg's rule - universe is quite a nasty place to live :-)
Sometimes however, such problems can be detected by source code analysis (when you have one, that's why opensource is not that bad).
So, just get me the source code for the universe and I shall solve your problem in a finite time :-)

[Stephen Scharf]

Experiments don't have to be hard if you do proper Design of Experiments (DOE).

I'm constantly surprised at the number of scientists and engineers that have never heard of, let alone were trained in, DOE.

[Hugh Crawford]

Heisenberg's Oscilloscope

There once was a certain high end piece of broadcast video equipment that has some permanently attached test leads. The manufacturer could never reproduce a particular intermittent problem when the board was hooked up to the scope but when they returned that model to customers the problems would recur. They never found the problem on the scope , but leaving a pair of test leads clipped in kept it from ever recurring so in they stayed.

[Craig]

Any problems we had in science labs was usually down to the water.

[V.I. Voltz]

I can make RC spots appear when I wash prints in tap water, which contains minerals. When I use MilliQ water for all processing steps, including the wash, I cannot make them appear. I suspect that the reaction may require a catalyst or initiator, and that UV alone can be insufficient to make the reaction occur.

Voltz

[Sarge]

Dear Ctein,

I just want to thank you belatedly for Post Exposure which you referred to me for further reading awhile back (in answer to a comment in your exposure range series). I didn't realize beforehand that you were giving away a book!

Although I'll probably never be shooting film, I find Post Exposure eminently useful when "developing" my digital photos in Lightroom. Being acquainted with the science behind post-processing is empowering, and encourages one to "experiment" while shooting and in post. To a layman, it's a very good read for its inclusive and "deathless" prose ;)

Thank you.

BTW, your web site is looking great with the new template.

[Jim in Denver]

Re: bronzing/silvering -

So that's what that is! I have quite a few RC prints that are bronzing. (Including one that I can't print digitally to save my life... go figure...) The bronzing is a very different effect from those that were improperly washed. I have been wondering what was going on, and now, after at least 10yr, finally know.

Thank You Ctein!!!

-Jim

[Joe]

This is also a big problem in software engineering. Many a vague bug report is rejected as "could not reproduce", for similar reasons. There are just too many variables. If QA or engineering can't see the problem, then they can't fix it.

[Semilog]

Ozone?

[Arne Croell]

Btw, for those of us still making silver gelatin prints, Sistan is still available, now under the Adox label as Adolux Adostab. Freestyle carries it in the US.

[Hans van Driest]

This is probably already mentioned, but to me the most logical explanation seems a problem with image stabilization. When it is parked in a wrong position, or started up in a wrong position, I can imagine the problem you found.

[ctein]

Dear semilog,

Well, don't know if it's ozone being photo-generated by the titanium dioxide in the paper, might just be triplet state oxygen. But, yeah, that's the prevailing wisdom, that oxygen is the intermediary and it's not a direct reaction between the titanium dioxide and the silver. It doesn't require UV, by the way. Visible light will do just fine… dammit.

It's also clearly extremely sensitive to processing conditions; prints usually do not silver out uniformly; there will be areas that do that are intermixed with areas that don't.

I was half tempted to subject some test prints to scanning electron microscopy and an electron microprobe (I've got a friend with that gear), but I didn't get around to it. My interest was of a practical bent–– could I nail down the proximate cause and could I find a solution to it. The answer seemed to be yes and yes, and after that it became more of abstract interest. Which pushed it way down the to-do queue, I'm afraid. One of those “in my copious spare time” things.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[ctein]

Dear J and John,

This was covered in the previous column, but since you posted the comments here…

Atmospheric distortion does not jump around in such a way that when you're doing a panoramic sequence it ONLY shows up in the right 25% of each frame and does so in each and every frame consistently.

Well, not unless the gods are really out to get you!

If it's a divine malevolence, not much I'm going to be able to do about it.

pax / cursed Ctein

[JohnMFlores]

Maybe it was an earthquake.

[Tom Kwas]

BTW Re: Bronzing...

The photographic department I was running back in the 90's was chugging along perfectly, everything copecetic, until the company replaced our carpeting with a nice, trendy colored, industrial carpeting. Within three months, everybody's RC prints they had thumbtacked to bulletin boards started bronzing! I had never seen anything like it, and always worked in commercial studios that had been using RC paper since it's introduction! I had RC stuff since the dawn of time just thumbtacked to stuff and never had a problem, even the older prints didn't bronze like the new stuff. They must have changed the formula in the paper...

[Kenneth Tanaka]

My own, previously unstated, opinion was that the blur might have been caused by a heat plume from an unseen roof vent between the lens and subject.

I occasionally get such blur patches photographing wide urban scenes from high vantages. Consider this wide scene from August. Now look at a detail of the men. Beam me up, Scotty!

Winds will make such plume blurs seem unreproducible.

[Ken N]

Diurnal Wander

[Kevin Purcell]

Six Stages of Debugging should always be kept in mind.

1. That can’t happen.
2. That doesn’t happen on my machine.
3. That shouldn’t happen.
4. Why does that happen?
5. Oh, I see.
6. How did that ever work?

The problem is business marketing folks see "issues" as stage 1 or 2 problems to be denied.

The engineers or scientists or software developers at least get to stage 3 and then work hard on stage 4 (the topic of this post) to get to stage 5.

[Chris Lucianu]

Re the defunct Agfa's Sistan: other than the above-mentioned Adolux-marketed Adostab, Ag Stab is commonly available over here in Europe. Same formulation as Sistan. Even the spec sheets linked in photochemical webshops are often old Agfa Sistan PDF docs.

Searching the ever-resourceful APUG forums for "homemade Sistan equivalents" (essentially K thiocyanate plus wetting agent), one can find a revealing post from 2005 by a contributor signed as "Photo Engineer", ominously located in Rochester, NY. It ends on the immortal line:

There is an article published by Ctein on this subject. I recommend it to you all.

http://www.apug.org/forums/viewpost.php?p=176584

These mighty words should be inscribed in bronze, silver, and titanium.

[ctein]

Dear Tom,

Fresh carpeting has all sorts of nasties outgassing from it. Classic case. Not a change in paper formulation, a change in the environment.

~~~~~~

Dear Ken,

Same problem with your hypothesis as with the rain/mist/turbulence theories. It has to magically appear in the right 25% of each frame, no matter what direction the camera's pointed in -- remember, I was photographing a pano sequence.

pax / Ctein

[John Robinson]

Software design, which I also really enjoy, is more like Mathematics....you have to envision what you have, see what you want, and imagine what the space in between looks like.....

But that's a whole other set of skills.....both activities are fun.....but in different ways.

For a long time, when I would hear people say that mathematics was like software development, I would argue against that with them....then one day I had a long chat with my brother, who was pursuing a PhD in mathematics, about exactly how one did pure research in math, and what thought processes were involved.....not too humble to say I was mistaken......

Designing software and finding new math is the same mental process.....

And it is mathematics.....not science....

[psu]

The difference between software and math is that software has to run on the machine in front of people and math doesn't. There are certainly similarities in the thought processes and in the nature of combining abstract ideas in the ether ... but actually shipping software is more of an engineering activity than anything else. Although a lot of actual engineers would also object to that statement given how fragile the stuff seems to be.