Can You Get Good ISO 6400 from an Olympus E-P1?

By Ctein

I've just hit upon something rather cool and somewhat unexpected. Some of you will recall a column of mine from a year or so back called "JPEG...Seriously?" Well, I've found a reason for photographing JPEGs with my Olympus Pen E-P1. I can use them to get much, much better results at ISO 6400 than I could pull out of a RAW file using every trick I know.

(As usual, you can click on any of these illustrations to see them larger.)

Fig. 1. Olympus Pen RAW photographs at ISO 6400 are not lovely things. The full frame is on top, with a 100% magnification crop below. Observe the serious chroma noise, especially the horizontal and vertical stripes. © 2010 by Ctein.

As I wrote at the beginning of the year in "What Tests Don't Tell You," the E-P1 has some serious plaid chroma noise problems above ISO 800. It's great up to there, and then quality just falls off a cliff. ISO 1600 is marginal by my standards, everything above that is crap due to the chroma noise. Figure 1 shows a particularly egregious example of ISO 6400 quality in RAW mode; there's no way that I've found to make this RAW into a palatable photograph.

Now, take a look at the same photograph saved as JPEG in the camera (figure 2) . It's not perfect; there's some large low-frequency blotchiness, but that's the kind of stuff I could dodge or burn away. The chroma quality is not utterly sucking. It's usually not even this bad in most of the ISO 6400 photographs; this is just about the worst example I could come up with.

Fig. 2. The same photograph as in figure 1, but saved as a JPEG in camera. Excellent reduction of the chroma noise, but overall the picture is soft and veiled, and the highlights and shadows are blocked up. © 2010 by Ctein.

I'm truly amazed what the internal data massaging in the camera can do. If I had to, I could live with this as an okay snapshot. I don't much like that overall veiled quality, plus it's rather a shame losing several stops of highlight and shadow detail, most evident in the illuminated and dark stained glass windows and the trees).

So I said to myself, wait a minute...the Olympus can record in RAW+JPEG mode; what if I do that and then mix the two photographs in Photoshop? Start with the JPEG as the base layer, convert the RAW to a grayscale image (which almost completely eliminates the plaid problem), paste it in as a second layer over the JPEG, and set the blend mode to Luminosity (figure 3). I wind up with a photograph that has the clean chroma of the JPEG but retains most of the fine detail structure and tonal range of the RAW file. It's also a lot grainier.

Fig. 3. This layer stack combines a monochrome, noise-filtered copy of figure 1 with the JPEG from figure 2. The result is figure 4.

So, I further said to myself, what if I use my most flexible noise reduction plug-in (Imagenomic NoiseWare) to get rid of majority of the grain in the RAW layer, but not so much that I'm seriously compromising fine detail? Which leads me to Figure 4. Hey, it works! It's grainy, but the grain is even and tight. It's a nice compromise mix of the best characteristics of both source photographs.

Fig. 4. This is not half bad! Chroma noise is much better controlled than in the RAW photograph, but it preserves most of the RAW's detail and tonal range. It also has an acceptable compromise between grain and fine detail. © 2010 by Ctein.

It prints out as a fairly decent 8x10 (Mike's seen prints and Oren and Carl have seen full-size JPEGs, so they can confirm this). Nothing I would call professional quality by my standards, but more than pretty enough. It would be even better if I did a little bit of dodging and burning in on the remaining blotches; I didn't because I'm trying to show what the file really looks like. But, it wouldn't take me long to clean it up a lot more.

Best of all is that I can turn at least 90% of this merging process into a Photoshop automation. Possibly all of it; I haven't experimented with enough photographs yet. Even by hand it's very fast, but an automation would save me a whole bunch of repetitive mouse clicks.

Here are some of the better ISO 6400 results I've gotten out of this new trick (figures 5–7). Is it perfection? Of course not. But seeing as I'd previously decided this camera wasn't any good above ISO 800 I'm a mighty happy camper.

Fig. 5. RAW at the top, merged JPEG+RAW on the bottom. It's a big improvement, and the print looks even better; it's genuinely pleasing. © 2010 by Ctein.

Fig. 6. The merged RAW+JPEG photo combines most of the exposure range of the RAW file and most of the noise reduction of the JPEG. Compare the 100% sections at the bottom, RAW on left, RAW+JPEG on right. © 2010 by Ctein.

Fig. 7. My best result so far, and a genuinely good-looking 8x10 print. © 2010 by Ctein.

Ctein

Ctein's weekly column appears every Thursday on TOP.

Featured Comment by Semilog: "I suspect that there are at least a couple of things going on here.

"First, Olympus has access to much more proprietary information about the sensor and its specific hardware implementation than even, say, DxO does.

"Second, as Andrew Molitor says, the in-camera software (firmware) can be tuned to deal not only with a specific chip design/implementation, but a specific sample of a chip. This is really important because consumer devices, unlike scientific devices, have rather heterogeneous pixels.

"(Scientific devices are usually "grade 0," i.e., zero defects beyond a rather stringent specification. Moreover, scientific devices are generally CCDs, which are usually both more linear in response, and more homogeneous pixel-to-pixel than CMOS sensors. Because grade 0 devices are generally the cream of the crop, selected from much larger runs of devices, they also command very a very steep price premium. Your garden-variety D90 or E-PL1 is not going to contain a grade 0 sensor!)

"The sample-specific dark current calibration done at the factory accounts for variation not only between pixels, but between rows of pixels. On many cameras, this calibration is re-doable through the menus, useful since cosmic ray hits generate new hot and cold pixels over time.

"This sample-specific calibration information is potentially really important when making a zeroed image to start with, prior to application of conventional NR algorithms. This information is not stored in a RAW file, and thus is not available to any of the third-party RAW developers, fundamentally compromising their performance no matter how good they are (unless the dark subtraction is done upstream of the RAW file, which many forum idiots elsewhere deride as not kosher, but which is in fact a great idea, for reasons that now should be obvious)."

Comments

[Jimbo]

Presumably this technique is better somehow than just reducing chroma noise from a converted RAW file? In camera JPEG processing is often very good, so that could be why you prefer blending JPEG+RAW? Or is it just a matter of convenience, to use the JPEG file to create a technically good image quicker? Let me know if I'm missing something because I feel like I am.

[John]

If I understand you correctly, you are working with the actual RAW file as one of the layers? Would this work just as well if you saved the RAW as an EPS file and used that rather than tampering with the original file? Just curious.

[David Ciechanowicz]

Amazingly interesting :) I wonder how much this effect has to do with quality of internal denoising algorithm and how much with let's call "image oversampling". Can you process the same image from RAW using any noise removing software and save it as a JPG and then repeat the procedure? I have feeling it's got nothing to do with Olympus processing, but it's rather effect of overlaying to JPG images - one developed for chroma data and one for luminance.

[James Bullard]

I must be missing something too. JPG files are just in-camera conversions done in accord with an algorithm designed by an engineer who works for the manufacturer. That algorithm doesn't have any data to work with except what's in the RAW capture. I don't understand how the JPG could be better that the best that could be gotten out of the original capture data (RAW).

[nugat]

I see you use cs4. Which version Camera Raw?
In Lightroom 3 (ACR6) when set to process 2010 the noise reduction is so good it seems no further tricks or noiseware is needed. Have you tried?

[e_dawg]

What RAW converter are you using? Most people find that noise reduction in LR3 has been significantly improved over LR2, and that it de-noises as good as anything out there whether it's straight from the camera or using a 3rd party NR plug-in.

[Geoff Wittig]

Very cool. I'll have to try this technique with an older D-SLR I've had converted to infrared use. It's great at low ISO, but above 200 it truly sucks.

As I read the description of this technique it started to sound familiar somehow, and it finally dawned on me just why. When I started inkjet printing in the paleozoic era of digital, I was scanning 35 mm slides and printing them...well, way too large. As you might predict, the grain in smooth continuous tone areas like skies was awful, desite using Provia 100F. The only way to get a decent large print was to scan twice; once for maximum sharpness in detailed areas (which resulted in horribly grainy skies), and a second time with noise reduction cranked up for smooth skies (but awfully mushy details). Once I stacked and blended the two files, I could get a 20x30" print that...well, okay, it was still pretty awful compared to what we can do nowadays.
But my mom liked 'em.

[antoine_k]

You don't say what noise reduction methods you used with the raw file to compare to the in camera. Is this better than RAW with Noise Ninja??

[Tim F]

Ctein, you are a hero. I cannot wait to get home and try this with my E-P1.

[Tim F]

Incidentally, I wonder whether this trick would also work on, say, a D90 at ISO 6400.

[peter]

perhaps we are getting to a point where jpegs are good enough making raw obsolete?

[Eric]

Did you use any noise reduction software on the RAW file? Seems like that would be quicker yet and can be fully automated.

[Jamie Pillers]

Ctein's post regarding combining JPEG and RAW files is similar to a concept I've been waiting for. And if Ctein can figure this out, then maybe manufacturers are not far behind.

Here's the concept. I'm waiting for a digital equivalent of, say, the Nikon compact film SLRs like the FM or FE series (or Olympus OM models). This digital camera would have a few simple control knobs to mimic the SLR's film speed, exposure +/-, and shutter cocking. And of course, an excellent optical viewfinder. It would have NO LCD screen! This is where Ctein's great idea comes in. If the camera could, internally, create excellent JAW files (JPEG + RAW with something like Ctein's process algorithm built in), then no image review is needed. Just compose, shoot, and then at the end of the day go home and download files to computer to see what you got!

Thanks Ctein.

[KeithB]

As Ctein wrote:
"I'm truly amazed what the internal data massaging in the camera can do. If I had to, I could live with this as an okay snapshot. I don't much like that overall veiled quality, plus it's rather a shame losing several stops of highlight and shadow detail, most evident in the illuminated and dark stained glass windows and the trees)."

I, too, am amazed at what the DSP in the camera can do. What I don't understand is why the computer can't do it, too.

[Tim]

I hit the same thing a couple of months ago with ISO 800 on a Canon G9. I thought I'd got the RAW processed about right, then on the spur of the moment I checked the on-camera JPEG and its SNR was way superior.

Now, I can't claim to be an expert on the matter but I'm not sure how you can tell that this merge-with-L-channel is better than diddling *carefully* around with the two sliders for luminosity and chroma noise in ACR. (Sample image #5 looks like the sort of effect I've seen that way.)

Looking at your sample images, I do like the effect on the last; but I beg to differ about the two microphone stands in #6. In film it is often said that grain appears to work "with" the image to improve its sharpness (at a relevant viewing distance/enlargement), and that applying NR algorithms to the noise costs sharpness. I get the same feeling about chroma in #6: by reducing chroma-noise, the colour information also appears to have suffered (big-time).

[XebastYan]

Hi Ctein,
thanx for the trick, i will try the layer trick.
I also figured out when it comes to High Iso the JPG's are better than my RAW Developer can do. Stunning, none of my other cameras does better JPG's than "self developed" RAW's. But the E-P1 does. It's the first System Camera i use where i can do JPG's that will satisfy. Now we know why the very good JPG performance of the PEN Series is famous. Incredible Olympus Engineers, they put a blasting and powerful JPG Engine in those "mickey-mouse-cameras."
Nicely XebastYan.

[Max]

This can be achieved by changing an image mode to lab colour and applying Gaussian or smart blur on the colour channels just to the point where the chroma noise is broken. It works great for film scans were you have that annoying chroma noise, especially on soft toned areas like skies, where the colour dots show. Eliminating the chroma noise has an impact, you can't quite see the colour noise in the print, but when it's gone and only the luminance channel noise remains the print looks a lot less "nervous".

[Amin]

Hi Ctein, would you consider sharing one of the RAW files for the rest of us to try our hand at processing? It's always a fun challenge.

[ctein]

Dear Jimbo,

No, you're not missing anything. You did understand what I was saying. The in-camera software does a far better job of reducing the chroma noise than any of the in-computer tools I have, even noise reduction plug-ins with settings specific for striped noise. The size of the stripes in the Olympus data are just too big for the usual noise reduction software to handle. I have quite a suite of such tools, and none of them could do a satisfactory job of getting rid of the banding, even at ISO 1600. It was entirely hopeless at 6400.

I stumbled upon the discovery that the in-camera software is so much better at this. Lucky me!

By the way, the Olympus Master 2 software can do almost as good a job of noise reduction as the in-camera software. It's very close. But it's no better. And working with two different programs really does complicate the workflow. Since my camera will capture both RAW and JPEG, I've chosen the easy route (which also happens to produce slightly better results).

But, if I find in looking through my older photographs that I have some RAW photographs made at high ISO that I gave up as hopeless before, I can try running them through the Olympus on-computer software to generate a low noise image which I can then merge with the RAW image and maybe get a satisfactory result.

Come to think of it, I should probably dig back further and see if the Fuji on-computer software will do a better job of noise reduction than my usual tools. I have quite a few Fujifilm S100 and 6500 camera RAW photographs made at high ISOs. Hmmm....

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Janne]

You should be able to get similar results by using wavelet-based denoising in LAB (or YCbYr) space, going heavy on the chroma channels, and careful in the luminance channel. Afterwards you will likely want to adjust the ends of the luminance curve a bit as the luminance filtering tends to bump the ends towards the center a bit.

In fact, the "veiling" and loss of dynamic range is a telltale sign that the Olympus software may in fact be using wavelet denoising or something very similar for their results. So if you just go lightly on the luminance filtering you should be able to get similar results without having to mix images.

[Kevin Bourque]

Nice work! I'll file it away for future reference.

You've probably already received a number of replies telling you that if you were using (insert camera brand of choice) you wouldn't need to play tricks like this. Why? Because they're PERFECT at ISO 6400 and what are you doing using one of these Olympus things anyway? Sigh.

[Andrew Molitor]

Camera JPEGs are often AMAZINGLY good, which isn't that surprising because the algorithms can be tuned to the camera model and, at least in theory, to this specific instance of the camera.

Some cameras also correct lens distortion in the JPEGing. Do they undistort for the RAW as well, if asked? If not, this trick isn't going to work at all!

[Ken Tanaka]

Nice practical experiment, Ctein. FIgure 7 is quite remarkable. You could almost claim that it was taken with one of today's shot-in-the-dark dslrs. I'm eager to give this trick a go with my E-P2.

There is actually a whole alternate universe of possibilities available through careful blending and masking with Photoshop. It's one of the little trade secrets of vocational retouchers.

I, too, have been very impressed by the quality of JPG files that I get from cameras these days. But I suppose that we shouldn't be surprised. After all, JPG processing is where the majority of r&d money is spent on these camera's firmware.

[Don Bryant]

Just curious, has anyone tried processing the ISO 6400 RAW files in Adobe Lightroom 3 or ACR 6.X? Adobe has substantially improved the NR in the new RAW processing engine.

[Mark S.]

Is that Joan Baez in the photo?

[David Bostedo]

So does the EP-1 do something with the pattern noise that's better than what you can do with the RAW file? Or are the noise reduction methods in the RAW converter just not as good as those in the camera JPG conversion? It would be interesting if the camera companies' knowledge of the sensor produced better noise reduction than you could massage out of something like Neat Image, Noise Ninja, DFine, or Noiseware.

[Robert Roaldi]

This reminds me of previous threads I've read (who knows where now) where people were annoyed that they could not reproduce the same jpg with their raw processor as the camera produced itself internally when shooting jpgs. It seems as if you should be able to.

[Dave Hodgkinson]

Two things I think are happening here:

1. The camera is applying some noise reduction when rendering to JPEG. Ah, yes, there's on-camera noise reduction.

2. The JPEG algorithm itself is reducing aberrant pixels. It is after all, a curve approximation for the changes of colour in an area. Hence "jaggies" on sharp edges.

I wager with the right combination of noise reduction and JPEG you can achieve the same results.

[David Bostedo]

"It would have NO LCD screen!"

Jamie Pillers - I'm not sure why the in-camera technique would have anything to do with whether or not you need an LCD screen. I use my screen to review exposure and composition. The JAW file concept (nice name by the way) would in no way affect my use of the screen.

Sounds to me like you want an old film camera, with a CF card instead of film. I don't think that's likely to be very popular, given the attachment (and for good reasons most times) people have to their LCDs.

[MJFerron]

Jamie says,

"Here's the concept. I'm waiting for a digital equivalent of, say, the Nikon compact film SLRs like the FM or FE series (or Olympus OM models). This digital camera would have a few simple control knobs to mimic the SLR's film speed, exposure +/-, and shutter cocking. And of course, an excellent optical viewfinder. It would have NO LCD screen!"

Yes and I don't get why Nikon doesn't build such a beast. And who needs a screen on a camera like that? I know it would sell big.

[ctein]

Dear folks,

Wow, a metric ton of semi-overlapping questions. I'm not going to try to address most of them individually. Shotgun response follows:

1) If you haven't read or don't remember the article I previously wrote on the Olympus noise problem, linked to in this article, please reread it before commenting. Some of the comments indicate unfamiliarity with what I've already written.

2) I own Neat Image, Noise Ninja, DFine, and Noiseware. All of which do a substantially better job of noise reduction than ACR, at least in my hands. I never depend on ACR for noise reduction. (For this particular problem, ACR's noise reduction capabilities are utterly inadequate. It barely makes a dent.)

I wouldn't call myself a true noise-reduction virtuoso, but I can make those four plug-ins jump through some pretty interesting hoops, and I do so on a routine basis. I can't get any of the five aforementioned programs to deal satisfactorily with the plaid chroma noise in high ISO Olympus Pen RAW photographs.

3) Success has nothing to do with the file format being a JPEG or with "oversampling". There's nothing in JPEG compression that magically cleans up such photographs, and there's no oversampling going on in this case. The magic comes from the camera throwing a whole bunch of built-in noise-reduction smarts at cleaning up the RAW file before creating a JPEG. The Olympus Master on-computer software can do almost as good a job and it yields a TIFF.

4) I'm not actually manipulating the RAW file, I'm working on the converted-from-RAW file. So far as I know, there is no way to work directly on RAW files in Photoshop. (Not that I'd want to.)

5) ** Max, no, your approach doesn't even come close to addressing this problem.

** Janne, ya gotta be effin' kiddin'! Why do I want to go through all that?!

** Amin, no, because I don't really have the time nor interest in looking at other people's experiments, and I know they will not be able to resist sending them to me.

** Andrew, yes, ACR applies the lens corrections to RAW; you are correct that this wouldn't work if it didn't.

** Mark, yes, that's Joan Baez.

6) This is entirely a side issue to the point of the column, but in-camera JPEG's are not a substitute for RAW, no matter how well done the JPEG. JPEG's have substantially less exposure range and substantially smaller color gamut. They also give you only 8 bit data to work with instead of 16. Discussed at length and answered in a previous column, referred to in the very first paragraph.

Whew! Did I miss anything important?

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[nugat]

Have you tried LR3 ? Can you compare LR3 with your method?

[Player]

Ken Rockwell's been right all along [incredulous giggle].

[XebastYan]

....ehm yes, how did you set the settings in camera menu for Gradation and noise reduction on your PEN?

thanX
XebastYan.

[Roberto C.]

Very nice pragmatic approach! What I find confusing (on what Olympus does, not you) is:
- Their own "Olympus Master" software is slightly worse that the in-body processor. Really strange, they should know what their DSP does, and that should be reproducible bit by bit. Off-line you have more computing energy so if anything you should get more refined results. Unless the in-body data flow to the jpeg processor is different from the data flow to the raw, and they are loosing something there. Possible but frankly unlikely.
- Companies should know that most (all?) people using raw will be using photoshop, aperture, lightroom. Why do they stick to they own image processing software (which they give out for free?). Why not to provide plugins for the mainstream photoimaging software (which they may even sell for a moderate amount of money)?
Well, nothing Ctein can do about that, I know...
R.

[John MacKechnie]

I must say that if I had stumbled across this post and comments 43 years ago, instead of leafing through a copy of Family of Man, I probably would not have spent the next 43 years obsessing about photography. My ignorance is bliss.

ctein, you are a force of nature. Thank you for your weekly contributions to TOP.

I just scored a Kearserge timer with a footswitch on Ebay. Oh....nevermind.

[Andy K]

Sounds like the two layers on Verichrome Pan :)

[Mike Johnston]

"I just scored a Kearsarge timer with a footswitch on Ebay."

I'm jealous.

Mike

[Alf]

Olympus released Viewer 2 a few weeks ago.
It is like Master, only better and free.
It has the features Studio had, better results, and much faster on multicore CPUs.

[richard]

Would it be interesting if Ctein could post an original RAW+Jpeg somewhere and we could all chip in and have a "noise reduction " competition?

(I can't figure out how I ever managed just pushing Tri-x to 800)

[David Dyer-Bennet]

I don't believe in the market for a digital camera (for photographers, professional or amateur; as opposed to specialized surveillance or scientific or whatever instruments) without an LCD screen. Giving that up loses you a good chunk of the benefit of going digital.

But if you really don't want those benefits, then DON'T CHIMP! You could even tape over the screen, once you got the basic configuration set.

Does anybody really believe that such a camera would sell, oh, more than 1% of the units the D700 has sold?

[Janne]

"Janne, ya gotta be effin' kiddin'! Why do I want to go through all that?!"

It's about three mouse clicks and a simple levels adjustment. But whatever makes us happy of course.

[Michael Barkowski]

"JPEG's have substantially less exposure range and substantially smaller color gamut."

I wonder to what extent it is theoretically possible, with today's technology, to work around these limitations in the camera.

Would it make sense for cameras to start having knobs for colour, dynamic range, and noise reduction?

[Will Frostmill]

Ah-ha!
This is relevant to my interests! Thanks Ctein, this will be exceptionally useful to me. I had been diligently saving my RAW files in addition to my jpegs for a long time, and wondering if I should bother, since the jpegs were more than good enough for my purposes most of the time. But this - this is awesome! I now have a whole slew of marginal pictures that will be usable!

Question: on my E-520, you can re-render a RAW file to jpeg. Would there be a benefit to doing so multiple times with different settings and then compositing in Photoshop?

Will

[Dave Polaschek]

Ctein, as Semilog pointed out, the in-camera calibration can do the equivalent of dark-frame subtraction to help tame the noise.

I haven't tried it recently, because de-noising software keeps getting better, but when I was first shooting long exposures with my Nikon D-50 back in 2006 or so, I would routinely shoot a dark-frame of the same duration and ISO as the long exposure, and then subtract it out. Made a big difference.

It would be interesting to try doing the same with the raw images from the E-P1. I suspect the internal software is doing more than that, but seeing the results might point a guy to other techniques that might be useful.

[Roberto C.]

Very interesting the featured comment by Semilog. But, the fact that the Olympus Master software can do an "almost as good job" seems to point more to better knowledge of the sensor, than to sample-specific sensor calibration done by the on-board firmware (which, I agree, should really go upstream the raw file generation as Semilog advocates).
R.

[robert e]

Ingenious! And very impressive examples. Thanks, Ctein.

This may be too stupidly obvious to say out loud, but this method is ultimately limited by the quality of the in-camera conversion, right? I ask because I've read more than a few reviewers and owners of recent Olympus cameras praising their extraordinarily good jpeg quality.

And thanks to Semilog and Andrew Molitor for some very illuminating insights.

[Greg Wostrel]

Ctein, you can use the RAW directly in Photoshop by opening in ACR, make adjustments desired and then click open as smart object. Then it is embedded in a PSD file. You can access the image through ACR at any point after that by double clicking the smart object icon on the layer. Pretty slick. Makes the file pretty big, too.

[José Martínez]

I have an EP1 also, and shot in raw+jpg almost always. How do you manage to develop the raw and get the exact same geometry found in the out of camera jpg.
I have calibrated my lens with both the jpg out from camera end from raw development with Oly's studio... the lens appears to be different as it should be due to the by-chip lens corrections.

[Paul logins]

I have a much simpler way of doing this. Trade in the E-p1 for
a Nikon D3s. How would this work on a picture taken at say
ISO 25k or 50k with a D3s?

[Dave C]

Ctein,

Following on from Semilog's comment, I wonder how much of the dark current 'image' would be in a JPEG of the back of the lens cap? Is there enough info in it to achieve the effect with one common JPEG?

If the Olympus SW can achieve similar with the RAW, the info is in there somewhere. Maybe dcraw plus some scripts could pull it out. ROI? Enough?

Very interesting technique anyway.

Dave

[ctein]

Dear Janne,

Please e-mail me off-line (ctein@pobox.com); this magazine shouldn't get cluttered up with technical discussions. Since I've already tried a whole bunch of noise reduction tools in a whole bunch of different modes, I don't know where your magic software is coming from. Explain it to me in private. Thanks!

~~~~~~~~

Dear Roberto,

Not strange that the in-camera and on-computer software don't produce exactly the same results. It surprised me how close they were. Remember that the camera has access to ALL the data; the on-computer software only has access to what has been put in the RAW file. As semilog has explained, those are not the same thing. Also, the on-computer software has very few controls and adjustments; the in-camera system could be fine-tuning all sorts of parameters for noise reduction based on the camera settings. Heavy noise reduction that doesn't badly compromise real image detail is a very, very complicated business.

pax / Ctein
==========================================
-- Ctein's Online Gallery http://ctein.com
-- Digital Restorations http://photo-repair.com
==========================================

[ctein]

Dear Alf,

I just downloaded Olympus Viewer 2, and while it is an overall more capable program (somewhere between Adobe Bridge and Lightroom in abilities, which makes it quite a bargain at $0!) it cannot get rid of the RAW noise well at all. Even with both noise settings set to maximum, the results are entirely inadequate. And much worse than what Master can do. Don't ask me why.

So, if you're trying to play with my trick and you don't have RAW + JPEG files to work with, Olympus Master is the way you have to go.

pax / Ctein

[John Krumm]

I second the recommendation for the new Olympus Viewer software, though it's still slow and clunky. It allows you to switch back and forth from camera mode to edit mode before saving. I've been pretty happy with the results from my 620 using it (with the 620 the results seem a bit better than in camera, but it has an older jpeg engine than the EP1).
Interesting to see your results, and love that last shot.

[Don Bryant]

Canon provides software for processing RAW files that will produce equivalent files to JPGs processed in the canera. The rendered file can then be saved as a TIF. Canon's Digital Photo Professional also supports all of the picture styles available from in camera processing. So it's no surprise that the EP-1 can spit out a JPQ with less noise than 3rd party software. I would have speculated that Olympus supplied something similar to Canon's DPP.

90 ways to skin the digital cat. Nice work flow though. I'm also betting PBASE's Capture One Pro 5 would do an outstanding job. In fact it would be worthwhile for someone to test all of the major RAW processing software.

[ctein]

Dear Michael,

I think the issue may be less technological than aesthetic. People basically like snappy, saturated photographs. JPEG's are supposed to be ready for use, out of camera, without any additional manipulation.

The situation can be improved to some extent, but since a RAW file conversion that contains the full range of the data looks inherently flat and undersaturated, I don't think we'll ever reach a point where JPEG conveys the same range as a RAW file.

~~~~~~~~~

Dear Will,

Multiple conversions might *possibly* be useful. The level and kind of noise varies greatly with the luminance level. The reason that NoiseWare is my preferred noise reduction tool for massaging the RAW image in the luminosity layer is that I can adjust its parameters to maximize noise reduction in the shadow, minimize it in the highlights, and tweak settings for maximum retention of real image detail. With less flexible tools, one could achieve the same ends by doing various conversions with different settings in different layers and then masking them, as you suggest.

~~~~~~~~~

Dear Greg,

Well that's cool. I didn't know about that. I haven't done much yet with smart objects. I'm going to have to master them sometime in the next two years (before I start writing the next edition of DIGITAL RESTORATION).

~~~~~~~~~

Dear José,

ACR in CS4 makes use of the geometry correction information, so my JPEG's and RAW files align just fine. If you're using a RAW converter that doesn't make use of such information in combination with a camera that does, yup, my method isn't going to work.

~~~~~~~~

Dear Paul,

I agree that's an extremely simple solution. So simple, in fact, that I'm sure you'll do lead to very small favors to make it possible. First, send me the difference in price between the Olympus Pen and the Nikon D3s, so I can buy one of the latter. Second, shrink the size of the Nikon by a factor of two, so it will fit in my shoulder bag in place of the Olympus.

See, small favors!

More seriously, DDB and I have been curious to know how this trick might work with his D700, which has awfully good high ISO performance.

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[ctein]

Dear XebastYan, Dave C. and Dave P.,

You fellas got me looking closely at the settings at the JPEG settings that I used and I found out an interesting thing.

First the settings I was using: gradation was set to "auto" which maximizes the exposure range presented, color space was set to Adobe RGB, noise filtering was standard, and noise reduction was on "auto," which means it doesn't kick in until the exposure time is more than around 5 seconds (a highly unlikely exposure in combination with ISO 6400).

I got curious what different combinations of the settings would do, so I tried noise filtering set to off, standard, and high, combined with noise reduction set to always off and always on -- six combinations in total, with an exposure time of about 1/60 of a second. Noise filtering set to standard did the best job. High was overkill; it did not further reduce chroma noise but it did smear detail out much more. It would be useful if the intent was to use the JPEG as is, but since it's primarily contributing chroma data to the final result, not a win.

Turning on noise reduction produced a result, in all cases, that seemed counterintuitive but it wasn't after a little thought. In every combination of settings, in both the RAW and JPEG photographs, noise reduction produced WORSE banding and plaid chroma noise.

The reason for that is because the high ISO chroma noise is not inherent in the sensor but is noise injected into the signal processing train, as I mentioned in my previous column on the Olympus Pen noise problem. Hence, it varies randomly from photograph to photograph. In which case, trying to subtract out the noise will actually increase the magnitude of the noise by an average factor of 1.4 X.

Possibly with long exposures, noise reduction would still be a win, but I can't come up with a circumstance where I would be combining that with a high ISO. Once I'm in the 5+ second range, I'm on a tripod, so I have no reason not to turn the ISO down to something sane like ISO 800 and just do a longer exposure.

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Sven W]

Personally, I put up with the image noise from my Canon point-and-shoot (even at ISO 800). But I do have a mild interest in this topic ... I think it's worth reading the experience of Andres Manessinger, on his blog. He's a very good photographer and is heavily into postprocessing as well.

http://manessinger.com/2010/05/1303-pink-noise-waltz-i.html

(There are 3 recent blog entries on de-noising.)

As an aside, I suspect Adobe could put out a crackerjack de-noising function, but are happy to let some smaller players add value to their Photoshop product.

[John Krumm]

Surprising you used auto-gradation in your shots. Over in the Oly dpr forums that seems to be the number one reason for new users to say their camera is producing too much noise in the shadows (and will get a pile of responses recommending only to use it at iso 100 if at all). Auto-gradation at iso 6400 is unheard of, but then not many people use that high of an iso anyway (and the EP-1 and 2 are the only Oly cameras that offer it so far). You are in uncharted territory! Keep experimenting--very interesting.

[Semilog]

For those of you who are interested, there's an absolutely wonderful description of how CMOS sensors work at Molecular Expressions, the superlative online textbook on optical micrsoscopy. Almost everything in the article is just as applicable to a DSLR as to a microscope.

[Semilog]

@ Ctein: "It surprised me how close they were. Remember that the camera has access to ALL the data; the on-computer software only has access to what has been put in the RAW file."

That is, of course, a strong indication that there is at least some DSP juju invoked on chip or at least in-camera, upstream of the so-called RAW file. Olympus would hardly be unique in doing this.

[XebastYan]

Hi again,

like John Krumm i am also a bit surprised that your gradation was set to auto. I also figured out that the auto setting increases the noise! I only use the auto setting when i set the iso manually to 100 or 160. If i go higher i turn it to normal (off).
Do you think the color space has effect on noise?
Thanx for Your Experiments!
Nicely
XebastYan.

[ctein]

Dear John,

Consider the context. As pointed out in my previous column, I never routinely photograph JPEG's, so the gradation setting is irrelevant, as it has no effect on RAW files. As for the JPEG's, what I'm interested in is the chroma information, and blocked up highlights or shadows have zero chroma information. The "auto" gradation setting produces substantially more highlight and shadow detail in the JPEG.

I don't know what effect "auto" has on noise levels in JPEG's photographed and used more normally (and I don't care) so you shouldn't take anything in this article as having any bearing on what photographers are discussing elsewhere about JPEG quality.

~ pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com
-- Digital Restorations http://photo-repair.com
======================================

[John Krumm]

CTein, it's been a while since I've tried the auto-gradation setting, but it does do one thing to raw files. It sets the exposure a little lower, so the raw files appear darker than the jpegs if you use something other than Olympus software to process the raws. At least it did two years ago... : )

[steven crichton]

Just tried this with my GF1 @ 3200 and I love the effect it gives, looks like a nice sharp pushed print film and much better on both the JPEG or a straight Raw conversion with adobe or aperture.

All we need now is some nice firmware updates to unleash this power with cameras. Sony is ahead of the game just now with the panorama drag and the exposure blending of their micro 4/3rds style cameras ... I'll cross my fingers and hope technology will march a bit quicker a bit sooner in the panasonic labs.

Steven

[Stephen Faust]

------------------------
Posted by: Paul logins;
I have a much simpler way of doing this. Trade in the E-p1 for a Nikon D3s. How would this work on a picture taken at say ISO 25k or 50k with a D3s?
-------------------------

You have to consider why one would choose the E-P1 or E-P2 in the first place :) Small, compact, decent image quality, etc. I could shoot my 1DsMkIII and get better results, but I couldn't fit the body alone in the bag I use to carry the E-P2, 35mm equivs for 40mm 1.7, 100mm 1.5, 150mm 2.5, and the kit 28-84.

I love the compactness of the E-P2 and what I can get from it when I don't want to carry the big guns. Thanks for taking the time to help me stretch the IQ at high ISO's with this camera!