PSDs and Permanence

This week's column by Ctein

One of the complaints I've seen raised about Adobe's new subscription-based license for Photoshop is that if you fail to keep up your subscription, years of your work will become unavailable to you. Which brings to my mind two questions:

1) Are you completely out of your friggin' mind?

2) If not, what unmitigated fool ever managed to convince you that Photoshop Document (.PSD, hereinafter rendered without the "dot") files were an appropriate way to be archiving your work?!

PSD's should not be considered an appropriate format for archiving. Especially not layered PSDs. Especially, especially not layered PSDs that include active elements like un-rendered type or smart filters or objects. No way, no how.

There is no guarantee that previous versions of Photoshop can open your current work product correctly. Equally, there is no guarantee that future versions will be able to open it correctly, or at least in exactly the same form it is now. Filter algorithms change. Type rendering changes. Just to give two examples. These complex structures you're building right now? They're intermediate work product, highly dependent upon the platform you're interpreting them with. It is nice and convenient when they endure; God knows, I take advantage of that. But they are not an archival format. There is no guarantee you will be able to read them in the future in the same way.

It's not just about a subscription-based license. What happens when Adobe goes out of business? It will. Very few companies and product lines of any sort make it to the age of 50. Only a minuscule number will make it to 100. None of them are immortal. Some of you reading this will be around and still making photographs when Adobe disappears.

Personally, I'm hoping it doesn't happen in my lifetime, and not 'cause I'm hoping for short life. But it will happen.

If you're treating these fragile and temporal constructs as your long-term archives, you're just asking for trouble.

There's an easy way out of this. So easy, in fact, that no one can be excused for not implementing it. It doesn't take terribly long or very much money, even for a very large base of PSD files. Which means that even if you don't do it now, like you should, you can do it when the time comes that you no longer feel able or willing to support a Photoshop subscription.

Here's the scheme:

Grab yourself a spare coupla-terabyte hard drive. Don't have one at hand? Go spend $100. Seriously, this is your "life's work" you're talking about; if it ain't worth that much to you, you've got nothing to whine about. Besides, you know you'll just need the new space in the next year, anyway. Admit it. Nature abhors an empty hard drive

Go through your existing inventory of work and copy anything with a .PSD extension to that drive. Specific details depend on exactly what directory structures and naming conventions you use for your work. You can figure it out; it is definitely not rocket science. If this takes you more than a day, no matter how large and complex your inventory is, I'll be amazed. Most of your time will be spent waiting for hundreds of gigabytes of data to get copied over.

Open up one of the files in Photoshop and create an action that does the following—first it flattens the file and second it saves it as an uncompressed TIFF. Yes, uncompressed. Less fragile, nearly universally readable and it will be for the rest of our mortal lifetimes, barring the complete collapse of technological civilization. (In which case, honestly, the disposition of your photographs is not going to be your biggest concern.)

Now tell Photoshop to batch-process all the files on your new hard drive using that action, and let 'er rip.

Come back an indeterminate amount of time later, and now you have rendered, universally (almost) readable versions of your work in its most current final state. Archive those, according to your taste.

That's it.

Oh, not quite. All of you complaining about being forced onto the upgrade treadmill so that you can read the RAW files from your newest camera? Adobe still distributes its free DNG Converter, which is kept as current as Adobe can, so you don't need the latest and greatest Lightroom or Photoshop to be able to access the files from your latest and greatest camera.

Now I'm done.

©2013 by Ctein, all rights reserved

Featured Comments from:

Jack: "According to John Nack, Adobe will be offering a way to use files once you've stopped paying the monthly tax. I'll believe it when I see it."

Mitch: "One copy on one hard drive isn't enough for archiving. Hard drives always eventually fail, so you should archive to at least two drives. Ideally, a) buy the drives at different times from different manufactures so the drives and components within aren't from any of the same production runs. This will reduce the likelihood of both drives failing around the same time; b) store the drives in different places, preferably in different geographic areas. If a major natural disaster or fire destroys your home, having a second drive at a friend or family member's home in another city/state means you're much less likely to lose work."

Ed Hawco: "Well, that seems very sensible. It doesn't fully apply to me, as I don't use Photoshop. I do everything in Lightroom, where I do all my adjustments on 'virtual copies,' which are essentially another locked-into-Adobe format. (In fact, they're not even a format proper; they're just descriptions of adjustments that are applied to previews and export files).

"My big worry is that I'll reach a point where I just cannot live with Adobe anymore (I'm not far from that now). The solution is obvious: Export all of those virtual copies (or at least the important ones) to TIFF.

"I think your idea makes some people uncomfortable (wrongly) because of its apparent finality. For those of us who do this more casually than professionally, we don't necessarily want to fully 'commit' our edits. We like to think we can always go back and tweak a bit more for the next print or export. We're not always convinced we got it right, so we don't want to commit.

"It's a completely bogus worry, of course. Just because you commit to a TIFF doesn't mean you can't still tweak the virtual copy/PSD, at least until such time as the plug is pulled on the format. But we're talking about fear of commitment, not practical methods.

"After reading this I'm going to make it a project to go through my Lightroom library and commit all of the images I've tweaked as virtual copies. The nice thing about Lightroom is that I can make those committed copies part of the library, grouped with the 'originals' so it will be easy to keep track of them.

"Thanks for the kick in the pants!"

Comments

[Richard Tugwell]

And if you make any modifications to your images? Save them again as flattened TIFFs? Overwrite original, re-test viability of backup etc etc

Rightly or wrongly, for purely photographic purposes I'm putting my faith in the DNG format (which is a fancy TIFF after all). It's years since I felt the need to use the PSD format for simple photographs. All raw files are converted to DNG and managed in Lightroom. For images where I do need to use PS (for graphic production, montage etc), then they are saved as TIFFs and are not very numerous compared to straight photos.

[Barbu]

I swear I could hear crickets... for about two microseconds. It takes a bit of guts to argument against the opinion of a well-respected person like Ctein, but somebody has to say it: point missed.

It's not like we don't archive our RAWs or (quasi-)completed works. It's about being locked out of *further* modifying previous work, with a program that we are required to "pay as you go" for using it. While big businesses are used to pay like that (but even then, usually for maintenance, not for simply using a proggie), individual users are upset by a mafia-style offer: take it or...
Anyway, I digress; once again, it's not about using .PSD for archival purposes
Side bit: I've seen discussions about still having access to a kind of read-only experience for those that paid for CC, but drop it. Still not enough; if we don't want to pay more, we still want to have full access to a program, and the usual style was to be locked out only of updates; so, if we don't pay more, we only have access to a fixed version, the current one at the time we last complied with the Adobe tax.
Combine that with the already jacked-up prices in Europe (and other non-USA places), and you'll see why not many would welcome any opinion that tries to explain Adobe's actions. We still have a tremendous respect for Ctein, but the present column is just a sidetrack, a column that many will try to forget as soon as possible. The only good that it does: reminds people to back up in a responsible way.

[Robert Roaldi]

This a sort off OT comment that concerns the going-out-of-business of companies that we think are too big to fail.

What's the world's backup to Google? Sooner or later, they're going to hire a wacko to run the place who'll bankrupt them. It's a matter of time. Or something better will come along, or ad revenue will dry up, like it did with television. What then?

Same with Adobe, maybe even Apple. When car companies and airlines went belly-up, we switched to other vendors, there's a semblance of competition in those industries, maybe more than a semblance. But it seems like this interweb thing, which was supposed to be provide easy entry for small nimble companies, more competition even, is turning out to be the home of really BIG CORP, world-wide monopolies, or nearly. What happens when they fizzle out?

I assume there are people thinking about these things.

[Michael Matthews]

Hear,hear! At last, a voice of common sense.

[Jack]

I've had to completely recreate composited photos with text and drop shadows and who knows what else because someone else had archived a flat tiff and we the client needed the text to be Spanish and not English (three years down the road) and I can tell you that there are times that things should be archived as psd. Or at least psd and tiff.
Also, remember that this is not just PSD but all of the CS programs, some of which don't have a tiff equivalent. Lots of freelancers make templates to save time and templates don't always work in generic files.

[rnewman]

Very good idea. Then all you need worry about is how long the terabyte drive will be readable. 8 inch floppys anyone?

[kirk tuck]

Bravo. A nice gust of common sense and practicality. I thought I was missing something when I read people bitching about never being able to open up their files again. Now I understand. It was just people being obtuse.

[Glenn C. Riffey]

Thanks for this great tip. Never gave much thought to the life of all of those .psd files that I have; until now.

Guess what I'll be doing, on and off, for the rest of the week. A simple suggestion resulting in a lifetime saved.

Thanks...

[RayC]

While I agree in spirit I think a common workflow that is missed here is the lightroom workflow. Probably even more of a transitory process than psd files as almost each new iteration of Lightroom seems to want to re-render your raw files, and then who knows how the other actions are performed in the database...

So as two additions to Cteins archiving strategy:
1) Do a search on all your 5 star images in Lightroom and export them as full Tiffs... to that new hard drive.
2) Clone that hard drive to another new drive (yup now you spent $200.00 to preserve your life's work). If you are really paranoid you could then store this drive in a different location but at least having two drives gives some coverage over drive failure or accidental erasure.

[Tony Roberts]

Terrific piece if only to puncture the balloon of hyperventilation we've been experiencing. For now, a 2T external drive with all my "good" work is safely stored in my Safety Deposit box down at the bank. When it needs to be updated to some other amazing storage device, that will be done. Meanwhile, Drobo takes care of me day-to-day.

[hugh crawford]

Tiff files are certainly anything but universal unless by universal you mean that anything you want can be wrapped up in tiff headers.

http://en.m.wikipedia.org/wiki/Tagged_Image_File_Format

TIFF stands for Tagged Image File Format but anyone who has had to write software to deal with them knows it realy means "Thousands of Incompatible File Formats" TIFF is realy more like a format for creating file formats. It is almost as misunderstood as DNG files many people think are some sort of standard data format.

Even photoshop will happily write tiff fils that the very same version of photoshop can't open thanks to confusion over signed integers.

[Albin]

I got curious about this a while ago and went to Smithsonian to see what format they use, figuring they're planning to keep their collection around for a while: sure enough, TIFF. RAW certainly is a problem but not sure DNG has the buy-in from the professional community as the standard it probably deserves to be.

[Marshall Smith]

" Adobe still distributes its free DNG Converter, which is kept as current as Adobe can, so you don't need the latest and greatest Lightroom or Photoshop to be able to access the files from your latest and greatest camera."

Yes, true, but I had to buy a new computer to access the latest version of the DNG Converter that would access the files from my latest camera.

If the treadmill doesn't get you in one way, it'll get you in another!

[Ian]

is there anything proprietary about psd files prohibiting third parties opening them ? If Adobe disappeared I am sure third party options for opening them would appear.

[Ben Syverson]

I save all of my large format scans as PSD—the original as the background, with any adjustment layers, masks, etc on top.

While the Photoshop file format is not what I would call "archival," it is publicly documented (http://www.adobe.com/devnet-apps/photoshop/fileformatashtml/). The core parts of PSD that relate to image layers, layer masks and adjustment layers (ie, the important stuff for photographers) haven't changed since 4.0 (1996).

The PSD format is like HTML—Adobe keeps adding features, but they're mostly optional. If you stick to the old features, you get foreward and backward compatibility. You can create a layered Photoshop file in CS6 and open it in 4.0, and vice versa. Here's the same PSD open in both versions:
http://bensyverson.com/images/2013/5/photoshop-4-cs6.jpg

[Gato]

Thanks. Somebody needed to say this.

It has been 2 or 3 years now since I did a really thorough cull and archive. I like to save a finished but unsharpened file as a best-quality JPEG - some slight loss, maybe, but IMO even more universal than TIF.

[RW]

While I am a longtime Photoshop user, I'm also a big open source advocate. This is a classic example of us users letting one company gain too much power. Open tools and file formats are so important in the digital age, so much so that photographers should be willing to give up some features for the freedom of "free" I'm sure crack is great, until you have to quit!

[Abraham]

Just playing devil's advocate here... But accessing PSDs isn't about accessing the finished products (the flattened tiff files) but about retaining access to the working file. This is the file that you have spent hours tweaking, adjusting, reworking and don't want to do it again.

However as a Designer this may be more important to me than a Photographer.

[Jed Wormhoudt]

No question that archiving in Tagged Image File Format is preferable to .PSD - But why start by flattening the file? Tiffs handle layers just fine for me. File size/storage capacity? As you correctly point out storage is pretty cheap these days.
Regards,
Jed

[John Robinson]

Good points all......but....there is already a free PSD viewer out there. Even if/when Adobe goes under, the OpenSource community will likely make sure the format exists and can be read/converted forever....

It may not be an optimal conversion, but I'd be willing to bet that if Adobe went under tomorrow, 99% of all PSD files would still be accessible long past that.....

And someone, somewhere, will likely acquire the IP and continue to market some form of the Photoshop...unless Google starts making it available as a free web service.... :)

[Severian]

Nice article! I always appreciate it when somebody clearly articulates the more subtle risks of our technological world and Ctein has done it well.

The way I think about this is that modern technology has done something very bifurcated to life's artifacts (pictures, documents - anything that might have previously only been on paper).

For the very careful, the preservation of important artifacts has never been better. You can trivially have multiple copies that could survive simultaneous disasters in two states. They won't fade, they won't burn up, they won't get water damaged, lost or blown away in a tornado, like a shoe box of pictures or a couple of albums can.

For the careless, the preservation of important artifacts has never been fraught with so much danger. You have all of your eggs in one electronic basket - and on average, many people aren't taking very good care of their basket.

Shoe boxes of photos don't become incompatible, they don't (usually) get accidentally deleted, you always have the program (fingers and eyes) to access them.

In the past, against the onslaught of time and chance, the difference in ultimate outcome (50+ years) of the very careful (multiple copies of prints, careful storage of negatives, acid free albums, etc) and the careless (a couple of shoe boxes) was not so wildly different.

Now, they can be and often will be very different.

BTW: This extends further than just pictures - recent personal finance articles were describing people that had done all the right things with wills and other end of life matters - but surviving spouses had no idea what ID or password to use for some of the online accounts. Same problem - different domain. Technology gives you huge leverage to do these things well, but also, all the rope you need to hang yourself.

[John Camp]

Part of the panic involving Adobe actually involves a lack of technical knowledge by people interested in photography. It's easy for you (Ctein) to write this, because you're a tech-head. When you say, "Grab yourself a spare coupla-terabyte hard drives," you've already lost me. I have a long list of questions just about the hard drive part, because I know *nothing* about this stuff. (Obviously, I can go down to Best Buy and get a hard drive -- or do I need more than one? And if so, how do I hook them together? Or do I treat each one separately, and have to do each backup process twice? -- but that's where it ends: with paying the money.) Do I need to worry about compatibility with the drives? Do I have to worry about brands, in terms of reliability? Should I back up each drive? How do I do that? And we haven't even gotten to the the whole DAM problem yet. So I dump a couple of terabytes of photos on my drives. How do I find the one I want? Do you have a DAM program for me? If not, how do I find out which one is best? Which one is the optimum choice considering price, performance, reliability and ease of use? Are there tutorials that teach me how to use them? How do you find the optimum one when you throw tutorials into the mix?

The fact is, Photoshop offered a seamless solution at a set price. I can set up my Mac and run it through a Time Capsule (which more or less sets itself up, and then gives me automated backup) and have one piece of software with which I'm familiar that not only allows me to modify my photos but also to organize them. What you're talking about requires me to spend time learning a bunch of stuff that I don't want to learn, because (a) I'm not interested in it, and (b) I'm really, really busy earning a living and any spare time I'd rather spend doing things I enjoy.

I know all this stuff can be done. You (Ctein) can do it without much problem. I can't. I'm pissed at Adobe because I had a good solution, and that's being changed in a way that I can't control.

I could just go ahead and pay Adobe's subscription price, because the fact is, the fee doesn't really have much impact on me. (I pay twice as much for a newspaper subscription.) But I object to paying a tax (which is what it amounts to) for using a program when I suspect that one reason for the tax in the first place is that Adobe is no longer able to deliver content that I'd be willing to buy separately. In other words, they can't convince me to buy their program on its merits, so they'll tax me to use it, knowing that they're forcing me to choose between their tax, on one hand, and great inconvenience (for me, not for you) on the other. I mean, at least the newspaper delivers new content every day...

Your answer to my plaint here would be along the lines of, "bite the bullet, learn what you need to live in the modern world." Well, I did. I spent quite a b it of time with Photoshop, and even took a Photoshop course at the Santa Fe Workshops to learn basic compositing techniques. Now, an arbitrary change makes all *that* work obsolete...

There's a lot more to this than the problem of flattening photos. It's that most people had a solution on which they relied, and to which they devoted a great deal of time to learn, and that's being changed in a way that creates a variety of practical or conceptual problems.

[Steven House]

Were you shouting? :)

[Grant]

The sanest rant that I've heard, in a long time. Thank you.

[MarkR]

The problem of course is that for most people psds and dng conversion aren't the real problem.

The real problem is that Adobe is changing their business model and that people don't like change. (see also: www.xkcd.com/1172.) Any attempt to reasonably overcome objections will continue to be met with new and even more outlandish objections until such time as Godwin's law gets invoked.

[Ed]

Complete Agreement (see that's why Ctein should live forever, his great advise), Gimp has made it's own formet .XCF the standard saveformat and all the rest is export (BAD, BAD, STUPID CHOICE). So I only export from 16 bit .tif to 8 bit uncompressed .tif (and I keep the 16 bit tiff and the RAW). XCF and PSD are "in between" save formats at best. When I go to the Loo (a place near Apeldoorn were an ant of our new king lives) and come back and wanna continue working on a file I save in XCF (or PSD if I use P-shop Elements at my dad's laptop). For the rest NEVER (in letters NEVER).

Greets, Ed.

[Michael Kay]

Mac users can open psd files with Apple's Preview application, don't Microsoft have something similar?

I take your points though. Opening old RAW files will be a greater problem as you indicate, I'm sure!

[RobinP]

Last things first - yes the .dng converter is free but I still had to upgrade to Photoshop CS5 for the raw converter to include the lens corrections option. It's been worth converting my raw files to .dng simply because they're half the size of the original .srw
Never heard of .srw ?? That's how I felt when you mentioned .psd , glad you reminded me that it's one of those proprietary formats I was lucky enough to avoid way back when these damned computers started taking over our lives.

[Dennis Fallon]

If people are actively working with PSDs and Adobe shuts down, then there's bound to be other software that will let users continue to work with PSDs. (And likely the writing will be on the wall earlier suggesting that migrating files might be advisable).
And if anyone wishes to switch at some point or stop paying or whatever, then I imagine that they can just pay for one additional month and during that month, migrate all their files (save to TIFF or whatever).
I imagine that serious users will just commit to a life long subscription. Those of us planning to just keeping using our perpetual license software are the ones who will have to plan for an eventual change, when our software no longer works on our new computer/OS or whatever.
Personally, I'm more concerned about the proprietary database I'm building up in Lightroom, whether based on original raw files (in my case) or DNG. If I move off Lightroom, I can always export everything to TIFF, but if I want to keep my non-destructive raw updates and the ability to go back in and tweak them, will any other software be able to understand the database ? (If I use sidecar XMPs will any other software understand those ?)

[Jim Roelofs]

Ahem, Ctein, as photographs go, you're right. Tiff'em!

But then there's my "other" life, as an illustrator. I create classic motor cars in PS. These files contain hundreds if not thousands of layers that cover such things as body colour etc. A client might order a particular drawing and specify a different colour, personal number plate, different styled wheels.

Therein lies my need to retain the integrity of each layer for future print orders. there's probably thousands of other PS users out there with similar issues.

Just sayin'.

[Ben Syverson]

To those who are wondering about "Layered TIFF": these files are simply Photoshop files that are wrapped in a TIFF container. TIFFs can contain nearly anything—EXIF is a kind of TIFF.

A PSD normally contains a flattened version of the image (this is the "Maximize Compatibility" checkbox). When Maximize Compatibility is on, apps like Preview can open the file. So in that way, a PSD is the best of both worlds, as it gives you the equivalent of a flattened TIFF and a fully editable layered image in one file.

[Jamie Pillers]

Ctein, as usual, thanks for the thoughtful article.

I have a thought, or maybe its really a question. I use Aperture. As you know, Aperture keeps the original RAW file, as well as the changes subsequently made. I'm the kind of guy that doesn't mind at all if the changes I made to a RAW file were lost, so long as I have the RAW file to start with again. After all, my tastes change over time and processing software changes over time as well.

So, for long-term archival storage, I was thinking that storing the RAW files would be enough. What do you think about this strategy?
Cheers from across the Bay,
Jamie

[Ed Richards]

As someone who had to write converters to move files from dead wordprocessors to live ones, I appreciate the problem. But as others have said TIFF is not standard, and DNG is both Adobe and non-standard. The key is that while Adobe will ultimately go away, it will not do it overnight. By the time it does, there will be new formats to switch to. Convert your files then, when you know where they need to go and what programs you need to use on them. Think of it as a long term process, not a final archive format.

[Ben]

Don't forget the Digital Obsolescence!
http://en.wikipedia.org/wiki/Digital_obsolescence

Someone should develop some small physical medium for archival; about 36x24mm should do it.

[Daniel S.]

With all due respect, I think it's more cost-efficient to use LZW compression then attach a plain ASCII file of an implementation in C alongside it, in case civilization collapses (and one image file in both compressed and uncompressed form to act as a Rossetta Stone if you're really paranoid). Just find a software engineering undergrad, pay him $30 to get it up and running in a modern machine, and off you go.

Uncompressed TIFFs, however, are *way* too unwieldy, if not in storage then certainly in terms of bandwidth—as in, that of your USB port and disk buffers.

In the same vein, while DNG is still the ideal choice for RAW archival, another viable option is to save the RAW files alongside a copy of the sourcecode for DCRaw or RawTherapee. In fact, do that regardless, just in case the specs for the DNG format are lost in time as well.

Open Source and Open Formats, don't you just love 'em? in fact, you could go even further, bundle an image of a bootable Linux distribution (complete with C compiler) with your data, and store the diagrams for a x86 CPU printed in archival-grade paper alongside your hard disk. Or just go all the way and engrave it on a gold plate—though that would make your data recoverable even by aliens, which might or might not be a good thing.

[Tom Kwas]

I can't believe that people actually store their valuable digital photos as PhotoShop files...it's crazy...

...and the other thing, is how many people have no concept of the value of TIFF files...I cannot believe the "pushback" and interweb hatred that's spouted any time someone talks about TIFF on the interwebs. I originally got into Nikon because it was one of the few camera systems that you could shoot native TIFF with, perfect for those of us who've shot transparencies for years, and work primarily in controlled lighting situation. Every time someone runs one of those "what file do you shoot and store in" stories on the web, the amount of vitriol expoused every time someone mentions TIFF is unbelievable! But it's got to be from people that just do not understand professional photography and the modern digital file system.

Sometimes I just get tired and sad trying to wade through photo sites that primarily have comments from idiot pro-sumers...

[Jeffrey Lee]

@ RW - "photographers should be willing to give up some features for the freedom of "free""

True enough, unless it's a feature that is crucial to your work. And what is crucial to me may be superfluous to the open source coders out there. I once used some open source replacements for Excel and Word and as appealing as free was (and sticking it to Microsoft allowed for some schadenfreude) each open source offering lacked a tool or tools that I needed badly. It slowed my work down tremendously. I'm not sure if it the missing tools were missing due to oversight or legal issues, but they weren't there and I had to move back to the paid stuff pretty quickly. I'm hopeful that open source coding is indeed the future, but I'm not optimistic. I'm writing this in version 21 of Firefox and I still have to "force quit" to make it quit.

[Robert Hudyma]

I attended a talk by Bill Reeves, who holds a PhD in Computer Science and also won an Oscar. He was one of the founders of Pixar.

http://en.wikipedia.org/wiki/William_Reeves_%28animator%29

The subject of his talk was about bit rot (loss of bits over time) and the incredible challenge in accessing old development tools and archival data.

At that time Pixar was re-working previous releases to become new 3D releases. Pixar had to make new software that would essentially deconstruct their finished products into a form that they could be worked on again.

This is a real problem with computers, how many of us can still read a floppy disk today?

To really be archival you need a way to print the image so that you have access to the pixels and at the same time see the image with no intervening technologies.

[Steve Jacob]

I have Powerpoints that new versions of Powerpoint can't open properly.

I have a lot of PSDs, but seldom save the "final cut" as a PSD, but a 16bit, uncompressed TIFF as suggested.

The PSD is more or less only there in case I need to do a last minute quick edit (because the print didn't look quite right). If I want to revisit an image in a more serious way I normally start from RAW.

However, many programs can open PSD files, as long as they don't contain vector graphics elements.

And I think that it will always be possible to open a Bayer-pattern RAW file using something like DCRAW. I'm not overly concerned, except with regards to my Fuji X cameras. You can convert them to DNG, but it's not much help.

Where things may get tricky, in say 5-10 years, is when more non-Bayer cameras hit the market, such as Sony and Canon's 3D sensors.

[RobinP]

These comments are a wonderful illustration of the myriad ways in which TOP readers understand and use their software - Mike & Ctein, well done for keeping such a diverse readership happy most of the time!

p.s. ....and yet again I feel so much better about the inability to get my head around using layers. I save (and occasionally back up) the OOC jpg, a dng and a tiff for shots which might be worth keeping.

[ctein]

Dear folks,

No points were missed, or even harmed, in the making of this column. This is not a “big picture” column. It was addressing one very specific plaint that came up repeatedly. I've written plenty of big picture columns, on planned obsolescence, the hazards of proprietary formats, etc. etc. Go back and read through them and you'll find that I am largely in agreement with the points you thought had to be argued with me.

But this ain't one of them columns.

You folks want to bring up the bigger picture? Fine by me! As I said, you're mostly preaching to the choir, here. But I'm not going to engage it this week.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[ctein]

Dear Richard,

Versioning certainly has to be dealt with when you're archiving. I didn't get into it, much as I didn't get into naming conventions, because everybody has a different way of handling this. But, yeah, people need to figure out for themselves how they want to handle that.

~~~~

Dear Hugh,

Quite so! I wrote sloppily. Any given tiff format is hardly universally accessible. What I meant to say, and should have said, that if you choose flattened, uncompressed 16-bit TIF, you're talking about a format that is pretty robust against damage and degradation and one which is so widely supported that it's not likely there will ever be a problem finding a program that can easily read them accurately.

~~~~

Dear Albin, Michael, Jamie, et al.

Archiving RAW files and archiving TIF files generated from your “finished” PSD files are two very different things. Archiving RAW files is like preserving your original negatives (in darkroom terms). Archiving the PSD-derived TIF file is like archiving a finished print that has the ability to replicate itself. You definitely want to keep both, unless you like the thought of recreating thousands of hours of "printing" work.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[ctein]

Dear Jeffrey,

This is why I am dubious about third-party solutions to the problem of opening complex PSD files. Simple stuff, curve adjustments, hue/saturation, that kind of thing, they'll probably get right. The more sophisticated the tricks, the worse the odds are that it will include a feature that they don't support. And some things, like typography, there's very little chance that a third-party viewer will render type EXACTLY the way Adobe does, although it may be satisfactorily close.

~~~~

Dear Robert,

Bit rot is one reason for preferring uncompressed storage formats (along with a wider range of software that can read them). Storage is cheap, bandwidths are entirely adequate. A bit error gets restricted to localized damage instead of propagating through the compression algorithm.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[robert harshman]

Actually, if you are worried about format changes, or Adobe going out of business, you would probably be better off generating loss-less jpeg's. TIFF format is a hang over from the printing industry. It's as likely to disappear in this digital age as Adobe is. Neither of which I think is any cause for concern. But, if you want a format proof archive, it's JPEG. JPEG is so embedded into the internet as a standard that's it's impossible that it will go away or be unsupported in your children's children's life time. TIFF, print, maybe not there anymore in your children's children's life time? But in my view much ado about nothing, the sky is falling as Chicken little noted :)

Robert

[Sergio Bartelsman]

The best way for your files to survive in the long run is to use a proven and well tested technology that lasts several lifetimes: paper.

[Bear.]

I spit on your 8 inch floppies! What do I do with my paper tapes?

[LakeshoreTech]

In addition to backing up on my own drives, I also backup to off-site "cloud" storage. Rather than go through someone like Dropbox or Bitcasa, I'd go directly to the source: Amazon S3. Takes a bit of "tech" knowledge to work with it, but by backing it up off-site, that gives an additional backup.

As for my own workflow, I save at least three versions of a photograph: the original as it came off the camera, the Photoshop file with my adjustments, and a rendered version after Photoshop processing. Being a pack rat, I keep all three versions in at least two places.

I definitely will not go with the online version of Adobe--I'll just update CS6 as much as possible and, perhaps, start migrating to something else I can use on my own computer once it comes along. Much as I've liked using Photoshop (since about 1997 or so), a bad corporate decision such as this subscription access may well be the mistake that sends the company tumbling.

[Joe Kashi]

Data obsolescence has been a problem since the beginning of the digital era 50 years ago and will continue to be a serious problem into the foreseeable future. Ctein is absolutely right to nail transient and proprietary hardware and data file formats as the major culprit.

This isn't a trivial problem. People worry about whether their prints have 100 year or 200 year archival permanence, yet the actual original, the data file out of the camera, will be lucky if it's usable in 20 or 30 years. Making the change is very feasible technically. It's likely that archivally processed black and white negatives from the 1940s will be printable at a later date than a lot of the digital photographs taken in 2013.

Even NASA continues to be troubled by rapid digital change, having experienced severe backward compatibility problems when attempting to compare current planetary probes data with 1970s baseline data contained on fragile mainframe-style magnetic tapes and old data formats. Even when it's possible to translate those old data formats into something modern and directly usable, the unavailability of older hardware requires some jury-rigged devices to mount those old tapes and run that old data. That's not always entirely successful.

Anyone storing business or other critical data on a computer system for any length of time has had to face this problem. I'm glad that Ctein has highlighted it, particularly in view of Adobe's withdrawal to ultra-proprietary systems. His observations and recommendations are technically on-point and by-the-book.

There's also no doubt that "cloud storage" is more fragile in the long-term compared to directly storing and controlling your own data, including photographs. When your memory's in the cloud, you have no control over what's done with it, how it's protected, and whether it's safe over the long-term. The past few years have seen all too many Internet data storage systems going out of business, sometimes without any warning at all and usually without adequate opportunity to retrieve your data, assuming that it's in a usable digital format. Even when there's some warning that a "cloud" vendor will be shutting down, have you considered how long it takes to copy a terabyte of data over the Internet to a local hard disk, assuming that you have an adequate hard disk handy? What about the vendor's soon-to-be-extinct system slowing to a crawl as every one of its users tries to simultaneously withdraw their data before it's too late? It's sort of like a run on a failing bank or stock brokerage.

Every business and many users have experienced these problems, even from companies as large as Microsoft My own law office web site, which included many legal technology articles and photo exhibit data, disappeared when Microsoft shut down its personal web site service. Even if I had decided to buy into Microsoft's premium service, I was required to manually rebuild my web site. Since that had to happen in any event, and since I owned the domain name, I moved to a more stable open-source ISP. At least I had the opportunity to download all of the tech articles and photos that I had posted over the years, all of which were in PDF and JPEG, so no harm done, just a lot of expensive hassle.

So, what's the best approach for the average photographer?

Bulk-convert all of your files to standard file formats, preferably several, while you can because we don't know what formats will survive over the long haul.

Lightroom can bulk-convert and save every file in a catalog to an external hard disk in TIFF, DNG, and super-fine JPEG, all of which are among the most likely to be usable 20 years down the road. DNG Converter will probably be updated for the foreseeable future and is also to bulk-convert older files to current data formats. Every camera vendor that doesn't use a standardized non-proprietary RAW format like DNG, and that's most of them, shares some of the blame here.

External SATA hard disks used with a USB 3 hard disk dock are really easy to install and use, while having the virtue of being universally compatible, inexpensive, and likely to be supported for the foreseeable future. I've found the Thermaltake BlackX docks to be the most reliable and among the least expensive external USB/eSATA external docks. If your computer supports eSATA external disk attachment, then use it. It's much faster than USB. I've found that eSATA devices can usually transfer a few hundred gigabytes per hour.

Store at least two verified backups in different physical locations.

Oh, and remember that this isn't a one-time project. You'll need to do it every few years to ensure that you're within the window of hardware and software backward compatibility. I assume that your photos are worth this much effort every four or five years.

[John Krumm]

The Tiff digital "keeper" is interesting and I might have to do that. Over the years I've kept my final images mostly as jpegs, and then if I want to make a print I will usually go back to the raw and re-process to some degree. At some point with a Lightroom and hard drive upgrade I lost most of my catalog information. One drawback to not using the XMP sidecar files (turned off by default) is that if your Library disappears, so does your history of adjustments). It wasn't that big a deal since version 4 was different enough that I like to re-process anyway, but still... if you have twenty or so little "dodge and burns" all over an image it's kind of nice to be able to see those as a sort of history. So I've gone back to sidecar files, right next to each raw.

[Dennis Fallon]

"This is a real problem with computers, how many of us can still read a floppy disk today?"

Just because not many can doesn't make it a real problem. How many of us want/need to ? If it were a real problem, there would be a real solution. (Well, assuming media integrity). I just threw out an external (USB) 3.5" floppy disc drive because I'll never use it.

I'm sure that over the years, most people will lose most of their photos. The same as has been happening for decades. In some cases, it's truly a loss to someone; in many cases, not so much. Hopefully people will think to somehow preserve a handful of important photos to pass on their children and grandchildren. I tend to think that quality prints are probably the best way to do that.

[Sareesh Sudhakaran]

I am a big fan of TIFF, and I think in today's day and age, it offers the safest bet for archiving both images and video.

I've written about it here: http://wolfcrow.com/blog/which-is-the-best-archive-file-format-for-video/

[psu]

I'm tired of this "oh, how many of you can still read a floppy disk" red herring. While the idea behind the statement has a kernel of truth to it (as media become obsolete mechanisms for reading it also die off) it's mostly irrelevant in the modern context for one very simple reason.

Reason: the number of people who used floppies is one or two orders of magnitude smaller than the number of people who now use hard drives and JPEG.

In any case, there is an easy workaround for this small bit of paranoia: make a new copy of all your pictures every year or two. If you have them on hard disks this will only take a few hours (for most people. People with large libraries, and by large I mean more than, say, 10TB, you need to do more planning) and it guarantees that your modern computer can still read/write the files.

In any case JPEG and TIFF are unlikely to suddenly disappear simply because they are in such wide use. The more people who actually use the format, the harder it is for the universe to unilaterally kill it off.

I have most of my pictures still in their RAW file format. All of the "good" ones are in JPEGs of various sizes in various online storage areas, and on 5 or 6 different hard drives and backups there of. I need to make a pass through all my Lightroom libraries and export everything as full sized non-raw images, but I have not gotten around to it yet.

I do keep some of my old PSDs, mostly as a reference for how some final picture should look that took a lot of work to get it looking right. But I don't expect to be able to read PSDs long term ... I have the same feeling about DNG, which while "open" is still owned by Adobe and thus suspect.

YMMV.

[Michael Barkowski]

It's true that I have yet to find a useful operating system that does not need security updates, or never breaks applications when it is updated. However, instead of giving up on the infinite life of PSDs, which by definition are digital and do not erode, the alternative is to use backups of the operating system and applications, and redundant hardware. I completely approve of the strategy laid out in this article, and it may be the most sensible, but it's probably not the only way to go.

On another tack, personally I think the big advantage to Adobe's recent change is that the future with them may be a continuous evolution. Probably that will be "just right" for many people and "just wrong" for many others. I'm getting used to using applications that continuously evolve by using "apps" on my phone and web browser, and for the types of things I use them for, it fits just fine. It is a problem, though, when some company like Flickr leads you to believe they have a continuous evolution strategy and then suddenly makes a huge transition. Why did they do that? Was it to get attention? Well, they certainly achieved that.

[Michael Barkowski]

@Daniel S.: I like the idea of storing the diagrams for a x86 CPU - perfect! Although it doesn't seem necessary, since I assume Vmware, QEMU, Virtualbox, Parallels, etcetera are here to stay.

[Ed]

MarkR has it dead right, people just want to portray themselves as victims. If Adobe don't want me as a customer, fine, my copy of CS 3 still works and I'll plan a migration.

[Kirk Thompson]

IMO the practical matters are the lesser part of the question. The Biggie is that because of Adobe's shift in marketing Photoshop, some (many?) users have stopped assuming it's their destiny to use it forever. Without predicting Adobe's demise, we've begun thinking of it as something we just might not choose to use in the future.

I'm sure that concerns about the future of PSDs wouldn't have received over forty comments prior to the announcement that from now on, we'll be 'renting.'

[Ronny A. Nilsen]

Ctein have a good point here. For my part much of the distress is not the subscription as such, that is "just" a matter of money. But this whole issue reminds me that Adobe some day may go out of business, and that I need to be prepared for that. The CC is Adobes bet on how to postpone that moment. But if they are wrong it may have the opposite effect... and that worries me.

[Tom Kwas]

BTW, as long as we're talking about file permanence, when I used to live in Washington DC, I was privy to some conversations by curators and conservationists concerning the ability to read any data stored electronically post EMF blow-out during a nuclear attack; there were people, very smart, very educated people, that said we should be backing up famous speeches on metal disc, because you could always rig up a turntable, and get a thorn off a rosebush, fashion some sort of cone with found materials, and at least hear the stuff...

Funny? They didn't think it was...

...and speaking of which, I had an informational interview at a famous government institution, of a museumish nature, and they were off storing photos on a Pioneer Laser Disc system!!! Which they adopted far before the future had been decided about storage at all. No one had been scanning and burning for a while 'cause they knew it was all crap now....warning against adopting any technology too early...one can only wonder how many millions the government spent going down that blind alley...

Film, sounding better and better!

[ctein]

Dear psu,

What you said.

~~~~

Dear robert,

You're right that JPEG is even less likely to become unreadable than TIFF, but the risk of uncompressed TIFF becoming unreadable is sufficiently small because of the large installed base of images, and TIFF has the advantage of allowing for 16-bit files (yeah, I know, technically so does the JPEG standard, but that is so nonstandard a form that I wouldn't count on being able to get a decoder for it 20 years from now). Which future-proofs the images some; e.g., if there ever proves to be a good reason for using 16-bit printing paths (yeah, it already exists, but I haven't seen that it makes one whit of difference) or if you need to make some future adjustments to the image for some reason (different printer/different profile/different taste).

~~~~

Dear Sergio,

That is so seriously wrong and flawed a statement that I should probably devote a whole column to debunking it, but I'm too busy. So you'll have to be satisfied with a relatively brief debunking [VBG].

Proof of principle has been demonstrated for black and white darkroom prints for multi-decadal life, but the extensive real-world data shows that only a minority of such prints actually survive in their original condition for many, many decades. The reality has proven to be that it is not anywhere close to a guarantee of providing you with longevity. If you're talking color, which is 99+% of all photographs made in the last 40 years, then the real-world testing data is even less favorable. Again, longevity has been demonstrated… But it's nowhere close to the norm.

If you're talking digital prints, the media haven't been around long enough to remotely call them proven and well tested. We are all just hoping that highly accelerated tests have accounted for all the real-world "gotchas."

If you're going to try to fall back on the highly-unrealistic “but if you do everything right…” argument, well, I can invoke that for digital files. In fact, given the choice of trying to establish a durable archive of digital files and one of prints, it will cost me less time, money, and energy, and I will be more sure of the results, if I do the latter.

I won't even accuse those of touting the durability of prints of living in a fool's paradise, because there isn't even a paradise there; it's long been proven to not exist.

pax \ Ctein
[ Please excuse any word-salad. MacSpeech in training! ]
======================================
-- Ctein's Online Gallery http://ctein.com 
-- Digital Restorations http://photo-repair.com 
======================================

[Hudson]

I have heard that people that backup lot of data offsite are now also including the hardware to process the data.

A pallet of Hard drives and a PC with a box of install disks all shrink wrapped together.

[Rob Rogers]

A fact little-known outside the storage industry is that hard disks are not well-suited for long term on-the-shelf storage. There are mechanical and magnetic media limitations. Since most of us can't afford "enterprise" tapes, a best practice is to rotate multiple hard drives so they spin regularly and (not "instead of") copy your files to "the cloud".

This article describes some of the issues:
http://serverfault.com/questions/51851/does-an-unplugged-hard-drive-used-for-data-archival-deteriorate