Picks and Nits in the Digital Age, Part 2

by Ctein

No matter how you pick at the data, those digital photos add up! A gigabyte of RAW files in a day isn't that hard to come by (though it's a very full day for me). When I get done messing around in Photoshop and adding layers and final corrections, I can easily end up with more than a DVD full of files a day. Writing duplicate copies to Kodak Gold media costs me $3–5. Pretty cheap compared to film, but anyone who says that storage is essentially free doesn't reckon on the multimegabytes of digital photography.

Which brings me to my next point: finding all those photos later. Another pleasant surprise—my old filing approach seems to work pretty well in this Brave New World! Namely, my traditional contact sheets and "catalog" numbers. A numbering scheme that dates back to my college days has worked well enough that I've never had need to revise it—(mmddyy-[roll#][strip letter][frame#]). E.g., (061607-#060) for the photo illustrating this column. I would claim genius on my part, but honestly I just got lucky. This scheme carries over just fine for digital. Photoshop makes proof sheets a snap. So, happy happy joy joy, one problem I don't have to deal with is a new filing system.

On the other hand...

Before they're archived and cataloged, bits are fragile.  I've already made the noob mistake of pulling a memory card out of my computer's card reader and sticking in a new one without dismounting the first card.  That leaves you with unreadable copies of the files on your computer and an unreadable card.  I've put the card aside while I'm looking for utility that can rebuild the file structure for me. Anyone out there wanna make some suggestions? Please??? The data is all there; it's the pointer structure that tells the computer which blocks of data are associated with which files that's messed up.

What really bothers me about this is not that I made that mistake; anyone who was worked seriously with IT for as many years as I have has done this more than once. It is a well-known "human failure mode" to forget to tell the computer to dismount a storage medium unit before physically removing that unit. The consequences are also well known. It's merely become more commonplace as ordinary end-users deal with removable storage.

What bugs me is that it would be trivially preventable: when you mount a chunk of storage, the OS should write a temporary ID byte to the device. Each time it accesses the device, it checks to see that the byte has the same value it did before. If it sees a different value (or no value), it doesn't do anything with the storage medium; instead it alerts you to reinsert the old unit so that it can be properly dismounted. When you tell the computer to dismount the storage unit, it erases the ID byte.

It's that simple. No unique identifier hard-coded into the storage medium (which would require bringing many standards groups and manufacturers into line and also freak the hell out of the paranoid). Just a small bit of code in the three major OS's and the problem would go away 99% of the time for 99.9% of the people out there. You'd think after 20+ years we could eliminate this particular failure mode, because it's not that hard to fix.

So why isn't it fixed? Beats me. Probably for the same reason there's no latch on the battery compartment on my camera and camera manufacturers can't ever be bothered to put a decent viewfinder in the camera. Because when designers aren't being brilliant, they can be just plain dumb.

____________

Ctein

Comments

[psu]

For first level backups, I find external disks to be more convenient than DVD. Buy 2 terabyte drives, keep one in a remote location, backup to the external drive immediately after processing the current shoot.

You buy new drives every couple of years and copy the data. You can do this in addition to burning DVDs, and it makes quick access to older files easier than digging out a DVD and mounting it IMHO.

As for device IDs, I can't really say except to say that it's complicated.

[Unbound]

Let me guess... you are using a Mac.
The problem is that Macs do not disable 'delayed write' on removable media, and there is nowhere (that I have been able to find after extensively searching) to set this parameter. On PCs the system automatically detects that media is removable, and disables 'delayed write', so there is virtually no problem just yanking out the card.
I have had to recover three flash cards and one entire hard drive for my Mac using wife... nothing we tried in Mac software seemed to do the job, but there are several utilities on the PC that solved the problem.

[Paul McEvoy]

Am I the only one who hasn't had a disaster like this with a card? I wonder if that's maybe a mac problem? I've straightened up my act but I've pulled cards out of cameras and readers with relative abandon in the past and never had a problem.

Like I said, I'm changing my act, but is this a pc problem too? It seems like macs are more picky about how you eject something.

Paul

[Janne]

Just a comment about removable devices: The particular failure you describe, with filesystem metadata meant for one device going to another, is as far as I know already prevented under Linux, and I'd be very surprised if it wasn't under OSX as well (which shares the same design philosophy). The machine detects that media has disappeared and doesn't assume that anything you plug in afterwards is the same unit again.

The more general problem of unplugging without unmounting is more common, more serious, and not trivially (or at all) preventable. The problem is that writes are not instantaneous. The system waits a while before writing data to a unit; writing is expensive in many ways, so it tries to make sure it has plenty of stuff to write before it begins. If it wrote stuff as it dribbled in, speed (and, for flash memory, media life) would plummet. SO it delays the write, you unplug the media and suddenly you sit with an inconsisten filesystem.

And even if you did write everything at once, you still can't prevent the user from unplugging it mid-write, causing even worse problems. The original Macintosh tried to solve it by preventing the user from removing the media - they did away with the eject button on the floppy drive. In the end, the frustration of users not being able to eject their floppies (remember having to use a bent paperclip to get a disk out?) won out, and later macs added the button back.

So how do you solve it? Basically, you don't. You can't stop people from yanking an USB cable at will, just as you can't really stop people from opening the passenger door in traffic, putting metal objects in the microwave or taking the boat out just before an impending hurricane.

[Dave Kosiur]

I've had success recovering images from corrupted cards and such using Lexar's Image Rescue (http://www.lexar.com/software/image_rescue.html).

[Mitch Alland]

Ctein

>>>Probably for the same reason there's no latch on the battery compartment on my camera and camera manufacturers can't ever be bothered to put a decent viewfinder in the camera.<<<

With a good LCD display on the back of the camera, like that on the Leica D-Lux 3 or the Ricoh GR-D, you don't really need a viewfinder. When I got my GR-D in July last year, I really got to like the camera and the way of shooting with it: at first I thought that I would use my external VC28 and Leica 21 external viewfinders in the GR-D's hotshoe, but I found that I like framing with the LCD monitor because it leads, or forces, me to a more "fluid" and "looser" shooting style than I had with my M6. In fact, during the few days I had an external viewfinder on the GR-D, I found that I always used the LCD monitor instead, something I never expected.

Insisting on a viewfinder on these little cameras is really sticking to the old paradigm. You havce to realize that a built-in viewfinder is always going to be of poor quality. This is the reason that Ricoh states that they did not build into the GR-D a viewfinder like they gave on the GR1.

Of course in bright sunlight the LCD is very difficult to use; but here again it's a matter of adapting: I only use the LCD to frame, that is, to establish the edges of the frame: when pressing the shutter I look at the subject, not at the LCD. You can see the results on my flickr site.

—Mitch/Potomac, MD

[Albano García]

This soft saved my soul when I thought I've lost my whole travel photos:
http://www.cgsecurity.org/wiki/PhotoRec#Known_file_format
Works well and its free.

[Aaron Shetland]

Photo Rescue is a fantastic tool for image recovery. You can download and try it out before you buy it, but at $30 I think you'll find it's a bargain.

http://www.datarescue.com/photorescue/v3/index.htm

[erlik]

Ctein, do try Restorer 2000, at

http://www.restorer2000.com/

I used it to restore photos from a _formatted_ card, used it to restore a complete 120GB disk Windows said was dead and the other day I pulled all the files from a USB stick that Windows were not recognizing anymore.

Frankly, I don't understand what you're saying about the ID cause I don't think it's relevant in this case.

What happened, if you use Windows, is that the transfer is not over when Windows say it's over. They can close down the Copy or Move windows but the files can still be in the process of being transferred. And if you just pull the reader plug out, two things can happen: nothing (most of the times) cause the files were really transferred; or, if you're unlucky, you're pulling the plug just as the Windows are accessing the stick/reader/whatever. In that case, you can damage a photo or photos, or you can kill the file allocation table, which seems to have happened to you.

(I could tell you about the time when I almost killed a hard disk by trying to format a floppy on a 286...)

So it's a good idea to dismount/stop USB devices before you unplug them.

And again, Restorer 2000 comes with my deepest admiration and a great big fat recommendation.

[Marc Rochkind]

I can't understand why removing the card prematurely would mess up the card, unless the card is being written to. Why would that be? Perhaps there is more to the story?

I have yanked cards out of various machines, running Windows and Mac OS X, and have never seen corruption of the card. But, I've never done it when the card was being written to, which it almost never is.

Anyway, any solution that involves writing to the card might make matters even worse. And, it wouldn't work if the card is read-only, as SD cards can be.

A better solution is to automate the ingestion, along with backups at ingestion time, and have the ingestion program automatically eject (unmount) the card. That's what ImageIngester does.

(I'm the developer of ImageIngester.)

--Marc

[witek]

Unfortunaltely I can't help you,
but would like to inform, that it's not fault of OS, but of primitive filesystem that exists on the card. It's M$'s fat32.
Modern (and free) filesystems (zfs from the Sun, jfs from the IBM, reiserfs and ext3 from the Linux, xfs from the SGI ...) contain journaling system, that can be easyly recovered, so maybe it's worth to sacrifice ~30MB of card space for the journaling system and format it with one of these systems before use?

As I said, I ca't tell you how to recover data from the card, but try goggle for something like that:

vfat OR fat32 OR "fat 32" OR fatfs recover OR recovery

[Gary]

Although I'm a hobbyist photographer, I use a computer to catalog other data for my work. I use a naming convention similar to yours, but because of the way computers sort numbers, I use: yymmdd_(then a descriptive filename)This way all the photos I take in a given year end up together. Your way, all photos taken in the same month, but in multiple years will be together. This may not make a difference in daily use, but when finding photos stored on a hard disk, or when another data storage medium is available, you may find this format more useful.

Gary

[Jernej]

Zero Assumption image recovery has done miracles for me where all others have failed. Need I mention it's free ;)

http://www.z-a-recovery.com/digital-image-recovery.htm

[John Banister]

When a hard drive I had failed, I successfully used _File Scavenger_ from www.quetek.com to get back thousands of files with filenames and the directory structure intact. This was especially a relief as when I took that hard drive in to a computer repair place they got most of my data back, but it was 10,000 files with no directory structure and helpful filenames like file7146.jpg, file7147.jpg, ... etc. The demo version is freely downloadable, and should allow you to determine whether it's worth purchasing in your case.

[RobinP]

I hate to praise Microsoft after the abominable mess they call Vista but Windows XP is quite happy for you to remove a memory card without "dismounting" it. Just as well 95% of Joe Public uses Windows or the present boom in digicam use would have hit a brick wall if stories of losing data by removing a card were rife.

Cheers, Robin

[mwg]

I'm not quite sure what you're doing there. yLord knows I've made my share of mistakes with storage, but this isn't a mistake I've made. I think. Maybe Windows actually handles this well for a change? That's where I do most of my image manipulation.

[Ctein]

Thanks to everyone who has suggested rescue tools for my memory, both here and in email. I'll let you know what happens as I pursue this to solution (or abandonment).

I'd like to caution against absolutist, binary thinking. ID-ing media is not suppoed to be failure-proof-- nothing involving humans using computers can be. But it is failure-resistant, and it cuts the losses to a small fraction of what they were.

My particular case did happen on a Mac running OSX, but over the years I've seen it happen on Macs and PCs with all kinds of media. Macs may be more prone (I don't know), but it's a known general failure mode.

Write-caching makes the problem a little more complicated, but not a lot. The principle: "Check that byte before you write" works on the caching level, too. System overhead is insignificant; the code is easy. Really, it is. Trust me on that. Hell of a lot easier than coding new features, lemme tell you.

It's easy for a designer to whine and complain about why they shouldn't be bothered to fix this. I'm of the old school of design-- patch bugs and holes BEFORE pursuing the next shiny bauble of featuritis. Yeah, it's just not as much fun. It's merely responsible.

pax / Ctein

[Ctein]

Re file storage and archiving, I'm right with psu on this. In fact, I'd just ordered a 500 GB external FW/USB2 drive from Fry's Online store, for just this purpose. $120 right now, free shipping, hard to beat. DVD's go offsite, the HD sits here. Faster, cheaper and better.

I agree with Gary, too; I name my computer photo folders year-month-day [title(optional)] for just that reason. Years ago, I wrote a parser macro for my spreadsheets that extracts the year field, for sorting purposes, so order of info in individual photo IDs isn't important to me there. But, yeah, if you're building a file system from scratch, do what Gary says from the get-go. Save ya lotsa grief later.

pax / Ctein

[mwg]

Witek, I don't think I'd recommend using another file system on a card your camera writes to. I don't think the camera would be able to write to it in the first place, and if it did I wouldn't trust it not to corrupt the data. (And I'm not ready to trust ZFS just yet on anything I care about.)

[chris]

all these comments on removing media, and only one person noted the more clear-cut issue--iso standards already tell us how to label our files (year-month-day) for the obvious reason that any computer system on the planet will sort them all into a usable order. it would be a shame if people out there copied your 'system' out of ignorance.

[Aubrey]

Hi,

Convar makes a free recovery utility for Windows that can be used to restore files on Flash media.

More information can be found on their site at http://www.pcinspector.de/smart_media_recovery/uk/welcome.htm

Hope this is of some help

Regards ......... Aubrey

[Michael Eckstein]

I use both Mac's and PC's. I use the Mac for my photo work and I have removed CF cards dozens and dozens of times without ejecting them properly and have never had a corrupted card under OS X. I only get the message that a device has been dismounted improperly.

[dyathink]

or, um, you could just shoot film and have a good old fashioned negative backup..sorry, just had to state the obvious :)

[John Lehet]

I've pulled the card out of the reader on my Mac running Tiger more times than I like to admit, and have never had a problem with the card. Usually I've done this after the files have copied, of course. When I get the message from the Mac telling me I shouldn't have done this, I usually shrug and think, "I'm about to reformat the card in the camera anyway."

[Craig Norris]

To be sure that Windows XP doesn't corrupt a randomly ejected removable storage device, one needs to go into Control Panel then System then Device Manager then Disk Drives and double click on the removable disk in question. Then in the new dialog box, click on the Policies tab, and select *Optimise for quick removal*

[Chris]

Ctein, to prevent such human failures in the future you can set the read-only switch on the card when you download files to your machine. This creates the inconvenience of having to toggle the switch, and reformatting the card in camera, but it is a physical and simple solution to this problem that works by preventing the data on the card from being modified.

In regards to your recovery problem, I have had success with Data Rescue II from www.prosofteng.com. Some flash cards come with data recovery software; the purchase of new media could give you new storage and a means to fix your old storage.

[John]

ctein: the code you describe (in my experience, we've called it "epoch numbering") is easy, but it does cause the more complicated problem of pinning writes in the cache. Obviously not a showstopper, but it does add complication. Ultimately, though, any significant changes to legacy code will get bottlenecked through management. ("Why should we change?" "Nobody else does it this way!" "Think of the impact on QA!")

And as for me, I'd rather have write caching turned on and have to remember to eject the card if I've been writing to it in the past three seconds than to deal with the significant UI performance lag of writing through to a flash device.

My personal goal in designing appliances and storage systems is to work on making the software more reliable than the hardware. Epoch numbering is a handy trick for that.

[Erez]

Why not invest in good storage system ? Something like a RAID array would do nicely for reliability issues and on-site backup (Saves you from a single point of hardware failure).
Regarding filing, every OS (and plenty of utilities) these days offer quick and fast file search, based on dates of creation, EXIF meta-data, last date of editing, etc. Why bother with complicated filing systems ?

[Barbu]

Another vote for Zero Assumption recovery: http://www.z-a-recovery.com/digital-image-recovery.htm

[witek]

>I don't think I'd recommend using another file system on a card your camera writes to. I don't think the camera would be able to write to it.

You're absolutely right!
As a film user I completely forgot that the thread
refers to the camera's memory card, I thought of it as of the storage media.
The camera woul'd not recognize the card, od would damage the file system on it definitely.

[Mauro]

The real point is why the cards do not have some kind of protection so they could be locked only if writing is going on (or formatting or what else) leaving them free to be extracted otherwise.
Even the ancient floppy has it. Maybe its time some seriour reviewer stands on.

Ciao

[Ctein]

Well, folks, I've recovered about 2/3 of the files, which are all that are gonna come back intact, so thanks for all your suggestions.

Chris, setting the write protect tab on media's been SOP for me... 'cept the xD cards don't have a write-protect tab!

dyathink-- if you start leaving useful messages instead of snarky ones, you'll start getting some meaningful replies. Just some friendly advice.

pax / Ctein

[John]

Erez:

I've posted this rant before, and I'm sure I'll do it again.

RAID is *not* a backup solution. RAID is entirely a reliability and business continuity system, and I regard it as almost entirely unnecessary in a home system. In fact, for most small and medium sized businesses, anything other than RAID-1 is a waste of money. (Actually, Google eschews all RAID for their entire storage infrastructure, instead relying on homegrown mirroring, and recent empirical data from CMU points to significant reliability problems with storage-based RAID implementations.)

RAID does nothing to protect you from the failures which are most common to home users. Using RAID, you will lose data if the user makes a mistake, the software makes a mistake (and ctein, I'd be more than happy to classify your problem as a software error, albeit an omnipresent one), your system gets infected with a virus, there's a power surge, there's a power failure, your house catches on fire, your house gets submerged in a flood, or somebody breaks in and steals your computer.

The only thing that RAID protects you from is the loss of a drive, which, in a single drive computer, can be expected to happen at a rate of about 0.005 drives a year. (Still significant, but not worth the expense.)

Back your data up. Send it to a remote site. Use something like Apple's new Time Machine, and make sure that there's a copy stored at your office.

Better solutions are coming, down the road, but RAID isn't one of them.

[R. Edelman]

I am sorry to pick this nit, but "data" is plural. The sentence should read: The data are all there...
I know that you are a perfectionist when it comes to photography, so I thought that you probably wouldn't mind some grammatical perfectionism!
I enjoy reading your articles. Please keep them coming.

[Ctein]

Dear Erez,

I'm not sure if your questions are directed at me or someone else (a problem with having non-threaded discussions), but I'll tackle them, anyway.

RAIDs are just fine, but they don't address either of the concerns brought up in this thread-- corruption of files from the get-go, and proper archiving of files after you've downloaded them. A RAID is reliable, but it doesn't provide you with offsite backup . 'Course, neither did film, so if your house got burglerized or flooded or burnt-down, you were hosed fer shure. But the great thing about digital files is being able to duplicate them and archive a set elsewhere. Whether it's two external HD's or a spindle of DVD's isn't that important-- both work and have comparable economics. The point is to get a set of data offsite.

As for filing systems, it all depends upon your needs. A file 'date of creation' doesn't always conform to the actual date you made the photograph. For example, all my recovered photos have a creation date of yesterday. Further, many folks wish to file by different criteria than mere computer-driven chronology. If it works for you, great. It wouldn't work for me, though. Honestly, if you think my system (a simple date code and proof sheets) is 'complicated' you'd find many folks' needs to be positively brain-busting.

pax / Ctein

[Matthew Miller]

xD cards seem particularly vulnerable to this problem. Not sure why exactly -- it seems like since xD puts the controlling circuitry in the reader/writer rather than on the card, they're well-positioned to be more *resilient*. But no -- they can be corrupted by removing them while mounted even if you're not writing anything to the card.

I have my Linux system set up to default to mounting cards read-only, which should avoid the problem (although I haven't explicitly tested it with xD).

[jim h]

I've solved my backup problem once and for all with Amazon S3 and Jungle Disk. Jungle Disk periodically sweeps up my new files and copies them to Amazon S3. From there I let the big boys worry about redundant backup servers.

[dasmb]

CTein: Welcome to digital!

Though not a solution necessarily for your card woes, I suggest getting yourself some decent Digital Asset Management software and bypass the file level operations. Lightroom, Aperture, iView, Capture One or even iPhoto; they all encapsulated the job of copying data from your cards, give you a nice interface for doing so with indications when the copy is complete (along with a message permitting you to eject the card directly, and wipe it if you wish).

But the nicest part of these programs is the ease of tagging, searching, displaying and backing up your data, combined with simple, straightforward solutions to common image manipulation tasks that should remind you of darkroom tasks. Got a whole roll underexposed by one stop? Fix the whole thing in one go without having to launch Photoshop!

I am a software engineer and am obsessive about my files...but I found that stopping worrying and learning to love the DAM app gave me more time to work with my photos.

Aperture's by far my favorite and it's got a free trial. Check it out, and visit us on apertureprofessional.com. iPhoto is surprisingly good for the money, and it streamlines the process of backing up to CD/DVD.

[Mike]

Sorry to pick nits in return, but data is NOT plural, unless you're an ancient Roman. The Latin root word is plural. The modern English word can be used as singular or plural with equal correctness.

Mike ("Editors Needed Everywhere") Johnston

[Ctein]

Dear Mike,

Yeah, I'm with you (sez the English major). 20 years ago, I was careful about "data are, datum is," but the language has changed since then. If I had to, I'd define "data" in present day use as "the collective noun for 'datums' [sic]" rather than "the plural of datum." That makes 'data is" the grammatical choice, since a collective noun is singular.

Words change with time, especially technical ones that come into common use. "Laser," for example, is no longer an acronym (I can cite examples, if anyone disbelieves).

(Paula just opined that anything that becomes a cat toy no longer is entitled to acronym status ).

pax / Ctein

[Mark Sirota]

I'm sure Ctein, like most of us photographers, hates it when computer engineers try to design cameras.

Likewise, I don't think the computer engineers would be happy with photographers or artists (even those with credentials like Ctein's) designing removable media interfaces.

If this problem were as easy to solve as Ctein suggests, it would have been solved.

[Ctein]

Dear dasmb,

Thanks for the belated welcome (it's about 35 years late, but appreciated nonetheless ).

I'm mulling over what kind of management I need or want. My volume of stuff has been so small that I really haven't needed DAM tools. That may change; I'll have to see if my photographing habits are markedly modified when I work digital. So far I'm still a pretty spare picture-maker, but who knows about the future?

I'm inclined towards either Aperture or Lightroom. On one hand, I like Apple stuff. OTOH, Adobe will *give* me Lightroom if I ask for it.

Just for clarity's sake, almost all my photo work is currently done under Win2K. I also have a Mac Powerbook, and I was transferring photos while traveling, which is why this whole mess happened under Leopard. But normally, I'm dealing with Win2K apps. That'll change with our next computer, which will be a MacIntel, but that's a bit in the future.

pax / Ctein

[dyathink]

ok, Ctein, next time i will belabor my point. It wouldn't make sense to do so in this thread because i really was just saying the neg thing as tongue in cheek, but next time, you're on. Just a friendly reply :)

[Matthew Brown]

RAID is insurance against disk failure, pure and simple. Sooner or later it'll save your ass, because sooner or later you will have a disk failure, guaranteed.

However, it's not a replacement for a backup system. It helps you not lose your changes since your most recent backup and means that your system stays up and available while you replace that disk, instead of going through a painful restore that can take a long-ass time.

It's also worth remembering that once you go beyond a one-drive system, your chances of being hit by a disk failure are much greater. Of course, the average or even much bigger than average home system isn't like the systems I work with; we get about 3-4 disk failures per week, across all our systems.

I notice that some older systems I used to work with, e.g. the Amiga and I seem to remember even MS-DOS to a degree, would actually tell you to put the damn drive back in so that it could finish writing to it. Why can't a modern system do that?

[dasmb]

Mike, thanks for the datum!

[Robert Roaldi]

I am currently making a living as a scientific copy-editor. In our publications, we continue to usually use data in the plural. It is a style decision on our part and largely based on context. But, sometimes "data" is used to mean a "collection" and not more than one datum values.

There are hard rules in English grammar but lots of soft ones too; English evolves and always has. It's best not to be too dogmatic.

There are still people who believe that you are not supposed to split an infinitive or end a sentence with a preposition. These have never been hard rules, people only thought they were.

[Ctein]

Dear Mark,

I've got a rather substantial background in the computer side of things... oh, say, 42 years worth/ More than 15 years ago, I started jokingly saying I was a "hacker emeritus" because I'd moved beyond the low-level stuff.

I've reviewed for computer journals as well as photo mags. One of my other jobs is technical writer, and I've documented gear on every level from I/O hardware bus specs and design to user interface principles.

This doesn't make me right, but I'm not in the habit of commenting on subjects I don't know a fair amount about.

Just because photography and art are my chosen passions does not mean they're my only areas of expertise.

pax / Ctein

[dasmb]

I think any workflow can benefit from a good DAM tool, regardless of size or frequency. They just cut out so much of the meaningless work involved with importing and tweaking. I haven't created an "adjustment layer" in two years and it feels great.

If/When you do choose one, I'd love to hear your thoughts on Lightroom/Aperture (the eternal struggle). I bought into Aperture from the crummy 1.0 version, and I've got too much bias to ever enjoy an Adobe product.

[John]

Matthew:

Oh, man. I hate geeking out on this site, especially when the topic is only tangentially related to photography, and I wouldn't blame anyone for not reading (or posting) this comment, but here goes:

I made the statement that I would hesitate to recommend RAID storage solutions, even for a small or medium sized business. That claim is supported by one of two recent long-term studies of actual storage reliability data in a variety of environments (paper at http://www.usenix.org/events/fast07/tech/schroeder.html).

A large number of interesting facts emerged from the study, but the most relevant right now is that the fundamental mathematical assumption that makes RAID work is not shown in the real world. RAID stands for Redundant Arrays of Independent Disks (originally Redundant Arrays of Inexpensive Disks, but that was hard on the marketing departments). The independence statement is the most important part, as it allows for us to treat failures of disks within a RAID set as isolated. If one disk fails, the others keep chugging away. That's the assumption that gives RAID those tremendous reliability numbers.

Unfortunately, what the study showed was that disk failures within a RAID set are *not* independent. If one disk fails, you are more likely to witness another disk fail, very soon. The probability of a second disk failure within the repair window approached 10% in some cases.

Another very interesting (and unsurprising) datum was that manufacturer claims of drive reliability are way, way off. The so-called "enterprise-quality" drives that cost hundreds of dollars each have reliability characteristics that aren't too much different from the SATA drives you buy at Walmart. *And* the assumption of an exponential distribution of drive failures is also incorrect, instead pointing to a steady, almost linear increase in drive failure rates as storage ages.

The upshot, combining the reliability of the single drives with the loss of the independence assumption, is that RAID provides only a marginal increase in reliability, even when compared to a *single* drive. Even a small-medium sized data center using RAID should expect to see about two instances of data loss due to array failure a year. Given the cost of RAID controllers, I don't see much value to RAID as a reliability solution when compared to simple, independent mirroring/replication. And my core message stays the same: backup, backup, backup. And always verify your backups.

Here, I'll tie it back to one of the themes of this blog (at least, a theme I perceive): RAID is an interesting case of conventional wisdom trumping experience. I've talked to a lot of people with large storage installations, and they all complained about the number of failures they see with RAID controllers. They all prepare elaborate systems to protect against those failures. And yet they continue to cite RAID as a reliability solution! They are always willing to believe the wisdom of the "experts" (like me) over the evidence of their own eyes. The failures and problems with their environments are all a result of some problem of their own. It's their fault, for running their systems too hard, or misconfiguring, or letting the labs get too hot. Nobody is ever ready to challenge the conventional wisdom.

Of course, the other lesson is: never trust a vendor to give you good reliability data.

[Janne]

John, basically agree. However, a RAID system can make a lot of sense if you're using it to improve access time or storage space, rather than for redundancy. Depending on the specific business case, that may make disk arrays a very sensible solution.

I note that you distinguish RAID from disk mirroring in the Google example; in my usage, RAID is more generally the idea of mirroring or striping disks, not specifically using a hardware disk controller interface to do so in one box. For me, continuous mirroring _is_ a RAID system, just not necessarily implemented at the hardware level. That's just an aside, of course.

[Stephen Best]

Forty nine comments on the file system of memory cards.

zzzzzzzzz

Get well soon Mike.

[John]

Important points, Janne, and that's why I tried to be careful with my wording. Although I would point out that a five-disk fibre channel RAID-5 config with 36G drives would cost about $1400 for the drives alone, yielding a usable capacity of 144G, whereas a three disk mirror based on 250G SATA drives would be about $300 for 250G, and the reliability would be exponentially higher. Of course, performance is where FC rules.

You're right, though, it all boils down to the specifics of the business case. RAID-5 and other erasure coding schemes are overused and oversold, but there are plenty of occasions where they *are* the right answer.

The RAID vs. mirroring terminology is a place where I draw a distinction based on common usage, since to many people, RAID implies a hardware controller. I entirely agree, mirroring is RAID-1, regardless of where it's implemented. *However*, the independence results suggest to me that a RAID-1 set inside an array would suffer from the same fault characteristics as a RAID-5 set, and I tend to use the terminology to try and highlight the difference.

And Stephen, this is my friggin' life. Pity me, please.

[Stephen Best]

Ctein,

Someone should do a study of the types drawn to photography, especially digital. The internet just feeds this mania for minutiae. I must admit to getting sucked into this myself but even I have my limits. I guess there were those that agonized over developer formulae (or whatever) in the past ... some of my buddies still do.

But since I'm here maybe I should offer my input. I've had my D80 for 6 months and haven't yet taken the card out of the camera ... but then again I'd don't shoot weddings or action. I use the USB cable that came with the camera. Works well. I delete the crap in-camera (there's a lot of it). Haven't lost anything yet.

Cheers,

Stephen