Grappling with Gigabytes
The Nature of The Problem
Digital camera files continue to increase in size due to the every-increasing size of imaging chips. A day's shooting with a camera like a Canon D60, Nikon D100 or Fuji S2 can produce several hundred RAW files, each of 6 — 9MB is size. That's about a Gigabyte in a single afternoon, without even trying. Process a dozen or so into TIFF or .PSD files at 24MB or more each and that's another few hundred Meg. People scanning film are used to 35MB to 150MB files from a single frame. If you're a professional photographer shooting thousands of photographs a month the issues really are serious.
But, almost regardless of how much you shoot before you know it you're drowning not in megabytes (that's old news), but Gigabytes. How to handle this, how to store them and how to archive them? These are pressing issues.
The short and medium term storage problem is, in my opinion, best handled by online storage. I use Firewire hard disks ranging in size from 80GB to 120GB and I just keep adding new ones as needed. This is more expensive than CDs, but much more convenient. Even so, making backups and archival copies is still a must, especially if you store a copy of everything off-site, as you should. Remember — Murphy was an optimist.
There are many different approaches to cataloging, and several good programs available. But a real challenge that no one has properly met until now is archiving Gigabytes of files across CDs that can only hold 650MB or so of data.
A single shoot or project can therefore span many disks. How do we tell where things are? How do you figure out how many files to put on what disk? Where did you stop copying and where do you start again. How do you find out later what's on what disk?
A Solution — At Last
PictureFlow LLC, the group that that markets YarcPlus, one of the best RAW image converters for Canon digital cameras, has now addressed a critical aspect of this problem — archiving your files. You can burn these Archives onto CDs (and now onto DVD as of the April 2003's V2.0 release...see Update below for details).
Archive Creator is based on a very simple concept. You select the files or directories that you would like to back up. You can select individual files, directories or drives and can turn on or off individual items. (See Fig. 2) below. You then simply instruct the program to begin, and it will automatically calculate the total size of the Archive to be created; will automatically break up the files and directories into the optimum number that will fit on the media to be used (650 MB per disk in the case of CD-R, for example) and will then proceed to automatically burn the disks. All that you need to do is load and unload them as they are burned.
But that's not all. It creates an HTML document that contains thumbnails of all of the files being archived and puts this document on each disk that's created. So if the Archive of your shoot in Yellowstone ends up spanning 6 CDs, an index file with a thumbnail, file name and the disk that it's stored on is created and placed on each CD in the set. Furthermore, viewing images (smaller JPEGs — down-samples of the original source files), are also created. By clicking on a thumbnail you see a viewing image of the size of your choice... even of RAW files. This means that whatever computer the disk is put in can display not only thumbnails but also viewing images, with no software needed on that machine except for Internet Explorer or Netscape.
Now that's cool!
Fig. 3 above shows some of the options that are available before you begin processing. I won't go into a detailed description of them all — most are self explanatory.
At the bottom of the main screen (see Fig. 1) there are four button: Setup / Process / Burn / Verify. After you have chosen what files and directories to Archive in the Setup screen you press Process.
In this example (Fig. 4) you see that I am about to Archive 1,454 files that will span 18 disks. The Index file alone will be 42MB in size. You can see as well that it will be placed on each disk and also that each disk has been automatically allocated the right number of files to fit within its capabilities. To create the index images for this many files takes quite some time (about 90 minutes at a few seconds a file), but for more reasonable sized archives the process is fairly swift.
Once the processing is over it's time to burn the disks. At this point you can specify the level of verification that you wish depending on how compulsive you are. That's it. Insert fresh disks when prompted and in a while you'll have a backup Archive of whatever you've thrown at the program.
I was particularly impressed with how clearly the program communicates with the user. Every step that is undertaken is shown onscreen with the program describing where in the process it is currently working. One can cancel out at any time and step backward to redo a step done in error. A very elegant bit of software design.
For the sake of clarity I should mention that any type of file can be Archived, and that the program knows how to create thumbnails for a great many types of formats, including all Canon and Nikon RAW / NEF formats as well as those from a large number of other digital cameras.
The final Verify function can be done to any CD, at any time. So, for example, if you still have the original files on your hard disk you can verify and compare them to what's on the CDs, even after burning. If the original files are not there the program can still test the CD for integrity. Nicely done.
Archive Creator is available from PictureFlow. The new V2.0 is a free upgrade to all registered users and is available to new purchasers for $40. You can download the program without charge and try it out for an unlimited time. The only limitation is that while an entire Archive Set can be created and viewed in the HTML Preview, only the first disk of a multi-disk set can be burned until you register the product.
This is a unique and valuable product that every photographer who needs to archive their growing collection of RAW and processed files will find to be an invaluable tool. PictureFlow also has a track record of providing outstanding customer support. What more could you ask for? Highly recommended.
Version 2.0 of Archive Creator has just been been released. Since my original review of V 1.0 of Archive Creator the folks at PictureFlow have added a significant number of features. The most important being the support for DVD formats including DVD-R/RW and DVD+R/RW. In addition, CD-RW is now also supported. This additional media support is complimented by quite a number of new or enhanced features:
— New Photo Index options - No Index, Index on all discs, on first disc only, on separate disc (first or last)
— HTML Photo Index Pages optimized for better space utilization
— HTML Index Pages now can be read on both Windows and Macintosh systems (late system 9 and above)
— Image files from previous session are reused if possible to minimize processing time
— Application can be resized
— New Separate Large Preview window (will even preview most RAW files)
— Additional RAW support for the HTML Photo Index. RAW Formats supported:
--Canon - D30/60/10D - 1D/1DS - G1/2/3 - S30/40/45 - Pro90 - (special free support files need to be installed)
--Nikon - D1/1h/1x/100
--Minolta - DiImage 5/7
--Olympus - E10/20
--Fuji - S2
--Kodak - 560/760 - (special free support files need to be installed)
— Matching THM files can be hidden in the HTML Photo Index
— A previous burn session can be continued at a later time (nothing is lost when the application is shut down)
— HTML Photo Index can be burned without having to archive source files
— New disc buffering system uses available system memory while burning
— EXIF data can now be displayed in the HTML Photo Index
— Archive Estimator on Setup page estimates number of disc in the archive set
— Master Index can maintain an Index of all previous indexes on the hard disk for a permanent record of the archive sets