Standard File Formats

[Schewe]

DNG is an open standard, royalty free, for anybody to use and a lot of 3rd party software has or will be adopting it such as:

Raw converters:
Adobe Photoshop CS (Win, Mac) with ACR 2.3 and 2.4
Adobe Photoshop Elements3 (Win, Mac)
Pixmantec Rawshooter 2005 (Win) -freeware-
Raw PhotoDesk : (Win) Improved support in V1.12
Programs that output to the DNG format:
Adobe DNG converter (mac, Win)
DxO Optics Pro v2.2 (Mac, Win)
Imacon Flexcolor v4.0b10 (beta version) (Mac, Win)
Image Viewers / Digital Assets Management tools:
BreezeBrowser 2.11 (Win)
BreezeBrowser Pro (Win)
CameraBits Photo Mechanic 4.3.3 (Mac, Win)
Cerious Thumbs Plus7, with the Digicam Raw Plug-in v 1.5 (Win)
dpMagic Plus since version 1.1.010
Extensis Portfolio7 (Win, Mac)
iView MediaPro 2.6.2 (Mac, Win) DNG 2.4 images in 2.6.3
Photools Imatch (Win)
Polybytes PolyView : since version 4.20 (Win)
Proshooters DigitalPro 3 : initial support in 3.1
Rawview (all OS: Java technology)
DNG support announced:
Iridient Digital Raw Developer (Mac): in the next dot release
Phase One Capture one (Mac, Win) in 2005
Qpict Media Asset Management: (Mac) in the next major release

Adobe has stated they will hand over DNG to an indepentent body such as the ISO if need be. Adobe allowed the ISO to adopt and adapt TIFF-6 for ISO's TIFF-EP standard file format. DNG is essentially a more speciified of TIFF-EP.

While DNG DOES allow private maker notes, not unlike the EXIF, the manner in which those private tags are wirtten is specificed. Currently there is no specification for the way private EXIF tags are written so software and can't understand the tags must strip them or else face the risk of corrupting the file. The DNG private tags could safely be passed on without corrupting the files since how and where thay are written is specificed. Thomas incporporated the private taggs so that camera companies could still have private tags that could pass along data to their software. Yes, that is less than ideal, but was done at a nod to the camera makers to remove a barrier for DNG adoption. Personally, I would prefer NO private tags and FULL documentation. But I'm realistic enough to know that will take time.

PSD is a documented file format with a complete SDK. Yes, you need to sign an NDA and be an Adobe developer to use it. But PSD is written and read by a lot of different applications. However, most Photoshop expert are seriusly reccomending user adopt TIFF (TIFF-7 to be specific) as a foward looking file format. It can be argued that PSD is the private native format of Photoshop, but layered tiffs can save EVEYTHING a PSD file can plus more. PSB, Adobe's new large file format is semi-proprietary only in so much as no other file format, including TIFF or PDF can save as LARGE a file as PSB.

Thomas has stated that he thinks it will be a 3-5 year battle to finally get to the stage where there are strict standards for raw file formats. At the moment, the _ONLY_ standard raw file format on the table _IS_ DNG. Until or unless something comes along to compete with DNG as a standard, I fully support the DNG initiative. I would encourage all others to do so as well. Could DNG be better? Yes. . .and I suspect Thomas would welcome any and all honest technical critiques and feedback. So far, there has been little or no comment by the camera compaies other than "they don't wanna" (expect for Hasselblad and Leica).

Regardless of what one might think about Adobe and DNG, the current situation will ONLY get worse without standards. There are already over 100 raw file formats and counting. The situation will get worse before it can get better. To allow camera companies to opperate outside ANY standards is simply not in the long term best interestes of the photo industry.

Nikon should be punished for encrypting their WB information.  . .and I believe they are getting punished in the marketplace, where it counts the most.

[giles]

Thanks Schewe for the excellent post.

I would add that while much of what Ian said is correct -- specifically that lockin to Adobe's PSD is also undesirable -- solving any part of the problem makes it easier to solve the rest.

DNG or another open format solves one part of the problem, that of archiving.  It helps for the problem of interoperable software, and over time we could expect to see layered standards to settle what private maker notes are used and what they mean.

The issue isn't one that will be solved overnight, but if we as customers keep encouraging vendors who move in the right direction and complaining to vendors who don't, we'll get there eventually.

Giles

[Schewe]

The biggest problem is that the camera companies are racing forward passing any and all standards as they go.

The whole "Private Maker Note" issue comes from the lack of enough defined public tags for EXIF. EXIF is a standard originally proposed by Kodak, adopted by the www.exif.org and then coopted by the JEITA (Japan Electronics and Information Technology Association). The last publicly available standard is EXIF 2.2 (April 2002) but I'm pretty sure most cameras are at 2.3 or 2.4. It seems that the JEITA has been pretty lax publishing the changes. . .but the problem is that new public fileds have not been defined so new cameras that contain new types of metadata are forced into using private maker notes-which is where the whole problem gets super sticky.

Then you move on to TIFF-EP which is the basis for most EVERY raw file format out there. . .the last published standard for TIFF-EP I can find is from 1998 or 1999. Well, a LOT of stuff has happened since then and the standard has simply not moved forward. . .

Why? There is no incentive for the camera makers to hold themselves to any standards. . .they don't WANT to be held to any standards.

There is another sinister reason perhaps. . .and you need to put on your conspiracy hat to delve into this speculation-and all it is is speculation mind you. I'm not making any accuations here, just speculations. . .

What if they have something to hide? What if by adopting a tough stance on proprietary and undomented raw file formats they (the camera makers) are hiding the fact that a whole bunch of patents are being steped on. If they can hide behind the undocumented, proprietary raw file format, they could be hiding the fact that they are violating a lot of exisiting patents. . .perhaps by Kodak or pehaps by a variety of other US and European companies. Nikon has been incredably "busy" the last couple of years putting a lot of patents in place for a lot of various aspects of digital photography.

The reason I wonder about this, and again, I stress this is speculation, a couple of years ago Microsoft started to asert the patents held for DOS and the whole compact flash card file system. You haven't heard much about that. . .recently there was some news about Forgent suing Microsoft over data compression techniques Forgent claims is used in the JPEG digital image standard and in wide use throughout the industry.

Forgent had sued 31 companies in April 2004 and several other companies after that. The company has reached licensing agreements with more than 35 companies and received more than $100 million in licensing revenue to this point.

That patent is due to expire 2006 or 2007. (can't remember which)

One wonders if the camera makers are just trying to buy time. . .to save money and avoid litigation. . .

[61Dynamic]

I've been converting my RAWs into a DNG with the original RAW embeded. Then I have a backup of everything; including the DNG converter. This way, if I ever need an original RAW image, I can load up the converter and extract it from the DNG file.

[IanS]

Michael,

This whole Nikon WB, ACR and DNG issue has annoyed the h-e-l-l out of me recently, and after your publishing of Jeff Schewe's comment I felt I had to give my opinion on this.

Whilst I don't think Nikon are being particularly helpful in this matter, I don't see that Adobe are so far being much better either. Currently Adobe are hiding behind DNG and trying to position themselves as doing a great favour to the photographic community by almost trying to bully Nikon into adopting it.

It strikes me that the people who stand to gain most from DNG are Adobe. They're obviously feeling the pressure of their decision to reverse engineer RAW formats as the camera manufacturers continue their onslaught of new models. IF everyone adopts DNG, this problem is largely solved for them. However looking at Mr. Schewe's comments I'm not convinced that DNG is the answer, or that the posturing of late by both Nikon or Adobe shows any sign of progress being made in the area of RAW formats.

In JS's quote he says "Nothing in the file should be in any way undocumented or in any way restrict the access to the image data by the author of the photograph." Now I know JS doesn't work for Adobe, but it seems Adobe have already broken this rule with DNG. They have put with CS2, application specific info into the DNG, which can only be read by ACR (and understood) and I doubt very much if any of it is documented.

Also in February Adobe updated the DNG standard to 1.1.0.0. Whilst they didn't start writing files to that format immediately, their software that was released at the same time, already used some of the new format tags and would read files in the new specification. Is this really an open standard? Strikes me that other software vendors are at a slight disadvantage here if the first time they see a DNG change is after Adobe have already implemented most of it. Surely for a true open standard everyone should get to see the proposed changes at the same time to create a level playing field.

JS goes on, "The raw file must belong solely to the photographer with no lien or encumbrance attached in any manner, what so ever." Pretty strong stuff, so, if Adobe are already dumping ACR specific metadata into DNG files, who's to say the rest won't follow? Do we really want a RAW file format that if processed in several different converters starts to accumulate application specific metadata on it's travels?? Not sure I do.

Of course, once you have a standard RAW file format, Adobe seem to lose interest quite abruptly. I'm sure Michael's workflow is not an umcommon one, convert RAW file, adjust in Photoshop (or some other application) using adjustment layers so that changes can be easily reversed. Save huge file to an open industry standard format that is fully documented as surely the image still belongs to the photographer right?? Oh, hang on, save to PSD, not open. But hey, that's OK, locks people into Photoshop. layered TIFF's are a bit of a crap-shoot support wise as is compression within them.

Next step is to create an output specific file, for print or for web or for whatever. Web isn't too much of a problem as JPEG's pretty standard but if you want a finished file without lossy compression, then TIFF is the only option but again it's not quite there. What format is best here? Not clear to me what final format will still be around in say 50 years time.

So, I'd say that DNG alone (or any equivalent) is not enough. We need:

1) A standard RAW format
2) A standard editing format that allows saving of layers that is properly supported.
3) A standard final format that is properly supported.

So whilst the camera manufacturers are clearly lacking so far on number one, the software companies haven't done a much better job at number 2 & 3. Yeah I know TIFF-EP is out there but how many times have we seen a complex TIFF written in one appliction cause another to choke??

I'd say no-one is whiter than white here and as users we shouldn't be concerned just with RAW formats because the latest one isn't supported in our software, we should be concerned with the whole end to end life cycle of our files. I also think any industry standard should be proposed by the industry as a whole and then owned by an independent standards body. Can't see that happening yet and as a result I'm not sure the manufacturers will ever buy into DNG.

Ian.

[Jonathan Wienke]

Well said, Jeff.

[IanS]

Hi Jeff,

This isn't meant to be sarcastic or rude, but thanks for your reply, it's the first one I've read by you on various forums that I actually for the most part agree with, not that it will matter to you of course!! :-)

I think Adobe should hand DNG over as soon as possible, keeping hold of it doesn't seem to serve much purpose and I'm sure would make the other software vendors happier. What I can't really see any answer too is getting the camera companies to play ball. People are not going to stop buying the cameras any time soon and one camera company seems to be hardening its stance on the subject.

If Thomas Knoll believes it will take 3-5 years for a RAW format to be established then it in some ways casts doubts over whether or not people should start converting all their RAW files to DNG (Even if they embed the original file in it's entirety...) Personally I see no reason why any EXIF info full stop should be 'private', EXIF is there to record useful information and if we can't read it what use does it serve? If EXIF information can't be written or copied without risk then the EXIF standard needs tightening up IMO as well.

With RAW data from the camera I can see more need for a maker-info type area as I don't think it is unreasonable for camera settings that affect in-camera processing to be recorded. This allows the camera makers software to use those settings to produce compression free pictures that look the same as in camera JPEGs if you like that 'look'. Also many of their settings are worthless to any other software, as for example, what exactly does sharpness -1 mean outside of the camera?

This is my view of where Nikon is coming from. Many reviews I've seen of the D2X for example have commented on its colour rendition ability. One can only assume that people have largely used NC to get the results or are looking at JPEG's straight out the camera. Now if we take ACR as the most relevant example, how it initially displays a file on-screen is quite different to the manufacturer view. On my (Canon 20D) camera I'd say that the ACR look was quite a neutral one and that the default Canon 'look' was more vivid and a little smoother.

I think what Nikon are trying to say is that 'Our Look' is a result of hard work in terms of in-camera auto white balance and our processing algorithms, and that they want to preserve that look for people who want to use their software. If, even with the AWB info, ACR still displays a quite different looking file on screen, is the Nikon derived AWB figure really that important? IF you want the Nikon look...

Whilst a RAW converter isn't a direct equivalent to a film emulsion, it can have a huge difference on how the image looks. The combination of digital sensor and RAW converter is almost like making your film choice. Velvia wasn't popular for it's blindingly accurate colours, but Fujifilm aren't going to tell eveyone else how to get Velvia colours. I don't see this as being a particular problem for the 3rd party converters as the better ones provide calibration capabilities anyway, so surely getting a consistent look that you like is possible. Also, as a user it's actually nice to know that you can use different methods to get different looks, much like you could buy different film to do the same thing.

You can argue over whether or not Nikons decision to sell their converter separately is a good or bad one, as long as Canon continue to give away their RAW processor it doesn't look good. In fact Canon give away two ways to convert your file, the ZoomBrowser/FVU/EVU method or the DPP method, each giving quite different results. No-one seems to be complaining that Canon's DPP 'faithful' colour tone isn't published, faithful to what exactly?? I quite like it, but if I want that look I just use DPP... perhaps that is ultimately Nikon's problem. People want the Nikon look but don't see why they should have to pay extra to process the files outside the camera...

None of which does anything to address the patchy implementation of post RAW formats such as TIFF in a way that gives us confidence that once we've converted our image we can store it properly going forwards, or points to a clear format for archiving processed files...

Ian.

[u56]

I know that some well-known photographers have decided to keep every photo in at least four copies: the RAW-file, a back-up copy of the RAW-file, a DNG-copy and a back-up copy of the DNG-copy. To me, this seems like a lot of disk space and a lot of work to keep updated.

So I wonder: What is the main objection to converting every "incoming" photo to DNG and just save that and a back-up copy of it? I.e. no RAW-files saved!

I am already experiencing the problem of having older RAW-files from cameras no longer sold and it seems that I will encounter problemes reading those files sooner rather than later. Using DNG across the board would at least take away the issue of using different cameras.

If I have to choose between RAW+copy and DNG+copy, what would you recommend? And why?