Page 1 of 5

Over Christmas 2008 I found myself in a somewhat unique situation. I had recently put together a Sony A900 camera system for long term testing, and had also received Canon 5D MKII and Nikon D3x cameras from their respective manufacturers. Both were on one month loans so that I could take them on the Antarctic Expedition / Workshop that I would be leading during the first three weeks of January.

As these three cameras started to ship during November and December there was a lot of sturm und drang online over the image quality (IQ) and high ISO noise characteristics of each of these cameras. Whose was better? Whose was worse? Would the good guys win – whoever the good guys happened to be?

What Are We Measuring?

This comparison is of sensor noise. Overall image quality is comprised of several factors, including sensitivity, dynamic range, noise, and colour fidelity. This test is of noise only.

Dynamic range and colour sensitivity are beyond the ability of someone without a lab to measure reliably, but noise is readily tested and compared. That’s what this test attempts to do – not to produce any quantitative results, but rather simply to do the type of comparison that any moderately knowledable photographer might do given the availability of these three cameras at the same time and in the same place.

The Perils of Comparison Testing

The challenge when doing any sort of photographic equipment testing is in trying to manage all of the variables. It’s bad enough when you’re just doing this to please yourself, but when a million or so other people are going to scrutinize the results things gets a bit daunting. This is doubly so because there are so many self-proclaimed experts online who like nothing better than todisswhatever someone else does, regardless of whether there are substantive issues with the experiment or not. I suppose this is a form of peer review (under the circumstances – pun intended).

Comparing different camera’s raw files presents a certain level of difficulty. One must of course use a raw converter. But, which one? Some people say that it’s only fair to use the manufacturer’s software, but this isn’t always the best choice. In the case of the Sony A900, for example, their provided raw converter is simply awful, and in the case of Nikon does one use the simple converter provided with the camera or their optional and pricey NX2?.

How about a popular standard like Camera Raw or Lightroom? Well, some people like them and some don’t. Aperture? A fine product, but Mac only, so 75% of the audience is going to feel like they can’t emulate the results without buying a new computer –which they should, but that’s another story.

In the end I decided to do all of the conversions in Camera Raw.No, as of late December Camera Raw does not officially support the D3x, but the engineers at Adobe were kind enough to provide me with a beta release that did. This made it the only available mainstream program which was able to process all three camera’s raw files.

Would I have gotten better (or at least different) results using different programs, ones which I or someone else had determined were the “best” at processing each particular raw file? Yes, probably. But multiply the 10 different raw processors that I have available by three different cameras and at least 7 different ISO’s and the permutations and combinations become astronomical. So I took the easy way out and used Photoshop CS4’s Camera Raw (beta version) for all three cameras and all ISO’s. Please feel free to do your own comparisons any way thatyouwish.

Figure 1

Since I was working at home rather than the studio during the holiday season I used natural daylight.
The set-up was intended to show a high contrast scene, but one with
indirectly lit mid tones, yet also blown highlights and hard shadows.

Oh yes – something sparkly and a standard colour reference were included as well
not to mention a miniature paparazzi for quasi skin tone .

Methodology

To my knowledge there is no one lens that can be mounted on all three cameras, and even if there were, what would that tell us? Fortunately I had matching lenses, and decided to use these. These were the Sony / Zeiss Vario-Sonnar 24-70mm f/2.8, the Canon 24-70mm f/2.8L and the Nikon 24-70mm f/2.8G.

The cameras were, of course, tripod mounted with both mirror lock-up and self timer activation. All in-camera noise reduction was turned off. Center point autofocus was activated for each frame and was centered on the brown square in the upper left corner of the mini-Macbeth Color Checker.

Each camera was set to Aperture priority mode at an aperture of f/8. Auto White Balance was used, but subsequently in Camera Raw the white point was set for all images on the third from left white square on the chart.

At first I stumbled over the fact that the three cameras metered the scene quite differently, producing quite dissimilar results. I ended up setting the cameras to the same optimized exposure, which lead to all frames in the test being exposed at the same shutter speed and F stop for each ISO.

I did screw up in one area though. Somehow, though it was my intention to have all three lenses at 70mm, the shots done with the Sony were at 60mm. Mea Culpa, though I don’t think this affects the results in any significant way since it has no effect on the evaluation of noise, which is the primary purpose of this comparison.

Also please note that a couple of the Nikon 100% crops seem a bit soft compared to others by the same camera, and also against the other two contenders. The only thing I can think of is that this is vibration caused my having to touch the camera when releasing the shutter manually, because on the Nikon MLU and self timer can’t be set and used at the same time –an old Nikon design complaint. In any event, this doesn’t affect noise analysis.

One more thing – all files were adjusted in Lightroom for optimum screen and print rendition. This includes sharpening. It was my intention to see how these three cameras compared in prints, and I don’t make (waste) 24X30″ prints by printing them unprocessed and unsharpened. Since I can’t show you the prints the closest that I can do is to share 100% crops on-screen.

The Comparisons

At this point you might wish to have a look at the various comparison files. This report is broken into five separate pages for viewing convenience.

–Sony A900 ISO Comparisons

–Canon 5D MKII ISO Comparisons

–Nikon D3x ISO Comparisons

–Three Camera Noise Comparisons

When you’re done pixel peeping please come back here to read my interpretation and conclusions.

Conclusion

Assuming that you’ve now pixel peeped to your heart’s content, especially the comparison page where all three cameras are shown at each matching ISO, it’s time for me to tell you my evaluation. It’s short and simple.

I see no significant noise difference between the Nikon D3x, Canon 5D MKII and Sony A900 up to and including ISO 800.

At ISO 1600 and higher the Sony falls behind the Canon and Nikon by about 1 EV. It’s simply a noisier camera at high ISO.

From ISO 1600 to ISO 6400 the Canon 5D MKII and the Nikon D3x are neck and neck. I think that the visible differences on-screen at I00% are a quibble, and on prints are completely irrelevant.

The Canon then goes the equivalent of two stops higher. But I see both ISO 12,500 and 25,600 on the Canon as being marginally usable at best, and then only for emergency documentation. My feeling is that they were added for marketing rather than for real world shooting purposes. “The D3 has ISO 25,000 so we have to as well, regardless of how it looks”.

So to repeat – one can quibble over the differences at 100% on screen, but in 24″ X 30″ or smaller prints these cameras are essentially indistinguishable in terms of comparative noise, up to ISO 1600 in the case of the Sony, and to 6400 in the case of the Nikon and Canon.

Other factors such as dynamic range and colour rendition are not being evaluated here, but I doubt that there is more than a half stop difference between them in terms of DR, and now with the availability of proper profiles and the profile editor in Camera Raw and Lightroom colour rendition differences between cameras are academic because these can be modified and made to match the manufacturer’s rendering, or anything else that pleases you.

As for how the Canon and Nikon comparisons would turn out if I had used other raw processors – all I can say is that I also did a small number of raw conversions of each of the camera’s files in their respective manufacturer’s software, and for the purposes of this test saw no significant differences that would have caused me to prefer something other than Adobe Camera Raw. If I was a real nit picker I would have done exhaustive comparisons of each camera at each ISO with several different processors and chosen the best one.In truth, I’m a pragmatist, so what you see is what you get. I believe though that what is shown here replicates what most people will do in the real world.

Summary

Though some fans of each of these cameras may turn out to be disappointed by these results, they shouldn’t be. All three cameras are remarkable.

The Sony is a very fine camera and offers exceptional value for the money, but it is known to be a noisy camera. Now we know how it compares to its primary competitors. As Sony did with the A700, which also had high ISO issues when first released, they may well offer a firmware upgrade which addresses this. In the meantime, used below ISO 1600 exceptional results can be achieved with the A900, fully comparable to the D3x or 5D MKII.

Some A900 users (myself included) have stated that Camera Raw and Lightroom do not do as good a job on high ISO Sony files as does Aperture or Capture One. After doing extensive comparisons for this test I have decided that when luminance and smoothing and chroma noise reduction are done properly almost equivalent noise characteristics can be achieved in most of these programs. It just take a bit of work.

The extent to which the Nikon D3x and Canon 5D MKII are equivalent in noise (or its lack) is almost uncanny. There simply isn’t anything to choose between them in this regard. Yes, the Canon goes to ISO 25,600, but that’s just for bragging rights and is unlikely to be used in the real world for anything other than experimentation or forensics.

The Nikon D3x is in a class by itself – the most full featured, the most robust, and need I add, the most expensive of the three by a factor of nearly 3X. The Canon 5D MKII is probably the all-around champ, offering a combination of image quality, features and especially price that is very hard to top. On the other hand, the Sony is comparably priced, has the best viewfinder, in-body stabilization, and access to some fantastic Zeiss lenses. Decisions, decisions.

Some readers may well ask – what about the Canon 1Ds MKIII? True – it matches the Nikon D3x in terms of build quality and ruggedness, and like the 5D MKII is a 21MP camera. Like the D3x it also is at the high end of the price spectrum.

All I can say is that based on more than a year of regular use, with the exception of very high ISO capability I would expect the results to be very similar in a noise test like this to that of the 5D MKII.

In the case of all three cameras what we come down to is that the real differences under typical shooting conditions are to be found in areas other than sensor technology. Handling, ergonomics, features, lens choices, shooting speed, price… and most of all individual user preferences are more germane factors than pixel peeping nit picking.

So – I’m done. I’m now ready to take all three of these cameras to Antarctica in January, 2009 and to have fun shooting with them. Other than at ISO 1600 and above, where I’ll leave the Sony in my cabin, I’ll shoot with all three cameras, choosing which one to usebasednoton presumptions about image quality, but having proven, to myself at least, that these amazing cameras are pretty much equivalent in image quality in typical use.

Other Pages

Other related pages are…

–Introduction, Methodology, and Analysis:Page 1 of 5 (this page)

–Sony A900 ISO Comparisons:Page 2 of 5

–Canon 5D MKII ISO Comparisons:Page 3 of 5

–Nikon D3x ISO Comparisons:Page 4 of 5

–Three Camera Noise Comparisons:Page 5 of 5

December, 2008