Introduction and Results Tutorial
This tutorial is based on a pre-release
version of DxO Analyzer.
Therefore, test methodologies may be altered prior to commercial release.
These tests will also include several customized charts in order to present complex data,
and I may decide to change these over time as my experience with this product grows.
How do a lens' or camera's optical capabilities affect image quality? Why spend $1,000 for a lens when there's one of similar focal length and aperture available for half the price? And what about digital sensors? These have optical characteristics as well, and when combined with any given lens together they produce a unique result.
Historically there have been three primary ways of measuring a lens' optical quality — the manufacturer's MTF chart, shooting a resolution chart, and using an optical bench. Each has its place, but none is the ideal solution. MTF charts are produced with sophisticated equipment and are usually only available from the lens manufacturer themselves. Also, MTF charts from different companies can't be compared because there is no standardized presentation format. Some manufacturers' MTF charts are theoretical for a particular lens design while others are based on real-world measurements. They're also difficult for the layman to understand.
Shooting a resolution chart, which many independent testers do, is fine as far as it goes — which isn't very far. All it tells you is about resolution. There's no information from such a test regarding vignetting, aberration, distortion, blur or any other lens characteristic other than resolution. And even then, measuring lines/mm isn't the complete story. Far from it.
It is also worth keeping in mind the difference between the concept of resolution — the smallest detail that gets imaged, and sharpness — how crisply image features are reproduced.
The use of an optical bench (expensive and difficult to use properly) tells a lot more, but due to price is limited in application. It also can only be used with a lens by itself. It can't be used to measure a digital camera and its sensor in combination with a given lens. And since a sensor and lens form a complete optical system, such test benches are now of declining usefulness.
But photographers as well as nature abhors a vacuum, and into this space we now have a new testing system — DxO Analyzer from DO Labs in France.
Canon EOS 1Ds with 135mm f/2.0L lens @ ISO 100
In mid-February, 2004 DO Labs announced at the PMA trade show in Las Vegas a new optical testing system designed specifically for measuring lenses and digital cameras. DxO Analyzer carries an annual license fee of approximately U.S. $20,000 / year. That's the bad news. The good news is that DO Labs is making it available to selected journalists, reviewers and magazines at a discounted price. The method to their madness is that as the acceptance of DxO Analyzer's test methodology grows and becomes something of an industry standard, lens and digital camera makers will wish to adopt this test suite themselves. Of course DO Labs' business plan is their affair, but the beneficiaries of this program are us — photographers — who now have a uniform, reliable and consistent test system. I anticipate that many major photographic magazines as well as web sites will adopt this system. Even at this early stage, the respected French magazine Chasseur d'Image and the Imaging Resource web site have adopted DxO Analyzer, as of course so has The Luminous Landscape. (A press release regarding the introduction of DxO Analyzer can be found here).
As to who DO Labs is — they are an applied mathematics and imaging software company. Their staff has quite a few PhDs and top-end mathematicians, and DxO Analyzer represents a great deal of effort by some of the best minds in the field. By way of full disclosure, I have been involved with Do Labs for about 5 months prior to the announcement and introduction of the system, having acted as something of a consultant to the company as well as an alpha and beta tester of the system. Am I biased? Yup. Very.
How it Works
The DxO Analyzer system consists of two components. The first is a large sheet of heavy plate glass onto which has been applied a series of precision dots. This is set up so that the camera / lens system is absolutely plane parallel to it and the chart fills the field of view. The target also has to be lit evenly to within one tenth of an F stop. The second part of the system is analysis software which runs on a Windows XP system.
To test a lens / camera one shoots the target at each aperture desired, and in the case of a zoom lens, at each focal length desired. Tests at every ISO setting are also needed when testing cameras. Thus even an evaluation at just 4 apertures, 3 sample focal lengths, and 4 ISO settings requires conducting 48 seperate exposures and measurements.
The images taken are then loaded into the software and individual tests are run. The output is a large series of charts and data tables which describe a range of optical characteristics, which are detailed and explained below. These can also be exported to HTML files as well as collated in an Excel spreadsheet.
Of course a natural question is — what about human error? Will two testers produce different results? The answer is that this is highly unlikely. The DxO software simply refuses to process a test file which hasn't been properly produced. If the lighting isn't completely uniform, or the target doesn't properly fill the frame, or the target and the lens aren't plane parallel, the software lets you know. It couldn't be more foolproof.
How consistent are the results? I am told that tests are repeatable within .02%. That's a level of precision that hasn't been seen before from an optical test system.
Before diving into a description and explanation of each of the tests, a few words are needed to explain the system's limitations as well as some concerns about this type of testing in general.
— DxO Analyzer can only be used to test lenses in combination with a digital sensor. A lens by itself can not be tested, and a lens in combination with a film camera can not be tested.
— In the case of DSLRs and interchangeable lenses, measurements are only meaningful for a specific type of combination. For example, a 135mm lens on a full-frame Canon 1Ds will measure differently than on a reduced frame Canon 10D. This of course is because much more of the lens is being used by the full-frame camera and therefore most measurements will differ due to the difference in coverage.
— Identical lenses, aren't. No two cameras or lenses are the same. Even ones off the same assembly line. It isn't unusual to find significant variance, and indeed some pros typically test a number of lenses of the same brand and type before choosing one.
— Only test results of lenses of similar focal lengths should be compared with each other. Long lenses are always going to perform as well as measure better than wide-angle lenses, regardless of brand or price.
— Don't compare zooms and prime lenses at the same focal length. With very few exceptions prime lenses will always be superior to zooms. The trade-off is convenience.
— Measurements don't tell the whole story. One camera / lens combination may perform better optically than another, yet because of design, handling and other non-measurable performance differences be less desirable for actually producing photographs than one that measures better.
— Small differences may not be important. Learn to read the results properly. Just because there is a small measurable difference between one system and another doesn't mean that this difference is necessarily visible on a print — regardless of size.
— Please don't ask me to perform measurements on any specific camera or lens. I will be using DxO Analyzer to test new lenses and digital cameras that come my way, beginning as of late February, 2004. I will also over time test all of the digital cameras and lenses that I already have available to me. But, I am not in the testing business, and don't intend to provide this capability as a service to others, other than as part of my usual reviews. There's a great deal of work involved in conducting these tests, and many hours go into producing, analyzing, organizing and publishing the results.
— The lens used in the tutorial below is a Canon 135mm f/2L used on a Canon 1Ds. I chose this lens because it is one of Canon's (and for that matter anyone's) best lenses, and therefore gives something of a benchmark to compare other lenses to. I used the Canon 1Ds because at the time of this writing (February, 2004) this is one of the highest resolution full-frame digital cameras on the market. During the past year or so it has become something of a benchmark itself for image quality, and it is in use by large numbers of professional photographers worldwide.
The Tests Explained
DxO Analyzer produces five separate measurements in four catagories. These are...
1: Distortion and Chromatic Aberration
Below we will look at each of these tests in turn and explain what it is that's being measured. All tests in the examples below were of a Canon 135mm f/2L lens used on a Canon 1Ds camera, taken at ISO 100, and with an aperture of f/5.6. A full test suite on a given lens / camera combination would show several different apetures, and in the case of zoom lenses a selection of focal lengths as well. Remember — this page is a tutorial, not a test report on this particular camera and lens combination. These are shown here simply for the purpose of illustration.
Distortion and Chromatic Aberration
In the green text chart above, under the Distortion heading, you will see a Maximum distortion reading of -0.19%. Because the number is negative it means that the distortion that this lens displays is of the pincushion variety. If the number had been positive the distortion will be of the barrel variety. The chart also shows that this lens displays an average distortion level of 0.09%.
The grid displayed to the right of the green text chart is a 5X magnification of the distortion displayed. This magnification is done to help us visualize the degree of distortion, and is there for visual reference purposes only. A 1:1 grid would not show any visible distortion except in the worst cases.
The way distortion is measured is relative to a straight line across the image. Points at either end are measured and then the worst point on that line is taken and a percentage variance is measured. In this test example the lens shows a worst case distortion of the pincushion type of less than two tenths of one percent, and an average across the lens' field of coverage of seven hundredths of one percent. Very small indeed.
The next four boxes; the yellow box, the blue / magenta coloured box, and the last two graphs, are used to describe Lateral Chromatic Aberration, usually referred to as CA.
Lateral CA is a measurement of the extent to which an optical system records all three primary colours (Red / Blue / Green) in exact registration with each other. If a lens is less than prefect (and likely no lens is perfect) then one can see colour fringing, especially around specular highlights and along the edges of strong brightness transitions.
On the green coloured text chart you'll see that this example lens shows a maximum of .32 pixels of CA, and an average across the lens of .20 pixels between any of the three primary colours.
The second set of numbers, in this case a Maximum of 0.07 / 1000 and an Average of 0.04 / 1000 is intended to help in comparing performance between different cameras that have different pixel sizes. In math-speak, the numbers have been normalized. It is these numbers that should be used when comparing cameras of differing resolutions.
The final set of numbers have been normalized to be the equivalent to a 35mm transparency by computing from pixels to millimeters. In this case since we're dealing with a full-frame sensor there is no difference.
The yellow chart is intended to show CA on a scale of 0 to 5. If the image is yellow, CA is low, and if it's dark red then CA is closer to 5. Since the average CA of this sample lens is just -0.19 pixels the chart is obviously uniformly yellow.
The Relative chart beside it shows how CA manifests itself for a particular lens. The dark blue and magenta areas show the relative performance of the lens between the maximum and minimum CA performance. In other words, the dark blue area is the area of maximum CA, while the magenta area shows the areas of minimum CA. This should not be used to compare lenses, since the colour differential will always be the same whether the worst case is 0.1 pixel widths of CA or 5 pixel widths of CA. It should be thought of as more like a CA histogram, because it shows relative not absolute values.
The last two graphs show the CA spread across the radius of the lens' coverage. The vertical scale is in pixels and the horizontal scale is in percentage distance from the center of the lens to the corner. The first chart shows the number of pixels by which the red plane varies from the green plane, and in the second chart we see how the blue pixel plane varies from the green pixel plane.
When comparing graphs between lenses please note any differences in the scaling, because DxO Analyzer automatically scales these graphs for maximum readability. Even the two graphs above for the same lens have different scales.
The above chart is not generated by DxO Analyzer, but rather is generated in Excel from the comprehensive chromatic aberration data. It shows measurements taken at three focal lengths, and at selected apertures. The vertical axis shows the average CA in pixel widths. Please note that this is the Average CA, not the maximum values that are reported in the other charts. (Also note that this chart does not relate to the example camera and lens shown in other examples on this page, but rather a digicam with built-in zoom lens).
The propose of the chart is to provide a simple visual presentation of a complex series of numbers. In this sample it makes instantly clear that at medium focal lengths this particular camera / lens combination has much lower Average CA than it does at either the wide angle or telephoto parts of its focal range.
The blue text chart for Vignetting shows that this lens has a maximum attenuation of 0.20 IL. IL stands for Illumination Index and is equal to one F stop. So this lens shows a maximum fall-off of two tenths of a stop. This is also shown as a percentage. In this case it is 13.09%.
It needs to be noted that vignetting is very much dependant on light level. Lenses will show illumination fall-off to a greater extent at low light levels than at high light levels. Since my tests are done in a studio under uniform and consistent lighting conditions these test results are valid between samples. But if another tester using this system does his or her measurements at dramatically different brightness levels (highly unlikely though), results may not be comparable.
The next chart shows a gray scale rendition of the pattern of vignetting that the lens produces. This is a highly exaggerated rendition so as to be visible here on-screen, and since each one will be manually adjusted for optimum visibility these charts should not be used to make comparisons between lenses. Use the numbers, not the chart.
The dot on the chart is the geometric center of the lens, while the cross shows the center of the vignetting. Note that this is not the point of least vignetting, but rather the center of the vignetting phenomena. The dX and dY numbers measure the percentage shift between the optical center of the lens and the Vignetting center, measured on both the X (horizontal) and the Y (vertical) axis.
The final chart is a Normalized Attenuation Profile. The horizontal axis is measured radially in pixels from the center of the image to the corner, and it simply shows in graphic form how smooth the fall-off of the lens is.
This text chart shows noise measurements for the Y — Luminance, R — Red, G — Green and B — Blue channels. Obviously this measurement pertains to digital sensors rather than lenses, but since with DxO Analyzer we are in fact measuring a complete optical system of both lens and sensor it is a highly relevant measurement. For those that are interested, the Luminance value in these tests is based on ratios of R=0.299, G=0.587, B=0.114.
GL refers to Gray Level. Obviously noise varies based on the relative brightness of the subject being measured. Bright areas will be less noisy than dark areas. This subject is explored in my essay titled Expose to The Right. In this example the image is being measured at an average GL of 164.5, which of course is on an 8 bit scale of 0-255, with 0 being black and 255 being white.
The Signal to Noise Ratio (SB) at this brightness level for this camera is 40db in the Luminance channel, and you can also see the SB for the separate R, G and B channels. A Standard Deviation of +/- .2 is shown, which tells us that this signal to noise ratio is found consistently within this brightness range.
NB: The current pre-release version of DxO Analyzer (Feb, 2004) basis its S/N readings on a single average Gray Level. Since this is difficult to accurately match between test cameras it is not yet appropriate to compare S/N readings between camera systems. It also does not provide sufficient information about how a particular camera's sensor responds to noise as the exposure value is varied.
The measurements up until now have been fairly standard and familiar. But with the introduction of DxO Analyzer, DO Labs has introduced a new unit of measurement called the Blur Experience Unit, or BxU.
Once again the Y reading is for the Luminance channel, and the R, G and B results as well are shown for both center and corner. The particular corner whose result is shown in this table is the top right hand corner.
In this example the lens is showing a BxU measurement of 2.12 +/- 5% at the center. The BxU is a relative unit, and the numbers are additive. So, it's easy to see that there is nearly 1 BxU difference between the center reading and the upper right corner when this lens is measured at f/5.6. How much is that? Simple. A BxU has been defined as the equivalent of one "Blur More" function in Photoshop. So, it's easy to see for yourself how much this represents visually by simply loading an image into Photoshop and selecting Filter / Blur / Blur More. The change that you see is +1 BxU.
The BxU is precise, can be repeatably measured, and is independent of resolution. Another way to visualize it is that a perfect circle formed by a point of light would have a BxU of 0.
In all, 17 areas of the screen are measured, and as can be seen from the second chart these areas are identified with numbers for 1-17, and are found by spiraling from upper right to upper left, then down and around. Each square shows the BxU measurement in its area, and in the center square (#0) we have a measurement for the center of the image area. These numbers are the same as shown in the orange table above except that the rest of the image's measurements are also shown. This gives you a measure of how even any lenses' performance is at 17 locations within its field of coverage.
The above chart is not generated by DxO Analyzer, but rather is generated in Excel from the comprehensive blur data. It shows measurements taken at three focal lengths, and at selected apertures. The vertical axis shows the Blur in BxU units. (Please note that this chart does not relate to the example camera and lens shown in other examples on this page, but rather a digicam with built-in zoom lens).
The propose of the chart is to provide a simple visual presentation of a complex series of numbers. In this sample we see that this particular sample camera / lens combination has its best resolution at wide-angle settings. Medium focal lengths are not as good, and the telephoto end is worse, and also quite variable.
One important thing that needs to be be borne in mind is that since JPG images are used for these measurements the Blur measurement is susceptible to any in-camera sharpening. This is usually user selectable on digicams, though it may not be on some DSLRs. Too much in-camera sharpening and unrealistic measurements can be produced. The tested numbers will look better, but the image won't necessarily.
A future version of DxO Analyzer will work on RAW files, but for now this needs to be taken into consideration by the tester and the reader. But even if RAW files are used, this won't provide a complete solution because some cameras even sharpen their RAW files in-camera, with no user control or ability to turn it off. Canon has confirmed that they are one of the companies that do this.
The French magazine Chasseur d'Image, which has also adopted DxO Analyzer for their testing, has decided to test cameras using each camera's default sharpening setting, whatever that may be. On the other hand I have decided to set in-camera sharpening to the lowest possible level during testing. Therefore this is one measurement where a direct comparison of test results between test sites will not correlate.
There's obviously a lot here to digest. Take your time, review all of the examples, and then refer back to the lens or camera system test on this site that you're interested in. Be sure that you understand what the tests are measuring. Also, beware of some of the caveats above. Also remember that most optical tests will vary considerably with aperture. Lenses are softest and show lowest contrast wide open. They sharpen up and show reduced vignetting and CA as they are stopped down, but then other optical degradation such as diffraction effects start to become bothersome as apertures become smaller. Photography is about compromise.
All lenses and digital camera systems are compromises. One has to balance off price, against features, against performance. One may find an optically superior lens that has poor dust sealing, or that falls apart under hard knocks. A digital camera or back may have superb characteristics but cost as much as a car, putting them out of reach. Conversely, sometimes inexpensive systems can provide tremendous value for the money, showing measurable but not necessarily visible differences in image quality.
And, even the best performing systems when it comes to optical tests may be dogs when it comes to how well they handle in the field, or how well supported they are by the company that makes them. Don't use these test results as the sole arbiter of a camera or a lens' goodness. There's more to photography than lens tests!
Invariably there will be readers who find fault with these tests. Before putting pen to paper (or fingers to keyboard), consider the following...
— DxO Analyzer was created by
highly talented PhD-level mathematicians, optical experts and software engineers
over many months, in consultation with industry experts and national testing organizations.
— If you have the educational
background and subject expertise to differ with this system's test methodology
you may wish to contact the company with your comments. I'm sure that DO Labs will be pleased to
hear from you, and to answer any questions that you may have.
— While operator error is of
course possible, my experience shows that the software always
catches faulty testing procedures. Every test result is Quality Assessed by the software before results are generated.
— Results from DxO Analyzer
can not be directly compared to those produced by other testing systems or procedures.
Just as different manufacturer's MTF charts can not be compared due to differing test methodologies.
— These test results are self-consistant.
That is, the same test done of the same camera / lens,
when done repeatedly over time, produce nearly identical results.
— Comparisons with data produced
by other testers of similar equipment appears to yield
very similar, though not always identical results. This is likely due to sample variations between products.
At this time (February, 2004) DxO
Analyzer is in pre-release, and S/N measurements are based on the average luminance
level of the test image.
This means that it is not possible to accurately compare cameras whose test shots were made at different luminance levels.
A new test methodology that produces S/N measurements at a range of luminance levels is being developed.
— While the majority of the
charts produced in these reports are generated directly by DxO Analyzer,
and are reproduced here without alteration, some 2-D and 3-D Excel-based charts are generated by me from measurement data.
I take sole responsibility for any errors of omissions that may be found in these.
Do not confuse DxO Analyzer with DxO
The product explained on this page, Analyzer, is designed to measure optical performance,
and to produce test results from those measurements. Optics Pro is a product for photographers to use
that corrects various forms of optical defects found in digital cameras, and lenses.
DxO Analyzer Test Results