Ad
Ad
Ad
Pages: « 1 2 3 [4] 5 »   Bottom of Page
Print
Author Topic: 3 Feb, 2009 - Eyes vs. Numbers - Which to Believe  (Read 26492 times)
Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #60 on: February 07, 2009, 05:06:10 PM »
ReplyReply


Quote
Something I had on my mind was flare and ghosting, they are difficult to quantify because they are highly dependent on the shooting conditions.

Erik,
Difficult to quantify in the absence of an internationally accepted standard and method of testing, but not necessarily difficult to test in relation to a specific set-up of direct lighting that someone might devise for the purpose of comparison. As long as the methodology is consistent, then comparative tests for flare and ghosting should reveal that Lens A is better than lens B. However, any single numerical rating resulting from one particular method of testing could not be compared with another single numerical rating that is the result of a different methodology.

Quote
Another issue is that there is often some esoteric quality attributed to certain lenses like "3D-look", I don't know what that means. I have two Sony lenses with blue labels saying Zeiss and I would say that they perform better than they test, at least in my own tests.

But we don't seem to have any DXO type tests of lenses available. DXO obviously must be making extremely thorough and detailed tests of the lenses they include as modules in their raw converter, otherwise their converter wouldn't be able to automatically correct for the lens deficiencies. But they don't make these tests public.

The MTF charts that are currently available for the lens itself seem to be mostly based on the theoretical performance of the lens design. Photozone's 'system resolution' tests seem to be the most informative tests currently available, but one always has to make a guess as to the role of the camera used.

Quote
Another such aspect is that knowledgeable and experienced people, like Michael Reichmann, say that the advantage of MFDBs is visible on relative small prints.

Michael has also said the opposite. A group of experienced photographers could not tell the difference between a G10 image and a P45+ image of the same scene, at A3+ size. My own view is that the nature of the subject being photographed and compared will often determine the differences that are noticeable.

Smooth sheets of ice in the Antartic might reveal very clearly the smoother tonality of the P45+ compared with the G10 at A3+ size.

Quote
There are a few comments on this forum essentially claiming that DxO measurements don't have results that fits expectations or experience and therefore they must be trash, In my opinion it's better to discuss what DxO measures and what is does not.

More than a few. There's almost hysteria   .
Logged
Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #61 on: February 07, 2009, 05:46:38 PM »
ReplyReply


Quote
Measure noise, and it might be ugly noise, or it might be a nicer looking noise. A single number doesn't tell you anything about the character of the noise. Given the prevalence of noise reduction software, how do you account for some types of noise being easier to remove than others in a single number?

Graeme,
I don't think that anyone is claiming that DXO type tests are a measurement of esthetics. Ugliness and beauty are very subjective matters. However, having identified the characteristics of noise which some people find ugly or beautiful, as the case may be, then I see no reason why it could not be tested and measured. DPP for example seems to produce a slightly finer type of noise than ACR which is noticeably blotchy by comparison, at the extreme pixel-peeping level. That difference didn't just happen by accident. It's a purposeful effect, or an unavoidable consequence of some other design feature of the raw converter.

In any case, DXO measurements are made at an earlier stage to the peculiar characteristic of a specific raw converter, aren't they?

Surely the goal is to have as little noise as possible. If you then want to add some for esthetic reasons, then that's possible, isn't it?
Logged
ErikKaffehr
Sr. Member
****
Online Online

Posts: 8031


WWW
« Reply #62 on: February 07, 2009, 07:56:03 PM »
ReplyReply

Ray,

I have a couple of issues with Photozone. The major one is that they compensate for field curvature, don't use the same focus for the centers, edges and corners. The other issue I have is that all their tests ae on APS-C or 4/3. Nothing wrong about that, except that the tests say nothing abut edge performance on full frame. Photozone testing is based on Imatest, the same program I use.

There are other lens tests on the net, SLR-gear tests are based on DxO technology and DPReview tests are based on "Slanted Edge", like Imatest. Both have relatively few tests.

My favorite tests are done by the Swedish monthly "Foto". They test at Hasselblad using their lens testing equipment. With Foto's tests I also have some issues, they are free for subscribers but others need to pay. So I guess it's hard to refer to those tests on the net. The other issue I have that they publish MTF for 20 LP/mm for full frame lenses, 30 for APS-C and 40 for 4/3. I would prefer the traditional 10/20/40 possibly completed by 60.

One great advantage of the "Foto" tests is that they match my experience.

A very nice aspect of the of the Photozone tests is that they also report on Bokeh and Longitudal Chromatic Aberration.

Best regards
Erik

Quote from: Ray
The MTF charts that are currently available for the lens itself seem to be mostly based on the theoretical performance of the lens design. Photozone's 'system resolution' tests seem to be the most informative tests currently available, but one always has to make a guess as to the role of the camera used.
Logged

Graeme Nattress
Sr. Member
****
Offline Offline

Posts: 582



WWW
« Reply #63 on: February 07, 2009, 08:46:27 PM »
ReplyReply

The demosaic can impart a particular character to the noise, and indeed, it does seem that DXO try to eliminate that. However, the noise is in the underlying data and will have a particular pattern or character to it. A PSNR measurement does not tell you that character though. Sure you could invent some measurement that perhaps tells you more, but that is tricky. Noise could be a mix of shot noise, read noise and fixed pattern noise. They all look different. A noise number just totals them up. On the whole, fpn is much more objectionable than the other as it makes for visible patterns in the extreme noise floor.

Given you'll always have noise, the precise character of that noise is very important.

Graeme

Quote from: Ray
Graeme,
I don't think that anyone is claiming that DXO type tests are a measurement of esthetics. Ugliness and beauty are very subjective matters. However, having identified the characteristics of noise which some people find ugly or beautiful, as the case may be, then I see no reason why it could not be tested and measured. DPP for example seems to produce a slightly finer type of noise than ACR which is noticeably blotchy by comparison, at the extreme pixel-peeping level. That difference didn't just happen by accident. It's a purposeful effect, or an unavoidable consequence of some other design feature of the raw converter.

In any case, DXO measurements are made at an earlier stage to the peculiar characteristic of a specific raw converter, aren't they?

Surely the goal is to have as little noise as possible. If you then want to add some for esthetic reasons, then that's possible, isn't it?
Logged

www.nattress.com - Plugins for Final Cut Pro and Color
www.red.com - Digital Cinema Cameras
Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #64 on: February 07, 2009, 11:03:45 PM »
ReplyReply

Quote from: Graeme Nattress
The demosaic can impart a particular character to the noise, and indeed, it does seem that DXO try to eliminate that. However, the noise is in the underlying data and will have a particular pattern or character to it. A PSNR measurement does not tell you that character though. Sure you could invent some measurement that perhaps tells you more, but that is tricky. Noise could be a mix of shot noise, read noise and fixed pattern noise. They all look different. A noise number just totals them up. On the whole, fpn is much more objectionable than the other as it makes for visible patterns in the extreme noise floor.

Given you'll always have noise, the precise character of that noise is very important.

Graeme

It would be an interesting exercise if someone could find a couple of cameras that DXO have tested, of similar pixel count and similar SNR ratings at a particular ISO, and show us some 100% crops demonstrating the ugliness of one camera's noise compared with the beauty of the other.

What you write sounds fine in theory, but in practice it might be of little significance outside of extreme pixel-peeping.

I wasn't aware that one could do anything about shot noise, so surely that should be out of the discussion. I recall that one of the attractive features of RSP, which I frequently used as my raw converter before Adobe absorbed them, was a slider for removing fixed pattern noise. I recall that it was very effective, but not without the usual penalty of noticeable loss of resolution.
Logged
Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #65 on: February 08, 2009, 12:05:40 AM »
ReplyReply

Quote from: ErikKaffehr
I have a couple of issues with Photozone. The major one is that they compensate for field curvature, don't use the same focus for the centers, edges and corners. The other issue I have is that all their tests ae on APS-C or 4/3. Nothing wrong about that, except that the tests say nothing abut edge performance on full frame.

Erik,
I wasn't aware that Photozone used different focussing for the centre and edges. I wonder if that has a practical significance that can be misleading, perhaps photographing brick walls from a close distance and at full aperture, for example.

The reason I have a high regard for the old Photodo tests as well as the newer Photozone tests is that whenever I've taken the trouble to test my own lenses that feature in either of those two bodies of tests, the results seem very similar to either Photodo's or Photozone's  tests in respect of resolution at the centre and edges and at the apertures they've tested.

A couple of examples would be:

(1) the relatively poor performance of the Canon 100-400 IS at 400mm and F5.6. Both Photodo and Photozone show that resolution is noticeably better at F8 and F11, and so it is with my copy of the lens.

(2) the stellar performance of the Canon EF-S 17-55/2.8. This lens did not exist when Photodo was producing MTF charts, but Photozone tests show this lens as being as sharp as the Canon 50/1.8 prime at F2.8, but as one would expect, not as sharp at the edges because the 50/1.8 prime is designed for full frame and the EF-S is designed for the cropped format.
 
My own comparisons with the 50/1.8 prime confirmed this result, but not initially. Focussing at infinity, the 50/1.8 prime at F2.8 looked noticeably sharper, causing me some initial disappointment. Perhaps I'd been unlucky and had bought a substandard copy of the 17-55. I tried again but this time using Live View on the 40D and focussing manually. Bingo! The results were almost exactly in line with Photozone's tests, or as close as matters. I sent the lens to Canon Repair for calibration under the warranty.

The fact that Photozone test their lenses on the APS-C format is not such a big disadvantage. I don't recall ever seeing an MTF chart that had a sloping response out to 12mm from the centre then suddenly a flat response from 12mm to the corners. If a lens has an excellent performance at the borders on an APS-C camera, it's likely to have at least a good performance at the borders on full frame. Likewise, a lens that has a noticeably poor performance at the borders on APS-C, is likely to be very poor at the borders on full frame, wouldn't you say?  

Logged
Quentin
Sr. Member
****
Offline Offline

Posts: 1123



WWW
« Reply #66 on: February 08, 2009, 06:23:45 AM »
ReplyReply

Isn't this really all about sacred cows and the fact we don't like having ours slaughtered?   DxO are guilty of the sin of independence from subjectivity.  Surely they should have realised that Photography is all about brand loyalty and preconceptions based on price, not real world performance    

Quentin
Logged

Quentin Bargate, ARPS, Author, photographer entrepreneur and senior partner of Bargate Murray, Law Firm of the Year 2013
ErikKaffehr
Sr. Member
****
Online Online

Posts: 8031


WWW
« Reply #67 on: February 08, 2009, 06:48:06 AM »
ReplyReply

Ray,

To begin with I have much respect for Photozone tests. Some lenses, like the Tamron 17-50/2.8, have significant field curvature. That means if you are focusing at the center the corners will not be sharp. The 17-50/2.8 I have is absolutely lousy in corners at full aperture at 17mm.

A few aberration increase with the square of the distance from the center, so sharpness can fall of rapidly. Many lenses have a "sweet spot" which often can cover the same area as APS-C, for these two reasons I don't feel that testing on APS-C gives any good information on border/corner performance when used on full frame.

Regarding the "old photodo tests", I agree fully. The present tests in "Foto" are essentially the continuance of those tests.

Lars Kjellberg was technical editor at "Aktuell Fotografi" and initiated those tests, after "Aktuell Fotografi" merged with "Foto" Lars Kjellberg started the original Photodo which originally was a publication on paper (bimonthly I think).

Best regards
Erik

Quote from: Ray
Erik,
I wasn't aware that Photozone used different focussing for the centre and edges. I wonder if that has a practical significance that can be misleading, perhaps photographing brick walls from a close distance and at full aperture, for example.

The reason I have a high regard for the old Photodo tests as well as the newer Photozone tests is that whenever I've taken the trouble to test my own lenses that feature in either of those two bodies of tests, the results seem very similar to either Photodo's or Photozone's  tests in respect of resolution at the centre and edges and at the apertures they've tested.

A couple of examples would be:

(1) the relatively poor performance of the Canon 100-400 IS at 400mm and F5.6. Both Photodo and Photozone show that resolution is noticeably better at F8 and F11, and so it is with my copy of the lens.

(2) the stellar performance of the Canon EF-S 17-55/2.8. This lens did not exist when Photodo was producing MTF charts, but Photozone tests show this lens as being as sharp as the Canon 50/1.8 prime at F2.8, but as one would expect, not as sharp at the edges because the 50/1.8 prime is designed for full frame and the EF-S is designed for the cropped format.
 
My own comparisons with the 50/1.8 prime confirmed this result, but not initially. Focussing at infinity, the 50/1.8 prime at F2.8 looked noticeably sharper, causing me some initial disappointment. Perhaps I'd been unlucky and had bought a substandard copy of the 17-55. I tried again but this time using Live View on the 40D and focussing manually. Bingo! The results were almost exactly in line with Photozone's tests, or as close as matters. I sent the lens to Canon Repair for calibration under the warranty.

The fact that Photozone test their lenses on the APS-C format is not such a big disadvantage. I don't recall ever seeing an MTF chart that had a sloping response out to 12mm from the centre then suddenly a flat response from 12mm to the corners. If a lens has an excellent performance at the borders on an APS-C camera, it's likely to have at least a good performance at the borders on full frame. Likewise, a lens that has a noticeably poor performance at the borders on APS-C, is likely to be very poor at the borders on full frame, wouldn't you say?  
« Last Edit: February 08, 2009, 07:10:32 AM by ErikKaffehr » Logged

Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #68 on: February 08, 2009, 05:28:51 PM »
ReplyReply

Quote
To begin with I have much respect for Photozone tests. Some lenses, like the Tamron 17-50/2.8, have significant field curvature. That means if you are focusing at the center the corners will not be sharp. The 17-50/2.8 I have is absolutely lousy in corners at full aperture at 17mm.

Eric,
Having just looked at the PZ review of the Tamron 17-50/2.8, I see what you mean. The graphs do not give any indication of the problem of field curvature, but at least Photozone mention this problem and the fact that at 17mm and even at 24mm it's pretty extreme with this lens.

The Canon 17-55/2.8 on the other hand, has only slight field curvature at 17mm, which PZ also mentions. I never considered getting the Tamron 17-50/2.8, not because I was aware it had strong field curvature, but because third party lenses lack IS and this zoom is a walk-around lens for me.

I know that some folks like to criticise MTF charts on the basis they don't tell you everything about a lens. And it's true, they don't. But that's no reason for ignoring them. They tell you what they tell you and hopefully if there are other issues such as field curvature or flare or excessive CA, the reviewer will mention such factors. If he doesn't, then other sources likely will.

Quote
Regarding the "old photodo tests", I agree fully. The present tests in "Foto" are essentially the continuance of those tests.

I'm interested. Is this magazine available in English?
« Last Edit: February 08, 2009, 05:31:03 PM by Ray » Logged
ErikKaffehr
Sr. Member
****
Online Online

Posts: 8031


WWW
« Reply #69 on: February 08, 2009, 07:00:50 PM »
ReplyReply

Sorry it is not.

Quote from: Ray
I'm interested. Is this magazine available in English?
Logged

BernardLanguillier
Sr. Member
****
Offline Offline

Posts: 8390



WWW
« Reply #70 on: February 08, 2009, 07:38:36 PM »
ReplyReply

Quote from: Quentin
Isn't this really all about sacred cows and the fact we don't like having ours slaughtered?   DxO are guilty of the sin of independence from subjectivity.  Surely they should have realised that Photography is all about brand loyalty and preconceptions based on price, not real world performance  

Is is apparently about things you are not skilled enough to see.

What remains to be confirmed is whether people blessed with these golden eyes are the same having the golden ears needed to feel comfortable using 20.000 US$+ speakers.

In the mean time I'll forge ahead using my inferior pano kit.  



Cheers,
Bernard
Logged

A few images online here!
Quentin
Sr. Member
****
Offline Offline

Posts: 1123



WWW
« Reply #71 on: February 09, 2009, 01:09:02 PM »
ReplyReply

Quote from: BernardLanguillier
Is is apparently about things you are not skilled enough to see.

True, but I know snake oil when I see it  

Quote
What remains to be confirmed is whether people blessed with these golden eyes are the same having the golden ears needed to feel comfortable using 20.000 US$+ speakers.

In the mean time I'll forge ahead using my inferior pano kit.  

Cheers,
Bernard

Well Bernard, I spent years of my time  and some serious money on exotic hi fi and there were moments of sublime sound, but not enough of them to outweigh the wasted time spent listing to the equipment and not to the music.  Now I have dumped the valve amps and imposing room cramping speakers, have a good but not absurdly expensive system and enjoy music a lot more.  Somewhere in there is a lesson for photo equipment junkies.

Quentin
Logged

Quentin Bargate, ARPS, Author, photographer entrepreneur and senior partner of Bargate Murray, Law Firm of the Year 2013
barryfitzgerald
Sr. Member
****
Offline Offline

Posts: 610


« Reply #72 on: February 09, 2009, 01:53:51 PM »
ReplyReply

Quentin, you are showering the forum with logic and sound thinking! Please stop it ;-)

I think we all share an interest in technology, and I am not going to deny that. I wouldn't slam the door on some super nice top end gear too.
The pursuit of testing/collecting equipment, for the sake of it, is a false prophet.
Music is a great comparison, I used to admire Jimi Hendrix when I was growing up (forget if you like his music, or the rock n roll lifestyle etc) This is the same guy who used a wooden broomstick to mimic playing on. The guy who used a one-stringed ukulele found in his father's garage. Now he moved onto Fenders, and Gibson's..the nice stuff..but the man was "obsessed with playing guitar". He even used to play in the toilet..with no amp.

What he was not..was "obsessed with technology, or guitars" He was not gripped with testing his equipment, he just used it (heck even smashed it up at times!). And it would be hard to ignore the pure talent, even if you did not care for his style.

The problem with many folks on this forum, is they are too busy trying to find the right guitar, for the "sound they like", and reading the tests that prove it's the best guitar out there. Not that it's unimportant, but hey..you have to call it a day sometime. And not enough time spent playing the damn thing!

So I don't want to hear about the "new guitar just out with a great sound" I want to hear about the tunes you knock out. Testing is pretty boring if we are honest..there will always be a new shiny guitar in the shop, that might be a bit better than the last model you bought.

In the spirit of non testing, I present a photo..well I just decided the other night to stop reading the tests, and take a photo. My apologies for using non high end stuff, I am sure I would get a sharper bigger print with Carl Zeiss optics, and a FF digital body. Sorry about that..shot on a super budget DSLR with a 70 lens, hey lets call it a slightly beaten up guitar with a not bad sound.. ;-)


[attachment=11419:Barry_Po...b_2009_8.jpg]


Have fun talking about testing! lol

ps..it passed my own personal test..aka using the thing!

Forgot to mention, jpeg straight from camera (bar USM and resize)...how dare I??? lol
« Last Edit: February 09, 2009, 02:04:39 PM by barryfitzgerald » Logged
BernardLanguillier
Sr. Member
****
Offline Offline

Posts: 8390



WWW
« Reply #73 on: February 09, 2009, 06:54:10 PM »
ReplyReply

Quote from: barryfitzgerald
The problem with many folks on this forum, is they are too busy trying to find the right guitar, for the "sound they like", and reading the tests that prove it's the best guitar out there.

Clearly so.  Another thing, closely related, is that you probably end up playing nicer if you use the same guitar all the time, since you'll get to really know it in depth.

As far as I am concerned, I have decided to stick to my current body for at least 2 to 3 years. Having understood that the world is not ready to acknowledge the fact that I own the best camera, I will focus on trying to be the best user of that camera in the world... that should keep me busy enough...

Cheers,
Bernard
Logged

A few images online here!
Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #74 on: February 09, 2009, 07:11:47 PM »
ReplyReply

Quote from: BernardLanguillier
As far as I am concerned, I have decided to stick to my current body for at least 2 to 3 years. Having understood that the world is not ready to acknowledge the fact that I own the best camera, I will focus on trying to be the best user of that camera in the world... that should keep me busy enough...

Cheers,
Bernard

Bernard,
I see that you are going through a phase of 'obsession with temple detail'. Is that right?  
Logged
BernardLanguillier
Sr. Member
****
Offline Offline

Posts: 8390



WWW
« Reply #75 on: February 09, 2009, 07:43:09 PM »
ReplyReply

Quote from: Ray
I see that you are going through a phase of 'obsession with temple detail'. Is that right?  

Temple are a fascinating subject!



120 megapixels HDR pano shot from D3x. The HDR work was done entirely by hand since I could not get satisfactory results with any of the automated tools I tried. Whitebalance was also difficult to handle well, and single conversions didn't do the trick.

Raw converson also involved different converters for differents parts of the image.

Cheers,
Bernard
« Last Edit: February 09, 2009, 08:17:21 PM by BernardLanguillier » Logged

A few images online here!
barryfitzgerald
Sr. Member
****
Offline Offline

Posts: 610


« Reply #76 on: February 09, 2009, 08:06:47 PM »
ReplyReply

What I was trying to say, in a round about way..was this.

Places like DxO (with all due respect), have little to no real benefit for photographers. It really is numbers and charts. There are no images to see (real or test shots), no comments on subjective area like handling and performance. Really not a great place to find out about cameras anyway.
You could argue review sites do similar things, but at least they have some worth.
Would you really care if they say x model has 0.3 extra DR over your current one? Even if that were true..it's unlikely to be field relevant.


Logged
Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #77 on: February 09, 2009, 10:16:12 PM »
ReplyReply

Quote from: barryfitzgerald
Would you really care if they say x model has 0.3 extra DR over your current one? Even if that were true..it's unlikely to be field relevant.

Yes. I would care very much if subjective reports from proud owners of such a camera claimed it had 2 stops greater DR than my current one. In the absence of objective reports from DXO, I might be tempted to buy such a camera. The fact that model x has only 0.3 stops of extra DR is useful information to have. It means that I can either exclude, or give low priority to that factor when considering the significance of the other features of camera x that I might find advantageous.
Logged
Ray
Sr. Member
****
Offline Offline

Posts: 8948


« Reply #78 on: February 09, 2009, 10:21:06 PM »
ReplyReply

Quote from: BernardLanguillier
120 megapixels HDR pano shot from D3x. The HDR work was done entirely by hand since I could not get satisfactory results with any of the automated tools I tried. Whitebalance was also difficult to handle well, and single conversions didn't do the trick.

Raw converson also involved different converters for differents parts of the image.

Cheers,
Bernard

Bernard,
I'm not so sure. You might save yourself a lot of work in front of the computer if you were to get yourself a P65+   .
Logged
BernardLanguillier
Sr. Member
****
Offline Offline

Posts: 8390



WWW
« Reply #79 on: February 09, 2009, 10:51:47 PM »
ReplyReply

Quote from: Ray
Bernard,
I'm not so sure. You might save yourself a lot of work in front of the computer if you were to get yourself a P65+   .

I don't think it would change that much... I would need a 3 images stitch anyway to reach this resolution, DR is said to be similar so that I would need to take at least 6 frames. The host cameras being heavier I would probably run into problems with the stability of my stitching head... I am really not sure that I would get superior results in fact.

Besides, since I don't charge myself any consulting fee, time is a lot cheaper than 32.000 US$...

Cheers,
Bernard
Logged

A few images online here!
Pages: « 1 2 3 [4] 5 »   Top of Page
Print
Jump to:  

Ad
Ad
Ad