Ad
Ad
Ad
Pages: « 1 [2] 3 4 5 »   Bottom of Page
Print
Author Topic: Eyes vs Numbers  (Read 21044 times)
Tomcat
Newbie
*
Offline Offline

Posts: 8


« Reply #20 on: February 03, 2009, 01:11:27 PM »
ReplyReply

Maybe I'm just an old crotchety skeptic, but I think some key things are being missed with this Dxo thing:

1. Dxo clearly listed they only tested the sensors, not actual image quality.  They did not process the raw files.  There is no way to judge actual resolving power, final color capability, etc. without this.

2. They compared the latest DSLRs against 2006-era MF backs.  They point out how much better the DSLR sensors have gotten, but they don't allow for the MF sensors to have the same opportunity.  

3. I can't help but notice that all of Dxo's test pages have a big ad for their raw processing software, which only supports DSLRs.  Sorry, but I'm having a hard time accepting that the results aren't slanted towards DSLRs because of this and #2.  They are getting a lot of free press from these results, and a lot of people of now viewing their software ad...

Color me skeptical.
Logged
otbricki
Newbie
*
Offline Offline

Posts: 3


« Reply #21 on: February 03, 2009, 01:34:29 PM »
ReplyReply

Interesting article, and especially the comparison with high end audio.

DXO is off to a start, but clearly they have some refinement to their measurements before people will take them as clear measures of the capabilities of the camera. It is important to realize that these sorts of tests are very valuable to the development process of the technology. If you are just going to rely on subjective opinions then you are going to get into trouble because you won't have a repeatable way of determining the quality of the camera. Now some things of course will always be subjective; preferences in color rendering for example. But the color match with the subject can be measured.

The comparison with high end audio is important, but not the way the article poses it. High end audio has a very bad reputation in the engineering community because nobody has been able to reproduce the claims of these so-called 'golden ears' in any measurement or controlled test. I would hate to see cameras head in this direction because it allows anyone to sell anything with totally outlandish claims that are totally unreproducible. Fortunately I don't think is likely to happen because a photograph is frozen in time, making it possible for everyone to examine the same result of the reproduction process. Audio suffers because of the ephemeral nature of the reproduction.
Logged
vjbelle
Full Member
***
Offline Offline

Posts: 201


« Reply #22 on: February 03, 2009, 01:47:32 PM »
ReplyReply

Quote from: hcubell
I do find it interesting that Phase One itself, when asked why someone should consider upgrading from a P45 to a P65, states  that the resolution advantage is relatively minor and the "real" advantages with the P65 are cleaner higher ISO files of 15mp and the POSSIBILITY of future upgrades(no indication of what upgrades or when). Well, for about 1/3 the price of the upgrade you could buy a D3X with 24mp files at 1600 ISO that will probably blow away those 15mp 1600 ISO P65 files. Plus, you get world class AF and a gorgeous LCD with the D3X.
I could afford to upgrade from a 39mp back to a 60mp back, but there is no way I am going for it unless I was convinced that, aside from the relatively minor resolution advantage and vague promises about good  things in the future, the new back offered significant improvements in dynamic range, a world class LCD and the ability to quickly and easily pull 20 different "looks" from a file like I could get by loading 20 different films into my old Pentax 67.

AMEN!!!  Couldn't agree with you more!  It will be interesting to see how the dust settles at the end of the year.  

Victor
Logged
fike
Sr. Member
****
Offline Offline

Posts: 1372


Hiker Photographer


WWW
« Reply #23 on: February 03, 2009, 01:49:45 PM »
ReplyReply

Quote from: otbricki
Interesting article, and especially the comparison with high end audio.

DXO is off to a start, but clearly they have some refinement to their measurements before people will take them as clear measures of the capabilities of the camera. It is important to realize that these sorts of tests are very valuable to the development process of the technology. If you are just going to rely on subjective opinions then you are going to get into trouble because you won't have a repeatable way of determining the quality of the camera. Now some things of course will always be subjective; preferences in color rendering for example. But the color match with the subject can be measured.

The comparison with high end audio is important, but not the way the article poses it. High end audio has a very bad reputation in the engineering community because nobody has been able to reproduce the claims of these so-called 'golden ears' in any measurement or controlled test. I would hate to see cameras head in this direction because it allows anyone to sell anything with totally outlandish claims that are totally unreproducible. Fortunately I don't think is likely to happen because a photograph is frozen in time, making it possible for everyone to examine the same result of the reproduction process. Audio suffers because of the ephemeral nature of the reproduction.

You can't automate the measurement of every intangible property of image quality.  We lack the tools to do it.  Human evaluation is the only way to aggregate all the liminal factors with all the subliminal factors.
Logged

Fike, Trailpixie, or Marc Shaffer
marcshaffer.net
TrailPixie.net

I carry an M43 ILC, a couple of good lenses, and a tripod.
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #24 on: February 03, 2009, 01:50:47 PM »
ReplyReply

I read following statements in Eyes vs Numbers:

Quote
I think though that the main problem is that DxO is analyzing raw camera data, which does not include any correction of fixed pattern noise. DSLR cameras remove these noise sources before storing their raw data. Medium format cameras do not. DSLRs also apply a lot of noise reduction algorithms before storing the data...Nikon, for example, does a lot of on-chip noise reduction to their D3 and D3x raw files

I feel following needs to be said, for Michael's statements may be taken by some readers too literally; this is not for or against any camera but for the better understanding of these issues.

1. All Nikon DSLR cameras cut off a portion of the pixel data, thereby removing pattern noise (one could not name this "noise reduction" without provocing the international Nikon community to violent responses). All Canon DSLRs keep the raw data "uncut". This difference may have caused the ridiculous assertion by DxO, that the D3X has two stops greater dynamic range than the 5D2.

2. Noise reduction in the sense most people understand it will be performed on raw data only by Sonys, at least by the A700 and A900. This step could not be disabled with an earlier firmware version of the A700, but now OFF means OFF, apparently.

3. On-chip (on-sensor) noise reduction occurs with all CMOS sensors, which are inherently more noisy than CCD. However, that is single-pixel noise reduction, which has absolute nothing to do with the noise reduction in the sense as it is done in raw processing software or by the mentioned Sony cameras, namely based on the context of pixels.

4. The raw data of the Phase One cameras is far from being really raw.

Finally, there is nothing wrong in analyzing raw data and making certain conclusions based on that; at least I am doing just that without feeling guilty. In fact, one can make certain findings only based on the raw data. However, one needs to be careful with those findings. I'm afraid, in the DxO evaluation of the P45+ they used ACR for the analysis; if so, that invalidates all of their findings.


Logged

Gabor
John Camp
Sr. Member
****
Offline Offline

Posts: 1252


« Reply #25 on: February 03, 2009, 02:10:04 PM »
ReplyReply

Quote from: otbricki
<snip> DXO is off to a start, but clearly they have some refinement to their measurements before people will take them as clear measures of the capabilities of the camera. <snip>

I honestly don't think this will ever happen. If you look at Sean Reid's review site [a pay site], you'll see that he often refers to a way a lens "draws." So in a review of high-end normal lenses, he gave excellent marks both to a Nikon and a Zeiss 50mm, said either one is capable of doing any kind of professional work, but he said that they draw differently and he prefers the Zeiss. So in this case, part of the "quality" equation is measuring Sean Reid's preferences, which may change from time to time. On the Online Photographer recently, Mike Johnston, who knows a lot about lenses, panned a lens because of its distortions, but actually liked the "look" it produced. In audio, you have large numbers of people who agree (apparently) that some amp/speaker systems sound better than others, despite measures to the contrary. It seems to me that the problem is with the engineers, not with the ears. Engineers have a bias toward measurement, which is a real bias, because if only 20% of a system is currently measurable and quantifiable, it's possible that the other 80% is more important in every way -- yet the engineers will tend to go with what they can measure, because that's their bias. It's the old "bumblebees can't fly" problem -- bumblebees CAN fly, the engineers just didn't have the ability to quantify the characteristics that allowed the bumblebees to do so.

The DXO problem is that they insist that they have a measure, but it's not a real measure. They have a whole bunch of measures, but those measures don't include the whole universe of possibilities, and without the whole universe, you can't reach a valid conclusion. And it may be inherently *impossible* to measure the entire universe of possibilities. So, if a lot of people with good, trained photographic eyes say that a large P65 print is better than a large D3 print, and a DXO measurement says it's not, then the eyes are right, just as they're right when they see a bumblebee fly.

JC
Logged
NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #26 on: February 03, 2009, 02:28:16 PM »
ReplyReply

Quote from: Panopeeper
I read following statements in Eyes vs Numbers:



I feel following needs to be said, for Michael's statements may be taken by some readers too literally; this is not for or against any camera but for the better understanding of these issues.

1. All Nikon DSLR cameras cut off a portion of the pixel data, thereby removing pattern noise (one could not name this "noise reduction" without provocing the international Nikon community to violent responses). All Canon DSLRs keep the raw data "uncut". This difference may have caused the ridiculous assertion by DxO, that the D3X has two stops greater dynamic range than the 5D2.

2. Noise reduction in the sense most people understand it will be performed on raw data only by Sonys, at least by the A700 and A900. This step could not be disabled with an earlier firmware version of the A700, but now OFF means OFF, apparently.

3. On-chip (on-sensor) noise reduction occurs with all CMOS sensors, which are inherently more noisy than CCD. However, that is single-pixel noise reduction, which has absolute nothing to do with the noise reduction in the sense as it is done in raw processing software or by the mentioned Sony cameras, namely based on the context of pixels.

4. The raw data of the Phase One cameras is far from being really raw.

Finally, there is nothing wrong in analyzing raw data and making certain conclusions based on that; at least I am doing just that without feeling guilty. In fact, one can make certain findings only based on the raw data. However, one needs to be careful with those findings. I'm afraid, in the DxO evaluation of the P45+ they used ACR for the analysis; if so, that invalidates all of their findings.


Gabor,

Thanks for your contribution. You confirmed much of what I was trying to say in this and other threads.

1. That issue of where Nikon are placing their zero point is well documented. It does affect pattern noise and is generally a good thing for all but the astrophotographers.

2. The Sonys, as you are saying, are the only cameras documented to perform real noise reduction (image processing in the digital domain for the purposes of reducing noise). I wonder where Michael got the feeling all dSLRs do that. And as you say, in the latest versions this can be switched off (and hopefully DxO have tested with this to off).

3. On chip 'noise reduction' cannot be called noise reduction.  A better word would be noise avoidance or measures to increase the S/N ratio. Any other noise avoidance technique (especially if performed in the analog domain) also cannot be called image processing or noise reduction. It is not a matter of semantics. These techniques only help to get cleaner digital data without compromising the image. If the MFDB manufacturers don't need, don't know or don't care to use such techniques has nothing to do with whether their raw data can be compared to dSLR raw.

4. I can't comment on that but if true it makes Michael's objection to comparing dSLR raw to MFDB raw even more irrelevant.One has to either accept that the concept of comparing raw is useful (for any camera) or not accept that and compare the output from the raw converters. Both approaches have caveats so both should be used IMO.
« Last Edit: February 03, 2009, 02:36:43 PM by NikosR » Logged

Nikos
Misirlou
Sr. Member
****
Offline Offline

Posts: 545


WWW
« Reply #27 on: February 03, 2009, 02:32:16 PM »
ReplyReply

Quote from: ErikKaffehr
Having excess DR, resolution or whatever is of little utility, but can be nice the day you photograph black cats in a coal cellar or need to print really big.

I hate to admit this, but I'd actually love to see a technically excellent photo of some black cats in a coal cellar.

Hey, if nothing else, I'd like to use Bernard's snowscape photo, and the black cats in a coal cellar photo to test the abilities of my printers and monitors.
Logged
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #28 on: February 03, 2009, 02:37:57 PM »
ReplyReply

Quote from: NikosR
3. On chip 'noise reduction' cannot be called noise reduction.  A better word would be noise avoidance or measures to increase the S/N ratio
I am usually cautious with this term, for most photographers understand "noise reduction" as the function performed by the raw converters or later. However, Canon is calling this "noise reduction"; following is from the Canon EOS 1D Mark III White Paper:

Additionally, the second-generation, on-chip noise removal circuit and the noise reduction technology explained below combine to remove noise effectively.
This has enabled the EOS-1D Mark III to be the first EOS DSLR camera to have ISO 3200 as part of its standard ISO speed range. Dynamic range at low ISO speeds is about the same with the 1D Mark III as it is with the EOS-1D Mark II and Mark II N despite the 1D Mark IIIís increased resolution. A second-generation, on-chip, noise-reduction circuit is provided.


The problem is, that most photographers are thinking of the context sensitive function when reading "noise reduction". "Single pixel based noise reduction" would be a correct term, but who would use it, and how many would understand its meaning?
Logged

Gabor
fike
Sr. Member
****
Offline Offline

Posts: 1372


Hiker Photographer


WWW
« Reply #29 on: February 03, 2009, 02:44:38 PM »
ReplyReply

Quote from: Panopeeper
I am usually cautious with this term, for most photographers understand "noise reduction" as the function performed by the raw converters or later. However, Canon is calling this "noise reduction"; following is from the Canon EOS 1D Mark III White Paper:

Additionally, the second-generation, on-chip noise removal circuit and the noise reduction technology explained below combine to remove noise effectively.
This has enabled the EOS-1D Mark III to be the first EOS DSLR camera to have ISO 3200 as part of its standard ISO speed range. Dynamic range at low ISO speeds is about the same with the 1D Mark III as it is with the EOS-1D Mark II and Mark II N despite the 1D Mark IIIís increased resolution. A second-generation, on-chip, noise-reduction circuit is provided.


The problem is, that most photographers are thinking of the context sensitive function when reading "noise reduction". "Single pixel based noise reduction" would be a correct term, but who would use it, and how many would understand its meaning?

How is this different than capture sharpening versus print sharpening. Both are sharpening.  You could consider it capture noise reduction (or hardware noise reduction) and printing/display noise reduction.
Logged

Fike, Trailpixie, or Marc Shaffer
marcshaffer.net
TrailPixie.net

I carry an M43 ILC, a couple of good lenses, and a tripod.
NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #30 on: February 03, 2009, 02:54:13 PM »
ReplyReply

Quote from: Panopeeper
I am usually cautious with this term, for most photographers understand "noise reduction" as the function performed by the raw converters or later. However, Canon is calling this "noise reduction"; following is from the Canon EOS 1D Mark III White Paper:

Additionally, the second-generation, on-chip noise removal circuit and the noise reduction technology explained below combine to remove noise effectively.
This has enabled the EOS-1D Mark III to be the first EOS DSLR camera to have ISO 3200 as part of its standard ISO speed range. Dynamic range at low ISO speeds is about the same with the 1D Mark III as it is with the EOS-1D Mark II and Mark II N despite the 1D Mark III's increased resolution. A second-generation, on-chip, noise-reduction circuit is provided.


The problem is, that most photographers are thinking of the context sensitive function when reading "noise reduction". "Single pixel based noise reduction" would be a correct term, but who would use it, and how many would understand its meaning?

I think that Canon refer to what you're saying as 'noise removal' and by 'noise reduction' they refer to their normal noise reduction for long exposures (dark frame subtraction) and for High ISO (in the jpeg engine).

Regardless, the fact remains that these 'noise removal' techniques cannot be implemented (as far as I understand it) in the raw converter. Thus, any noise reduction that is performed there, be it at the pre or post demosaicing stage is a true noise reduction (context sensitive as you put it). If MFDB manufacturers insist on us performing such a function before looking at their results then one should compare their results with dSLR output having been also de-noised with comparable techniques in the raw converter.
« Last Edit: February 03, 2009, 03:12:28 PM by NikosR » Logged

Nikos
NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #31 on: February 03, 2009, 02:57:44 PM »
ReplyReply

Quote from: fike
How is this different than capture sharpening versus print sharpening. Both are sharpening.  You could consider it capture noise reduction (or hardware noise reduction) and printing/display noise reduction.

This is not the correct analogy. What Gabor and I are referring to does not by definition have any adverse effect on the quality of the image data since they are noise removal or noise avoidance techniques at the pixel level. You don't remove useful data in the process of removing noise. You just make sure that the individual pixel level is 'correct'.
« Last Edit: February 03, 2009, 02:58:47 PM by NikosR » Logged

Nikos
fike
Sr. Member
****
Offline Offline

Posts: 1372


Hiker Photographer


WWW
« Reply #32 on: February 03, 2009, 03:11:45 PM »
ReplyReply

Quote from: NikosR
This is not the correct analogy. What Gabor and I are referring to does not by definition have any adverse effect on the quality of the image data since they are noise removal or noise avoidance techniques at the pixel level. You don't remove useful data in the process of removing noise. You just make sure that the individual pixel level is 'correct'.


hmmm...okay.  I can accept that, but I don't understand how any algorithm can be so certain that it is only removing unwanted data--noise.  That is kind of the big problem with signal processing and noise.  It is very difficult to discern what is valid data and what is noise.  There are a lot of images that can really cause problems for noise reduction/removal, etc...:  astronomy, microscopy, macros of sand. Heck, even a highly detailed tile mosaic can conceivable cause significant problems to any noise reduction algorithm.  So, in the end, you are changing the image, although perhaps to make it more appealing to the human eye.  We all know the plastic look of excessive noise reduction as well as the crunchy look of too much sharpening.  Everything we and our cameras do changes the data.

What am I missing here?
Logged

Fike, Trailpixie, or Marc Shaffer
marcshaffer.net
TrailPixie.net

I carry an M43 ILC, a couple of good lenses, and a tripod.
NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #33 on: February 03, 2009, 03:17:28 PM »
ReplyReply

Quote from: fike
hmmm...okay.  I can accept that, but I don't understand how any algorithm can be so certain that it is only removing unwanted data--noise.  That is kind of the big problem with signal processing and noise.  It is very difficult to discern what is valid data and what is noise.  There are a lot of images that can really cause problems for noise reduction/removal, etc...:  astronomy, microscopy, macros of sand. Heck, even a highly detailed tile mosaic can conceivable cause significant problems to any noise reduction algorithm.  So, in the end, you are changing the image, although perhaps to make it more appealing to the human eye.  We all know the plastic look of excessive noise reduction as well as the crunchy look of too much sharpening.  Everything we and our cameras do changes the data.

What am I missing here?

You are missing the fact that we are not talking about noise reduction the way you understand it, that's why it's no good to call it noise reduction in the first place. We are talking about making sure there is minimum error in the reading of an individual pixel. It is the sum of the errors of individual pixels that make up 'noise' as you understand it. So, in effect, such techniques make sure there is less noise to deal with rather than trying to remove noise that already exists in the context of the image.


PS. FWIW I quote bobn2 from dpreview who gives a conscise description for what a CMOS on chip noise avoidance circuit can amount to (if you can bear reading it):

'The reason that CMOS needs 'NR' circuitry (actually misnamed) is that the random access CMOS cell can't provide a sample of the reset value, which is needed for correlated double sampling to reduced (sic) reset and pattern 'noises'. The NR circuitry is an additional transistor which allows the reset value to be read by the in-cell amplifier before the photodiode reading is made.'
« Last Edit: February 03, 2009, 03:44:51 PM by NikosR » Logged

Nikos
fike
Sr. Member
****
Offline Offline

Posts: 1372


Hiker Photographer


WWW
« Reply #34 on: February 03, 2009, 03:49:06 PM »
ReplyReply

You know, I even catch myself slipping down into the bowels of discussing the camera sensor design and measurement(about which I no very little)  instead of focusing on the camera system.  I think the point of all of this is that it is the subjective results that matter, not abstract measurement.   I needn't care whether there is noise reduction or removal and where in the pipeline it is done.  I should care about the quality of the ouput--prints.

This stuff is interesting, but not really incredibly relevant to determining whether a camera is good or not.  Or, perhaps I should say, can you explain to me why this is useful or relevant information for the pro or amateur consumer to understand?
« Last Edit: February 03, 2009, 03:52:43 PM by fike » Logged

Fike, Trailpixie, or Marc Shaffer
marcshaffer.net
TrailPixie.net

I carry an M43 ILC, a couple of good lenses, and a tripod.
NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #35 on: February 03, 2009, 04:01:30 PM »
ReplyReply

Quote from: fike
You know, I even catch myself slipping down into the bowels of discussing the camera sensor design and measurement(about which I no very little)  instead of focusing on the camera system.  I think the point of all of this is that it is the subjective results that matter, not abstract measurement.   I needn't care whether there is noise reduction or removal and where in the pipeline it is done.  I should care about the quality of the ouput--prints.

This stuff is interesting, but not really incredibly relevant to determining whether a camera is good or not.  Or, perhaps I should say, can you explain to me why this is useful or relevant information for the pro or amateur consumer to understand?


If you're referring to the noise removal discussion, it is not. It is being discussed in the context of Michael's objection to comparing raw data from dSLRs to raw data from MFDBs due to dSLRs purportedly performing internal noise reduction before writing out the raw data while MFDBs do that in the raw converter.

If you are referring to DxO's test (or any other objective measurements) I believe they are relevant because a. may partially assist someone making decisions and b. may assist someone understand why he is seeing or not seeing things and what exactly it is he is seeing.
« Last Edit: February 03, 2009, 04:02:41 PM by NikosR » Logged

Nikos
Mort54
Sr. Member
****
Offline Offline

Posts: 590


WWW
« Reply #36 on: February 03, 2009, 04:17:05 PM »
ReplyReply

Quote from: Panopeeper
1. All Nikon DSLR cameras cut off a portion of the pixel data, thereby removing pattern noise (one could not name this "noise reduction" without provocing the international Nikon community to violent responses). All Canon DSLRs keep the raw data "uncut".
I find this comment hillarious, given that the D3X delivers more detail in deep shadows than either the 1DsIII or the 5DII at lower ISOs. So tell me again about cutting off a portion of the pixel data Huh? Sounds like a rationalization to me.
Logged

I Reject Your Reality And Substitute My Own
BernardLanguillier
Sr. Member
****
Offline Offline

Posts: 7523



WWW
« Reply #37 on: February 03, 2009, 04:43:09 PM »
ReplyReply

Quote from: michael
A lovely image Bernard!

I'd live to see a big print. The web hardly does it justice.

Thanks. The image just underwent a recall though, I found some small issues that need fixing.

I should be able to show you a large print next time you visit Tokyo.

Cheers,
Bernard
Logged

A few images online here!
Chris_Brown
Sr. Member
****
Offline Offline

Posts: 729



WWW
« Reply #38 on: February 03, 2009, 04:51:33 PM »
ReplyReply

Michael R.,

Great topic! Thanks for stirring the pot.

For me, a camera is more than a sensor and its supporting electronics. Even though the DxO evaluations are informative, they don't tell us the whole story. Considerations of body design, lenses, portability and software weigh significantly on our choices as photographers (pros & amateurs alike). Some of these factors are given more weight in our decisions than others.

One thing I'd like to point out is that, at present, all camera manufacturers don't seem to be listening to the global user base very well. They seem to be focusing on cost per pixel at the moment, which is quickly coming to a dead end (as Michael's topic illustrates).
Logged

~ CB
jrmintz
Newbie
*
Offline Offline

Posts: 18


« Reply #39 on: February 03, 2009, 04:53:02 PM »
ReplyReply

Michael's analogy to high-end audio reproduction is interesting. As a producer of music and an amateur photographer I've found many similarities, and one huge difference: while photography is about capture and reproduction, audiophilia (is that a word?) is only about reproduction. There is no link between audio production gear (capture) and reproduction gear analogous to the common calibration of monitors and the profiling of printers by serious amateur and professional photographers. There are no standards like proRGB to calibrate to. You can create a listening area with reasonably flat frequency response through acoustic treatment and electronic EQ, like most good recording studio control rooms. There is still no basis on which to make a judgment of the 'accuracy' of playback equipment, high- or low-end, unless you were at the recording session. If you think your gear sounds accurate, great! I have often encountered great gear in lousy-sounding rooms. The job of the mastering engineer is to make a recording 'translate' to as many different playback situations as possible. The good ones make a bundle and are worth every penny.
Logged
Pages: « 1 [2] 3 4 5 »   Top of Page
Print
Jump to:  

Ad
Ad
Ad