More Color Checker questions WRT accurate color

[Tim Lookingbill]

Wow! Why does that Nikon D5xx curve look so familiar? Looks pretty close to my attempt at getting exact Lab numbers for the sample image posted above.

Bob said:

If there's a tone curve in the DNG as a result of the profile used in the Calibration pane (or if there's one baked into the DNG Converter) then it's not the linear camera data anymore.

After the voltage readings off each sensor pixel site goes through the A/D converter, all sensor data is interpreted by software at that point. Remember it's just grayscales assigned RGB colorants. Even demosaicing is an interpretation. Even if you were to work with linear data each converter will deliver its own flavor/interpretation of what that looks like or how it's calculated.

I've compared three previews of "linear" Raw data from what was claimed by each Raw converter to be a linear setting based preview and they were all different from each other.

In the end it's really just individually colored tiny little squares crammed tightly next to each other. It's the display preview that's really controlling how you see and edit the severely anti-aliased preview depending on zoom level. Who cares what each individual pixel looks like. They all look the same at 1600% zoom level in Photoshop.
 

[BobFisher]

Wow! Why does that Nikon D5xx curve look so familiar? Looks pretty close to my attempt at getting exact Lab numbers for the sample image posted above.

Bob said:
After the voltage readings off each sensor pixel site goes through the A/D converter, all sensor data is interpreted by software at that point. Remember it's just grayscales assigned RGB colorants. Even demosaicing is an interpretation. Even if you were to work with linear data each converter will deliver its own flavor/interpretation of what that looks like or how it's calculated.

I've compared three previews of "linear" Raw data from what was claimed by each Raw converter to be a linear setting based preview and they were all different from each other.

In the end it's really just individually colored tiny little squares crammed tightly next to each other. It's the display preview that's really controlling how you see and edit the severely anti-aliased preview depending on zoom level. Who cares what each individual pixel looks like. They all look the same at 1600% zoom level in Photoshop.
 

No quibble with any of that, Tim.  What I'm trying to figure out - and as yet can't get an answer - is how to get the best starting point for creating the camera profile.  It would seem that the linear (or near linear) data out of the camera would be that best starting point.  But a file with a tone curve baked in seemingly wouldn't be.  If that (near) linear data is the best starting point and if the tone curve from the camera calibration pane of LR/ACR is getting baked into the DNG during conversion because that part of the rendering can't be turned off and if that default tone curve isn't backed out of the file when it's opened up in DNG PE or run through something like XRite's Passport software to bring the file back to linear then it seems like a fruitless exercise.  But no one's yet said if that (near) linear camera data is the best starting point and why or why not.  No one's yet said whether the tone curve of the profile used as the default in the camera calibration pane gets baked in or not.  No one's yet said whether that tone curve gets backed out along with any other user edits in DNG PE or the Passport software (user edits get backed out in the Passport software too because I loaded a b&w DNG and it opened in colour).  And I'm fully on board with the fact that the camera profile isn't going to be perfect (I've said that all along) and that further 'tweaking' will be necessary.  But it almost seems as though it's coming down to the point that going through the step of creating a camera profile is essentially useless. 

[madmanchan]

Hi Bob,

Conversion to DNG doesn't involve any application of a tone curve.  The tone curve gets applied when rendering a DNG to an output (e.g., for display or print purposes). 

There is a very significant difference between "converting" and "rendering."  As an example, consider opening a PSD file in Photoshop and re-saving it as a TIFF.  All you've done is a conversion (of file format).  There is no change in rendering.  Now, consider taking that same image, and applying a tone curve to it (e.g., using Curves panel).  In that case, you're changing the rendering. 

Converting a raw file (like CR2, NEF, etc.) to DNG isn't rendering.  It's just converting to another file format / container for the purposes of holding the image & associated metadata.  The image data is still unrendered (scene-referred) e.g., no white balance, no color profile, no tone curve, etc.  The rendering takes place when you open that DNG into a raw converter software and then save it out as something else (e.g., JPEG, TIFF), in a so-called output-referred format.

[madmanchan]

No quibble with any of that, Tim.  What I'm trying to figure out - and as yet can't get an answer - is how to get the best starting point for creating the camera profile.  It would seem that the linear (or near linear) data out of the camera would be that best starting point.

Correct.  And that's how the DNG Profile Editor works (same with X-Rite's ColorChecker Passport software).  The color tables that are created using DNG PE and Passport transform the linear scene-referred RGB values.  You don't need to do anything special to make this happen.  Raw DNG files are (by definition) scene-referred and linear.  

The part that may be confusing is the fact that the default preview of the image as shown in DNG PE (and in ACR/LR) is not scene-referred.  Similarly, if you try to take RGB readouts in ACR (or open the image into Ps and take Lab readouts), you'll be reading values that aren't scene-referred.  That's because of the default tone curve and black subtraction, as I explained earlier.  You can turn all that stuff off, if you want to check the numerical readings against published values (with the practical caveats that I mentioned earlier about illumination, and various other factors).

[madmanchan]

Sandy, deejjjaaaa,

It is true that DNG Converter doesn't have all of ACR/LR engine in it.

It is also true that DNG Converter needs enough of it to build thumbnails & embedded previews (and of course, to read non-DNG raw files).

The public DNG SDK does contain a lot of the essentials, such as linearization, tone curves, white balance, and color profiles.  Both DNG Converter and ACR/LR heavily use the DNG SDK.  (That is, the DNG SDK isn't just some separate piece of code we put out there!    )

In terms of the file format, the DNG SDK contains everything needed to read & write DNGs.  So a software developer could, for example, take the dcraw source code and DNG SDK and essentially write their own "DNG converter software" if desired.  The bit about rendering thumbnails & embedded previews is often convenient for users, but totally optional. 

[BobFisher]

Thanks Eric.  That answers the questions.  I do understand the difference between rendering and converting/scene-referred and output-referred and I know I used 'render' in my last remark but if a tone curve were applied that would be rendering and not just converting. 

Understood about the comparison between sampled and published values.  I'm not trying to do that.  But if one were going to do that, the best way to try and compare the values would be to set the curve to Linear in the Tone Curve tab of PE.  Right? 

[madmanchan]

Understood about the comparison between sampled and published values.  I'm not trying to do that.  But if one were going to do that, the best way to try and compare the values would be to set the curve to Linear in the Tone Curve tab of PE.  Right? 

There are a couple of ways to do this.

One way is to leave the Tone Curve tab of PE alone (unchanged), and instead set ACR's Brightness Contrast, and Blacks (in Basic panel) to zero, and also set ACR's Point Curve to Linear.  Thus, when you apply your color profile, you are performing the color adjustments, but turning off ACR's default tone curve rendering.

Another way is to do as you mentioned, i.e., set the curve to Linear in DNG PE's Tone Curve tab.  Then in ACR, you would need to set Blacks to zero, and leave Brightness, Contrast, and the Point Curve at their respective defaults of 50, 25, and Medium Contrast.

In case you're wondering why:  ACR is set up so that ACR will produce the profile's embedded tone curve, at ACR's default curve settings.  ACR's default curve settings are Brightness, Contrast, and Point Curve = 50, 25, and Medium Contrast (respectively).  If you've set your profile's embedded tone curve (via DNG PE's Tone Curve panel) to Linear, then that's exactly what you'll get at these ACR defaults. 

[BobFisher]

Got it.  Thanks, Eric.

[deejjjaaaa]

Sandy, deejjjaaaa,

It is true that DNG Converter doesn't have all of ACR/LR engine in it.

It is also true that DNG Converter needs enough of it to build thumbnails & embedded previews

thank you for clarifications ! noted.

PS: now we need to find out what is missing though 

[Tim Lookingbill]

thank you for clarifications ! noted.

PS: now we need to find out what is missing though 

What I suspect is missing is based on examination of that Nikon D5xx tone curve and the similarities to the one I created to get exact Lab numbers of a perfectly exposed (within specs) CCchart.

I suspect we're seeing the effects of sensor gain on linearity brought about by the nature of electronics in response to a REAL D50-ish light source. Just throwing it out there.

Hopefully Eric Chan will set me straight on that suspicion.

[BobFisher]

I suspect we're seeing the effects of sensor gain on linearity brought about by the nature of electronics in response to a REAL D50-ish light source. Just throwing it out there.

How would the electronics respond differently to different light sources with respect to sensor gain?

[Tim Lookingbill]

How would the electronics respond differently to different light sources with respect to sensor gain?

I'm basing my suspicion on the increased amount of electrons bombarding a sensor just before peak saturation which looks to be around 200RGB region. The pulled down section in both mine and the Nikon D5xx curve hint at a need to correct in that area as a way to compensate for the non-linearity caused by this electrical phenomenon.

I doubt the electronics of Bayer sensors built into a digital camera system are as high quality with regard to electronic tolerances as the electronics built into a spectrophotometer used to measure the CCchart.

I'm only guessing on this based on my observing how some rheostat switches in home lighting (LCD brightness adjusts) and volume attenuators in home audio amplifiers behave as loudness or lighting is increased. The behavior of increase starts out smooth but suddenly becomes less smooth and eratic the higher it goes just at the point before maxing out. There's variances between manufacturers in the quality of these switches and how they behave to attenuation.

It's just a guess.

Got to remember a digital camera was never meant to be a copier of reality by the numbers, only an interpreter. The electronics has to be the weak link that prevents it from being that precise.

At least it's a lot more precise than shooting and scanning film.

Here's a speech given by the inventor of the CMOS sensor...

http://www.dpreview.com/news/2011/10/28/ericfossumspeech