Ad
Ad
Ad
Pages: [1] 2 »   Bottom of Page
Print
Author Topic: ACR BaselineExposure offset  (Read 12280 times)
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« on: January 01, 2009, 11:39:35 AM »
ReplyReply

A not so well documented feature of Adobe Camera Raw is its use of a BaselineExposure offset. For details see page 32 of the DNG specification. When one converts a raw file to a DNG, the BaselineExposure is written in a tag of the file and can be read with an EXIF reader. ACR uses the same BaselineExposure offset, and if you are not aware of this offset, it can confound your efforts to expose to the right, with exposure being judged in the ACR histogram.

For the Nikon D3, the BaselineExposure offset is +0.5 EV, which means a positive exposure compensation will be applied to NEFs from this camera. Because of this offset, the highlights may be blown in the ACR histogram when then are intact in the raw file. To investigate this offset, I took exposures of a Stouffer wedge such that step one was just short of clipping. This is what I consider the optimum ETTR. The raw histogram as shown by Rawnalize is shown below:

[attachment=10673:05_Rawnalize.png]

One can also look at the histogram of the Green1 channel by using the split_cfa command of Iris to split off the green1 channel for analysis by an editor. I used ImageJ (a freeware program available from the US National Institutes of Health). The histogram of the green channel is shown below. It is just below clipping

[attachment=10674:03_green1_histo.gif]

If one looks at the histogram of the raw file in ACR with default settings and the Adobe Standard camera profile, the highlights appear blown:

[attachment=10675:ScreenSh..._Exp_0.0.png]

By using a negative exposure compensation of - 0.6 EV, one can eliminate the histogram.

[attachment=10676:ScreenSh..._Exp_0.6.png]

I then rendered the NEF to an aRGB 16 bit TIFF using various ACR settings and split out the green channel in ImageJ and used the Plot Profile command to examine the rightmost 5 patches of the wedge. The image in the top shows that the two brightest patches are completely blown out; instead of five levels, one sees only three. The middle image shows the image rendered with an exposure of -0.5 EV to undo the +0.5 EV offset. One can now see all five levels, but the brightest patch is slightly blown. With an exposure value -0.6 EV, one has a bit of headroom.

[attachment=10677:AdobeComposite.png]

The take home point is that if you want your ACR readouts and histograms to reflect the values in the raw file, you need to compensate for the baseline offset. Otherwise, your images will appear overexposed and you might try compensate for this by using less exposure in the camera and would not get the full benefit of ETTR.

Bill
Logged
NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #1 on: January 01, 2009, 12:40:33 PM »
ReplyReply

Quote from: bjanes
A not so well documented feature...

That's an understatement if I've ever read one. BTW LR works in the same way. Big Brother Thomas knows what's best for you.
« Last Edit: January 01, 2009, 12:41:35 PM by NikosR » Logged

Nikos
Eyeball
Jr. Member
**
Online Online

Posts: 93


« Reply #2 on: January 01, 2009, 03:04:16 PM »
ReplyReply

Bill,

I find your work interesting but I am curious.  Did you happen to make comparisons against Nikons raw software and/or in-camera Jpegs?  Your implication seems to be that only Adobe is applying the Basline Exposure offset.  Is that what you are saying?

Also, it is not completely clear to me if you are claiming that ACR in effect is overexposing all images by 0.6 or if you're just saying that this characteristic makes it necessary to make adjustments for shoot-to-the-right techniques.  If it is the latter, I am not sure that is a major revelation.  You would expect to have to adjust your raw developer when using STTR, no?  If you commonly used approx. the same amount of STTR adjustment, you could also create a preset or default for it.  It would be interesting to see how much, if any, ACR compensates for this when using Auto.

Also, while I would consider myself generally a "shoot-to-the-righter", it seems to me that depending on that last 0.6 of exposure could get kind of iffy.  I know I have read heated forum threads in the past over how wise it may be to stretch too far into the highlights for STTR when the sensor characteristics may not be as linear there as what we may assume.  I would be interested to hear your thoughts.
« Last Edit: January 01, 2009, 03:14:12 PM by Eyeball » Logged
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« Reply #3 on: January 01, 2009, 03:42:03 PM »
ReplyReply

Quote from: Eyeball
Bill,

I find your work interesting but I am curious.  Did you happen to make comparisons against Nikons raw software and/or in-camera Jpegs?  Your implication seems to be that only Adobe is applying the Basline Exposure offset.  Is that what you are saying?

Also, it is not completely clear to me if you are claiming that ACR in effect is overexposing all images by 0.6 or if you're just saying that this characteristic makes it necessary to make adjustments for shoot-to-the-right techniques.  If it is the latter, I am not sure that is a major revelation.  You would expect to have to adjust your raw developer when using STTR, no?  If you commonly used approx. the same amount of STTR adjustment, you could also create a preset or default for it.  It would be interesting to see how much, if any, ACR compensates for this when using Auto.

Also, while I would consider myself generally a "shoot-to-the-righter", it seems to me that depending on that last 0.6 of exposure could get kind of iffy.  I know I have read heated forum threads in the past over how wise it may be to stretch too far into the highlights for STTR when the sensor characteristics may not be as linear there as what we may assume.  I would be interested to hear your thoughts.

The exposure is achieved in the camera and the ACR exposure control merely affects the rendering of this exposure. If you expose fully to the right with the D3, the image will appear blown in ACR unless you use -0.5 EV of exposure correction.

Here is the same image processed with CaptureNX, which reads the camera settings and gives the same results as the in camera JPEG.  This is using the D3 standard picture control. The image is slightly clipped. I've found that the camera histogram with the Standard Picture Control and Adobe RGB give a sufficiently accurate and slightly conservative reflection of the raw file. Some photographers use sRGB and a low contrast setting, since they feel those settings give a better preview.

[attachment=10678:NX.PNG]

Bill
« Last Edit: January 01, 2009, 03:45:18 PM by bjanes » Logged
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #4 on: January 01, 2009, 05:41:32 PM »
ReplyReply

Quote from: Eyeball
while I would consider myself generally a "shoot-to-the-righter", it seems to me that depending on that last 0.6 of exposure could get kind of iffy
Not if you can rely on the in-camera histogram and clipping indication.

Quote
I know I have read heated forum threads in the past over how wise it may be to stretch too far into the highlights for STTR when the sensor characteristics may not be as linear there as what we may assume
The pixel intensity is linear, as long as the saturation is not reached. In fact, the higher the exposure, the closer to true linearity.

The myth of non-linearity has too sources:

1. the pixels of some cameras, particularly Nikons (like the D200 and the D3) don't have the same saturation level; the difference can go into the hundreds. However, each pixel delivers linear values if treated properly;

2. if pixel saturation occurs, but it affects only one or two of the raw channels in an inspected area, then the resulting RGB values are not linear.
Logged

Gabor
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« Reply #5 on: January 02, 2009, 08:17:23 AM »
ReplyReply

Quote from: Panopeeper
Not if you can rely on the in-camera histogram and clipping indication.


The pixel intensity is linear, as long as the saturation is not reached. In fact, the higher the exposure, the closer to true linearity.

The myth of non-linearity has too sources:

1. the pixels of some cameras, particularly Nikons (like the D200 and the D3) don't have the same saturation level; the difference can go into the hundreds. However, each pixel delivers linear values if treated properly;

2. if pixel saturation occurs, but it affects only one or two of the raw channels in an inspected area, then the resulting RGB values are not linear.


The behavior of the D3 at saturation is interesting. Shown below are histograms of the Green2 and Blue channels at clipping. The Green2 channel has 4 distinct peaks, all below the 14 bit limit of 16383, whereas the blue channel clips mostly at 16383 with a few stragglers to the left. The numbers below the histogram show the number of pixels in bins of the histogram near clipping.

[attachment=10694:002_g2_b...stograms.png]

If one looks at the blue channel one step short of clipping, there is a nice bell shaped curve with a standard deviation of 161 ADU.

[attachment=10695:002_r_histogram.png]

The total variance (standard deviation squared) can be attributed to shot noise, PRNU (pixel response nonuniformity), variations in the target density due to dust and imperfections, and a very small amount of read noise. According to studies of Bill Claff, the full well electron capacity of the D3 is 63,600 electrons and the gain is 3.97 electrons per 14 bit ADU. The mean of the red channel in the second histogram is 14180, representing 56293 electrons (14180 * 3.97). The shot noise is sqrt(14180) =  electrons, or 237 electrons. In terms of ADUs the shot noise would be 60 ADU. The total variance is 161^2 = 25842 and the variance due to shot noise is 60^2 = 3572 (the calculations are done with full precision, not rounded numbers). Subtracting, one gets a residual variance of 22271, which is a standard deviation of 149. Thus, near saturation noise sources other then shot noise constitute the majority of the noise, contrary to what is normally presented. Emil Martinec has a masterful paper explaining these noise sources. Be sure to look at his rollover figure 7.

Target noise and PRNU are fixed pattern noise and can be eliminated by taking two identical shots and subtracting them, leaving only random noise. This would be an exercise for the future.

The Nikon D3  has 12 channel readout and this could contribute to PRNU if the channels are not perfectly balanced. I think that this accounts for the peaks in the green channel at clipping. The amplifier gain for the green channel is below the amount that would result in clipping in the ADU, whereas the gain in the blue channel is high enough so that most of the clipping occurs in the ADU.

Bill
« Last Edit: January 02, 2009, 08:19:24 AM by bjanes » Logged
Eyeball
Jr. Member
**
Online Online

Posts: 93


« Reply #6 on: January 02, 2009, 10:03:46 AM »
ReplyReply

Thanks for the additional info on the Nikon software and Jpegs, Bill.  Have you heard any comments from the Adobe team as to why the offset is being applied?

Also, I went back in the Adobe Forum archives and refreshed my memory on the non-linearity discussion.  I was laughing to myself because I had forgotten that you were the culprit that started the whole bru-ha-ha.
Logged
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« Reply #7 on: January 02, 2009, 10:24:58 AM »
ReplyReply

Quote from: Eyeball
Thanks for the additional info on the Nikon software and Jpegs, Bill.  Have you heard any comments from the Adobe team as to why the offset is being applied?

Also, I went back in the Adobe Forum archives and refreshed my memory on the non-linearity discussion.  I was laughing to myself because I had forgotten that you were the culprit that started the whole bru-ha-ha.


Yes, I have. See this ACR Fourm Thread. The baseline exposure offset is helpful for photographers who use different cameras and want to get consistent results. However, if you are using only one camera, the baseline offset is not a good idea IMHO. However, once you are aware of it, you can easily compensate for its effects and I don't think that it is a big deal.

Bill
Logged
JeffKohn
Sr. Member
****
Offline Offline

Posts: 1671



WWW
« Reply #8 on: January 02, 2009, 11:27:19 AM »
ReplyReply

Quote from: bjanes
Yes, I have. See this ACR Fourm Thread. The baseline exposure offset is helpful for photographers who use different cameras and want to get consistent results.

IMHO Eric's response in that thread is weak at best. Even if this is their supposed goal, it would only make sense for users who are using different cameras with a single light meter. The fact that my D300 might expose 1/2 stop less than a 5D at any given iso/shutter/aperture is of no interest to me whatsoever, because I would never use the 5D's meter to determine exposure on my D300. Adobe is basically causing headache and confusion for a large majority of their users just so a few that use multiple cameras with the same meter can get consistent results?

Quote
However, once you are aware of it, you can easily compensate for its effects and I don't think that it is a big deal.
It's a big deal because Adobe is being so sneaky about it. There's no indication in ACR that it's even happening, let alone a way to turn it off.  Yes, you can apply Exposure Comp adjustment to undo it, but you have to know how much to use, and if your camera happens to be one that gets a different baseline adjustment at different ISO's, you can't just change the camera default;  you're going to have to create presets and be sure to apply the correct one to each shot based on its ISO.

The whole thing is just dumb. As Gabor has pointed out, there are some cases where a baseline exposure adjustment makes sense, as in the case of Canon's Highlight Tone Priority mode. But the way they're using it for Nikon DSLR's is of no benefit at all to most users, in fact quite the opposite considering how many users are completely in the dark about this issue. I hate to think how many people have dialed in a permanent -2/3 EV compensation in the camera so that their files look the way they want them to in ACR because they don't know about this.
Logged

Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #9 on: January 02, 2009, 01:00:58 PM »
ReplyReply

Quote from: bjanes
Shown below are histograms of the Green2 and Blue channels at clipping
Bill, for the sake of this discussion (which is a bit departing from the baseline exposure compensation), do you mind uploading this file, the 002? I have #003, #005, #005b, but not this one.

Thanks
Logged

Gabor
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« Reply #10 on: January 02, 2009, 01:52:59 PM »
ReplyReply

Quote from: Panopeeper
Bill, for the sake of this discussion (which is a bit departing from the baseline exposure compensation), do you mind uploading this file, the 002? I have #003, #005, #005b, but not this one.

Thanks

Gabor,

I think this is the file you are looking for. Others can have a look too, if interested before the download limit expires.

Stouf0003.DNG

Here are some BaseLine offsets for which I have data. Perhaps you and others can add to the list. It is better to have data rather than bitch about Thomas Knoll's decision to implement the feature. I agree that the "feature" should be better documented.

[attachment=10700:Snap1.gif]
Logged
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #11 on: January 02, 2009, 03:26:50 PM »
ReplyReply

Quote from: bjanes
I think this is the file you are looking for. Others can have a look too, if interested before the download limit expires
Thanks, Bill. For the unlikely but not impossible case that someone else too is interested for these technical details, I post here my findings, instead of sending them to you.

Quote
The Green2 channel has 4 distinct peaks, all below the 14 bit limit of 16383
The D3's green channels clip from 15750. There are no distinct peaks there; the peaks you see are caused by the distribution of the saturated pixels.

It would be interesting to see a totally saturated image, i.e. all pixels saturated.

Bill, if you create one, pls send it to me as well, but rather in raw format, not in DNG. It would be even better to create several such shots at different ISOs (200, 400, 800), at different times, in order to verify if the pattern is really constant with a given copy.

Quote
whereas the blue channel clips mostly at 16383 with a few stragglers to the left
The "stragglers" to the left are unclipped pixels from strip 2. The D3 red and blue channels saturate at a single level; unfortunately, this is not a universal constant. The blue is 16383 with your copy of D3 and with several others, but I found 16263 with two different copies. Darn! At least the red seems to be constant.

The green saturation is interesting. Even more intersting, that this occurs with other cameras as well (CCDs as well, and some Canons too). There is a detailed demo in the Rawnalyze guide under Exposure (from your D200), I post5 here only a single capture showing the strange change of saturation points.

Attached is the full histogram of the posted DNG image; the white level is 15892 (decided by the DNG converter, which is the same as what ACR is thinking). This is wrong left and right: the red and blue clip much above this levels, while the green starts clipping sooner. This demonstrates an error in the DNG specification: it provides only for a single saturation level.

The fine histogram shows, that the indicated saturation level 15892 "misses" a part of the factual green saturation. (Rawnalyze assumes 15750 for the green clipping).

The third capture shows the column-wise clipping; the displayed pixels are between 15700 and 15898.
« Last Edit: January 02, 2009, 03:30:26 PM by Panopeeper » Logged

Gabor
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #12 on: January 02, 2009, 03:29:36 PM »
ReplyReply

Quote from: bjanes
I agree that the "feature" should be better documented.
The "documentation" of ACR is close to worthless: it is a shame.

The auto-adjustment should be displayed with warning, so that users are alerted. Even better: it should be eradicated.
Logged

Gabor
stewarthemley
Guest
« Reply #13 on: January 03, 2009, 06:30:21 AM »
ReplyReply

Bill and Gabor I want to thank you for bringing this to light (excuse the pun!). It's something I had noticed when comparing various raw converters and not had the time or technical knowledge to investigate. I agree that this certainly should be explained very clearly in LR/ACR - or removed. Half a stop exposure can ruin a shot if, say, it is meant to be deliberately "under" or "over" exposed. Not wishing to have too much of a go at Adobe (generally I think they are fairly responsive to their user base) it strikes me as being similar to Capture One applying sharpness, etc without informing the user. I like to make my own decisions concerning such aspects, otherwise I would be happy to use jpgs.
Logged
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« Reply #14 on: January 03, 2009, 08:21:00 AM »
ReplyReply

Quote from: stewarthemley
Half a stop exposure can ruin a shot if, say, it is meant to be deliberately "under" or "over" exposed. Not wishing to have too much of a go at Adobe (generally I think they are fairly responsive to their user base) it strikes me as being similar to Capture One applying sharpness, etc without informing the user. I like to make my own decisions concerning such aspects, otherwise I would be happy to use jpgs.

Stewart,

The exposure is made in the camera and the ACR settings do not the actual exposure, but only the rendering of the image. If you want to darken or lighten the image for artistic effects, you can easily so so with the exposure slider. The raw file is linear, and if you want the rendered image to appear underexposed, it is still desirable to expose normally and then render the image to the desired tones. Here is a good article on the rendering of images.

Bill
Logged
Ben Rubinstein
Sr. Member
****
Offline Offline

Posts: 1733


« Reply #15 on: January 03, 2009, 10:11:17 AM »
ReplyReply

Would creating a custom profile with the DNG Profile Editor enable a 'true' level of brightness/saturation?
Logged

Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #16 on: January 03, 2009, 10:22:49 AM »
ReplyReply

Quote from: pom
Would creating a custom profile with the DNG Profile Editor enable a 'true' level of brightness/saturation?
No, that is a different level of control.

One can include the respective counter-adjustment in the ACR camera profile. However, one has to be careful with certain ISO settings, for

1. the automatic adjustment is different for the lowest ISO in those cases, when that is not the native ISO. Examples: the Nikon D3 and D300 (the adjustment is +0.5 EV from 200, -0.5 EV @ 100, I don't have any files with 125 and 160. The 40D is messed up thoroughly..

2. the adjustment is justified in some cases, unjustified in otehrs, and justified but incorrect in certain cases.
Logged

Gabor
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« Reply #17 on: January 03, 2009, 08:25:20 PM »
ReplyReply

Quote from: Panopeeper
It would be interesting to see a totally saturated image, i.e. all pixels saturated.

Bill, if you create one, pls send it to me as well, but rather in raw format, not in DNG. It would be even better to create several such shots at different ISOs (200, 400, 800), at different times, in order to verify if the pattern is really constant with a given copy.

I have two totally saturated images, one at IS0 200 and another at ISO 6400

[attachment=10726:Sat_200.png]

[attachment=10727:Sat_6400.png]

These NEFs and several others at 200 and 6400 are in this ZIP.

Bill






Logged
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #18 on: January 04, 2009, 01:33:39 PM »
ReplyReply

Quote from: bjanes
I have two totally saturated images, one at IS0 200 and another at ISO 6400

This is a frustrating camera; it is inconsequential. Although it is not surprizing, regarded that analogue events are part of the game, it goes a bit too far.

First, I have to revoke my statement, that the blue saturates at once. I have not seen this before, but here I see a saturation range of about 100 levels. On the other hand, the red does saturate at once.

However, the green is the really fancy one. The ISO 200 shots are very close to each other, the difference between the 4sec and 30se is very small (though large enough to make a "template shot" useless), but the ISO 6400 shots are varying much more. The 1/8sec follows the ISO 200 shots, but they change dramatically from 1/4sec (see below).



Now, the really interesting side: the pattern of saturation. The actual levels are not constant, but the pattern is the roughly the same except the longer exposed ISO 6400 shots. Following capture shows the green saturation to the level 15858. The orangy selection shows a vertical line, which is a separation between blocks of patterns. There are three such separations in the entire image, their locations are fixed. If you check out the transitional saturation of the blue channel, you find that those "early saturated" blues are just at those lines.

All this is, as I see it now, useless from the point of raw image interpretation, but I would like to know the underlying reasons for this behaviour.


Logged

Gabor
bjanes
Sr. Member
****
Offline Offline

Posts: 2779



« Reply #19 on: January 04, 2009, 06:50:47 PM »
ReplyReply

Quote from: Panopeeper
This is a frustrating camera; it is inconsequential. Although it is not surprizing, regarded that analogue events are part of the game, it goes a bit too far.

First, I have to revoke my statement, that the blue saturates at once. I have not seen this before, but here I see a saturation range of about 100 levels. On the other hand, the red does saturate at once.

However, the green is the really fancy one. The ISO 200 shots are very close to each other, the difference between the 4sec and 30se is very small (though large enough to make a "template shot" useless), but the ISO 6400 shots are varying much more. The 1/8sec follows the ISO 200 shots, but they change dramatically from 1/4sec (see below).



Now, the really interesting side: the pattern of saturation. The actual levels are not constant, but the pattern is the roughly the same except the longer exposed ISO 6400 shots. Following capture shows the green saturation to the level 15858. The orangy selection shows a vertical line, which is a separation between blocks of patterns. There are three such separations in the entire image, their locations are fixed. If you check out the transitional saturation of the blue channel, you find that those "early saturated" blues are just at those lines.

All this is, as I see it now, useless from the point of raw image interpretation, but I would like to know the underlying reasons for this behaviour.

Gabor,

The behavior is indeed interesting. As I mentioned previously, the D3 uses 12 channel readout, which is partially documented here. The sensor is read in columns by the sensor. Examination of a 100 x 100 pixel area from the green1 channel at saturation shows this pattern. It appears as if the pixels are being read by columns by different channels.

[attachment=10738:sat_200_10_g1.png]

Individual pixels can be measured in ImageJ. In a given column, the pixel values are plus or minus one, indicating that they are likely being read by a single ADC. In the horizontal direction, the values vary markedly in a periodic fashion, indicating that they are being read by different ADCs.

[attachment=10739:sat_200_..._measure.png]

However, the blue channel shows more variation within one column and the others are completely clipped.

[attachment=10740:Sat_200_..._measure.png]

What do you think? Others are invited to speculate.

Bill
« Last Edit: January 05, 2009, 06:51:14 AM by bjanes » Logged
Pages: [1] 2 »   Top of Page
Print
Jump to:  

Ad
Ad
Ad