Ad
Ad
Ad
Pages: [1] 2 3 ... 6 »   Bottom of Page
Print
Author Topic: Camera's histogram reliable to the RAW data  (Read 258256 times)
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« on: January 07, 2008, 01:22:11 PM »
ReplyReply

We all know that camera's display in terms of histogram and blown highlight is based on a in-camera JPEG version of the RAW file, and is pesimistic with respect to the real RAW data.

One can set a low contrast value in the camera parameters, no saturation, and so on,... but those tricks have little effect in making the camera's more reliable to the RAW information.

Thanks to playing with DCRAW I learnt some time ago how critical is the white balance in transforming the RAW's histogram, by overexposing one or even 2 channels by 1 or even 1.5 f-stops.

I have tried to find out the way to neutralise my 350D's white balance setting a neutral (1.0 scaling in all 3 channels) white balance. Images will display in a wrong colour (of course RAW data remains intact), but the histogram and blown blinking highlights should improve.

Tell me what do you think.
The detailed article can be found here:
UNIWB. MAKE CAMERA DISPLAY RELIABLE


Steps:

1. I create a colour chart representing all possible RGB proportions (I chose to fix G=64). Over this chart I shot and I will try to find out which tone becomes pure gray in the RAW file when no WB at all is applied (that is camera's "true gray"):

 :



This is the RAW magenta chart that can be used on any 350D to achieve this specificic WB: UniWB350D.cr2

What do you think?
« Last Edit: April 05, 2009, 11:56:47 AM by GLuijk » Logged

bansal98
Newbie
*
Offline Offline

Posts: 5


« Reply #1 on: January 08, 2008, 10:22:20 PM »
ReplyReply

I've been doing a similar thing for gamma correction. The idea being to get as close to the raw histogram as possible. See here.
Logged
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #2 on: January 08, 2008, 10:40:19 PM »
ReplyReply

GENIAL!!!

I will have the chart printed and do the same with my cameras.
« Last Edit: January 08, 2008, 10:41:46 PM by Panopeeper » Logged

Gabor
Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #3 on: January 08, 2008, 10:49:26 PM »
ReplyReply

Btw, how did you make the basis color chart? If I make a graduation from (0,64,255) - (255,64,0), then I don't get any green, like you got in the upper left corner.
Logged

Gabor
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #4 on: January 09, 2008, 08:28:35 AM »
ReplyReply

Quote
Btw, how did you make the basis color chart? If I make a graduation from (0,64,255) - (255,64,0), then I don't get any green, like you got in the upper left corner.
[{POST_SNAPBACK}][/a]

I set (almost) arbitrarily G=64 since I knew the higher sensitivity of the sensor for the G channel would produce seeing a pure gray at some R&B dominant point.

Regarding the gradation, R and B are independent of each other: R runs 0 to 255 left to right, and B runs 0 to 255 up to down, so top left corner is pure green (0,64,0), and (64,64,64) is the gray point for this test chart. All that is clearer looking at the Hue distribution of the test chart:




Find here the test chart in 1024x1024 size: [a href=\"http://img520.imageshack.us/img520/6944/cartag64zs5.jpg]CARTA DE PRUEBA G64[/url]

I don't recommend you to print the charts but shoot them straight on your monitor instead.
This will not invalidate the procedure at all as long as the magenta chart is shot under the same conditions (monitor settings + room light conditions) as the test chart.
Thus your monitor does not need to be calibrated at all (mine is not), and your room's light conditions can also be any as long as they remain constant (be careful with sunlight entering your window, these experiments are best done at night).
You will even be more precise this way, and you will save time and ink!

BTW I have been told Nikon cameras (from D200) can do this by simply configuring the so called 'UniWB' white balance mode in the camera. No idea how difficult is that.

Regards.

PS: this is a sample shot using the custom WB achieved, comparing the result of developing the same RAW file:
- Left: with its embedded WB from camera (i.e. our specific neutral WB)
- Middle: no WB at all (forcing 1.0 multipliers in DCRAW)
- Right: Daylight WB preset



First 2 images match very well as expected.

In my first quick tests, blinking areas in the camera's display (350D) are very close to real RAW blown areas. In fact, surprisingly they are a little bit more optimistic than a strict partial saturation detection over the RAW data. So watch out!.
Whether this happens or not depends solely on the 350D's implementation of the blinking highlights; the WB calculation procedure is conceptually flawless.
« Last Edit: July 18, 2008, 05:05:38 PM by GLuijk » Logged

Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #5 on: January 09, 2008, 08:48:58 AM »
ReplyReply

Quote
I've been doing a similar thing for gamma correction. The idea being to get as close to the raw histogram as possible. See here.
[a href=\"index.php?act=findpost&pid=166044\"][{POST_SNAPBACK}][/a]

I have your article in mind since I saw it some days ago. Will have a close look tonight.
Logged

Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #6 on: January 09, 2008, 04:46:56 PM »
ReplyReply

Guillermo,

I carried out the "neutral WB calibration" (I too came to the idea of doing it with the monitor). The result is as good as I can imagine it; the blinking clipping indication of the camera reflects the clipping down to a tiny clipping, barely perceivable. This is much better than what I got with setting the color temperature and color correction (that method would certainly lead to an exact adjustment, but that is very tiresome).

Now I can rely on the blinking on the camera's display. The histogram itself is only a rough indication because it is too tiny and one can't see if it really reaches the right end.

Are you the originator of this idea? I want to spread it and want to give the well-deserved credit to the originator.

Btw, I copied the template on every CF card, so that if it becomes necessary to change the custom balance, one can return to this setting any time. Of course I protected the template from accidental erasure.
Logged

Gabor
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #7 on: January 09, 2008, 04:54:37 PM »
ReplyReply

Quote
I carried out the "neutral WB calibration" (I too came to the idea of doing it with the monitor). The result is as good as I can imagine it; the blinking clipping indication of the camera reflects the clipping down to a tiny clipping, barely perceivable. This is much better than what I got with setting the color temperature and color correction (that method would certainly lead to an exact adjustment, but that is very tiresome).

Now I can rely on the blinking on the camera's display. The histogram itself is only a rough indication because it is too tiny and one can't see if it really reaches the right end.

Are you the originator of this idea? I want to spread it and want to give the well-deserved credit to the originator.

It's nice, isn't it?

Strangely, I am the originator of the idea. Last Monday was bank holiday in Spain and I spent the whole Sunday night shooting my monitor lol. Didn't want to go to bed until I published the article.

Feel free to distribute the idea but from my experience, don't expect too much feedback or good vibrations, few have the knowledge to appreciate such intriguing camera fine tunings.

Just curious, what multipliers did you achieve? I was surprised at my 5,6% error, remarkably low since I was really sleepy and didn't take any care to be precise. Wanted to go to bed!
« Last Edit: January 09, 2008, 05:00:18 PM by GLuijk » Logged

Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #8 on: January 09, 2008, 05:23:52 PM »
ReplyReply

I am not familiar with DCRaw and have not used that for the calibration; I picked the grey point from the shot of the original template with Rawnalyze, and verified the resulting grey too with it. But I was curious re the factors, so I found in DCRaw that there is an option -i for information. It displayed (1057, 1045, 1083, 1045) - this is 3.6%.

Re shooting the monitor you need to think of followings:

1. defocus and do not go too close, otherwise you record the "gaps" between the monitor pixels as well (Moire-like effect),

2. do not go too far, for the "grey to be" area (R=162, G=64 B=104 for your camera) has to cover the circle on the focusing screen. Be generous, because that area is involved in the custom WB setting. I increased the size, so that I got a larger image.

Guillermo, you deserve a big THANK for this idea.

Btw, if you want to distribute it, you can upload the final raw file for the WB setting, that depends only on the camera but not on the illumination any more. I will offer this on the Rawnalyze page, with due credit to you (I am at the middle of the new manual, it is done in a few days).
Logged

Gabor
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #9 on: January 09, 2008, 05:46:47 PM »
ReplyReply

Quote
I am not familiar with DCRaw and have not used that for the calibration; I picked the grey point from the shot of the original template with Rawnalyze, and verified the resulting grey too with it. But I was curious re the factors, so I found in DCRaw that there is an option -i for information. It displayed (1057, 1045, 1083, 1045) - this is 3.6%.

Re shooting the monitor you need to think of followings:

1. defocus and do not go too close, otherwise you record the "gaps" between the monitor pixels as well (Moire-like effect),

2. do not go too far, for the "grey to be" area (R=162, G=64 B=104 for your camera) has to cover the circle on the focusing screen. Be generous, because that area is involved in the custom WB setting. I increased the size, so that I got a larger image.

Guillermo, you deserve a big THANK for this idea.

Btw, if you want to distribute it, you can upload the final raw file for the WB setting, that depends only on the camera but not on the illumination any more. I will offer this on the Rawnalyze page, with due credit to you (I am at the middle of the new manual, it is done in a few days).
[a href=\"index.php?act=findpost&pid=166215\"][{POST_SNAPBACK}][/a]

Grrr, so you beat me. What cameras do you have?
With dcraw -v -w pic.cr2 DCRAW will display the 1.0 based multipliers.

I already put the 350D UniWB RAW file for download at the end of the first post of the thread. BTW I have just been sent a chart shot using the 40D from other user and the magenta chart (we could call it 'wine chart') for that camera and conditions happened to be:

Canon 40D:


R=170
G=64
B=115

My 350D was:
Canon 350D:
R=162
G=64
B=104

Of course comparing is a bit nonsense since the differences are in this case not only for the cameras but also include my and the other user's conditions (monitor + lighting conditions), so I will send him the wine chart to obtain the final UniWB40D.cr2 RAW file.
Anyway, looking at the hue transformations in both cameras, if the 350D's well know issue of getting "oranged" reds, what about the 40D? could it even be worse looking at the distribution?:

« Last Edit: January 09, 2008, 06:39:04 PM by GLuijk » Logged

Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #10 on: January 09, 2008, 09:53:48 PM »
ReplyReply

My "grey to be" for the 40D is (120, 64, 75). Of course, this does not mean anything.

I don't understand your "oranged red" subject. I did not make that step at all. Instead of generating a non-WBd file by DCRaw, I displayed the shot of the rainbow chart (what you posted) with Rawnalyze, no WBing, and looked for grey point (simply by clicking around, and when close, I saw from the numbers, in which direction I have to go to get even closer to true grey). I transformed the co-ordinates of that point over the original (simply by measuring it with a ruler on the screen and interpolating), so I got the RGB of the "grey to be". I created that square.

Then I shot that "winechart", though my one is not really "winy" and checked the color in Rawnalyze. It was a bit off grey, I corrected directly the winechart (for example the red was too much, so I reduced it in the winechart), and shot it again. I made three iterations and then shot a serie of the scenery outside with 1/3 stops, and compared the clipping indication with that in Rawnalyze (it produces something like ACR's clipping indication). Since then I have been happy. I don't touch it, because it can become only worse :-)

My 40D template can be downloaded from

http://www.cryptobola.com/PhotoBola/WB_00040.CR2

It can be stored and used on the CF card with this name (both the 40D and the 20D are picky about accepting files with names different from the original scheme).
Logged

Gabor
NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #11 on: January 10, 2008, 02:58:32 AM »
ReplyReply

Maybe I'm missing something, but how is this method original and different than the well known Uniwb advocated by Iliah Borg on dpreview and nikoncafe for some years now and actually quite extensively used by Nikon photographers by setting up unitary WB coefficients in their Nikon cameras?

(PS. BTW, unitary wb coefficients are not handled gracefully in ACR/LR and some other raw converters so one has to use 'not-quite unitary' coefficients if one wants to work in these converters).
« Last Edit: January 10, 2008, 03:02:55 AM by NikosR » Logged

Nikos
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #12 on: January 10, 2008, 04:25:01 AM »
ReplyReply

Quote
I don't understand your "oranged red" subject. I did not make that step at all.

What I mean by the "orange reds" in the Canon is that many Canon user complain at the red colours their cameras capture, which trend to become a bit orange (this is a fact I have often read about in the forums).

And lookig at the RGB -> HSV transformations of the original (test chart) and the results captured by the camera sensors, we can notice 2 effects:

1. Displacement of the gray (R=G=B ) point: this was expected due to the different relative channel sensitivities.

2. But also a colorimetric effect, which makes tone (Hue) change. You can see that the angle of Hues that could be considered as acceptably "red" in the original test chart, becomes a much narrower angle in the 350D and specially the 40D.
Of course this was influenced by the lilghting conditions present in the scene (a calibrated monitor and 100% neutral ambience lighting would be needed to properly confirm). But anyway I wonder if also the camera sensor could have an influence on this.
After all R channel is just a band pass filter which can behave far from ideally, and could happen that in these sensors the R channel trend to have more gain on the frequency band close to oranges.




BTW your UniWB RAW yields in DCRAW multipliers  1.016330 1.000000 1.033622.
Do you mind if I offer it for download in my website and other forums?
(I will reference you of course)

One question: by opening it I see it display not only the chart but also a screenshot of PS. Does this mean the camera only takes into account a centred portion of the RAW to calculate the custom WB? I didn't know that. Do you know exactly what size (%) of the total image surface?

Thanks
« Last Edit: January 10, 2008, 11:32:19 AM by GLuijk » Logged

Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #13 on: January 10, 2008, 04:29:26 AM »
ReplyReply

Quote
Maybe I'm missing something, but how is this method original and different than the well known Uniwb advocated by Iliah Borg on dpreview and nikoncafe for some years now and actually quite extensively used by Nikon photographers by setting up unitary WB coefficients in their Nikon cameras?

(PS. BTW, unitary wb coefficients are not handled gracefully in ACR/LR and some other raw converters so one has to use 'not-quite unitary' coefficients if one wants to work in these converters).
[a href=\"index.php?act=findpost&pid=166284\"][{POST_SNAPBACK}][/a]

I knew that Nikons from the D200 could be configured for UniWB, but thought they had just this config option in the camera. Is the method for Nikons the same as explained here?    then why was it assumed to be valid just for Nikons? I always thought they had something special that made UniWB much easier in them. I have a feeling I could have reinvented the wheel here hehe.


Of course the unitary converters are not to be fed into your RAW developer, that's nonsense. This is only a white balance for capture, not for developing. Commercial RAW developers find problems in handling certain WB multipliers, probably when converted they fall in too extreme Temperature/Tone parameter values.

Just curious: a friend of mine turned his old 300D into an IR camera. It worked fine in the capture, but he found that ACR was not capable to achieve the needed Temperature for a proper development. He used DCRAW instead, which has no limitations on the multipliers, and he got his pics OK then.
« Last Edit: January 10, 2008, 04:37:14 AM by GLuijk » Logged

NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #14 on: January 10, 2008, 04:38:42 AM »
ReplyReply

Quote
I knew that Nikons from the D200 could be configured for UniWB, but thought they had just this config option in the camera. Is the method for Nikons the same as explained here? then why was it assumed to be valid just for Nikons? I always thought that.

Of course the unitary converters are not to be fed into your RAW developer, that's nonsense. This is only a white balance for capture, not for developing.
Just curious: a friend of mine turned his old 300D into an IR camera. It worked fine, but he found that ACR was nos capable to achieve the needed Temperature for a proper development. He used DCRAW instead, which has no limitations on the multipliers, and he got his pics OK then.
[{POST_SNAPBACK}][/a]

With Nikon cameras that support uploading the coefficients directly (e.g. D2 cameras) you can do it directly. For other cameras you just load a custom white balance (examples of which Mr. Borg and others have provided) created, I suppose, in a similar fashion as you do here.

[a href=\"http://www.pochtar.com/Uni.zip]http://www.pochtar.com/Uni.zip[/url] provides you with a UniWB custom setting and a 'not-quite-uni'WB for raw converters that cannot handle coefs of exactly 1.0



So your work here might be original with regards the specific Canon implementations but the idea is far from original. A Google search for UniWB will point you to a lot of discussions.

I'm not saying that one feeds these coefficients in the converter. I'm just saying that if a RAW file with coeffs of exactly 1.0 (at least for Nikon cameras) is fed into ACR/LR, ACR/LR will not treat them correctly when you subsequently try to change WB. This has been acknowledged by Thomas Knoll in Adobe forums. http://www.adobeforums.com/cgi-bin/webx/.3bb6a85c.3bbfce73/2

Other converters (e.g. Nikon Capture and NX) are known to crash if an attempt at adjusting the exposure is done in the converters. BTW, such adjustment even if possible would not lead to the benefits one might expect when using normal WB coefficients as there is no artifically clipped channels in the raw conversion as rendered by the converter in the first place! That's the benefit of UniWB.
« Last Edit: January 10, 2008, 04:47:47 AM by NikosR » Logged

Nikos
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #15 on: January 10, 2008, 04:46:19 AM »
ReplyReply

Quote
I'm not saying that one feeds these coefficients in the converter. I'm just saying that if a RAW file with coeffs of exactly 1.0 is fed into ACR/LR, ACR/LR will not treat them correctly when you subsequently try to change WB. This has been acknowledged by Thomas Knoll in Adobe forums.

I cannot find a logic in this. Even if ACR or any RAW developer cannot handle the 1.0 multipliers (which is understandable), they should not fail later to apply a correct "regular" WB since pure RAW data are not affected by the UniWB. Could you please provide me that Thomas Knoll reference?

Regarding the copyright/original idea matter, believe me, I am 0% interested in being acknowledged as the inventor of anything. I just had fun configuring my camera for a neutral WB last night, that is all.
People waste too much time in the forums discussing who is the owner or the copier of new ideas, instead of being positive and trying to get the best of our hobbies/work.
« Last Edit: January 10, 2008, 04:47:02 AM by GLuijk » Logged

NikosR
Sr. Member
****
Offline Offline

Posts: 622


WWW
« Reply #16 on: January 10, 2008, 04:50:40 AM »
ReplyReply

Quote
I cannot find a logic in this. Even if ACR or any RAW developer cannot handle the 1.0 multipliers (which is understandable), they should not fail later to apply a correct "regular" WB since pure RAW data are not affected by the UniWB. Could you please provide me that Thomas Knoll reference?

Regarding the copyright/original idea matter, believe me, I am 0% interested in being acknowledged as the inventor of anything. I just had fun configuring my camera for a neutral WB last night, that is all.
People waste too much time in the forums discussing who is the owner or the copier of new ideas, instead of being positive and trying to get the best of our hobbies/work.
[a href=\"index.php?act=findpost&pid=166294\"][{POST_SNAPBACK}][/a]

For your first comment pls. see my updated post above.

For the second, I have no reason not to believe you, but I thought that setting the record straight was the correct thing to do, since there were posters in this thread assuming exactly what you were not interested in. Blame it on my scholarly background.
« Last Edit: January 10, 2008, 04:54:20 AM by NikosR » Logged

Nikos
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #17 on: January 10, 2008, 05:01:20 AM »
ReplyReply

Quote
For your first comment pls. see my updated post above.

"Using the "UNIWB" hack is confusing Camera Raw into thinking the image is a double exposed image, and Camera Raw using a different processing path."

I understand. This is an elegant way to admit that ACR has an undesired behaviour (i.e. a bug)  
Everytime I find more happy with Dave's  DCRAW.

PS: Panopeeper, I am terribly sorry for making you believe I was the originator of this idea (if it is true someone thought of it before I did, although I couldn't find a similar procedure anywhere valid for any camera model and brand), and I hope you can forgive me some day for such an offence. lol
« Last Edit: January 10, 2008, 11:40:30 AM by GLuijk » Logged

Panopeeper
Sr. Member
****
Offline Offline

Posts: 1805


« Reply #18 on: January 10, 2008, 04:00:25 PM »
ReplyReply

Gullermo,

1. of course I have no problem if you offer anyone the 40D WB template I created,

2. I made a search for uniwb and did not find anything re how that has been achieved. There are several ways, for example there is a huge color checker box from Gretag with 120 or so colors. When one shots that from the monitor, one can find a color, which is quite close to grey without WB (non-demosaiced, of course). From there, it requires I guess perhaps five to eight iterations (only guessing) to arrive at the proper color.

The core of the idea is using an "inverted color", and for me you are the inverter.

3. Anyway, if you disseminate the method and/or the raw files, you should explain, that the contrast, saturation, sharpness and color tone have to be "neutralized". How to achieve that depends on the camera. For example the 40D shows a "0" at the appropriate places: at the left end of sharpness and at the middle of the other ones.
Logged

Gabor
Guillermo Luijk
Sr. Member
****
Offline Offline

Posts: 1290



WWW
« Reply #19 on: January 10, 2008, 04:18:26 PM »
ReplyReply

Quote
2. I made a search for uniwb and did not find anything re how that has been achieved. There are several ways, for example there is a huge color checker box from Gretag with 120 or so colors. When one shots that from the monitor, one can find a color, which is quite close to grey without WB (non-demosaiced, of course). From there, it requires I guess perhaps five to eight iterations (only guessing) to arrive at the proper color.

The core of the idea is using an "inverted color", and for me you are the inverter.

3. Anyway, if you disseminate the method and/or the raw files, you should explain, that the contrast, saturation, sharpness and color tone have to be "neutralized". How to achieve that depends on the camera. For example the 40D shows a "0" at the appropriate places: at the left end of sharpness and at the middle of the other ones.
[a href=\"index.php?act=findpost&pid=166387\"][{POST_SNAPBACK}][/a]

This is a good point, I want to make tests in my camera to find out since I wonder:
- Contrast: would be 0 value a good idea or a negative one? (I imagine negative is an inverted S curve).
- Saturation: I guess the right value here is 0, non-modifying saturation value.
- Sharpness: I don't think it will affect too much, but 0 should be OK.
- The big question: what about sRGB/AdobeRGB? since sRGB expands more the histogram, do you think it could be a good idea to set AdobeRGB? or we could even fall into being too optimistic with the neutral WB?

I would really like to know which is the camera's criteria to blink. I think it is partial saturation (any channel saturated) but I am not sure if there are more hidden rules.

Regarding the authority, I really don't care. I had this idea Sunday night by myself, I had fun putting it into practice, and if someone already had it before is fine.

PS: I recall this question above: One question: by opening it [your RAW winechart] I see it displays not only the chart but also a screenshot of PS. Does this mean the camera only takes into account a centred portion of the RAW to calculate the custom WB? I didn't know that. Do you know exactly what size (%) of the total image surface?
« Last Edit: January 10, 2008, 04:19:19 PM by GLuijk » Logged

Pages: [1] 2 3 ... 6 »   Top of Page
Print
Jump to:  

Ad
Ad
Ad