Ad
Ad
Ad
Pages: « 1 2 3 [4] 5 6 ... 8 »   Bottom of Page
Print
Author Topic: Camera Profiling - DNG, ICC and alternative methods  (Read 40594 times)
samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #60 on: March 01, 2013, 01:06:46 AM »
ReplyReply

Here's another quickie. Canon 5D2, 2 Solux lamps run at 14 Volt, custom WB of the day.
All displayed in ID, working and output profile set to ProPhoto, rendering intent to rel.col. Otherwise ID defaults except the profiles

Hening, have you tried them in ID's linear mode - all sliders zeroed? These look too contrasty and are not close to the reference values. ID's default settings probably introduces hue and saturation shifts on top of tone shifts that is going to be evaluation tricky.
Logged
samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #61 on: March 01, 2013, 01:59:17 AM »
ReplyReply

Andrew, I think you in particular might find this interesting. I read about your discussion here some time ago and I was interested too how different can two cameras be. Perhaps other folks may also be interested.

Here are four photos from 2 different 5D Mark IIs. I photographed both the ColorChecker and the QPcard. All of them were made in under daylight illumination. the ColorChecker was in direct sunshine, the QP was when the sun got obscured by cloud. Light was rather consistent when switching between cameras.

They are not identical but I am surprised by how close they are. Yes, they are only two units, and I won't mind getting my hands on a few hundred or more 5Ds.
Logged
Hening Bettermann
Sr. Member
****
Offline Offline

Posts: 578


WWW
« Reply #62 on: March 01, 2013, 06:54:25 AM »
ReplyReply

Here are the same 3, everything linear. Obviously, the removal of the camera curve in ID has had no effect on the DNG. So this is not comparable. Also I think it is not a valid procedure just to remove a curve from a profile that is created to have one.  At least this is as I understand Brian.
 
The images are too contrasty? You sound like you have measured them, but visually, I find them close to the target, displayed in the same Solux light to the same exposure value (8 1/3) at which they were shot. Whereas the linear images look muddy as exspected.

Anyway, this was just a quick aside for me. What I really am exploring right now is ColorPerfect, which I find theoretically most intriguing, and - so far - very convincing in a few non-calibrated roll film scans.
Logged

Vladimirovich
Sr. Member
****
Offline Offline

Posts: 1320


« Reply #63 on: March 01, 2013, 09:39:03 AM »
ReplyReply

Light was rather consistent when switching between cameras.
and what was that supposed to prove or disprove ? I can understand when you illuminate sensors in 2 different cameras (the same model) the same way with monochromator in a controlled lab env... but "rather consistent" light outdoors Huh
Logged
samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #64 on: March 01, 2013, 11:01:59 AM »
ReplyReply

The images are too contrasty? You sound like you have measured them, but visually, I find them close to the target, displayed in the same Solux light to the same exposure value (8 1/3) at which they were shot. Whereas the linear images look muddy as exspected.

I should have mentioned to adjust the exposure slider to patch the white patch to it reference value.

Here's my shot of the QP with a custom dcp profile for it. All sliders in ACR zeroed, tone curve linear. Exposure slider adjusted to match the white patch to its reference value. White balanced from the second neutral patch from the left. I find it much closer to the reference values provided by Robin Myers.

Also I think it is not a valid procedure just to remove a curve from a profile that is created to have one.  At least this is as I understand Brian.

Depends on what you are after, I guess. You can strip the curve from the DNG profile manually. Vladimirovich suggested doing so - I gather he prefers it that way.
« Last Edit: March 01, 2013, 11:03:36 AM by samueljohnchia » Logged
samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #65 on: March 01, 2013, 11:10:13 AM »
ReplyReply

and what was that supposed to prove or disprove ? I can understand when you illuminate sensors in 2 different cameras (the same model) the same way with monochromator in a controlled lab env... but "rather consistent" light outdoors Huh

Nothing if you look at it that way. The sun basically didn't move much between shots that were some seconds apart, and no clouds came in to affect the light. I'm surprised that they are this similar, that's all. I was expecting to see much bigger differences since a number of variables were out of my control (actually I would rather it were significantly different, because it might further justify all this custom profiling work). If I may be so bold as to say from two 5D Mark IIs (the cameras were bought months apart) that the cameras are very consistent with they way they capture color, the Adobe generic profile should do a good job, and basically reduces the need for a custom camera profile, except for non-daylight/tungsten light. This could go some way to support your stand that the Standard profiles are enough, and I don't deny that.
Logged
Hening Bettermann
Sr. Member
****
Offline Offline

Posts: 578


WWW
« Reply #66 on: March 01, 2013, 04:46:30 PM »
ReplyReply

So here the 2 ICC renderings are with exposure adjusted. I can not do to DNG profiles what Vladimirovich can, which implies compiling and decompiling.
My white balance  is from an external card. Here are the Lab values for the white patch as the Mac DigitalColorChecker reads them on the screen:
ID: 94-2.137-3.176   QP: 94-1.707-3.340
and in Photoshop:
ID: 94-2-3   QP: 94-1-3
but the a and b values vary a good deal between 1 and 4
Reference: 94-0-2.8
What I wrote about linear vs gamma-encoded profiles was what I understood from Brian, which referred to the color management processing pipeline in ID. If the input profile is e.g. linear, the pipeline exspects linear data, and you can not just change that under ways. Conversion to a gamma encoded image is done on output. However, my understanding may be lacking. Also, that was about gamma encoding, not a linear adjustment of the white point.
Logged

Hening Bettermann
Sr. Member
****
Offline Offline

Posts: 578


WWW
« Reply #67 on: March 03, 2013, 12:18:40 PM »
ReplyReply

Hi
I am in doubt about the purpose/state of this thread. Samuel, are you breeding on a common testing procedure ?

In the meantime, I have fiddlet a little with Iridient Digital and ColorPerfect, with the question: If I want to use CP, am I restricted to MakeTiff as the raw converter, or can I use e.g. Iridient?

Screen shots of the results in the zip.
My conclusion is yes I can use ID with my own linear profile. The canned profile of CP is good, as is my own, but not IDs default if linearised.
Logged

samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #68 on: March 03, 2013, 09:07:18 PM »
ReplyReply

Hi
I am in doubt about the purpose/state of this thread. Samuel, are you breeding on a common testing procedure ?

In the meantime, I have fiddlet a little with Iridient Digital and ColorPerfect, with the question: If I want to use CP, am I restricted to MakeTiff as the raw converter, or can I use e.g. Iridient?

Screen shots of the results in the zip.
My conclusion is yes I can use ID with my own linear profile. The canned profile of CP is good, as is my own, but not IDs default if linearised.


Hi Hening, apologies for my cyber silence. I got swamped with work this week but I managed to put some time in here and there to look deeper into camera profiles.

Let me first address ColorPerfect. I am struggling with its UI. But I am seriously shocked by how accurately its renders colors at its default settings (with the correct WB and white point settings), even with photographs of color targets - with no internal profile to skew colors to a few reference values, leaving us to think what's happening then to the rest of the colors. This I think we both agree on. There are other issues I have with CP. One of them that stopped me from going further with CP is that I cannot set it to render a completely linear tone curve. By design, it applies an automatic black point (BP). Look at this example:



I carefully set the White point correctly in both ACR and CP for this test, not so easy to do in CP! The entire image was remarkably similar tonally except in the shadows, and color rendering. The first image crop on the left is ACR's linear mode. The rightmost image is CP with its recommended BP setting, smartly calculated, may be unique to each image. The third image on the left is when I manually set its BP to zero. Second image from the left is ACR in linear mode, but with the Blacks slider at 5. Coincidentally, the default for Blacks in PV2010 is very similar to the minimum BP setting in CP. On checking with David, he informed me that I should never set the BP to less than 0.001, because of imperfections in image quality from sensors near the noise floor. With sensors performing as well as those on the latest Nikon, Sony and Pentax cameras, I'm not convinced that the BP clipping need be so high. Many photographers will disagree with me on this, and often they would want a strong black in their images and that's ok. For me I would rather decide whether to clip or not upon deciding my output medium, since printers are getting better and better at reproducing tones near black, and things get interesting there. Jon Cone has an interesting article on that.

You can certainly feed ID linear files into ColorPerfect, but David of course recommends MakeTiff. You must be able to get ID to output a linear gamma, unity white balance, non-profile tif to take advantage of Color Perfect's "color integrity", as it was designed.

Back to the profiles. Yes, your latest collection of CC photos all look quite similar, but with white point setting too high, I had to darken all the images to match the white patch reference value. That darkening may have skewed the colors a bit, since I had to work on a gamma corrected image, not a linear gamma version. Also, since this is a screenshot, your display profile may be doing unexpected things to the color.

I stress again that to properly make comparisons, we must first match the white patch to its reference value by setting the correct white point (figuring out which slider in different raw converters does this is not as hard as I thought initially), and set a custom white balance after applying the profile, using the second lightest neutral patch of the CC. It will be unique for every profile (even in the same raw converter) but it is necessary so that the neutral patches are that - neutral. Then let everything else fall where they may.

But here is a synthetic CC overlay of your 2C=ID,-Canon_lin,-WP-adj-in-ID file:


Here is one of mine from ACR in "linear mode" and a custom profile from the DNG PE:


Here is another from Raw Therapee in "linear mode" with a custom profile that I built in Argyll using the arguement "colprof -v -y -qh -am -nc -u" in command line:


I'm not convinced that RT can handle dual table DNG profiles. It messed up with the same DNG profile that I used in ACR. One can set the preferred profile to Daylight, Tunsten, fluorescent or flash, possibly indicating that it is only using one table at one time. Btw this setting does not affect ICC profiles. This one is the "daylight" option:


This one is the "tungsten" option:


Caveat: These are all converted to sRGB for the web, and the cyan patch in row three is not correctly represented. Here is a tif file (10MB), with all the examples in labeled layers, ProPhoto RGB, 16 bits.

I don't have access to a mac at the moment, but I think ID will handle DNG profiles correctly. I must try it out later. I would only be able to run ID in demo mode and cannot make a custom ICC profile for it. Might need some help with that.

I am surprised by how well the ICC profile performed. The ICC profile that I created was a simple matrix profile, with the -u flag in Argyll to "avoid setting the white point to that of the profile chart". Additional information is here. This, I am thinking, makes the ICC profile behave similarly to a DNG profile, assuming you strip the DNG profile of its LUT table. Not the tone curve, because the ICC profile will scale the patch values to the reference values in the matrix curves, effectively a "tone curve" as well.

I'm not sure what the DNG Profile Editor is using for its ColorChecker reference values, but I know for Argyll. It could be that that is causing the subtle differences we see here. Differing tradeoffs in designing the algorithms to map colors could also be another factor. Raw converter handling of camera profiles could also be involved. Many other factors that are not immediately obvious too.

Then there are many ways that one can mess up profile building doing the capture process (solarj is having this problem in another thread), and the profile building process. In the course of my testing, I discovered that the DNG PE is extremely sensitive to the placement of the four control points - slighly changing their positions will result in a visibly different profile. Sometimes hard to tell which is better, because some areas improve while others shift away from ideal. ICC profiling is too complicated to detail in this long post right now.

At this point I don't know if all this work is ever going to be justified. We still don't have luminosity based curve adjustments in the commonly used raw converters, including all the tone affecting sliders that affect a general image tone curve. I also disagree wholly with the results I am getting from ColorPerfect's "scientifically correct" method of saturating images, and I find Joseph Holmes's color variants to be far superior in many ways, not just the visible results. This all might work out in the end assuming that it works as it should, and the image processing pipeline does not introduce hue shifts at all, to invalidate the camera profile. Tough for the stars to align this way.
« Last Edit: March 03, 2013, 09:12:23 PM by samueljohnchia » Logged
solarj
Newbie
*
Offline Offline

Posts: 25


« Reply #69 on: March 04, 2013, 12:29:21 AM »
ReplyReply

samueljohnchia, have you noticed that DNG PE never generate any matrix? It just took the matrix from the profile you selected when you start the chart wizard and generate a 90x25x1 LUT on top of it
Logged
samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #70 on: March 04, 2013, 01:57:50 AM »
ReplyReply

samueljohnchia, have you noticed that DNG PE never generate any matrix? It just took the matrix from the profile you selected when you start the chart wizard and generate a 90x25x1 LUT on top of it

Yes, I am aware of that. The matrices cannot be edited in the DNG PE at the moment. I'm not sure about the consequences of such a design, and one is tied to the LUTs to enjoy the corrections by the profile. If your preference in to go with the X-rite solution, by all means go ahead.
Logged
Hening Bettermann
Sr. Member
****
Offline Offline

Posts: 578


WWW
« Reply #71 on: March 04, 2013, 09:48:51 AM »
ReplyReply

Hi Samuel,
thank you for your extensive answer. There is no reason for apologies for your silence. You are under no obligation. I just wondered about the state of the project.
Let me know what I can do on the Mac. I in turn may need help with procedure and software handling to achieve what you want. I'll start trying to improve my profiles (Uff - I who thought I had a reasonable one!) using the new shots that just fill about 1/2 of the frame and see if that helps.
Logged

samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #72 on: March 05, 2013, 03:49:24 AM »
ReplyReply

samueljohnchia, have you noticed that DNG PE never generate any matrix? It just took the matrix from the profile you selected when you start the chart wizard and generate a 90x25x1 LUT on top of it

Writing this in response to solarj. DNG profiles can contain the following tables, but not profile building/editing software fill all table values.

1. CameraMatrix1
2. CameraMatrix1
3. ForwardMatrix1
4. ForwardMatrix2

If the raw converter follows the DNG spec in the way it handles DNG profiles, it will use the ForwardMatrix table values instead of the CameraMatrix table values. If the Forwardmatrix is not included (table values =0) then a rather convoluted method is used to map colors.

The Adobe Standard profile in general contains information in all four tables. - this is by default the base profile used by the DNG PE, so your new custom profile will also inherit these matrices.
X-rite Passport contains only 1. for single illuminant profiles. For dual illuminant I'm assuming it contains 1. and 2. It does not include the ForwardMatrices. Too bad.
QPcard contains only 1. and 3.

I spent considerable time to eliminate all the LUTs in the profiles except for the matrices, so see which one looks best in comparison. I hope this information may be useful to others. Here is a ProPhoto 16 bit tif with all the important versions on separate layers, overlayed with a reference CC target.

Additional info about the layers:
"Adobe Standard Matrices, DNGPE LUT"is Adobe Standard profile matrices with custom LUT from the DNG PE built with a photograph of my CC.
"X-rite Passport software matrices, DNGPE LUT" is the X-rite custom matrices with with custom LUT from the DNG PE built with a photograph of my CC
"Adobe Standard ColMmatrix1" is the Adobe Standard matrices stripped off all except for the D65 illuminant ColorMatrix table.
"Adobe Standard ColMatrix12 ForMatrix12" is all the Adobe Standard color matrices in play
"Xrite ColMatrix1" is the custom X-rite ColorMatrix generated when building my custom DNG profile
"QP ColMatrix1 ForMatrix1" is the QPcalibration generated matrices when building a custom DNG profile from a QPcard 203

Observations about the color matching. The "Adobe Standard ColMatrix1" is closer in the blues and purples, while the "Xrite ColMatrix1" is closer for the warm colors. The profiles which include the DNG PE custom LUTs are much closer. While the X-rite version lost out slightly in the red and light skin patch, it won visibly in the oranges in row two. The rest look more or less the same.

I'm not sure which I'd rather go with at the moment. If X-rite would also create the ForwardMatrices, I might be more inclined to use that as the base profile in the DNG PE.
Logged
Hening Bettermann
Sr. Member
****
Offline Offline

Posts: 578


WWW
« Reply #73 on: March 05, 2013, 09:22:11 AM »
ReplyReply

Samuel, thank you for sharing the result of this extensive effort!
Logged

samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #74 on: March 05, 2013, 12:27:53 PM »
ReplyReply

Samuel, thank you for sharing the result of this extensive effort!

Fixed - Link to wrong file.

You are welcome! If you need help to get the profile to map the target's colors closer to the reference values, I'll do my best.

I managed to "trick" Irident Developer into not putting the watermark (I ran the demo) right in the centre of the ColorChecker image by cropping it off-center. It worked!! I am glad that I got the CC to only fill about 1/4 of the frame. Otherwise I would have to restart from the beginning because all the custom profiles must be generated from the same image. That was many hours of work.

This is what we've all been waiting for. DNG vs ICC, across Adobe Camera Raw (DNG only), Raw Therapee and Irident Developer. Here is the tif file, ProPhoto RGB, 16 bits.

Just for fun I also made RT process out a photo using the custom ICC profile from Raw Therapee. That is the "RT with ID custom ICC" layer. It is surprisingly close to the RT specific custom ICC profile, which is the "RT with custom ICC" layer. This is probably because of my ICC profile building choices that I explained earlier. Very interesting. Cross raw converter compatible ICC profile anyone?

ID does seem to work well with the latest DNG profiles. Compare layers "ID with same DNG profile as ACR" and "ACR DNGPE Adobe Standard base custom LUT". It was extremely difficult to get the exact exposure and white balance between the two raw converters to match, but I think I got it close enough, especially since the mac I borrowed was a laptop, and my work station is PC. I probably made about twenty subtle iterations for this match. I may be wrong but I think that the color readouts in ID are not the same as ACR/PS. Made matching the patches by the numbers a lot more difficult. There may also be slightly different methods in handling the color mapping that is resulting in slight differences in the color patches. Not as ideal as I would expect, and not much better than using ICC profiles for another raw converter either.

RT on the other hand is not handling DNG profiles properly. I'm not sure if it is also ignoring the LUTs in the profile, not enough time to test that. It may also be using 1 matrix table at a time.

I haven't had the time to process out a number of real world color images with ICC and DNG profiles to know which works better in practice.

Well, that's all for now, its past 2am here.
« Last Edit: March 05, 2013, 08:04:06 PM by samueljohnchia » Logged
Hening Bettermann
Sr. Member
****
Offline Offline

Posts: 578


WWW
« Reply #75 on: March 05, 2013, 01:27:51 PM »
ReplyReply

Wow!
Hope you had a good nights sleep by the time you read this...
Obviously you were tired... the TIF does not match your description.
Here is a screen shot of the layers:
Best regards - Hening
Logged

digitaldog
Sr. Member
****
Online Online

Posts: 9222



WWW
« Reply #76 on: March 05, 2013, 01:49:21 PM »
ReplyReply

Thanks to the neatness of the layered doc, I was able to extract this into ColorThink Pro, take any of this as you wish <G>. I have the pixel files and color lists if anyone wants to play further in CTP.





ColorPatches vs. ACRwDNG_Custom:
--------------------------------------------------

dE Report

Number of Samples: 4320

Delta-E Formula dE2000

Overall - (4320 colors)
--------------------------------------------------
Average dE:   0.58
    Max dE:   4.55
    Min dE:   0.00
 StdDev dE:   0.79

Best 90% - (3887 colors)
--------------------------------------------------
Average dE:   0.39
    Max dE:   1.73
    Min dE:   0.00
 StdDev dE:   0.58

Worst 10% - (433 colors)
--------------------------------------------------
Average dE:   2.21
    Max dE:   4.55
    Min dE:   1.73
 StdDev dE:   0.49

--------------------------------------------------
--------------------------------------------------
Color Patches vs. RtwArgyll_ICC:
--------------------------------------------------

dE Report

Number of Samples: 4320

Delta-E Formula dE2000

Overall - (4320 colors)
--------------------------------------------------
Average dE:   0.78
    Max dE:   6.56
    Min dE:   0.00
 StdDev dE:   1.09

Best 90% - (3887 colors)
--------------------------------------------------
Average dE:   0.53
    Max dE:   2.41
    Min dE:   0.00
 StdDev dE:   0.78

Worst 10% - (433 colors)
--------------------------------------------------
Average dE:   3.10
    Max dE:   6.56
    Min dE:   2.41
 StdDev dE:   0.68

--------------------------------------------------
--------------------------------------------------
Color Patches vs. RtwDNG_Day:
--------------------------------------------------

dE Report

Number of Samples: 4320

Delta-E Formula dE2000

Overall - (4320 colors)
--------------------------------------------------
Average dE:   0.84
    Max dE:   6.23
    Min dE:   0.00
 StdDev dE:   1.20

Best 90% - (3887 colors)
--------------------------------------------------
Average dE:   0.55
    Max dE:   2.60
    Min dE:   0.00
 StdDev dE:   0.82

Worst 10% - (433 colors)
--------------------------------------------------
Average dE:   3.47
    Max dE:   6.23
    Min dE:   2.60
 StdDev dE:   0.80

--------------------------------------------------
--------------------------------------------------
Color Patches vs. RtwDNG_Tung:
--------------------------------------------------

dE Report

Number of Samples: 4320

Delta-E Formula dE2000

Overall - (4320 colors)
--------------------------------------------------
Average dE:   0.84
    Max dE:   6.23
    Min dE:   0.00
 StdDev dE:   1.20

Best 90% - (3887 colors)
--------------------------------------------------
Average dE:   0.55
    Max dE:   2.60
    Min dE:   0.00
 StdDev dE:   0.82

Worst 10% - (433 colors)
--------------------------------------------------
Average dE:   3.47
    Max dE:   6.23
    Min dE:   2.60
 StdDev dE:   0.80

--------------------------------------------------
--------------------------------------------------


Note: the Color Patches is synthetic (certainly very smooth), the other's are not so the numbers here could be quite different if more sampling or smoothing were done on the images. But I suspect if done correctly, we'd see the same differences with just lower values overall. Each color patch fed to CTP was 8x8.
« Last Edit: March 05, 2013, 05:15:25 PM by digitaldog » Logged

Andrew Rodney
Author “Color Management for Photographers”
http://digitaldog.net/
digitaldog
Sr. Member
****
Online Online

Posts: 9222



WWW
« Reply #77 on: March 05, 2013, 02:01:02 PM »
ReplyReply

One item that's interesting to view in CTP are the various color lists plotted 3D with vectors: the directions they follow differently compared to the Color Patches of which they are compared.

Logged

Andrew Rodney
Author “Color Management for Photographers”
http://digitaldog.net/
samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #78 on: March 05, 2013, 08:04:51 PM »
ReplyReply

Wow!
Hope you had a good nights sleep by the time you read this...
Obviously you were tired... the TIF does not match your description.
Here is a screen shot of the layers:
Best regards - Hening


Oops. Thanks Hening. I fixed the link now.
Logged
samueljohnchia
Sr. Member
****
Offline Offline

Posts: 327


« Reply #79 on: March 05, 2013, 08:08:16 PM »
ReplyReply

Thanks to the neatness of the layered doc, I was able to extract this into ColorThink Pro, take any of this as you wish <G>. I have the pixel files and color lists if anyone wants to play further in CTP.

...

Note: the Color Patches is synthetic (certainly very smooth), the other's are not so the numbers here could be quite different if more sampling or smoothing were done on the images. But I suspect if done correctly, we'd see the same differences with just lower values overall. Each color patch fed to CTP was 8x8.

Andrew, thank you very much for doing the comparisons in CTP. They are very helpful in visualizing the color differences. Could you pass me a copy of the pixel files and color lists? I'll send you a PM with my email.

Yeah, I could have done better to rotate the target to avoid glare from the sunlight. It would probably improve things a bit.
Logged
Pages: « 1 2 3 [4] 5 6 ... 8 »   Top of Page
Print
Jump to:  

Ad
Ad
Ad