We understand from the previous article that rendering color during raw conversion essentially means mapping raw data in the form of triplets into a standard color space via a Profile Connection Space in a two step process
The first step white balances and demosaics the raw data, which at that stage we will refer to as , followed by converting it to Profile Connection Space through linear projection by an unknown ‘Forward Matrix’ (as DNG calls it) of the form
Determining the nine coefficients of this matrix is the main subject of this article.
The second step projects the resulting image information from to the ‘output’ colorimetric color space chosen by the photographer, say sRGB or Adobe RGB. The necessary linear matrices for this transformation are standardized and readily available online.
9 Equations and 9 Unknowns
So how do we determine the nine coefficients of the Forward Matrix in Equation 1?
The 9 unknown coefficients operate on white balanced and demosaiced data to transform it linearly into data. It follows that if we had the values from three pixels and we knew their corresponding triplets we could solve for the coefficients. The results would be valid for the given hardware, scene and illumination.
For instance we could capture in the raw data 3 patches of uniform diffuse reflectance illuminated by a known light source with the camera whose matrix we want to determine, thus obtaining three sets of values; then measure with a spectrophotometer or similar instrument the reflectance of the 3 patches and illuminant spectral power distribution; and calculate the values that the reflectance and illuminant imply.
All that would be left to do then is assemble the 3×3 pairs of and corresponding data in the form of 9 equations and 9 unknowns to solve for the coefficients of the relative Forward Matrix . Below the ‘s that make up are the unknowns, the X,Y,Z’s and ‘s would be known:
Looks easy? In fact it is much easier than that.
1 Capture, 72 Equations
X-Rite produces a number of ColorChecker 24 patch standard targets whose reflectance information is published and well thumbed. For this example I will use their handy Passport Photo version.
The 24 patches in the ColorChecker target carry reflectances that are supposed to be representative of everyday photographic subjects, as found in skin, foliage and sky colors. The color scientists at BabelColor.com have measured a sample of 30 ColorChecker targets over the years and compared them to published specifications (pre-November 2014 formulations shown, as my unit is older than that):
Note that 1 is supposed to represent a just noticeable color difference, so you can see that with the exception of purple and white the patches do seem to provide a reasonably stable reference. This information, as well as average patch reflectance from 380 to 730nm at 10nm increments, is available in a spreadsheet from the relative link above.
Bingo! the L*a*b* color space is a simple transformation away from . With the data above and a single raw capture of a ColorChecker Passport Photo target in clear noonish sunlight (call it D50) we’ve got 24 as opposed to just 3 sets of raw and reference data triplets needed to solve for the coefficients of the Forward Matrix in Equation 1. And we can use the larger data set to make sure that the coefficients fit a wider number of potential photographic subjects – that’s why they call it a compromise color matrix. Shutter Release.
Computing the Coefficients by the Normal Equation
Ok, so now we have the cc24 target illuminated by a roughly D50 source captured in the raw file of a Nikon D610+24-120mm/4. Next we read the mean values in the three color channels for each of the 24 patches with a tool like RawDigger . Then we white balance them based on the third gray patch from the right in the neutral bottom row and demosaic them. The result is now the white balanced and demosaiced raw data set that we need, one triplet per patch.
We could then obtain the reference corresponding to each patch by measuring them or via ColorChecker values published by X-Rite or BabelColor as above and solve the 72 equations for the coefficients in matrix that best fit the available data using the Normal Equation as follows:
with representing the transpose and the inverse of the relative arrays. Voilà, by the magic of linear algebra Equation 3 will produce the 3×3 compromise color matrix that will result in the smallest sum of square differences to reference values for the target at hand. However that turns out not to be the best way to solve for because the method gives equal weight to all values, while the Human Visual System tends to be more sensitive in some parts of the XYZ color space than others.
A Better Color Difference Metric: dE2000
In fact in what follows I will use the standard color difference (CIEDE2000) as the minimization criterion. We can set up a spreadsheet with the 24 triplets and 9 cells representing the coefficients in the forward matrix as arrays. Seed the coefficients to, say, 1/3 each. Matrix multiply the triplets by the seed matrix, convert to L*a*b*, compute differences to the reference values in Figure 1 above and let Excel Solver figure out what values of the 9 coefficients of minimize the sum of the differences. The resulting matrix will be the best compromise found for the 24 patches under that illuminant, taking HVS color sensitivity in consideration therefore giving each patch approximately equal perceptual importance.
The Compromise Forward Matrix
I effectively followed that last procedure but using Matlab/Octave instead of Excel. Excellent toolkit OptProp and its built-in ColorChecker reference data lent a helping hand. This is the Forward Matrix obtained for my setup:
Cool. These are the differences for each patch resulting from the transformation using OptProp’s ColorChecher reference data.
The average is 1.5 and maximum is 3.8 in the light skin patch. Recall that 1 is a just noticeable difference. I repeated the exercise this time using the BabelColor 30 database as reference and this is the resulting forward matrix then:
Looks like the differences to reference data from BabelColor’s database are more evenly distributed: average is still 1.5 but Maximum is a lower 3.1 in the ‘green’ patch.
To do it properly I should have measured the Spectral Power Distribution of the illuminant and the reflectance of the patches with a spectrophotometer around the time that the capture was taken (didn’t have one, X-Rite ColorMunki Photo/i1 Studio on the way, see post scriptum). Using published reference SPDs and reflectances pollute the data and contribute to degrade the results – but you get the idea.
Correcting Matrix ‘Errors’: Profiles
Note however that the compromise matrix is just that, a compromise, and even if the procedure had been perfect it could result in relatively large errors. For better overall color rendering performance the preferred method is to correct such errors through optional nonlinear color profile adjustments that are typically introduced while in via ProPhoto RGB HSV lookup tables. The tables and application methods are usually referred to as camera ‘profiles’ (ICC and DCP for instance), a subject beyond the scope of this article.
Calculating Sensitivity Metamerism Index (SMI)
While the data is out we can calculate the Sensitivity Metamerism Index, which for us is equal to 100 minus 5.5 times the average (not ) of just the 18 color patches. The shown values (misnamed CRI above) are not necessarily maximized for the given setup because the matrix finding routine is built upon minimizing . Still my D610 with its 24-120mm/4 around D50 shows SMIs of 80 in the first case and 83 in the second. They jump to 82 and 86 respectively if the routine is setup to minimize instead. Not bad, although I know some people do not give much credence to this metric – and I can understand why.
Step 2: Matrix to Output Color Space
Now that we have done the hard work of determining the Forward Matrix to convert white balanced raw data to the PCS with this illuminant, we need a standard matrix to move it on to the colorimetric color space chosen by the photographer for output. In this case we will map it to by multiplication with the following transform matrix obtained from Bruce Lindbloom’s site
The product of this and the forward matrices calculated earlier produces the following combined white balanced raw linear transforms for a D50ish illuminant. My results are shown top and bottom, with DXOmark’s for the D610 at D50 in the center for reference:
Pretty close, which confirms the illuminant of my captures was indeed near D50.
So that’s where color matrices come from and why we need them. Next, a closer look and putting them to work.
I got myself an X-Rite ColorMunki Photo spectrophotometer (it’s identical to the recently rebranded i1 Studio), it’s a blast. It comes with its own mini ColorChecker 24 Classic, I assume with the new formulation. I measured it with the Munki and captured it in the raw data in somewhat similar conditions to Figure 2. Correlated Color Temperature was about 5050K. These are the matrices that are produced from it (white balanced off my WhiBal card):
Pretty similar and SMI is now 85. In fact if I set the routine up to minimize instead of SMI is 86. The sum of rows in the XYZ matrix should represent the white point of the target illuminant, which in this case was just the reference data for D50. The matrix white point is [0.9609 1 0.8214] which has a correlated color temperature of 5016K, according to Bruce. Good, because D50 has a CCT of 5002K.
Here are the differences, measured vs captured:
Now average is 1.17, there are only three patches above 2 and the rest seem to be better controlled overall. I guess pushing down the outliers is the reason why linear matrices are not enough and profiles are necessary. To get even closer I should have brought the Munki with me and measured the illuminant. Well, next time.
Notes and References
1. Lots of provisos and simplifications for clarity as always. I am not a color scientist, so if you spot any mistakes please let me know.
2. The ColorChecker pages at BabelColor.com can be found here. Careful that there was a change in formulations in November 2014.
3. RawDigger can be found here and dcraw here.
4. optprop can be found here.
5. For an example of profile implementation see for instance Adobe’s Digital Negative Specification Version 22.214.171.124.
6. See here for a description of the Sensitivity Metamerism Index and here for DXO’s take on it.
7. See this page at Bruce Lindbloom’s site for precise matrices from XYZD50 to many colorimetric color spaces.
8. DXOmark.com D610 color measurement can be found here by clicking on the ‘color response’ tab.