Color: Determining a Forward Matrix for Your Camera

We understand from the previous article that rendering color during raw conversion essentially means mapping raw data represented by RGB triplets into a standard color space via a Profile Connection Space in a two step process

    \[ RGB_{raw} \rightarrow  XYZ_{D50} \rightarrow RGB_{standard} \]

The process I will use first white balances and demosaics the raw data, which at that stage we will refer to as RGB_{rwd}, followed by converting it to XYZ_{D50} Profile Connection Space through linear transformation by an unknown ‘Forward Matrix’ (as DNG calls it) of the form

(1)   \begin{equation*} \left[ \begin{array}{c} X_{D50} \\ Y_{D50} \\ Z_{D50} \end{array} \right] = \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{bmatrix} \times \left[ \begin{array}{c} R_{rwd} \\ G_{rwd} \\ B_{rwd} \end{array} \right] \end{equation*}

Determining the nine a coefficients of this matrix is the main subject of this article[1].

The second step maps the resulting image information from XYZ_{D50} to the ‘output’ colorimetric color space chosen by the photographer, say sRGB or Adobe RGB.  The relative linear matrices are standardized and readily available.

9 Equations and 9 Unknowns

So how do we determine the nine coefficients of the Forward Matrix in Equation 1?

The 9 unknown coefficients operate on white balanced and demosaiced RGB data to transform it linearly into XYZ_{D50} data.  It follows that if we had 3 sets of RGB_{rwd} values and their corresponding XYZ_{D50} triplets we could solve for the a coefficients.  The results would be valid for the given hardware and illumination.

For instance we could capture with the camera whose matrix we want to determine 3 targets of uniform diffuse reflectance illuminated by a known light source; measure with a spectrophotometer or similar instrument the reflectance of the targets; calculate the  XYZ_{D50} values that the reflectance and illuminance imply; all that would be left to do then is assemble the 3×3 pairs  of RGB_{rwd} and corresponding XYZ_{D50} data  in the form of 9 equations and 9 unknowns to solve for the coefficients of the relative Forward Matrix.  Below the a‘s are the unknowns, the X,Y,Z’s and R,G,B’s would be known:

    \begin{align*} X_1 &= a_{12}R_1 + a_{12}G_1 + a_{13}B_1 \\ Y_1 &= a_{21}R_1 + a_{22}G_1 + a_{23}B_1 \\ Z_1 &= a_{31}R_1 + a_{32}G_1 + a_{33}B_1 \\ X_2 &= a_{12}R_2 + a_{12}G_2 + a_{13}B_2 \\ \vdots \\ Z_3 &= a_{31}R_3 + a_{32}G_3 + a_{33}B_3 \\ \end{align*}

Looks easy?    In fact it is much easier than that.

1 Capture, 72 Equations

X-Rite produces a number of ColorChecker 24 patch standard targets whose reflectance information is published and well thumbed.  For this example I will use their handy Passport Photo version.

The 24 patches in the ColorChecker target carry reflectances that are supposed to be representative of everyday photographic subjects, as found in skin, foliage and sky colors.  The color scientists at BabelColor.com have measured a sample of 30 ColorChecker targets over the years and compared them to published specifications (pre-November 2014 formulations shown, as my unit is older than that)[2]:

babel-cc24
Figure 1.  Average measurements of 30 pre-2015 ColorChecker 24 targets  by BabelColor.com

Note that 1 \Delta E_{00} is supposed to represent a just noticeable color difference, so you can see that with the exception of purple and white the patches do seem to provide a reasonably stable reference.  This information, as well as average patch reflectance from 380 to 730nm at 10nm increments, is available in a spreadsheet from the linked page above.

Bingo!  the L*a*b* color space is a simple transformation away from XYZ_{D50}.  With the data above and a single raw capture of a ColorChecker Passport Photo target in clear noonish sunlight (call it D50) we’ve got 24 as opposed to just 3 sets of raw and reference data triplets needed to solve for the coefficients of the Forward Matrix in Equation 1.  And we can use the larger data set to make sure that the coefficients fit a wider number of potential photographic subjects.  Shutter Release.

wbdsc_3378
Figure 2. Pre-2015 X-Rite ColorChecker Passport Photo captured by a Nikon D610 mounting a 24-120mm/4 at 120mm, 1/800s, f/5.6, ISO100 in early September at 11:27 on a clear mountain day.  The Correlated Color Temperature is in the 5000-5200K range.

Computing the Coefficients

Ok, so now we have the cc24 target illuminated by a roughly D50 source captured in the raw file of a Nikon D610+24-120mm/4. Next we read the mean RGB_{raw} values of each of the 24 patches with a tool like RawDigger [3].  Then we white balance each patch triplet based on the second gray patch from the left in the bottom row.  Then we demosaic it.  This is now the white balanced and demosaiced raw data set that we need, RGB_{rwd} .

We could then compute the corresponding reference XYZ_{D50} ColorChecker values published by X-Rite or BabelColor, brush up on linear algebra skills and solve the 72 equations for the coefficients that best fit the available data.  However, linear algebra was never my favorite subject so why go through all that hard work when computers can do it much faster and painlessly for us?

We can set up a spreadsheet with the 24 RGB_{rwd} triplets and 9 cells representing the coefficients in the forward matrix as arrays.  Seed the coefficients to, say, 1/3 each.  Matrix multiply the  RGB_{rwd} triplets by the seed matrix, convert to L*a*b*, compute \Delta E_{00} differences to the reference values in Figure 1 above and let Excel solver figure out what values of the 9 coefficients minimize the sum of the differences.  The resulting matrix will be the best compromise found for the 24 patches under that illuminant, giving each patch equal importance.

The Compromise Forward Matrix

I effectively followed that last procedure but using Matlab/Octave instead of Excel.   Excellent toolkit optprop and its built-in ColorChecker reference data lent a helping hand[4].  This is the forward matrix obtained for my setup:

Figure 3.

Cool.  These are the \Delta E_{00} differences for each patch based on optrop’s ColorChecher reference data.

optprop-database-d2000
Figure 3.  CIEDE2000 difference between the measured raw values from Figure 2 transformed to L*a*b* via XYZD50 by the computed Forward Matrix and the ColorChecker reference values built into optprop that were used to obtain it.

The average \Delta E_{00} is 1.5 and maximum is 3.8 in the light skin patch.  Recall that 1 is a just noticeable difference.  I repeated the exercise this time using the BabelColor 30 database as reference and this is the resulting forward matrix then:

forwardmatrix-d610-babel

Looks like the differences to reference data from BabelColor’s database are more evenly distributed:  average \Delta E_{00} is still 1.5 but Maximum is a lower 3.1 in the ‘green’ patch.

babel30-database-d2000
Figure 4. CIEDE2000 difference between the measured raw values from Figure 2 transformed to L*a*b* via XYZD50 by the computed Forward Matrix and the BabelColor database ColorChecker reference values that were used to obtain it.

To do it properly I should have measured the Spectral Power Distribution of the illuminant and the reflectance of the patches with a spectrophotometer around the time that the capture was taken (didn’t have one, ColorMunki Photo on the way).  Using published reference reflectances pollute the data and contribute to degrade the results – but you get the idea.

Correcting Matrix ‘Errors’: Profiles

Note however that the compromise matrix is just that, a compromise, and even if the procedure had been perfect it could result in quite large errors.  For better overall color rendering performance the preferred method is to correct such errors through optional nonlinear color profile adjustments that are introduced while in XYZ_{D50} via HSV lookup tables.  The tables and application methods are usually referred to as camera ‘profiles’ (ICC and DCP for instance), a subject beyond the scope of this article[5].

Calculating Sensitivity Metamerism Index (SMI)

While the data is out we can calculate the Sensitivity Metamerism Index, which is equal to 100 minus 5.5 times the average \Delta E (not \Delta E_{00}) of just the 18 color patches[6].  The shown values (misnamed CRI above) are not necessarily maximized for the given setup because the matrix finding routine is built upon minimizing \Delta E_{00}. Still my D610 with its 24-120mm/4 around D50 shows SMIs of 80 in the first case and 83 in the second.   They jump to 82 and 86 respectively if the routine is setup to minimize \Delta E instead.  Not bad, although I know most people do not give much credence to this metric – and I can see why.

Step 2: Matrix to Output Color Space

Now that we have done the hard work of determining the forward matrix to convert white balanced raw data to the PCS with this illuminant, we need a standard matrix to move it on to the colorimetric color space chosen by the photographer for output.  In this case we will map it to sRGB_{D65} by multiplication with the following transform matrix obtained from Bruce Lindbloom’s site[7]

lindbloom1

The product of this and the forward matrices calculated earlier produces the following combined RGB_{rwd}\rightarrow RGB_{sRGB} linear transforms for a D50ish illuminant.  My results are shown top and bottom, with DXOmark’s for the D610 at D50 in the center for reference[8]:

tosrgbdxo2
Figure 5. Forward Matrices from white balanced raw data to XYZD50 for a Nikon D610+24-120/4  in approximately D50 light. Top: matrix computed through the procedure in this article from the raw data in Figure 2 with optprop’s built in ColorChecker reference values. Middle: DXOmark.com’s D50 matrix. Bottom: matrix computed in this article from the raw data in Figure 2 with BabelColor 30 database ColorChecker reference values.

Pretty close, which confirms the illuminant of my captures was indeed near D50.

So that’s where color matrices come from and why we need them. Next, a closer look and putting them to work.

 

PS:  I got myself an Xrite ColorMunki Photo spectrophotometer, it’s a blast.  It comes with its own ColorChecker 24 mini, I assume with the new formulation.  I measured it with the Munki and captured it in the raw data in somewhat similar conditions to Figure 2.  Correlated Color Temperature was about 5050K.  These are the matrices that are produced from it (white balanced off my WhiBal card):

Pretty similar and SMI is now 85.  Here are the  \Delta E_{00} differences, measured vs captured:

Figure 6. CIEDE2000 obtained from ColorChecker mini measured with a ColorMunki spectrophotometer and captured in the raw data in late december at about 1700m, 2pm, sunny day with snow on the ground.  CCT is about 5050K.

Now average \Delta E_{00} is 1.17, there are only three patches above 2 and the rest seem to be better controlled overall. I guess pushing down the outliers  is the reason why linear matrices are not enough and profiles are necessary.

Notes and References

1. Lots of provisos and simplifications for clarity as always.  I am not a color scientist, so if you spot any mistakes please let me know.
2. The ColorChecker pages at BabelColor.com can be found here. Careful that there was a change in formulations in November 2014.
3. RawDigger can be found here and dcraw here.
4. optprop can be found here.
5. For an example of profile implementation see for instance Adobe’s Digital Negative Specification Version 1.4.0.0.
6. See here for a description of the Sensitivity Metamerism Index and here for DXO’s take on it.
7. See this page at Bruce Lindbloom’s site for precise matrices from XYZD50 to many colorimetric color spaces.
8. DXOmark.com D610 color measurement can be found here by clicking on the ‘color response’ tab.