We’ve seen how humans perceive color in daylight as a result of three types of photoreceptors in the retina called cones that absorb wavelengths of light from the scene with different sensitivities to the arriving spectrum.
A photographic digital imager attempts to mimic the workings of cones in the retina by having different color filters arranged in an array (CFA) on top of its photoreceptors, which we normally call pixels. In a Bayer CFA configuration there are three filters named for the predominant wavelengths that each lets through (red, green and blue) arranged in quartets such as shown below:
It is the quality of the combined filtering part of the imaging system (lenses, UV/IR, CFA, sensor etc.) that determines how accurately a digital camera is able to capture color information from the scene. So what are the characteristics of better systems and can perfection be achieved? In this article I will pick up the discussion where it was last left off and, ignoring noise for now, attempt to answer this question using CIE conventions, in the process gaining insight in the role of the compromise color matrix and developing a method to visualize its effects.
Just be aware that when I say CFA for brevity in this article I really mean the combined effect of its spectral sensitivity functions, responsivity of the detector, filtering effects of lenses, infrared, ultraviolet and/or any other filter present.
Eye CFA Proxy = CMF
Since we are sticking with CIE conventions we might as well start with Color Matching Functions, those miraculous curves determined experimentally in 1931 that, multiplied wavelength-by-wavelength by irradiance from the scene , transform color information into XYZ coordinates.
Coordinates in the CIE XYZ color space are said to have one-to-one correspondence to colors perceived by the average person, represented by the CIE Standard Observer. If two coordinates are the same, the two colors will look identical to such a person, if they are sufficiently different they will look different.
In other words, if the observer and a digital camera were co-located at a scene and if the camera were able to transform captured image data to the exact same XYZ values as those produced by a Standard Observer present at the scene we would have the perfect CFA, capable of capturing any tone the observer might be able to see.
The (CIE) CFA for Perfect Color
How is the objective of having image data match one-to-one the observer in XYZ achieved? The most straightforward way is to have the color filters in the CFA match the Observer Color Matching Functions – and let a linear color matrix take care of translating the captured raw data to XYZ, as described in a series of articles starting here. The process can be summarized as follows:
- In order to know what colors the Observer would see from the scene all one needs to do is convert spectral irradiance from the scene to CIE XYZ by multiplying it by the respective color matching function in Figure 3.
- if the CFA of a digital camera looked like Figure 3 in relative energy units, the captured raw data would represent CIE XYZ colors natively; we could say that the camera meets the Luther-Ives condition, its color space is CIE XYZ and all tones would match perfectly those seen by the average (CIE) observer.
- Since the CFA is alas fixed but the illumination changes from scene to scene, a linear matrix would in theory then be needed just to correct for different illuminants.
So here is the perfect CFA under spectrally flat, equi-energy per small wavelength interval illuminant . From this point on I will use the new and improved 2006 two-degree Color Matching Functions shown in Figure 4 instead of the classic 1931 ones in Figure 3.
And in fact if we let the optimum matrix finding algorithm described earlier loose we obtain a perfect SMI score of 100 assuming such a CFA (Sensitivity Metamerism Index, a measure of color accuracy). Here the job of the color matrix would simply be to account for the illuminant, with its diagonal equal to the latter’s XYZ coordinates (which in the case of happen to be 1,1,1):
Reading the Matrix
Recall that the matrix effectively performs an invertible linear projection from one space to the next. In the context of this post it transforms white balanced raw data (r,g,b) to the XYZ CIE color space:
Projection of white balanced raw intensity (r,g,b) to XYZ is performed by multiplying each raw value triplet into the matrix, row by row as follows
In the case of Matrix 1 above the diagonal terms and are all ones while all others are zero. That is good because, for instance, we want the raw data from a uniform neutral patch of the brightest white (1,1,1 when normalized to the 0->1 range) to map to the coordinates of the white point in the projected space, which in this case are also (1,1,1) for equi-energy illuminant
Obtaining the Perfect CFA
A manufacturer could produce a CFA with Spectral Sensitivity Functions in the shape of gaussians with means and standard deviations as follows
sprinkling about 20% blue in the red CFA to obtain that bump in the blue wavelengths – and come out with color close to perfection, an SMI of 98.3 (dots are the Gaussian CFA sensitivities, lines are CMFs for comparison):
Again the job of the matrix here would mainly be to adjust for the current illuminant. It wouldn’t be as clean as that corresponding to Figure 5 because there are some small differences, but close enough:
Note that the terms along the diagonal are all about 1.0 (for equi-energy illuminant ) and those off diagonal close to zero.
While we have the matrix out, though, shouldn’t we be able to use it to also generate the red bump around 450nm? After all it seems to be centered near the blue CFA peak – and we have seen how the matrix’s job is to add together different proportions of energy going through the individual CFA channels.
The Perfect CFA, Take II
So couldn’t we just have a single red peak in the relative CFA curve and let the matrix take care of adding in some blue? This is what the perfect color CFA would look like then:
Now the perfect CFA has just three peaks, one for each channel. Red looks taller than in the previous figure because it lost the bump around 450nm and the curves are designed to all have the same area, that is to integrate to the same raw value under equi-energy illuminant . Letting Matlab’s lsqnonlin search routine find the best compromise color matrix for this CFA produces the following result:
Virtually identical to Matrix 2 – except for the redimensioned red component top left and the new coefficient that popped up top right, the blue component in the red channel: that’s the bump we got rid of in the CFA, reinstated by the optimizing routine.
Visualizing the effect of the Matrix
In fact we can visualize the CFA directly in XYZ space by applying the matrix to the CFA spectral curves themselves, wavelength by wavelength – it’s a linear system after all. In other words, once multiplied by the matrix under illuminant the resulting red curve is the result of 0.8341 times the red CFA curve in FIgure 6 minus 0.0030 times the green curve plus 0.1623 times the blue curve, and so for the others. If we perform those operations on each of the three curves we effectively transform them to the perceptual XYZ color Space. There we can compare them directly to the respective experimentally derived Color Matching Function:
Voilà, once transformed to XYZ by the matrix the red peak has been brought back to size (83.41% of its value in Figure 6) and the bump in the blue wavelengths is back (+16.23% blue) despite the fact that in the real world the CFA has now just three peaks.
Note however that the r,g,b curves in Figures 7 and 5 aren’t exactly equal: for instance the blue curve in XYZ is slightly raised around 550nm, the result of adding 4.45% green per the relative matrix coefficient – an addition that is of course not there in the CMF. That’s a result of the compromises necessary to obtain the best matrix possible in the given conditions. Nothing to worry about at this stage though, SMI is still an unheard of 98.3.
Bumps are Red, Violets are Blue
It becomes then apparent that it is not strictly necessary for the red filter to leak into blue wavelengths in order to be able to reproduce violet, as once suspected. All one needs are properly positioned CFA peaks and a well tuned compromise matrix. Easier said than done I guess, otherwise we would all be walking around with cameras producing virtually perfect color.
On the other hand with the red bump baked into the CFA, a sensor could collect a wider bandwidth signal in the red channel with potential white balance benefits and no penalty in color accuracy, so if it’s feasible and it meets design objectives why not have it? In fact this is not an all or nothing affair: one could choose to have just a little red bump in the blue wavelengths if that worked better with other imager compromises and let the matrix take over from there. I wonder if that’s why the CFAs of many current cameras show just such little compromises, albeit not in as nearly a well behaved form as in the figures above.
CFA IQ in the XYZ Color Space
I found this exercise useful to better understand what is happening when searching for the optimal matrix (by using raw values captured off a known target and compared to published reference data). In the end it turns out that the objective of the linear color system of a digital camera is to make linear combinations of the three CFA spectral sensitivities look as much as possible like CIE Color Matching Functions under the given illuminant, in XYZ space. The recipe for the combinations is contained in the compromise color matrix.
IQ, as far as color is concerned in a digital camera ignoring noise, is determined by how closely CFA curves look like color matching functions in XYZ after transformation by a linear matrix. The closer they are to CMFs the more accurate the color.
How hard would it be for a manufacturer to produce a CFA with Spectral Sensitivity Functions in the shape of gaussians centered at 596.8, 560.2 and 447.1nm of standard deviation 33.0, 44.0 and 23.5nm for the r, g and b channels respectively – taking into consideration other filtering usually present in digital photography (lens, hot mirrors etc.)? A question for the materials science guys -but we know that there are many compromises to be made, especially as it pertains to noise. This is the reason why I suspect I have personally never seen an SMI of 90 or above in practice. Until then here is one proposed ‘perfect’ color CFA in more familiar form, normalized so that all Spectral Sensitivity Functions peak at 1.
Of course since we are starting with white balanced raw data, which is proportional to the area under the curves, if we put this CFA through the routine that computes the optimum Compromise Color Matrix we will get Matrix 3 again and the CFA in XYZ will look exactly as in Figure 7 – the only difference being the coefficients required to white balance the raw data, which effectively reset the curves to their natural state seen in Figure 6.
In a future article we will look at the SSFs of the CFA of a few recent cameras transformed to XYZ space to see how closely they are able to match CMFs.
Notes and References
1. Lots of provisos and simplifications for clarity as always. I am not a color scientist, so if you spot any mistakes please let me know.
2. A camera or colorimeter is said to be colorimetric if it satisfies the Luther condition (also called the “Maxwell-Ives criterion”), if the product of the spectral responsivity of the photoreceptor and the spectral transmittance of the filters is a linear combination of the Color Matching Functions. See here (in german, right click to translate it).
3. Note that the CFA curves in this article are in relative energy units and cannot be used to read off Quantum Efficiency or be compared without conversion to diagrams in quantal units, such as those found in manufacturer spec sheets for instance. These curves are often also shown with their peaks normalized to one but for the purposes of this post their scale is determined by having the same area. Ignoring noise for now, what counts most for color is their shape, not their size, because the matrix optimizing routine adjusts size linearly and automatically to achieve the best possible color accuracy.
4. The means and standard deviations of the synthetic CFA’s gaussian sensitivity functions were obtained by minimizing wavelength-by-wavelength root mean square differences to the indicated CMFs.