Color correction using least square method
Question:
I have tried color correcting an image using the least square method. I don’t understand why it doesn’t work, this is supposed to be the standard way of color calibration.
First I pull in the above image in CR3 format, convert it to RGB space then crop out the four color patches using the OpenCV boundingRect and inRange functions, saving these four patches in an array called coloursRect. vstack is used so that the array storing each pixel’s colour transforms from 3d to 2d. So, for example, colour0 stores every pixels’ RGB value of the ‘red patch’.
colour0 = np.vstack(coloursRect[0])
colour1 = np.vstack(coloursRect[1])
colour2 = np.vstack(coloursRect[2])
colour3 = np.vstack(coloursRect[3])
lstsq_a = np.array(np.vstack((colour0,colour1,colour2,colour3)))
Then I declare the original reference colours in RGB.
r_ref = [240,0,22]
y_ref = [252,222,10]
g_ref = [30,187,22]
b_ref = [26,0,165]
ref_patches = [r_ref,y_ref,g_ref, b_ref]
The number of each reference color is multiplied according to the number of pixels in that actual image color patch, so, for example, r_ref is multiplied by the length of colour0 array. (I understand this is a bad way to manipulate the data, but this should work theoretically)
lstsq_b_0to255 = np.array(np.vstack(([ref_patches[0]]*colour0.shape[0],[ref_patches[1]]*colour1.shape[0],[ref_patches[2]]*colour2.shape[0],[ref_patches[3]]*colour3.shape[0])))
Least square is computed, and multiplied with the image.
lstsq_x_0to255 = np.linalg.lstsq(lstsq_a, lstsq_b_0to255)[0]
img_shape = img.shape
img_s = img.reshape((-1, 3))
img_corr_s = img_s @ lstsq_x_0to255
img_corr = img_corr_s.reshape(img_shape).astype('uint8')
However this color correction method does not work and the colours in the image are incorrect.
May I know what is the problem?
Edit: using RGB instead of HSV for the reference colours
Answers:
Ignoring the fact that the image ICC profile is not properly decoded here, this is the expected result given your reference RGB values and using Colour:
import colour
import numpy as np
# Reference values a likely non-linear 8-bit sRGB values.
# "colour.cctf_decoding" uses the sRGB EOTF by default.
REFERENCE_RGB = colour.cctf_decoding(
np.array(
[
[240, 0, 22],
[252, 222, 10],
[30, 187, 22],
[26, 0, 165],
]
)
/ 255
)
colour.plotting.plot_multi_colour_swatches(colour.cctf_encoding(REFERENCE_RGB))
IMAGE = colour.cctf_decoding(colour.read_image("/Users/kelsolaar/Downloads/EKcv1.jpeg"))
# Measured test values, the image is not properly decoded as it has a very specific ICC profile.
TEST_RGB = np.array(
[
[0.578, 0.0, 0.144],
[0.895, 0.460, 0.0],
[0.0, 0.183, 0.074],
[0.067, 0.010, 0.070],
]
)
colour.plotting.plot_image(
colour.cctf_encoding(colour.colour_correction(IMAGE, REFERENCE_RGB, TEST_RGB))
)
The main functions, available in this module are as follows:
def least_square_mapping_MoorePenrose(y: ArrayLike, x: ArrayLike) -> NDArray:
"""
Compute the *least-squares* mapping from dependent variable :math:`y` to
independent variable :math:`x` using *Moore-Penrose* inverse.
Parameters
----------
y
Dependent and already known :math:`y` variable.
x
Independent :math:`x` variable(s) values corresponding with :math:`y`
variable.
Returns
-------
:class:`numpy.ndarray`
*Least-squares* mapping.
References
----------
:cite:`Finlayson2015`
Examples
--------
>>> prng = np.random.RandomState(2)
>>> y = prng.random_sample((24, 3))
>>> x = y + (prng.random_sample((24, 3)) - 0.5) * 0.5
>>> least_square_mapping_MoorePenrose(y, x) # doctest: +ELLIPSIS
array([[ 1.0526376..., 0.1378078..., -0.2276339...],
[ 0.0739584..., 1.0293994..., -0.1060115...],
[ 0.0572550..., -0.2052633..., 1.1015194...]])
"""
y = np.atleast_2d(y)
x = np.atleast_2d(x)
return np.dot(np.transpose(x), np.linalg.pinv(np.transpose(y)))
def matrix_augmented_Cheung2004(
RGB: ArrayLike,
terms: Literal[3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22] = 3,
) -> NDArray:
"""
Perform polynomial expansion of given *RGB* colourspace array using
*Cheung et al. (2004)* method.
Parameters
----------
RGB
*RGB* colourspace array to expand.
terms
Number of terms of the expanded polynomial.
Returns
-------
:class:`numpy.ndarray`
Expanded *RGB* colourspace array.
Notes
-----
- This definition combines the augmented matrices given in
:cite:`Cheung2004` and :cite:`Westland2004`.
References
----------
:cite:`Cheung2004`, :cite:`Westland2004`
Examples
--------
>>> RGB = np.array([0.17224810, 0.09170660, 0.06416938])
>>> matrix_augmented_Cheung2004(RGB, terms=5) # doctest: +ELLIPSIS
array([ 0.1722481..., 0.0917066..., 0.0641693..., 0.0010136..., 1...])
"""
RGB = as_float_array(RGB)
R, G, B = tsplit(RGB)
tail = ones(R.shape)
existing_terms = np.array([3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22])
closest_terms = as_int(closest(existing_terms, terms))
if closest_terms != terms:
raise ValueError(
f'"Cheung et al. (2004)" method does not define an augmented '
f"matrix with {terms} terms, closest augmented matrix has "
f"{closest_terms} terms!"
)
if terms == 3:
return RGB
elif terms == 5:
return tstack(
[
R,
G,
B,
R * G * B,
tail,
]
)
elif terms == 7:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
tail,
]
)
elif terms == 8:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R * G * B,
tail,
]
)
elif terms == 10:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
tail,
]
)
elif terms == 11:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
tail,
]
)
elif terms == 14:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**3,
G**3,
B**3,
tail,
]
)
elif terms == 16:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**3,
G**3,
B**3,
]
)
elif terms == 17:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**3,
G**3,
B**3,
tail,
]
)
elif terms == 19:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**2 * B,
G**2 * R,
B**2 * G,
R**3,
G**3,
B**3,
]
)
elif terms == 20:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**2 * B,
G**2 * R,
B**2 * G,
R**3,
G**3,
B**3,
tail,
]
)
elif terms == 22:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**2 * B,
G**2 * R,
B**2 * G,
R**3,
G**3,
B**3,
R**2 * G * B,
R * G**2 * B,
R * G * B**2,
]
)
def matrix_colour_correction_Cheung2004(
M_T: ArrayLike,
M_R: ArrayLike,
terms: Literal[3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22] = 3,
) -> NDArray:
"""
Compute a colour correction matrix from given :math:`M_T` colour array to
:math:`M_R` colour array using *Cheung et al. (2004)* method.
Parameters
----------
M_T
Test array :math:`M_T` to fit onto array :math:`M_R`.
M_R
Reference array the array :math:`M_T` will be colour fitted against.
terms
Number of terms of the expanded polynomial.
Returns
-------
:class:`numpy.ndarray`
Colour correction matrix.
References
----------
:cite:`Cheung2004`, :cite:`Westland2004`
Examples
--------
>>> prng = np.random.RandomState(2)
>>> M_T = prng.random_sample((24, 3))
>>> M_R = M_T + (prng.random_sample((24, 3)) - 0.5) * 0.5
>>> matrix_colour_correction_Cheung2004(M_T, M_R) # doctest: +ELLIPSIS
array([[ 1.0526376..., 0.1378078..., -0.2276339...],
[ 0.0739584..., 1.0293994..., -0.1060115...],
[ 0.0572550..., -0.2052633..., 1.1015194...]])
"""
return least_square_mapping_MoorePenrose(
matrix_augmented_Cheung2004(M_T, terms), M_R
)
def colour_correction_Cheung2004(
RGB: ArrayLike,
M_T: ArrayLike,
M_R: ArrayLike,
terms: Literal[3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22] = 3,
) -> NDArray:
"""
Perform colour correction of given *RGB* colourspace array using the
colour correction matrix from given :math:`M_T` colour array to
:math:`M_R` colour array using *Cheung et al. (2004)* method.
Parameters
----------
RGB
*RGB* colourspace array to colour correct.
M_T
Test array :math:`M_T` to fit onto array :math:`M_R`.
M_R
Reference array the array :math:`M_T` will be colour fitted against.
terms
Number of terms of the expanded polynomial.
Returns
-------
:class:`numpy.ndarray`
Colour corrected *RGB* colourspace array.
References
----------
:cite:`Cheung2004`, :cite:`Westland2004`
Examples
--------
>>> RGB = np.array([0.17224810, 0.09170660, 0.06416938])
>>> prng = np.random.RandomState(2)
>>> M_T = prng.random_sample((24, 3))
>>> M_R = M_T + (prng.random_sample((24, 3)) - 0.5) * 0.5
>>> colour_correction_Cheung2004(RGB, M_T, M_R) # doctest: +ELLIPSIS
array([ 0.1793456..., 0.1003392..., 0.0617218...])
"""
RGB = as_float_array(RGB)
shape = RGB.shape
RGB = np.reshape(RGB, (-1, 3))
RGB_e = matrix_augmented_Cheung2004(RGB, terms)
CCM = matrix_colour_correction_Cheung2004(M_T, M_R, terms)
return np.reshape(np.transpose(np.dot(CCM, np.transpose(RGB_e))), shape)
I would probably recommend using Colour directly as there are multiple methods that gives different result depending on the training set. That being said, I would not expect great results given that you really only have 4 chromatic colours and none achromatic. The minimum recommended chart for that kind of calibration is the ColorChecker Classic with 24 patches.
I have tried color correcting an image using the least square method. I don’t understand why it doesn’t work, this is supposed to be the standard way of color calibration.
First I pull in the above image in CR3 format, convert it to RGB space then crop out the four color patches using the OpenCV boundingRect and inRange functions, saving these four patches in an array called coloursRect. vstack is used so that the array storing each pixel’s colour transforms from 3d to 2d. So, for example, colour0 stores every pixels’ RGB value of the ‘red patch’.
colour0 = np.vstack(coloursRect[0])
colour1 = np.vstack(coloursRect[1])
colour2 = np.vstack(coloursRect[2])
colour3 = np.vstack(coloursRect[3])
lstsq_a = np.array(np.vstack((colour0,colour1,colour2,colour3)))
Then I declare the original reference colours in RGB.
r_ref = [240,0,22]
y_ref = [252,222,10]
g_ref = [30,187,22]
b_ref = [26,0,165]
ref_patches = [r_ref,y_ref,g_ref, b_ref]
The number of each reference color is multiplied according to the number of pixels in that actual image color patch, so, for example, r_ref is multiplied by the length of colour0 array. (I understand this is a bad way to manipulate the data, but this should work theoretically)
lstsq_b_0to255 = np.array(np.vstack(([ref_patches[0]]*colour0.shape[0],[ref_patches[1]]*colour1.shape[0],[ref_patches[2]]*colour2.shape[0],[ref_patches[3]]*colour3.shape[0])))
Least square is computed, and multiplied with the image.
lstsq_x_0to255 = np.linalg.lstsq(lstsq_a, lstsq_b_0to255)[0]
img_shape = img.shape
img_s = img.reshape((-1, 3))
img_corr_s = img_s @ lstsq_x_0to255
img_corr = img_corr_s.reshape(img_shape).astype('uint8')
However this color correction method does not work and the colours in the image are incorrect.
May I know what is the problem?
Edit: using RGB instead of HSV for the reference colours
Ignoring the fact that the image ICC profile is not properly decoded here, this is the expected result given your reference RGB values and using Colour:
import colour
import numpy as np
# Reference values a likely non-linear 8-bit sRGB values.
# "colour.cctf_decoding" uses the sRGB EOTF by default.
REFERENCE_RGB = colour.cctf_decoding(
np.array(
[
[240, 0, 22],
[252, 222, 10],
[30, 187, 22],
[26, 0, 165],
]
)
/ 255
)
colour.plotting.plot_multi_colour_swatches(colour.cctf_encoding(REFERENCE_RGB))
IMAGE = colour.cctf_decoding(colour.read_image("/Users/kelsolaar/Downloads/EKcv1.jpeg"))
# Measured test values, the image is not properly decoded as it has a very specific ICC profile.
TEST_RGB = np.array(
[
[0.578, 0.0, 0.144],
[0.895, 0.460, 0.0],
[0.0, 0.183, 0.074],
[0.067, 0.010, 0.070],
]
)
colour.plotting.plot_image(
colour.cctf_encoding(colour.colour_correction(IMAGE, REFERENCE_RGB, TEST_RGB))
)
The main functions, available in this module are as follows:
def least_square_mapping_MoorePenrose(y: ArrayLike, x: ArrayLike) -> NDArray:
"""
Compute the *least-squares* mapping from dependent variable :math:`y` to
independent variable :math:`x` using *Moore-Penrose* inverse.
Parameters
----------
y
Dependent and already known :math:`y` variable.
x
Independent :math:`x` variable(s) values corresponding with :math:`y`
variable.
Returns
-------
:class:`numpy.ndarray`
*Least-squares* mapping.
References
----------
:cite:`Finlayson2015`
Examples
--------
>>> prng = np.random.RandomState(2)
>>> y = prng.random_sample((24, 3))
>>> x = y + (prng.random_sample((24, 3)) - 0.5) * 0.5
>>> least_square_mapping_MoorePenrose(y, x) # doctest: +ELLIPSIS
array([[ 1.0526376..., 0.1378078..., -0.2276339...],
[ 0.0739584..., 1.0293994..., -0.1060115...],
[ 0.0572550..., -0.2052633..., 1.1015194...]])
"""
y = np.atleast_2d(y)
x = np.atleast_2d(x)
return np.dot(np.transpose(x), np.linalg.pinv(np.transpose(y)))
def matrix_augmented_Cheung2004(
RGB: ArrayLike,
terms: Literal[3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22] = 3,
) -> NDArray:
"""
Perform polynomial expansion of given *RGB* colourspace array using
*Cheung et al. (2004)* method.
Parameters
----------
RGB
*RGB* colourspace array to expand.
terms
Number of terms of the expanded polynomial.
Returns
-------
:class:`numpy.ndarray`
Expanded *RGB* colourspace array.
Notes
-----
- This definition combines the augmented matrices given in
:cite:`Cheung2004` and :cite:`Westland2004`.
References
----------
:cite:`Cheung2004`, :cite:`Westland2004`
Examples
--------
>>> RGB = np.array([0.17224810, 0.09170660, 0.06416938])
>>> matrix_augmented_Cheung2004(RGB, terms=5) # doctest: +ELLIPSIS
array([ 0.1722481..., 0.0917066..., 0.0641693..., 0.0010136..., 1...])
"""
RGB = as_float_array(RGB)
R, G, B = tsplit(RGB)
tail = ones(R.shape)
existing_terms = np.array([3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22])
closest_terms = as_int(closest(existing_terms, terms))
if closest_terms != terms:
raise ValueError(
f'"Cheung et al. (2004)" method does not define an augmented '
f"matrix with {terms} terms, closest augmented matrix has "
f"{closest_terms} terms!"
)
if terms == 3:
return RGB
elif terms == 5:
return tstack(
[
R,
G,
B,
R * G * B,
tail,
]
)
elif terms == 7:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
tail,
]
)
elif terms == 8:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R * G * B,
tail,
]
)
elif terms == 10:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
tail,
]
)
elif terms == 11:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
tail,
]
)
elif terms == 14:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**3,
G**3,
B**3,
tail,
]
)
elif terms == 16:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**3,
G**3,
B**3,
]
)
elif terms == 17:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**3,
G**3,
B**3,
tail,
]
)
elif terms == 19:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**2 * B,
G**2 * R,
B**2 * G,
R**3,
G**3,
B**3,
]
)
elif terms == 20:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**2 * B,
G**2 * R,
B**2 * G,
R**3,
G**3,
B**3,
tail,
]
)
elif terms == 22:
return tstack(
[
R,
G,
B,
R * G,
R * B,
G * B,
R**2,
G**2,
B**2,
R * G * B,
R**2 * G,
G**2 * B,
B**2 * R,
R**2 * B,
G**2 * R,
B**2 * G,
R**3,
G**3,
B**3,
R**2 * G * B,
R * G**2 * B,
R * G * B**2,
]
)
def matrix_colour_correction_Cheung2004(
M_T: ArrayLike,
M_R: ArrayLike,
terms: Literal[3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22] = 3,
) -> NDArray:
"""
Compute a colour correction matrix from given :math:`M_T` colour array to
:math:`M_R` colour array using *Cheung et al. (2004)* method.
Parameters
----------
M_T
Test array :math:`M_T` to fit onto array :math:`M_R`.
M_R
Reference array the array :math:`M_T` will be colour fitted against.
terms
Number of terms of the expanded polynomial.
Returns
-------
:class:`numpy.ndarray`
Colour correction matrix.
References
----------
:cite:`Cheung2004`, :cite:`Westland2004`
Examples
--------
>>> prng = np.random.RandomState(2)
>>> M_T = prng.random_sample((24, 3))
>>> M_R = M_T + (prng.random_sample((24, 3)) - 0.5) * 0.5
>>> matrix_colour_correction_Cheung2004(M_T, M_R) # doctest: +ELLIPSIS
array([[ 1.0526376..., 0.1378078..., -0.2276339...],
[ 0.0739584..., 1.0293994..., -0.1060115...],
[ 0.0572550..., -0.2052633..., 1.1015194...]])
"""
return least_square_mapping_MoorePenrose(
matrix_augmented_Cheung2004(M_T, terms), M_R
)
def colour_correction_Cheung2004(
RGB: ArrayLike,
M_T: ArrayLike,
M_R: ArrayLike,
terms: Literal[3, 5, 7, 8, 10, 11, 14, 16, 17, 19, 20, 22] = 3,
) -> NDArray:
"""
Perform colour correction of given *RGB* colourspace array using the
colour correction matrix from given :math:`M_T` colour array to
:math:`M_R` colour array using *Cheung et al. (2004)* method.
Parameters
----------
RGB
*RGB* colourspace array to colour correct.
M_T
Test array :math:`M_T` to fit onto array :math:`M_R`.
M_R
Reference array the array :math:`M_T` will be colour fitted against.
terms
Number of terms of the expanded polynomial.
Returns
-------
:class:`numpy.ndarray`
Colour corrected *RGB* colourspace array.
References
----------
:cite:`Cheung2004`, :cite:`Westland2004`
Examples
--------
>>> RGB = np.array([0.17224810, 0.09170660, 0.06416938])
>>> prng = np.random.RandomState(2)
>>> M_T = prng.random_sample((24, 3))
>>> M_R = M_T + (prng.random_sample((24, 3)) - 0.5) * 0.5
>>> colour_correction_Cheung2004(RGB, M_T, M_R) # doctest: +ELLIPSIS
array([ 0.1793456..., 0.1003392..., 0.0617218...])
"""
RGB = as_float_array(RGB)
shape = RGB.shape
RGB = np.reshape(RGB, (-1, 3))
RGB_e = matrix_augmented_Cheung2004(RGB, terms)
CCM = matrix_colour_correction_Cheung2004(M_T, M_R, terms)
return np.reshape(np.transpose(np.dot(CCM, np.transpose(RGB_e))), shape)
I would probably recommend using Colour directly as there are multiple methods that gives different result depending on the training set. That being said, I would not expect great results given that you really only have 4 chromatic colours and none achromatic. The minimum recommended chart for that kind of calibration is the ColorChecker Classic with 24 patches.