Why does the camera calibration in opencv-python takes more than 30 minutes?
Question:
I was trying to replicate this camera calibration code.
When I run my script, the calibration process won’t stop for about 30 minutes (I expected the intrinsic camera matrix for the output). What is wrong with my code? Why does the calibration process take so long?
This is my code
import os
import cv2
import numpy as np
# DUMMY TEST
FilepathCalib = './data'
# Declaring global variable
img_calib_set = []
points_3d_sample = np.zeros((6*7,3), np.float32)
points_3d_sample[:,:2] = np.mgrid[0:7,0:6].T.reshape(-1,2)
points_2d = []
points_3d = []
# Loading calibration image dataset
for img_calib in os.listdir(FilepathCalib):
img_calib = cv2.imread(FilepathCalib + '/' + img_calib, 0)
img_calib_set.append(img_calib)
# Finding checkerboard corners pattern in image dataset
for img in img_calib_set:
ret_val, corner = cv2.findChessboardCorners(img, (7,9), None)
if ret_val == True:
points_3d.append(points_3d_sample)
points_2d.append(corner)
# Extracting camera parameters
ret_val, intrinsic_mat, dist_coef, rot_vector, tran_vector =
cv2.calibrateCamera(points_3d, points_2d, img_calib_set[0].shape[::-1], None, None)
print(intrinsic_mat)
And here’s my sample calibration image
Answers:
The cv2.findChessboardCorners() function’s processing time depends a lot on the input image.
30 minutes on 1K-4K images is not abnormal.
Have you tried writing the processing time to the console output or a debug log?
This way you can find out if certain images take a lot longer than the others. I had a case in which most images took 10-20 seconds, but a handful took 3-5 minutes. The pictures that took a lot of time to process often had blurry checkerboard patterns (motion blur), or had other rectangular background patterns (closets, blinds, window frames, etc…).
Judging your input image, I can definitely see the paper cardboard is slightly curved as a result of holding it. I also had a hard time sticking the calibration pattern to a truly flat surface. In the end I used an LCD monitor to display the image, and moved the camera around for the calibration images (make sure you don’t scale the image on the monitor; 1 pixel on the image should be 1 pixel on the monitor, and it doesn’t have to be full screen).
This gave me the lowest re-projection error.
IS your sample image lower quality than your actual footage? You can clearly see that the JPEG compression algorithm creates a lot of compression artifacts on the edges in the image. This is not ideal. If this is your actual footage, I would recommend setting the compression quality to maximum, or use a lossless format like PNG.
I was able to cut the calibration time in half by replacing some calibration images with better candidates.
add an argument flags = cv2.CALIB_USE_LU then it will much faster.
such as:
cv2.calibrateCamera(points_3d, points_2d, img_calib_set[0].shape[::-1], None, None,flags = cv2.CALIB_USE_LU)
To add more to @yulong ‘s answer, using the CALIB_USE_LU uses LU decomposition to solve the mathematical problem while matching the points, instead of SVD decomposition which is the default. The difference of the two algorithms is well summarized here.
I used my realsense D455 for comparing the difference of the two.
With 210 images of identical 1280×720 640×480 image sets for both, each took:
774.77 seconds for SVD decomposition
9.47 seconds for LU decomposition
The manufacturer provided intrinsics are
[ 1280x720 p[649.005 363.69] f[643.648 642.881] Inverse Brown Conrady [-0.0569014 0.0683799 -0.000436772 0.00101618 -0.0218782] ]
SVD produced
[[399.20488414 0. 321.01594445]
[ 0. 399.92750648 243.45452012]
[ 0. 0. 1. ]]
[[-0.06073074 0.06658484 -0.00025374 0.00013107 -0.01865239]]
with rms of 0.833252711657677
LU produced
[[399.20488387 0. 321.01594447]
[ 0. 399.92750621 243.45452006]
[ 0. 0. 1. ]]
[[-0.06073074 0.06658484 -0.00025374 0.00013107 -0.01865239]]
with rms of 0.8332527116576773
with 60 images, it took
15.27 seconds for SVD and 0.21 seconds for LU, with similar results.
According to an experiment by Witek, the execution time increases in rather highly polynomial shape, if not exponential.
I was trying to replicate this camera calibration code.
When I run my script, the calibration process won’t stop for about 30 minutes (I expected the intrinsic camera matrix for the output). What is wrong with my code? Why does the calibration process take so long?
This is my code
import os
import cv2
import numpy as np
# DUMMY TEST
FilepathCalib = './data'
# Declaring global variable
img_calib_set = []
points_3d_sample = np.zeros((6*7,3), np.float32)
points_3d_sample[:,:2] = np.mgrid[0:7,0:6].T.reshape(-1,2)
points_2d = []
points_3d = []
# Loading calibration image dataset
for img_calib in os.listdir(FilepathCalib):
img_calib = cv2.imread(FilepathCalib + '/' + img_calib, 0)
img_calib_set.append(img_calib)
# Finding checkerboard corners pattern in image dataset
for img in img_calib_set:
ret_val, corner = cv2.findChessboardCorners(img, (7,9), None)
if ret_val == True:
points_3d.append(points_3d_sample)
points_2d.append(corner)
# Extracting camera parameters
ret_val, intrinsic_mat, dist_coef, rot_vector, tran_vector =
cv2.calibrateCamera(points_3d, points_2d, img_calib_set[0].shape[::-1], None, None)
print(intrinsic_mat)
And here’s my sample calibration image
The cv2.findChessboardCorners() function’s processing time depends a lot on the input image.
30 minutes on 1K-4K images is not abnormal.
Have you tried writing the processing time to the console output or a debug log?
This way you can find out if certain images take a lot longer than the others. I had a case in which most images took 10-20 seconds, but a handful took 3-5 minutes. The pictures that took a lot of time to process often had blurry checkerboard patterns (motion blur), or had other rectangular background patterns (closets, blinds, window frames, etc…).
Judging your input image, I can definitely see the paper cardboard is slightly curved as a result of holding it. I also had a hard time sticking the calibration pattern to a truly flat surface. In the end I used an LCD monitor to display the image, and moved the camera around for the calibration images (make sure you don’t scale the image on the monitor; 1 pixel on the image should be 1 pixel on the monitor, and it doesn’t have to be full screen).
This gave me the lowest re-projection error.
IS your sample image lower quality than your actual footage? You can clearly see that the JPEG compression algorithm creates a lot of compression artifacts on the edges in the image. This is not ideal. If this is your actual footage, I would recommend setting the compression quality to maximum, or use a lossless format like PNG.
I was able to cut the calibration time in half by replacing some calibration images with better candidates.
add an argument flags = cv2.CALIB_USE_LU then it will much faster.
such as:
cv2.calibrateCamera(points_3d, points_2d, img_calib_set[0].shape[::-1], None, None,flags = cv2.CALIB_USE_LU)
To add more to @yulong ‘s answer, using the CALIB_USE_LU uses LU decomposition to solve the mathematical problem while matching the points, instead of SVD decomposition which is the default. The difference of the two algorithms is well summarized here.
I used my realsense D455 for comparing the difference of the two.
With 210 images of identical 1280×720 640×480 image sets for both, each took:
774.77 seconds for SVD decomposition
9.47 seconds for LU decomposition
The manufacturer provided intrinsics are
[ 1280x720 p[649.005 363.69] f[643.648 642.881] Inverse Brown Conrady [-0.0569014 0.0683799 -0.000436772 0.00101618 -0.0218782] ]
SVD produced
[[399.20488414 0. 321.01594445]
[ 0. 399.92750648 243.45452012]
[ 0. 0. 1. ]]
[[-0.06073074 0.06658484 -0.00025374 0.00013107 -0.01865239]]
with rms of 0.833252711657677
LU produced
[[399.20488387 0. 321.01594447]
[ 0. 399.92750621 243.45452006]
[ 0. 0. 1. ]]
[[-0.06073074 0.06658484 -0.00025374 0.00013107 -0.01865239]]
with rms of 0.8332527116576773
with 60 images, it took
15.27 seconds for SVD and 0.21 seconds for LU, with similar results.
According to an experiment by Witek, the execution time increases in rather highly polynomial shape, if not exponential.