How does OpenCV projectPoints perform transformations before projecting?

Question

I have two 3d points that I am trying to project onto a 2d image plane using cv::projectPoints(). Originally the points are not in the cameras frame of reference so I have to transform them. I am testing the validity of the method’s transformations.

First I manually apply a translation to my points as well as a -90 degree rotation about the x axis via rotation matrix matr.

import numpy as np
import math
import cv2

# center of projection
cop = np.array([-14.45194, 34.59882, 19.11343])

# get rotation
r_vec = cv2.Rodrigues(np.array(matr))[0]
print([round(math.degrees(i),2) for i in r_vec])

# init arrays
coords = np.array([[4.27874, 115.15968, 18.1621], [27.52924, 113.3441, 17.70207]])
transformed_coords = np.zeros(coords.shape)

# transform coords
for b, _ in enumerate(coords):

    arr = np.array([0,0,0])

    # translate
    for r in range(3):
        arr[r] = coords[b][r] - cop[r]

    # rotate
    transformed_coords[b] = np.dot(matr, arr)

Next I pass in the transformed coords into projectPoints() and compare the resulting 2d points with the points I get by passing the transformation into the method.

points_2d = cv2.projectPoints(np.array(transformed_coords), np.array([0.0,0.0,0.0]), np.array([0.0,0.0,0.0]), cam_matrix, distortion)[0]
print("Manual Transformation Projection: ")
print(points_2d )

points_2d = cv2.projectPoints(np.array(coords), np.array(r_vec), np.array(cop), cam_matrix, distortion)[0]
print("nOpenCV Transformation Projection: ")
print(points_2d )

Output:

[-90.0, 0.0, 0.0] # matr rotation 

Manual Transformation Projection: 
[[[596.41419111 538.38054858]]

 [[159.74685131 557.65317027]]]

OpenCV Transformation Projection: 
[[[1101.1539809  -274.07081182]]

 [[ 738.45477039 -281.42273082]]]

Why are they different?

By the way heres the cam matrix and distortion if you want to recreate it:

cam_matrix = np.array([[1561.9015217711233, 0, 944.3790845611046], [0, 1557.8348925840205, 538.3374859400157], [0, 0, 1]])
distortion = np.array([-0.2136432557736835, 0.20055112514542725, 0.00054631323043295, -0.00067835485282051, -0.07781645541334031])

Asked By: i_sniff_ket

||

Source

Answer 1

to clarify a few things:

projectPoints expects rvec and tvec to be the object’s pose in the camera’s coordinate system/frame (transforming points from object frame to camera frame)
If you already transformed your points and they’re camera-local, you should pass empty/all-zero rvec and tvec
The rvec is NOT Euler angles. It is an axis-angle encoding. Its length equals the amount of rotation (in radians, not degrees). The vector itself is the axis of rotation.

Issues you should investigate:

Assuming your cop array is supposed to be the tvec, you are supposed to add that, not subtract it.
Present matr. It is not contained in your code.
use numpy effectively. Don’t iterate and assign individual elements of arrays.

If you have a world frame, and both your camera and object are defined in the world frame, then you have T_world_cam and T_world_obj.

For projectPoints(), you need T_cam_obj, or ^camT_obj (math notation).

This transformation…

transforms geometry from the object frame (input, right side) into the camera frame (output, left side)
expresses the object’s pose relative to the camera frame

The equations:

T_cam_obj = T_cam_world @ T_world_obj
T_cam_obj = inv(T_world_cam) @ T_world_obj # np.linalg.inv

Write utility functions that convert between rvec,tvec (for OpenCV) and 4×4 matrix representation (for calculating without going insane):

def rtvec_to_matrix(rvec=(0,0,0), tvec=(0,0,0)):
    "Convert rotation vector and translation vector to 4x4 matrix"
    rvec = np.asarray(rvec)
    tvec = np.asarray(tvec)

    T = np.eye(4)
    (R, jac) = cv.Rodrigues(rvec)
    T[:3, :3] = R
    T[:3, 3] = tvec.squeeze()
    return T

def matrix_to_rtvec(matrix):
    "Convert 4x4 matrix to rotation vector and translation vector"
    (rvec, jac) = cv.Rodrigues(matrix[:3, :3])
    tvec = matrix[:3, 3]
    return rvec, tvec

Answered By: Christoph Rackwitz

How does OpenCV projectPoints perform transformations before projecting?

Question:

Answers: