Triangulated points are shifted and rotated but reasonable otherwise

I have two different views of a football game, I use a 2d pose estimator on both views to get 2d poses and then I map the different 2d poses(currently by hand) in the two different cameras such that I know the corresponding poses. I think the 3d reconstruction looks very reasonable, however it’s rotated and shifted in space, and I can’t figure out what’s causing this.

Here’s an image of my triangulation:

My triangulated pose

To calibrate the camera I handmark known points on the field, for instance corners, penalty spot etc, mostly planar points but I also use the cross of the goal to get some nonplanar points. I have tried a bunch of different ways to calculate estimated intrinsic parameters, the best result I have is using the planar points to estimate the intrinsic matrix with the cameraCalibrate function.

def new_guess(self, image_points, real_worlds):
        planar_img, planar_3d = Homo_est.get_planar_points(image_points, real_worlds)
        mat, ds = self.create_camera_guess(planar_img, planar_3d)
        real_worlds = np.array([list(tup) for tup in real_worlds])
        image_points =[list(tup) for tup in image_points]
        objPts = []
        imgPts = []
        
        objPts.append(real_worlds)
        imgPts.append(image_points)
        objPts = np.float32(objPts)
        imgPts = np.float32(imgPts)
        dist_coeffs = np.zeros((4,1)) 
        gray = cv2.cvtColor(self.img, cv2.COLOR_BGR2GRAY)
        #mat = self.second_camera_guess() # bästa so far
        criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30000, 0.0001) # att ha 
        ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objPts, imgPts, gray.shape[::-1], mat,             ds, flags=cv2.CALIB_USE_INTRINSIC_GUESS, criteria=criteria)
        return mtx, dist

The new_guess function essentially calculates my intrinsic parameters. I then use the intrinsic matrix calculated as input to the solvePNP function where I calculate my P matrix. I verify the result using cv2.projectPoints(), where I have an error of at most 10 pixels and most times less which I believe is accurate?

def calibrate_solvePNP(self, image_points, real_worlds):
        planar_img, planar_3d = Homo_est.get_planar_points(image_points, real_worlds)    
        mat, dist = self.new_guess(image_points, real_worlds)
        real_worlds = np.array([list(tup) for tup in real_worlds])
        image_points =[list(tup) for tup in image_points]
        objPts = []
        imgPts = []        
        objPts.append(real_worlds)
        imgPts.append(image_points)
        objPts = np.float32(objPts)
        imgPts = np.float32(imgPts)
        dist_coeffs = np.zeros((4,1)) 
        gray = cv2.cvtColor(self.img, cv2.COLOR_BGR2GRAY)
        (succsess, rvec, t) = cv2.solvePnP(objPts, imgPts, mat, dist_coeffs)
        res, jac = cv2.projectPoints(real_worlds, rvec, t, mat, dist_coeffs)

        R, jac = cv2.Rodrigues(rvec)
        Rt = np.concatenate([R,t], axis=-1) # [R|t]
        P = np.matmul(mat, Rt)
        return P 

Following this I triangulate my points:

def triangulate_pts(self):
        P1_points = Homo_est.player_view1() # pose coordinates view 1
        P2_points = Homo_est.player_view2() # pose coordinates view 2
        homo = Homo_est()
        P1, P2, K1, K2, d1, d2 = homo.generate_cameras() # essentially calls my calibrate_solvePNP # and new_guess to create these values
        dist_coeffs = np.zeros((4,1)) 
        P1_new = np.matmul(np.linalg.inv(K1), P1) # unsure if this is good or bad
        P2_new = np.matmul(np.linalg.inv(K2), P2)

        # doesn't seem to make a difference using d1 or d2
        P1_undist = cv2.undistortPoints(P1_points, cameraMatrix=K1, distCoeffs=dist_coeffs)
        
        P2_undist = cv2.undistortPoints(P2_points, 
                                   cameraMatrix=K2,
                                   distCoeffs=dist_coeffs)
        
        triangulation = cv2.triangulatePoints(P1_new, P2_new, P1_undist, P2_undist)

        homog_points = triangulation.transpose()
        
        euclid_points = cv2.convertPointsFromHomogeneous(homog_points)

The euclid_points is what I then use to plot and draw my 3d pose!

I’ve tried playing around with the distortion coefficients in all possible ways, it doesn’t seem to make much of a difference. I tried normalizing one camera such that I have one camera as [I 0] and converted the other camera as well, didn’t work but there could’ve been some innaccuracy in this attempt. I’ve followed all steps of this(unless i’ve made some misstake that I can’t spot): Is cv2.triangulatePoints just not very accurate?

I have tried basically everything I can think of, intuitevly I feel like there has to be some way to undo this rotation and shift since the points do resemble the pose i’m estimating. In worst case it would be enough to just be able to get rid of the rotation since I have an accurate homography for points that are on field! Anyone has any idea of what could be the reason?