How to map correlation between two frames to tracking window of a selected object in object tracking

Question

I am trying to implement an object tracking algorithm using phase correlation, based on the paper titled "A Robust Image Tracker based on Phase Correlation and Fourier-Mallin Transform". I am at the initial stages of implementation where I select an object in a video and form a tracking window around it. Now upon finding the phase correlation between the previous and current frame I would like to move the tracking window as suggested in the algorithm. However whafter finding the phase correlation output I am unable to understand what it means and how I should move the tracking bar according to this output.

I find the sub-pixel displacement term mentioned in the paper for this purpose but can not get around how to implement this.

Any help, suggestions or sources will be appreciated.

In my code after an object is selected then the flag1 variable changes which initiates the tracking rectangle around the object. Then I tried directly adding the output of phasecorrelate function to the track box.

import cv2 as cv
import numpy as np

capture = cv.VideoCapture('PHASE_CORRELATION/VIDEOS/flying_bird1.mp4')
cv.createHanningWindow((960,540),cv.CV_32F)
flag = 0
flag1 = 0

a,b = 0,0
def mouseHandler(event, x, y, flags, params):
    if event == cv.EVENT_LBUTTONDOWN:
        flag1 = 1
        global a
        global b
        a = round(x)
        b = round(y)
cv.namedWindow('Window',cv.WINDOW_NORMAL)
cv.setMouseCallback('Window',mouseHandler)


while True:

    istrue,frame = capture. read()

    frame = cv.resize(frame,(960,540),interpolation=cv.INTER_AREA)

    frame = cv.cvtColor(frame,cv.COLOR_BGR2GRAY)


    if (flag==0):
        curr = frame
        flag = 1
    else:
        prev = curr
        curr = frame

        
        shift = cv.phaseCorrelate(np.float32(prev),np.float32(curr))

        if (flag1==1):
            frame = cv.rectangle(frame,(x+30,y+30),(x-30,y-30),[255,0,0],2)
            a = round(a+shift[0][0])
            b = round(b+shift[0][1])
   
            flag1 = 2
        
        if (flag1==2):
            a = round(a+shift[0][0])
            b = round(b+shift[0][1])
            frame = cv.rectangle(frame,(x+shift[0][0]+30,y+shift[0][1]+30),(x+shift[0][0]-30,y+shift[0][1]-30),2)




        cv.imshow('Window',frame)

    if cv.waitKey(20) & 0xFF==ord('d'):
        break

capture. release()
cv.destroyAllwindows()

Asked By: NIHAR

||

Source

Answer 1

Here is a simple code for doing phase correlation in Python/OpenCV.

The values returned from doing that are (a tuple of a tuple and a constant), namely:
((Offset_x, Offset_y), phase_corr_score)

where the offsets are the shifts in x and y between the two images and the constant is the phase correlation peak value. Actually (from the documentation), "if non-zero, the [correlation peak] parameter is computed as the sum of the elements within the 5×5 centroid around the peak location. It is normalized to a maximum of 1 (meaning there is a single peak) and will be smaller when there are multiple peaks."

Input 1:

Input 2:

import cv2
import numpy as np

# read image 1 as grayscale
img1 = cv2.imread('a.png', cv2.IMREAD_GRAYSCALE)

# read image 2 as grayscale
img2 = cv2.imread('b.png', cv2.IMREAD_GRAYSCALE)

# do phase correlation
((shift_x, shift_y), phase_corr_score) = cv2.phaseCorrelate(np.float32(img1), np.float32(img2))

print("Xoffset:", shift_x)
print("Yoffset:", shift_y)
print("Correlation Score:", phase_corr_score)

Resulting Values:

Xoffset: 20.208096095390133 (px)
Yoffset: 22.46002747057109  (px)
Correlation Score: 0.5970434197084403 (in range 0 to 1)

img1 would need to be shifted 20 px to the right and 22 px down to be in the same position as img2

I think sub-pixel results are being returned already. From the documentation: "Finally, it computes the peak location and computes a 5×5 weighted centroid around the peak to achieve sub-pixel accuracy."

With regard to your target window, perhaps the pixel data within the target windows are what you are supposed to phase correlate. That is take sub-section crops of the two images within the bounds of the target windows and phase correlate those crops (ROIs). Sorry, I do not have access to that reference and you have not provided a link to it.

Answered By: fmw42

How to map correlation between two frames to tracking window of a selected object in object tracking

Question:

Answers: