cv2.calcOpticalFlowPyrLK Adding New Points

The Problem

I am trying to add custom points for tracking in Lucas Kanade optical flow tracking method. But I get an Assertion failed error when I call this function after inserting my custom points to the points array that was automatically calculated.

OpenCV Error: Assertion failed ((npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0) in calcOpticalFlowPyrLK, file /opt/opencv-3.1.0/opencv/modules/video/src/lkpyramid.cpp, line 1114 
Traceback (most recent call last):
  File "/opt/pycharm-community-2016.2.3/helpers/pydev/pydevd.py", line 1580, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/opt/pycharm-community-2016.2.3/helpers/pydev/pydevd.py", line 964, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/opt/pycharm-community-2016.2.3/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"n", file, 'exec'), glob, loc)
  File "/work/12_Kyloren/Repository/kyloren/firmware/Python/modules/kanade.py", line 45, in <module>
    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)
cv2.error: /opt/opencv-3.1.0/opencv/modules/video/src/lkpyramid.cpp:1114: error: (-215) (npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0 in function calcOpticalFlowPyrLK

The Code

I first sample points on mouse click like this:-

def sample_track_points(event, x,y, flags, param):
    global user_points
    if event == cv2.EVENT_LBUTTONDBLCLK:
        user_points = np.empty([1, 1, 2], dtype=float)
        user_points[0][0] = [x,y]


cv2.namedWindow("frame")
cv2.setMouseCallback("frame",sample_track_points)

Then I concatenate it to the main points array that is automatically created from goodFeaturesToTrack

p0 = cv2.goodFeaturesToTrack(old_gray, mask=None, **feature_params)

This is how I concatenate

if len(user_points) > 0:
    p0 = np.concatenate([p0, user_points])
    user_points = np.empty([0,1,2])

It fails at this line

    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

I have stepped and debugged and fixed all numpy array mismatch issues. So the issue is with some internal mechanism.

Can anyone help?

The Full Code

I have modified this program here (the first one) and below is my fully modified code.

import numpy as np
import cv2
cap = cv2.VideoCapture("/work/12_Kyloren/Videos/WC5_Door_Front.mp4")
# params for ShiTomasi corner detection
feature_params = dict( maxCorners = 100,
                       qualityLevel = 0.3,
                       minDistance = 7,
                       blockSize = 7 )
# Parameters for lucas kanade optical flow
lk_params = dict( winSize  = (15,15),
                  maxLevel = 2,
                  criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_COUNT, 10, 0.03))


# Create some random colors
color = np.random.randint(0,255,(100,3))
# Take first frame and find corners in it
ret, old_frame = cap.read()
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
p0 = cv2.goodFeaturesToTrack(old_gray, mask=None, **feature_params)

# Create a mask image for drawing purposes
mask = np.zeros_like(old_frame)


# add points to the existing array on mouseclick
user_points = np.empty([0,1,2],dtype=float)
def sample_track_points(event, x,y, flags, param):
    global user_points
    if event == cv2.EVENT_LBUTTONDBLCLK:
        user_points = np.empty([1, 1, 2], dtype=float)
        user_points[0][0] = [x,y]


# set the mouse call back
cv2.namedWindow("frame")
cv2.setMouseCallback("frame",sample_track_points)

# start the processing
while(1):
    ret,frame = cap.read()
    frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    if len(user_points) > 0:
        p0 = np.concatenate([p0, user_points])
        user_points = np.empty([0,1,2])

    # calculate optical flow
    p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)
    # Select good points
    good_new = p1[st==1]
    good_old = p0[st==1]
    # draw the tracks
    for i,(new,old) in enumerate(zip(good_new,good_old)):
        a,b = new.ravel()
        c,d = old.ravel()
        mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)
        frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)
    img = cv2.add(frame,mask)
    cv2.imshow('frame',img)
    k = cv2.waitKey(30) & 0xff

    if k == 27:
        break
    # Now update the previous frame and previous points
    old_gray = frame_gray.copy()
    p0 = good_new.reshape(-1,1,2)
cv2.destroyAllWindows()
cap.release()

Reproduce Error

Double Clicking on the preview window will cause it to crash with an assertion fail error.