How to get the latest frame from capture device (camera) in opencv
Question:
I want to connect to a camera, and only capture a frame when an event happens (e.g. keypress). A simplified version of what I’d like to do is this:
cap = cv2.VideoCapture(device_id)
while True:
if event:
img = cap.read()
preprocess(img)
process(img)
cv.Waitkey(10)
However, cap.read seems to only capture the next frame in the queue, and not the latest. I did a lot of searching online, and there seems to be a lot of questions on this but no definitive answer. Only some dirty hacks which involve opening and closing the capture device just before and after grabbing (which won’t work for me as my event might be triggered multiple times per second); or assuming a fixed framerate and reading a fixed-n times on each event (which won’t work for me as my event is unpredictable and could happen at any interval).
A nice solution would be:
while True:
if event:
while capture_has_frames:
img = cap.read()
preprocess(img)
process(img)
cv.Waitkey(10)
But what is capture_has_frames? Is it possible to get that info? I tried looking into CV_CAP_PROP_POS_FRAMES but it’s always -1.
For now I have a separate thread where the capture is running at full fps, and on my event I’m grabbing the latest image from that thread, but this seems overkill.
(I’m on Ubuntu 16.04 btw, but I guess it shouldn’t matter. I’m also using pyqtgraph for display)
Answers:
If you don’t want to capture the frame when there is no event happening, why are you preprocessing/processing your frame? If you do not process your frame, you can simply discard it unless the event occur. Your program should be able to capture, evaluate your condition and discard at a sufficient speed, i.e. fast enough compared to your camera FPS capture rate, to always get the last frame in the queue.
If not proficient in python because I do my OpenCV in C++, but it should look similar to this:
vidcap = cv.VideoCapture( filename )
while True:
success, frame = vidcap.read()
If Not success:
break
If cv.waitKey(1):
process(frame)
As per OpenCV reference, vidcap.read() returns a bool. If frame is read correctly, it will be True. Then, the captured frame is store in variable frame. If there is no key press, the loop keeps on going. When a key is pressed, you process your last captured frame.
I think the solution mentioned in the question, namely having a separate thread that clears the buffer, is the easiest non-brittle solution for this. Here reasonably nice (I think) code for this:
import cv2, queue, threading, time
# bufferless VideoCapture
class VideoCapture:
def __init__(self, name):
self.cap = cv2.VideoCapture(name)
self.q = queue.Queue()
t = threading.Thread(target=self._reader)
t.daemon = True
t.start()
# read frames as soon as they are available, keeping only most recent one
def _reader(self):
while True:
ret, frame = self.cap.read()
if not ret:
break
if not self.q.empty():
try:
self.q.get_nowait() # discard previous (unprocessed) frame
except queue.Empty:
pass
self.q.put(frame)
def read(self):
return self.q.get()
cap = VideoCapture(0)
while True:
time.sleep(.5) # simulate time between events
frame = cap.read()
cv2.imshow("frame", frame)
if chr(cv2.waitKey(1)&255) == 'q':
break
The frame reader thread is encapsulated inside the custom VideoCapture class, and communication with the main thread is via a queue.
I posted very similar code for a node.js question, where a JavaScript solution would have been better. My comments on another answer to that question give details why a non-brittle solution without separate thread seems difficult.
An alternative solution that is easier but supported only for some OpenCV backends is using CAP_PROP_BUFFERSIZE. The 2.4 docs state it is "only supported by DC1394 [Firewire] v 2.x backend currently." For Linux backend V4L, according to a comment in the 3.4.5 code, support was added on 9 Mar 2018, but I got VIDEOIO ERROR: V4L: Property <unknown property string>(38) not supported by device for exactly this backend. It may be worth a try first; the code is as easy as this:
cap.set(cv2.CAP_PROP_BUFFERSIZE, 0)
On my Raspberry Pi 4,
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
does work and was all that I needed for my pi camera to give me the latest frame, with a consistent 3+ second delay between the scene in front of the camera and displaying that scene in the preview image. My code takes 1.3 seconds to process an image, so I’m not sure why the other 2 seconds of delay are present, but it’s consistent and works.
Side note: since my code takes over a second to process an image, I also added
cap.set( cv2.CAP_PROP_FPS, 2 )
in case it reduces any unneeded activity, since I can’t quite get a frame a second. When I put cv2.CAP_PROP_FPS to 1, though, I got a strange output of all my frames being almost entirely dark, so setting FPS too low can cause an issue
Here’s a simplified version of Ulrich’s solution.
OpenCV’s read() function combines grab() and retrieve() in one call, where grab() just loads the next frame in memory, and retrieve decodes the latest grabbed frame (demosaicing & motion jpeg decompression).
We’re only interested in decoding the frame we’re actually reading, so this solution saves some CPU, and removes the need for a queue
import cv2
import threading
# bufferless VideoCapture
class VideoCapture:
def __init__(self, name):
self.cap = cv2.VideoCapture(name)
self.lock = threading.Lock()
self.t = threading.Thread(target=self._reader)
self.t.daemon = True
self.t.start()
# grab frames as soon as they are available
def _reader(self):
while True:
with self.lock:
ret = self.cap.grab()
if not ret:
break
# retrieve latest frame
def read(self):
with self.lock:
_, frame = self.cap.retrieve()
return frame
EDIT: Following Arthur Tacca’s comment, added a lock to avoid simultaneous grab & retrieve, which could lead to a crash as OpenCV isn’t thread-safe.
Its also possible to always get the latest frame by using cv2.CAP_GSTREAMER backend. If you have gstreamer support enabled in cv2.getBuildInformation(), you can initialize your video capture with the appsink parameters sync=false and drop=true
Example:
cv2.VideoCapture("rtspsrc location=rtsp://... ! decodebin ! videoconvert ! video/x-raw,framerate=30/1 ! appsink drop=true sync=false", cv2.CAP_GSTREAMER)
I want to connect to a camera, and only capture a frame when an event happens (e.g. keypress). A simplified version of what I’d like to do is this:
cap = cv2.VideoCapture(device_id)
while True:
if event:
img = cap.read()
preprocess(img)
process(img)
cv.Waitkey(10)
However, cap.read seems to only capture the next frame in the queue, and not the latest. I did a lot of searching online, and there seems to be a lot of questions on this but no definitive answer. Only some dirty hacks which involve opening and closing the capture device just before and after grabbing (which won’t work for me as my event might be triggered multiple times per second); or assuming a fixed framerate and reading a fixed-n times on each event (which won’t work for me as my event is unpredictable and could happen at any interval).
A nice solution would be:
while True:
if event:
while capture_has_frames:
img = cap.read()
preprocess(img)
process(img)
cv.Waitkey(10)
But what is capture_has_frames? Is it possible to get that info? I tried looking into CV_CAP_PROP_POS_FRAMES but it’s always -1.
For now I have a separate thread where the capture is running at full fps, and on my event I’m grabbing the latest image from that thread, but this seems overkill.
(I’m on Ubuntu 16.04 btw, but I guess it shouldn’t matter. I’m also using pyqtgraph for display)
If you don’t want to capture the frame when there is no event happening, why are you preprocessing/processing your frame? If you do not process your frame, you can simply discard it unless the event occur. Your program should be able to capture, evaluate your condition and discard at a sufficient speed, i.e. fast enough compared to your camera FPS capture rate, to always get the last frame in the queue.
If not proficient in python because I do my OpenCV in C++, but it should look similar to this:
vidcap = cv.VideoCapture( filename )
while True:
success, frame = vidcap.read()
If Not success:
break
If cv.waitKey(1):
process(frame)
As per OpenCV reference, vidcap.read() returns a bool. If frame is read correctly, it will be True. Then, the captured frame is store in variable frame. If there is no key press, the loop keeps on going. When a key is pressed, you process your last captured frame.
I think the solution mentioned in the question, namely having a separate thread that clears the buffer, is the easiest non-brittle solution for this. Here reasonably nice (I think) code for this:
import cv2, queue, threading, time
# bufferless VideoCapture
class VideoCapture:
def __init__(self, name):
self.cap = cv2.VideoCapture(name)
self.q = queue.Queue()
t = threading.Thread(target=self._reader)
t.daemon = True
t.start()
# read frames as soon as they are available, keeping only most recent one
def _reader(self):
while True:
ret, frame = self.cap.read()
if not ret:
break
if not self.q.empty():
try:
self.q.get_nowait() # discard previous (unprocessed) frame
except queue.Empty:
pass
self.q.put(frame)
def read(self):
return self.q.get()
cap = VideoCapture(0)
while True:
time.sleep(.5) # simulate time between events
frame = cap.read()
cv2.imshow("frame", frame)
if chr(cv2.waitKey(1)&255) == 'q':
break
The frame reader thread is encapsulated inside the custom VideoCapture class, and communication with the main thread is via a queue.
I posted very similar code for a node.js question, where a JavaScript solution would have been better. My comments on another answer to that question give details why a non-brittle solution without separate thread seems difficult.
An alternative solution that is easier but supported only for some OpenCV backends is using CAP_PROP_BUFFERSIZE. The 2.4 docs state it is "only supported by DC1394 [Firewire] v 2.x backend currently." For Linux backend V4L, according to a comment in the 3.4.5 code, support was added on 9 Mar 2018, but I got VIDEOIO ERROR: V4L: Property <unknown property string>(38) not supported by device for exactly this backend. It may be worth a try first; the code is as easy as this:
cap.set(cv2.CAP_PROP_BUFFERSIZE, 0)
On my Raspberry Pi 4,
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
does work and was all that I needed for my pi camera to give me the latest frame, with a consistent 3+ second delay between the scene in front of the camera and displaying that scene in the preview image. My code takes 1.3 seconds to process an image, so I’m not sure why the other 2 seconds of delay are present, but it’s consistent and works.
Side note: since my code takes over a second to process an image, I also added
cap.set( cv2.CAP_PROP_FPS, 2 )
in case it reduces any unneeded activity, since I can’t quite get a frame a second. When I put cv2.CAP_PROP_FPS to 1, though, I got a strange output of all my frames being almost entirely dark, so setting FPS too low can cause an issue
Here’s a simplified version of Ulrich’s solution.
OpenCV’s read() function combines grab() and retrieve() in one call, where grab() just loads the next frame in memory, and retrieve decodes the latest grabbed frame (demosaicing & motion jpeg decompression).
We’re only interested in decoding the frame we’re actually reading, so this solution saves some CPU, and removes the need for a queue
import cv2
import threading
# bufferless VideoCapture
class VideoCapture:
def __init__(self, name):
self.cap = cv2.VideoCapture(name)
self.lock = threading.Lock()
self.t = threading.Thread(target=self._reader)
self.t.daemon = True
self.t.start()
# grab frames as soon as they are available
def _reader(self):
while True:
with self.lock:
ret = self.cap.grab()
if not ret:
break
# retrieve latest frame
def read(self):
with self.lock:
_, frame = self.cap.retrieve()
return frame
EDIT: Following Arthur Tacca’s comment, added a lock to avoid simultaneous grab & retrieve, which could lead to a crash as OpenCV isn’t thread-safe.
Its also possible to always get the latest frame by using cv2.CAP_GSTREAMER backend. If you have gstreamer support enabled in cv2.getBuildInformation(), you can initialize your video capture with the appsink parameters sync=false and drop=true
Example:
cv2.VideoCapture("rtspsrc location=rtsp://... ! decodebin ! videoconvert ! video/x-raw,framerate=30/1 ! appsink drop=true sync=false", cv2.CAP_GSTREAMER)