Pipe raw OpenCV images to FFmpeg
Question:
Here’s a fairly straightforward example of reading off a web cam using OpenCV’s python bindings:
'''capture.py'''
import cv, sys
cap = cv.CaptureFromCAM(0) # 0 is for /dev/video0
while True :
if not cv.GrabFrame(cap) : break
frame = cv.RetrieveFrame(cap)
sys.stdout.write( frame.tostring() )
Now I want to pipe the output to ffmpeg as in:
$ python capture.py | ffmpeg -f image2pipe -pix_fmt bgr8 -i - -s 640x480 foo.avi
Sadly, I can’t get the ffmpeg magic incantation quite right and it fails with
libavutil 50.15. 1 / 50.15. 1
libavcodec 52.72. 2 / 52.72. 2
libavformat 52.64. 2 / 52.64. 2
libavdevice 52. 2. 0 / 52. 2. 0
libavfilter 1.19. 0 / 1.19. 0
libswscale 0.11. 0 / 0.11. 0
libpostproc 51. 2. 0 / 51. 2. 0
Output #0, avi, to 'out.avi':
Stream #0.0: Video: flv, yuv420p, 640x480, q=2-31, 19660 kb/s, 90k tbn, 30 tbc
[image2pipe @ 0x1508640]max_analyze_duration reached
[image2pipe @ 0x1508640]Estimating duration from bitrate, this may be inaccurate
Input #0, image2pipe, from 'pipe:':
Duration: N/A, bitrate: N/A
Stream #0.0: Video: 0x0000, bgr8, 25 fps, 25 tbr, 25 tbn, 25 tbc
swScaler: 0x0 -> 640x480 is invalid scaling dimension
- The captured frames are definitely 640×480.
- I’m pretty sure the pixel order for the OpenCV image type (IplImage) is GBR, one byte per channel. At least, that’s what seems to be coming off the camera.
I’m no ffmpeg guru. Has anyone done this successfully?
Answers:
Took a bunch of fiddling but I figured it out using the FFmpeg rawvideo demuxer:
python capture.py | ffmpeg -f rawvideo -pixel_format bgr24 -video_size 640x480 -framerate 30 -i - foo.avi
Since there is no header in raw video specifying the assumed video parameters, the user must specify them in order to be able to decode the data correctly:
-framerate Set input video frame rate. Default value is 25.-pixel_format Set the input video pixel format. Default value is yuv420p.-video_size Set the input video size. There is no default, so this value must be specified explicitly.
And here’s a little something extra for the power users. Same thing but using VLC to stream the live output to the web, Flash format:
python capture.py | cvlc --demux=rawvideo --rawvid-fps=30 --rawvid-width=320 --rawvid-height=240 --rawvid-chroma=RV24 - --sout "#transcode{vcodec=h264,vb=200,fps=30,width=320,height=240}:std{access=http{mime=video/x-flv},mux=ffmpeg{mux=flv},dst=:8081/stream.flv}"
Edit:
Create a webm stream using ffmpeg and ffserver
python capture.py | ffmpeg -f rawvideo -pixel_format rgb24 -video_size 640x480 -framerate 25 -i - http://localhost:8090/feed1.ffm
Took me an hour to figure out that by default, windows pipes are not binary. This causes some bytes (specifically newlines) to be modified/omitted, and the resulting video is slowly shifting because the frame size is not constant.
To work this around, the modified python file:
"""
videoCapture.py
"""
import cv2, sys
import time
if sys.platform == "win32":
import os, msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
cap = cv2.VideoCapture(0) # 0 is for /dev/video0
while True :
ret, frm = cap.read()
sys.stdout.write( frm.tostring() )
To test if piping the raw video is successful, use ffplay. Make sure you specify a higher framerate than what is coming from the pipe, otherwise the video will start to lag
python videoCapture.py | ffplay -f rawvideo -pix_fmt bgr24 -s 640x480 -framerate 40 -i -
Not sure if this is Mac OS-specific, or python3-specific, but I needed to cast the frame to a string in order for this to work for me, like so:
sys.stdout.write(str(frame.tostring()))
I’m Kind of late, But my powerful VidGear Python Library automates the process of pipelining OpenCV frames into FFmpeg on any platform. Here’s a basic python example:
# import libraries
from vidgear.gears import WriteGear
import cv2
output_params = {"-vcodec":"libx264", "-crf": 0, "-preset": "fast"} #define (Codec,CRF,preset) FFmpeg tweak parameters for writer
stream = cv2.VideoCapture(0) #Open live webcam video stream on first index(i.e. 0) device
writer = WriteGear(output_filename = 'Output.mp4', compression_mode = True, logging = True, **output_params) #Define writer with output filename 'Output.mp4'
# infinite loop
while True:
(grabbed, frame) = stream.read()
# read frames
# check if frame empty
if not is grabbed:
#if True break the infinite loop
break
# {do something with frame here}
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# write a modified frame to writer
writer.write(gray)
# Show output window
cv2.imshow("Output Frame", frame)
key = cv2.waitKey(1) & 0xFF
# check for 'q' key-press
if key == ord("q"):
#if 'q' key-pressed break out
break
cv2.destroyAllWindows()
# close output window
stream.release()
# safely close video stream
writer.close()
# safely close writer
You can check out VidGear Docs for more advanced applications and features.
Hope that helps!
After multiple days of efforts!!!!!!!!!!!
I got to know that if you pass bgr8 OpenCV frames ; Still you need to set
-pix_fmt bgr24 in ffmpeg pipe.
!!!!Bingo now everything will works!!!!
I don’t know why? but yeah its working now
Here’s a fairly straightforward example of reading off a web cam using OpenCV’s python bindings:
'''capture.py'''
import cv, sys
cap = cv.CaptureFromCAM(0) # 0 is for /dev/video0
while True :
if not cv.GrabFrame(cap) : break
frame = cv.RetrieveFrame(cap)
sys.stdout.write( frame.tostring() )
Now I want to pipe the output to ffmpeg as in:
$ python capture.py | ffmpeg -f image2pipe -pix_fmt bgr8 -i - -s 640x480 foo.avi
Sadly, I can’t get the ffmpeg magic incantation quite right and it fails with
libavutil 50.15. 1 / 50.15. 1
libavcodec 52.72. 2 / 52.72. 2
libavformat 52.64. 2 / 52.64. 2
libavdevice 52. 2. 0 / 52. 2. 0
libavfilter 1.19. 0 / 1.19. 0
libswscale 0.11. 0 / 0.11. 0
libpostproc 51. 2. 0 / 51. 2. 0
Output #0, avi, to 'out.avi':
Stream #0.0: Video: flv, yuv420p, 640x480, q=2-31, 19660 kb/s, 90k tbn, 30 tbc
[image2pipe @ 0x1508640]max_analyze_duration reached
[image2pipe @ 0x1508640]Estimating duration from bitrate, this may be inaccurate
Input #0, image2pipe, from 'pipe:':
Duration: N/A, bitrate: N/A
Stream #0.0: Video: 0x0000, bgr8, 25 fps, 25 tbr, 25 tbn, 25 tbc
swScaler: 0x0 -> 640x480 is invalid scaling dimension
- The captured frames are definitely 640×480.
- I’m pretty sure the pixel order for the OpenCV image type (IplImage) is GBR, one byte per channel. At least, that’s what seems to be coming off the camera.
I’m no ffmpeg guru. Has anyone done this successfully?
Took a bunch of fiddling but I figured it out using the FFmpeg rawvideo demuxer:
python capture.py | ffmpeg -f rawvideo -pixel_format bgr24 -video_size 640x480 -framerate 30 -i - foo.avi
Since there is no header in raw video specifying the assumed video parameters, the user must specify them in order to be able to decode the data correctly:
-framerateSet input video frame rate. Default value is 25.-pixel_formatSet the input video pixel format. Default value is yuv420p.-video_sizeSet the input video size. There is no default, so this value must be specified explicitly.
And here’s a little something extra for the power users. Same thing but using VLC to stream the live output to the web, Flash format:
python capture.py | cvlc --demux=rawvideo --rawvid-fps=30 --rawvid-width=320 --rawvid-height=240 --rawvid-chroma=RV24 - --sout "#transcode{vcodec=h264,vb=200,fps=30,width=320,height=240}:std{access=http{mime=video/x-flv},mux=ffmpeg{mux=flv},dst=:8081/stream.flv}"
Edit:
Create a webm stream using ffmpeg and ffserver
python capture.py | ffmpeg -f rawvideo -pixel_format rgb24 -video_size 640x480 -framerate 25 -i - http://localhost:8090/feed1.ffm
Took me an hour to figure out that by default, windows pipes are not binary. This causes some bytes (specifically newlines) to be modified/omitted, and the resulting video is slowly shifting because the frame size is not constant.
To work this around, the modified python file:
"""
videoCapture.py
"""
import cv2, sys
import time
if sys.platform == "win32":
import os, msvcrt
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
cap = cv2.VideoCapture(0) # 0 is for /dev/video0
while True :
ret, frm = cap.read()
sys.stdout.write( frm.tostring() )
To test if piping the raw video is successful, use ffplay. Make sure you specify a higher framerate than what is coming from the pipe, otherwise the video will start to lag
python videoCapture.py | ffplay -f rawvideo -pix_fmt bgr24 -s 640x480 -framerate 40 -i -
Not sure if this is Mac OS-specific, or python3-specific, but I needed to cast the frame to a string in order for this to work for me, like so:
sys.stdout.write(str(frame.tostring()))
I’m Kind of late, But my powerful VidGear Python Library automates the process of pipelining OpenCV frames into FFmpeg on any platform. Here’s a basic python example:
# import libraries
from vidgear.gears import WriteGear
import cv2
output_params = {"-vcodec":"libx264", "-crf": 0, "-preset": "fast"} #define (Codec,CRF,preset) FFmpeg tweak parameters for writer
stream = cv2.VideoCapture(0) #Open live webcam video stream on first index(i.e. 0) device
writer = WriteGear(output_filename = 'Output.mp4', compression_mode = True, logging = True, **output_params) #Define writer with output filename 'Output.mp4'
# infinite loop
while True:
(grabbed, frame) = stream.read()
# read frames
# check if frame empty
if not is grabbed:
#if True break the infinite loop
break
# {do something with frame here}
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
# write a modified frame to writer
writer.write(gray)
# Show output window
cv2.imshow("Output Frame", frame)
key = cv2.waitKey(1) & 0xFF
# check for 'q' key-press
if key == ord("q"):
#if 'q' key-pressed break out
break
cv2.destroyAllWindows()
# close output window
stream.release()
# safely close video stream
writer.close()
# safely close writer
You can check out VidGear Docs for more advanced applications and features.
Hope that helps!
After multiple days of efforts!!!!!!!!!!!
I got to know that if you pass bgr8 OpenCV frames ; Still you need to set
-pix_fmt bgr24 in ffmpeg pipe.
!!!!Bingo now everything will works!!!!
I don’t know why? but yeah its working now