cv2.VideoWriter issues

Question:

I’m looking to record a Twitch Livestream by feeding it the direct livestream url using streamlink.streams(url) (which returns a .m3u8 url). With this, I have no problem reading the stream and even writing a few images from it, but when it comes to writing it as a video, I get errors.

P.S.: Yes, I know there’s other options like Streamlink and yt-dwl, but I want to operate solely in python, not using CLI… which I believe those two are only dealing with (for recording).

Here’s what I currently have:

if streamlink.streams(url):
    stream = streamlink.streams(url)['best']
    stream = str(stream).split(', ')
    stream = stream[1].strip("'")
    cap = cv2.VideoCapture(stream)
    gst_out = "appsrc ! video/x-raw, format=BGR ! queue ! nvvidconv ! omxh264enc ! h264parse ! qtmux ! filesink location=stream "
    out = cv2.VideoWriter(gst_out, cv2.VideoWriter_fourcc(*'mp4v'), 30, (1920, 1080))
    while True:
        _, frame = cap.read()
        out.write(frame)

For this code, I get this error msg:

[tls @ 0x1278a74f0] Error in the pull function.

And if I remove gst_out and feed stream instead as well as moving cap and out into the while loop like so:

if streamlink.streams(url):
    stream = streamlink.streams(url)['best']
    stream = str(stream).split(', ')
    stream = stream[1].strip("'")
    while True:
        cap = cv2.VideoCapture(stream)
        _, frame = cap.read()
        out = cv2.VideoWriter(stream, cv2.VideoWriter_fourcc(*'mp4v'), 30, (1920, 1080))
        out.write(frame)

I get:

OpenCV: FFMPEG: tag 0x7634706d/'mp4v' is not supported with codec id 12 and format 'hls / Apple HTTP Live Streaming'

What am I missing here?

Asked By: c0nfluks

||

Answers:

The fist part uses GStreamer syntax, and OpenCV for Python is most likely not built with GStreamer.
The answer is going to be focused on the second part (also because I don’t know GStreamer so well).

There are several issues:

  • cap = cv2.VideoCapture(stream) should be before the while True loop.
  • out = cv2.VideoWriter(stream, cv2.VideoWriter_fourcc(*'mp4v'), 30, (1920, 1080)) should be before the while True loop.
  • The first argument of cv2.VideoWriter should be MP4 file name, and not stream.
  • For getting a valid output file, we have to execute out.release() after the loop, but the loop may never end.

  • It is recommended to get frame size and rate of the input video, and set VideoWriter accordingly:

     width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
     height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
     fps = int(cap.get(cv2.CAP_PROP_FPS))
    
     video_file_name = 'output.mp4'
    
     out = cv2.VideoWriter(video_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (width, height))  # Open video file for writing
    
  • It is recommended to break the loop if ret is False:

     ret, frame = cap.read()
    
     if not ret:
         break
    
  • One option to end the recording is when user press Esc key.
    Break the loop if cv2.waitKey(1) == 27.
    cv2.waitKey(1) is going to work only after executing cv2.imshow.
    A simple solution is executing cv2.imshow every 30 frames (for example).

     if (frame_counter % 30 == 0):
         cv2.imshow('frame', frame)  # Show frame every 30 frames (for testing)
    
     if cv2.waitKey(1) == 27:  # Press Esc for stop recording (cv2.waitKey is going to work only when cv2.imshow is used).
         break
    

Complete code sample:

from streamlink import Streamlink
import cv2

def stream_to_url(url, quality='best'):
    session = Streamlink()
    streams = session.streams(url)

    if streams:
        return streams[quality].to_url()
    else:
        raise ValueError('Could not locate your stream.')


url = 'https://www.twitch.tv/noraexplorer'  # Need to login to twitch.tv first (using the browser)...
quality='best'

stream_url = stream_to_url(url, quality)  # Get the video URL
cap = cv2.VideoCapture(stream_url, cv2.CAP_FFMPEG)  # Open video stream for capturing

# Get frame size and rate of the input video
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = int(cap.get(cv2.CAP_PROP_FPS))


video_file_name = 'output.mp4'

out = cv2.VideoWriter(video_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (width, height))  # Open video file for writing


frame_counter = 0
while True:
    ret, frame = cap.read()
    
    if not ret:
        break

    if (frame_counter % 30 == 0):
        cv2.imshow('frame', frame)  # Show frame every 30 frames (for testing)

    out.write(frame)  # Write frame to output.mp4

    if cv2.waitKey(1) == 27:  # Press Esc for stop recording (cv2.waitKey is going to work only when cv2.imshow is used).
        break

    frame_counter += 1

cap.release()
out.release()
cv2.destroyAllWindows()

Testing the setup using FFplay and subprocess module:

from streamlink import Streamlink
import subprocess

def stream_to_url(url, quality='best'):
    session = Streamlink()
    streams = session.streams(url)

    if streams:
        return streams[quality].to_url()
    else:
        raise ValueError('Could not locate your stream.')


#url = 'https://www.twitch.tv/noraexplorer'  # Need to login to twitch.tv first (using the browser)...
url = 'https://www.twitch.tv/valorant'
quality='best'

stream_url = stream_to_url(url, quality)  # Get the video URL

subprocess.run(['ffplay', stream_url])

Update:

Using ffmpeg-python for reading the video, and OpenCV for recording the video:

In cases where cv2.VideoCapture is not working, we may use FFmpeg CLI as sub-process.
ffmpeg-python module is Python binding for FFmpeg CLI.
Using ffmpeg-python is almost like using subprocess module, it used here mainly for simplifying the usage of FFprobe.


Using FFprobe for getting video frames resolution and framerate (without using OpenCV):

p = ffmpeg.probe(stream_url, select_streams='v');
width = p['streams'][0]['width']
height = p['streams'][0]['height']
r_frame_rate = p['streams'][0]['r_frame_rate']  # May return 60000/1001

if '/' in r_frame_rate:
    fps = float(r_frame_rate.split("/")[0]) / float(r_frame_rate.split("/")[1])  # Convert from 60000/1001 to 59.94
elif r_frame_rate != '0':
    fps = float(r_frame_rate)
else:
    fps = 30  # Used as default

Getting the framerate may be a bit of a challenge…

Note: ffprobe CLI should be in the execution path.


Start FFmpeg sub-process with stdout as pipe:

ffmpeg_process = (
    ffmpeg
    .input(stream_url)
    .video
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
    .run_async(pipe_stdout=True)
)

Note: ffmpeg CLI should be in the execution path.


Reading a frame from the pipe, and convert it from bytes to NumPy array:

in_bytes = ffmpeg_process.stdout.read(width*height*3)
frame = np.frombuffer(in_bytes, np.uint8).reshape([height, width, 3])

Closing FFmpeg sub-process:
Closing stdout pipe ends FFmpeg (with "broken pipe" error).

ffmpeg_process.stdout.close()
ffmpeg_process.wait()  # Wait for the sub-process to finish

Complete code sample:

from streamlink import Streamlink
import cv2
import numpy as np
import ffmpeg

def stream_to_url(url, quality='best'):
    session = Streamlink()
    streams = session.streams(url)

    if streams:
        return streams[quality].to_url()
    else:
        raise ValueError('Could not locate your stream.')


#url = 'https://www.twitch.tv/noraexplorer'  # Need to login to twitch.tv first (using the browser)...
url = 'https://www.twitch.tv/valorant'
quality='best'

stream_url = stream_to_url(url, quality)  # Get the video URL

#subprocess.run(['ffplay', stream_url])  # Use FFplay for testing

# Use FFprobe to get video frames resolution and framerate.
################################################################################
p = ffmpeg.probe(stream_url, select_streams='v');
width = p['streams'][0]['width']
height = p['streams'][0]['height']
r_frame_rate = p['streams'][0]['r_frame_rate']  # May return 60000/1001

if '/' in r_frame_rate:
    fps = float(r_frame_rate.split("/")[0]) / float(r_frame_rate.split("/")[1])  # Convert from 60000/1001 to 59.94
elif r_frame_rate != '0':
    fps = float(r_frame_rate)
else:
    fps = 30  # Used as default

#cap = cv2.VideoCapture(stream_url, cv2.CAP_FFMPEG)  # Open video stream for capturing
#width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
#height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
#fps = int(cap.get(cv2.CAP_PROP_FPS))
################################################################################


# Use FFmpeg sub-process instead of using cv2.VideoCapture
################################################################################
ffmpeg_process = (
    ffmpeg
    .input(stream_url, an=None)  # an=None applies -an argument (used for ignoring the input audio - it is not required, just more elegant).
    .video
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
    .run_async(pipe_stdout=True)
)
################################################################################


video_file_name = 'output.mp4'

out = cv2.VideoWriter(video_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (width, height))  # Open video file for writing


frame_counter = 0
while True:
    #ret, frame = cap.read()    
    in_bytes = ffmpeg_process.stdout.read(width*height*3)  # Read raw video frame from stdout as bytes array.
    
    if len(in_bytes) < width*height*3:  #if not ret:
        break

    frame = np.frombuffer(in_bytes, np.uint8).reshape([height, width, 3])  # Convert bytes array to NumPy array.

    if (frame_counter % 30 == 0):
        cv2.imshow('frame', frame)  # Show frame every 30 frames (for testing)

    out.write(frame)  # Write frame to output.mp4

    if cv2.waitKey(1) == 27:  # Press Esc for stop recording (cv2.waitKey is going to work only when cv2.imshow is used).
        break

    frame_counter += 1

#cap.release()
ffmpeg_process.stdout.close()  # Close stdout pipe (it also closes FFmpeg).
out.release()
cv2.destroyAllWindows()
ffmpeg_process.wait()  # Wait for the sub-process to finish

Note:
In case you care about the quality of the recorded video, using cv2.VideoWriter is not the best choice…

Answered By: Rotem