Playing a video with captions in Jupyter notebook

Question:

How to play a video with captions in Jupyter notebook?

With code snippets from these post, I’ve tried to play a video inside jupyter notebook:

from IPython.display import HTML

# Show video
compressed_path = 'team-rocket.video-compressed.mp4'
mp4 = open(compressed_path,'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()


HTML("""
<video width=400 controls>
      <source src="%s" type="video/mp4">
</video>
""" % data_url)

[out]:

enter image description here

When I tried to add a .vtt file as caption, the option appears

from IPython.display import HTML

# Show video
compressed_path = 'team-rocket.video-compressed.mp4'
mp4 = open(compressed_path,'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()


HTML("""
<video width=400 controls>
      <source src="%s" type="video/mp4">
      <track src="team-rocket.vtt" label="English" kind="captions" srclang="en" default >
</video>
""" % data_url)

[out]:

enter image description here

But the subtitles/captions are not present when playing the video. How to play a video with captions in Jupyter notebook?

The files used in the examples above are on:

Asked By: alvas

||
Source

Answers:

It is a combination of security issues. Modern browsers forbid using local data (local subtitles file in this case) and Colab prevents you rendering embedded videos through "insecure" http protocol. So in order to display a video with subtitles you will need to create an iframe and embed there a video that is served through https (both video and subtitles).

You can test this by creating a cell with the following content:

%%html
<iframe width="800" height="500" src="https://iandevlin.com/html5/webvtt-example.html"
frameborder="0" allow="encrypted-media;" allowfullscreen />

So as I see it you will need to create a self signed certificate, add it to your browser for it to become legit, run a TLS web server in a thread to not block your notebook, create a small HTML file with the <video> tag which (the HTML file) your web server will serve over HTTPS and its URL will be the source of the <iframe> in the notebook. Whoa, this is really complicated.

If we try to load an full HTML into an iframe that has the video + track assets stored non-locally, these works:

%%html
<iframe width="800" height="500" src="https://iandevlin.com/html5/webvtt-example.html"
frameborder="0" allow="encrypted-media;" allowfullscreen />

or

HTML("""<iframe width="800" height="500" src="https://iandevlin.com/html5/webvtt-example.html"
frameborder="0" allow="encrypted-media;" allowfullscreen /> """)

[out]:

enter image description here

But when you try to take the HTML from https://iandevlin.com/html5/webvtt-example.html and load it into the iframe through srcdoc as follows, it fails to load the captions/subtitles:

%%html
<iframe width="800" height="500" 
srcdoc='
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset=utf-8>
    <title>WebVTT Video Subtitles Example for Chrome</title>
    <meta name="keywords" content="webvtt,webvtt example,chrome,html5" />
    <meta name="description" content="Sample HTML5 code that shows how Chrome supports WebVTT subtitles" />
    <style>
        body { 
            padding:10px;
            font-size:80%;
            font-family:verdana, serif;
        }
        figure { margin:30px 0; }
    </style>
</head>
<body>
    <header>
        <h1>WebVTT Video Subtitles Example for Chrome</h1>
        <p><a href="http://www.iandevlin.com/blog/2012/06/html5/google-chrome-supports-webvtt-subtitles">Read the full article</a></p>
        <p>To view this in Google Chrome, you must first enable the <code>track</code> element:</p>
        <ol>
            <li>Go to the configuration tools in Chrome by typing <code>chrome:flags</code> in the address bar</li>
            <li>Search for "Enable the &lt;track&gt; element" and activate it</li>
            <li>Restart the browser</li>
        </ol>
    </header>
    <figure>
        <video controls preload="metadata" poster="http://html5multimedia.com/media/elephants-dream.title.jpg" width="422" height="253">
            <source src="http://html5multimedia.com/media/elephants-dream-medium.webm" type="video/webm">
            <track label="English subtitles" kind="subtitles" srclang="en" src="https://iandevlin.com/html5/english-subtitles.vtt" default>
        </video>
        <figcaption>
            <small><a href="http://www.elephantsdream.org">Elephant's Dream</a> &ndash; &copy; copyright 2006, Blender Foundation / Netherlands Media Art Institute</small>
        </figcaption>
    </figure>
</body>
</html>' />

And taking a closer look at the console following @atalay-k’s comment, you will see:

Unsafe attempt to load URL https://iandevlin.com/html5/english-subtitles.vtt from frame with URL about:srcdoc. Domains, protocols and ports must match.

which looks like a pesky issue from Security error loading subtitles on HTML video

Answered By: igrinis

Try this:

from IPython.display import HTML
from base64 import b64encode

video_path = 'team-rocket.video-compressed.mp4'
captions_path = 'team-rocket.vtt'

with open(video_path, 'rb') as f:
    video_data = f.read()
    video_base64 = b64encode(video_data).decode()

with open(captions_path, 'r') as f:
    captions_data = f.read()
    captions_base64 = b64encode(captions_data.encode('utf-8')).decode()

video_html = f"""
<video width="640" height="360" controls>
    <source src="data:video/mp4;base64,{video_base64}" type="video/mp4">
    <track src="data:text/vtt;base64,{captions_base64}" kind="captions" srclang="en" label="English" default>
</video>
"""

HTML(video_html)

For some reason, explicitly streaming in the video + captions/subtitle while specifying the encoding would bypass the security issues that @igrinis’ answer pointed out.

  • "data:video/mp4;base64,{video_base64}"
  • "data:text/vtt;base64,{captions_base64}"
Answered By: Mazhar