Building a Media Player #4: DASH Manifests and an Offline Prototype

PAUL LEWIS: Hello again, folks Now let’s talk about media And I think maybe two episodes ago I explained about DASH and MSE and HLS– right So I’ve been making another prototype In fact, to be honest with you, I’m quite a way on with the build But I was away at BlinkOn in the States last week, and so the diaries are lagging behind reality a little bit But we’ll come onto that in a while What I want to do is I want to talk, as I say, about media I talked about DASH and MSE and HLS And I’ve been making a prototype using the Shaka Player So let’s talk about that first of all Why would I use Shaka Player? Because if you’ve got MSE and you’re swapping between different quality versions of your video, you have to do things like tracking the bandwidth You have to figure out where you can switch and get the next segment of your video and all these kinds of things And it’s actually a tremendous amount of work that I don’t think is worth doing when you can get the Shaka Player for 43 kilobytes gzipped Now it feels to me like that’s really small for what is actually doing on your behalf You should actually have a look through all the docs and everything In fact we’ll link to it in the show notes about the Shaka Player and where you can get it and have a look at the demo and all these kind of things There’s loads It’s really quite small for what it is And I felt like, you know what, actually it’s got loads of unit tests and it’s just– it’s properly done So that’s what I’ve been using– right So with that in place, how actually would we use the Shaka Player? That’s what has been on my mind and that’s what I’ve been working on So let’s have a look at what’s on my screen So if I’ve got a bunch of FFmpeg commands I should explain what I’m going to try and do I’m going to start with a video I’ve been starting with one video that’s running at HD And then what I wanted to do was make different quality versions of this So I’m using FFmpeg You see on screen I’ve got the FFmpeg commands Which is just– I mean, they’re sort of reverse engineered from looking around at documentation and kind of other ways of doing things I just chose like a 720, a 540, and a 360 But let me just show you this There is a page on the YouTube help that covers live encoder settings, but nonetheless, encoder settings, bitrates, and resolutions So for example, they’re saying if you going to do 1080p video, you probably want it 1,920 by 1,080– big surprise But bitrate would be 3,000 to 6,000 bits per second So were I doing this for real and not just kind of in placeholder mode– sort of like oh, let’s just do a few versions, just see how this works– I would probably be following something like this to kind of go, right, that’s what a 1080p looks like That’s what 720p should have, 480, and so on all the way down to the ones that I care about the most So, I’m just running FFmpeg, just making a bunch of these, like three different variants of my video The next thing I’m using is a thing called the Shaka Packager And what the Shaka Packager does– and it is quite a mouthful to say, Shaka Packer It feels like it just has a nice little, “taka, taka, ta.” I really like it I’m going to say it one time for you Shaka Packager, there you are What it does is it takes the videos that you’ve got, all the different variations, like the 720p, the 540, 360, and you give it a profile, so whether it’s live streaming or video on demand I’m doing on demand, not even thought about live streaming You tell it where you want it to output the DASH manifest, because that’s what it’s going to do for you It can take in all the video and so on, and it’s going to split things out to video tracks, audio tracks, different qualities, blah, blah, blah And you tell it how much, for example, you want it to buffer, which in my case I’m saying three seconds would be good Just get me three seconds up front Further down the line, there will probably be some dancing that I need to do there around prebuffering video content so that, for example, we can give you a flying start with video But I haven’t really got my head around that space yet Like, how do we get the first five seconds of the video primed by a service worker so that we can just play instantly? Haven’t really thought about that And in the DASH manifesto you can just say, hey you know what, when you first load this video, just try and get the first three seconds off the network That’s one thing And then segment duration that you see here is, as I understand it, is how often you can switch qualities So I’m saying every three seconds, if you need to, you can switch qualities This to the Shaka Player through the DASH manifest, you can switch every three seconds What’s important about that is when you reverse back up and look at the video calls, you see here that there’s this keyint, min-keyint, no-scenecut What we need to do when we take our original video and we turn it into the different quality versions, we actually have to put these things called iframes– not iframes like we would–

they’re different iframes They’re iframes but they’re not iframes, you know what I mean? They’re just– they’re video frames They’re keyframes That’s what they are They’re keyframes So we need a keyframe So in my case I’m doing it every 24 frames, which means, because it’s a 24 frames per second video, I’m getting a keyframe every second, right? So every time we hit one of these keyframes is a savepoint at which we could switch to another quality So the segment duration should, if I understand this correctly, be a multiple of the keyframes So if I’ve got a keyframe every second, I could switch every two seconds, four seconds, five seconds, as long as it’s a multiple of what my keyframe value is Does that make sense? I hope so But I have a segment duration of three seconds That’s what I’m saying What I also do, as you’ll see here, is I actually do three more packages, one for the 720, one for the 540, one from the 360, because I was just interested in whether I might need separate manifests for the offline stuff, which I’m going to come onto shortly– right Woo So, taken the video, turned it into three different quality versions It’s the same production you’d do– you’d go through that YouTube help and just kind of be, like, OK, I need like five or 10 different versions and so on And you probably just set up a pipeline for processing videos And then we’re going to package them up into a DASH manifest or have a DASH manifest which points to all these things Let me show you one of those looks like– sample-manifest-full, that’s the one So this is what the Shaka Packager outputs for us And you can see that there’s– I mean, there’s loads in here, and it’s a bunch of XML But here’s the minimum buffer time, which is that three seconds that I set How long the video is– so the media presentation duration is 71 seconds in this case And then inside here, you’ll see that there’s these things, these adaptation sets and these representations But you’ll see what’s crucially important here is that you’ve got a representation for the 640 by 360, 960 by 540, and 1,280 by 720 So you can start to see that it’s listing out all the different qualities of video that you can actually use for playback, which tallies entirely with what I passed to the packager in the first instance And it has really interesting things like the amount of bandwidth that that particular video quality is going to need, the codecs it’s using, a bunch of other stuff as well And to be honest there’s a bunch of stuff in here I don’t fully understand, like the initialization range Not quite sure what that is It’s quite exciting I’m sure you’ll agree Maybe not I don’t know And then further down, we also have audio configurations as well So again, we’ve got 360, 540, and 720 audio I have a feeling that in reality, you probably only just want like one audio track, something that’s like an MP3 or an AAC or– probably the video quality, I reckon, is going to change a lot more and be the one that’s the more bandwidth sensitive than the audio, is my guess So you might just be like, well, it’s going to take one audio track, and I’ll have loads of different qualities of video But I don’t know, I’m still figuring a lot of this stuff out for myself Right, so let’s take a look at some of the code to actually take that DASH manifest and put it into use with the Shaka Player So this is the code, which is not particularly neat I’ll give you the heads up It is very much prototype code But hopefully we can just kind of pick out the bits that make sense for the particular context Here we go So we have this initPlayer function that I’ve got here And what we do is we wrap a video element in the shaka.Player It kind of makes a wraparound and starts kind of giving us additional things that we can do But we can also talk to the video directly if we need to and tell to dot play and seek and all those kind of– like dot current time and all those things that you normally do with a video So it’s kind of nice actually It’s a really nice way of working that you just kind of still predominantly can just listen to events from the video player But when you’re actually setting it up and doing playback, the Shaka Player stuff kicks in and switches all the different qualities for you It’s really cool and I actually really like it So, the thing we do is we give it the manifest So the manifest that we created with the Shaka Packager we can hand to the player, the Shaka Player And a lot of the API is promise-based So we call it with the URL for the manifest And then when it’s loaded, it loads the manifest and the first little bit of video across all the qualities listed So it kind of gets the first few seconds of the 720 and the 540 and the 360, whatever versions you go Does that, and then it fires the callback for the– fires the dot then of the promise that says,

OK, you’re good to go So when you get that– now you can see I actually have a bit more code going on here, which we’ll talk about in a minute, because I’ve been trying to get it to work offline as well Been successful in that, at least partially successful So we’ll talk about that as well But, you can then start calling dot play, and that will work So let me just show you what that looks like So– [VIDEO PLAYBACK] – Hello, I don’t know– PAUL LEWIS: Don’t talk Paul So as you can see, on my screen, there is the video And actually if I go to the Network tab, you can see it’s requesting– there’s a moment in time it’s requesting the 360 video, 540, 720 It’s decided that my bandwidth, because it’s coming from localhost, my bandwidth can support this 720, and off it goes [END PLAYBACK] PAUL LEWIS: Let me show you this as well This is quite interesting It makes what we call range requests So the request itself only has a request and therefore only has a request No, the request has a range in there, which is rather than saying give me all the video at once, it’s saying, I want this byte range to that byte range And the Shaka Player figures out that range request itself based on, I guess, the bandwidth and how big the segments are And it goes, well, I guess I need that number bytes next to make that next chunk of the video So it makes this range request There are other ways to do it You can, as I understand it, instead of having one big MP4 file or whatever video file you’ve got, WebM or whatever, you can also do it in chunks and actually have individual files And that’s also a way to do it I’ve gone for one big file because I think I’m going to have to pass one big video file across for chromecasting later on So it just made sense for me I was just like, I’m just going to have one big video and make little chunks, range requests off that And as I say, that’s working So we make one of these range requests And the server– the response from the server is– it’s over here, and it looks pretty tiny But it’s basically just those– you can see the size of it I mean, the original file is much bigger than that But you can see that you just get, say, 477k of the video or however much of the video it thinks it needs So that all works And you can see as well that the bar here in the controls only fills up with the amount it’s buffered So all good That’s working just fine Now let me show you what actually happens if we try it again But this time, I’m going to throttle the network connection back down so regular 3G So probably, as a guess, it will start asking for the 720 Yeah, you see it start asking for the 720 video But soon enough I reckon it’s going to drop down and ask for the 360 when it figures out that actually the bandwidth isn’t so good And again, this is what Shaka Player is going to do for you So I don’t have to do that, which is brilliant, because it’s code I don’t want to write Now you can see the requests are for the 360 video and the 360 audio That’s what it’s decided I actually want– cool So that’s kind of like the DASH side completely sorted Let’s have a quick chat about the HLS as well In my script further down, I take those 720, 540, and 360 videos, and I call this media file segmenter, which is something you can get downloaded from the Apple Developer Portal And what that does is it will slice and dice the different videos into a particular formats for you This M3U8, you can this is like a playlist file If we actually go and have a look– let’s have a look, 720.m3u8 I don’t really know what I’m looking at other than I can see that this is like three seconds and it tells you which sequence of files And you see it makes all these files in the process of running media file segmenter So that does that bit And then after that, I move some stuff around But then you call variantplaylistcreator, which is another one of these tools that comes with the Apple downloads that you get And you pass it– the three playlists in my case, but all the playlists that you’ve made, and it makes one master playlist which looks like this And that is essentially, I guess, roughly equivalent to the DASH manifest I mean, on the technical side it won’t be But for Android and desktop Chrome and desktop Safari and whatever, you can just use MSE, and that will just work So you’ve got the DASH manifest for those But for iOS Safari today, you do need to create one of these M3U8 It’s the equivalent, I guess, of the M3U8 On the code side, if, for whatever reason, we get an error from loading the manifest– which should happen,

I believe on iOS, because it will go oh, I tried to load the DASH manifest and I tried to do all that stuff and, it’s just not working, I have a fallback that just says, OK, switch across to HLS, which is where you get that M3U8 file and you just say video.source equals M3U8 So it’s actually very, very straightforward to use, very convenient to use And iOS Safari will just kind of go, oh sure, and it’ll start adapting and getting the right things for you So you don’t you have to do an awful lot with HLS other than point your video element at the M3U8 playlist file I think the upside of DASH is that Shaka Player can manage the bandwidth and you get to have a little bit more of a say on what bandwidth you think it should have and which track you think it should lock into and all those kinds of things, if you want to With HLS, it’s much more like the user-agent just kind of goes, I’ve got less And if it decides to download the entire video, there isn’t much, as the developer, that you can say other than, sure So it is worth knowing that that’s what’s going to happen I mean there’s nothing you can do about it particularly The only way I’ve found actually to at least tweak it a little bit is to say on the video element that you do preload equals metadata If you don’t set the preload attribute, it will go off and download all of the content, which may or may not be what you want So there you go– right I’m aware that I’ve taken up quite a bit of your time, but I still want to show you the offline stuff So I hope you stick around I hope you’re enjoying this Let’s have a quick look at what it does, what we require for the offline stuff to work So at the moment everything’s just going off to the network And the idea was, what if we could prefetch everything into the cache– into a cache, so that when the request goes out for that part of the MP4 file, say in the DASH version, that the service worker could be like, oh yeah, I got this This is already in the cache I’ll just create that I’ll handle that range request I’ll get the item out of the cache, get that bit of the file, and then just send it back So it’s kind of transparent as far as Shaka is concerned We just kind of go, just get that video and it will go off and do it Enough chatter Let’s look a little bit of– hang on, let me get my words right Let’s look at some code There we are You wouldn’t have thought it was so difficult to say, but apparently, for me, yes Here we are Service worker– this is my service worker as it is today And this is really the interesting bit So imagine that this fetch is called on every time a request is made from the page OK, cool What we do is we do a match against the caches And if there’s a match, we’ll handle it If not we’ll just do like a pass through So what we need to do is figure out, well, at what situation would we call the caches.match and get something back? And the answer is, when there is something in the caches So if we went off and got the video and audio and everything else that we’d made and put those into the cache, then we’d get a cache hit One thing I discovered– and I was very grateful to Jake, Jake Archibald If you don’t know him, you should follow him on Twitter and all that And he was helping me figure out how I should actually handle the range response Remember I said that when we make the request, it’s a range request We only ask for a certain byte range When we get a cache match, we get a cache match for the whole thing So if we just say, well, return whatever the cache has, rather than returning just those bytes, you’d actually return the entire video, which is not what was actually asked for We were only asked for bytes from, say, zero to a thousand, for example So we actually have to do that ourselves So what I do is I’m taking Jake’s code here And what we do is we create a ranged response So that takes in the request And we find it from the cache But what we do is we call dot array buffer on it And that gets us the entire video We’ll come back to that in a moment And then what we have is some code that looks through the range header and goes, OK, which bytes did they actually want? And then that gets sliced out from that array buffer We set the status to 206, which I think is something like partial content, or partial response, or something along those lines that says, you’re not getting the whole thing You’re just getting that little bit that you asked for with the range request And we set a couple of other headers like the content-length and the content-range And then we return that instead For me, this is a bit unpleasant, not only because there’s a bunch of code in here that could error, I guess, could go wrong, but mostly because calling dot array buffer at the moment on the response would get me the whole video So if it’s like a one gig video, calling dot array buffer will get me a one gig array buffer in memory for every request And the solution here, I think, is

going to be to slice and dice the video Instead of one big file in the cache, actually have lots of different files, like smaller chunks inside the cache And then just pull out the chunks that we need to make the ranged response We’ll see how that goes when we actually get to production and how big the videos are and whether it’s actually a thing But I fully expect to actually have to do that I should also mention, by the way, that there is an offline implementation already in Shaka which uses IndexedDB to do its thing So that takes care of it But it doesn’t use a service worker today It’s all in the foreground It populates IndexedDB But it also does all the bits and pieces like if you’ve got [INAUDIBLE] licenses– which again, I don’t understand fully– it will store those for you as well– right Let me just show you, though what it actually looks like when you do this So when I hit cache video, hopefully– oh, it’s set to the regular 3G Let’s do no throttling Oh if only life was actually like that where you just go no throttling, phew You see now we’ve got the cached video So if I just clear that and start running it, now all the requests are coming from the service worker Essentially it’s doing its job The service worker is stepping in, and rather than going out to the network, it’s actually creating the response from the cache So that’s a way to transparently provide the video to the Shaka Player So all this code– in fact the whole site code at the moment, let me show you this, is up on GitHub You can get it at media pwa We will link to that in the notes below If you want to have a play around with the prototype, I’ll add a prototypes folder to this so you can at least see, roughly speaking, what I’ve been doing and go from there In the meanwhile, have fun I will be back very, very soon with an exploration of what I’ve been doing on the server side Catch you later [MUSIC PLAYING] Hey folks, thanks for watching Don’t forget that there is more content that you can find kind of over here-ish And if you want to subscribe, there’s probably a button, well, maybe there, maybe somewhere around there Click that if you’ve not done that Brilliant [MUSIC PLAYING]