Recently we released a game for one of our clients, Red Bull. The game feature two awesome breakdancers which you control. Hitting the markers at the beat of the music gives you performance and accuracy points.
The biggest challenge we faced during the R&D was to play video of the dancers whilst allowing the player to interact. Currently, there is no straightforward way of doing this on iPhone devices. Any playing video tag will automatically startup in fullscreen using the built in media player. This is where our little hack came useful.
Through extensive testing and research we managed to extract image-data from a video-tag. Setting the currentTime property will stealthily move the playhead of the video and make that frame available for rendering onto a canvas. Realizing this we created our own custom player capable of extracting 25 frames a second, move to defined clips and perform loops. All on the “incompatible” iPhone.
See the Pen 6ec5f3c83c52ee78d1d4eaa2007a2299 by Carl Calderon (@carlcalderon) on CodePen.
Video compression played a huge role on the outcome. Usually videos enter keyframes around each second of video material. This also means that seeking in videos often snap to those keyframes. We did however discover that our video-material got even smaller in file-size setting the keyframes to every frame and allowed us to seek freely.
Performance is a pain and video buffers doesn’t make it any easier. Video-elements only automatically buffer a certain amount of data. Jumping back and forth in a video shifts the buffers around while playing resulting in lag and dropped frames. This wasn’t good enough for gaming. Solution; Blobs. Loading the complete video into a Blob, from which the video-player can read from, removed the delays making video access instant. Now, this can be solved using several different techniques, for instance Application Cache, but we came to the conclusion that Blobs and trust in the regular browser cache would be sufficient and optimal in terms of performance.
/Carl
Comments
2 responses to “Inline video in Mobile Safari on iOS”
Interesting. Which phone and video resolution was used to get 24 fps playback?
In our tests this approach performs very badly with fullhd resolution and older phones. We get only 15 fps on iPhone 6, but the 6S can achieve up to 30 fps.
Hi Mikko!
We got the most performance boost from using Blobs but we did not render FullHD in the end. We settled for 540×960 resolution since it rendered good enough for our purpose. The threshold we set was iPhone 5S with satisfactory FPS. The game relies heavily on jump-cuts so dropped frames in between the cuts was not that big of an issue.
/Carl