Recently we released a game for one of our clients, Red Bull. The game feature two awesome breakdancers which you control. Hitting the markers at the beat of the music gives you performance and accuracy points.
The biggest challenge we faced during the R&D was to play video of the dancers whilst allowing the player to interact. Currently, there is no straightforward way of doing this on iPhone devices. Any playing
video tag will automatically startup in fullscreen using the built in media player. This is where our little hack came useful.
Through extensive testing and research we managed to extract image-data from a
video-tag. Setting the
currentTime property will stealthily move the playhead of the video and make that frame available for rendering onto a
canvas. Realizing this we created our own custom player capable of extracting 25 frames a second, move to defined clips and perform loops. All on the “incompatible” iPhone.
Video compression played a huge role on the outcome. Usually videos enter keyframes around each second of video material. This also means that seeking in videos often snap to those keyframes. We did however discover that our video-material got even smaller in file-size setting the keyframes to every frame and allowed us to seek freely.
Performance is a pain and video buffers doesn’t make it any easier. Video-elements only automatically buffer a certain amount of data. Jumping back and forth in a video shifts the buffers around while playing resulting in lag and dropped frames. This wasn’t good enough for gaming. Solution; Blobs. Loading the complete video into a Blob, from which the video-player can read from, removed the delays making video access instant. Now, this can be solved using several different techniques, for instance Application Cache, but we came to the conclusion that Blobs and trust in the regular browser cache would be sufficient and optimal in terms of performance.