My Adventure Book

Thesis: Virtual Cinematography - February 2012

Another month of progress developing the Mobile Virtual Camera System. With the hardware side of the system developed for testing, it was important to focus on what I can achieve on the software end that would make this form factor functional. What I hope to achieve, at the beginning, is to rebuild the core functionality of freespace motion and record/playback which I achieved separately. The following includes some of my experiments and observations as I begin to build software functionality to the MobileVCS:

Failed Experiments

Building software for the system did prove challenging, especially as I begin converting my projects to the latest Unity4 update. Experimentations began on how I could leverage some of the core iPad functionality such as real-time depth of field on mobile, taking screenshots and saving it to the Photos app, and accessing raw back-facing camera to be applied as a video texture in Unity.

I envisioned a functionality within the application where I would be able to mimic the tap-to-focus functionality that is so popular on mobile phone cameras. First I began experimenting in Motion Builder with real-time depth of field with the traditional Virtual Camera System. Even there, the depth of field would work, but would cause significant performance hits when the camera was in motion, understandably so. I was hoping to have better luck in Unity, with my fingers crossed that it was optimized for Mobile. Unity4 included a more robust script for calculating depth of field, something that was enhanced using Direct X11. With it came a binary visualizer where you can see the depth plane viewed from the camera perspective. Building the visualizer into the iPad proved easy to implement and worked flawlessly in real-time, with the ability to dial in the focal plane on the fly. Unfortunately once enabling the full rendered view, much like Motion Builder, caused the framerate of the system to drop greatly with the camera moving in real-time with the Move. That being said, with the visualizer working on mobile in real-time, it is possible to build the two-color visualizer in the app to establish the focal plane for a camera and output the to a program such as Maya for final rendering.

Next up, I looked into enabling some core iPad functionality into the app such as access to the back-facing camera as well as taking programmatic screenshots with a push of a GUI button.

The purpose of enabling the back-facing camera was to potentially be able to, in real-time, overlay the 3D world over the real world. This would allow you to physically see how far you need to move in to achieve a particular camera move. While this is possible using Unity’s WebCamTexture, the camera stream proved not optimized for mobile use. I looked into and began testing with another Unity script that did potentially enable access to iOS camera as well as recording/playback functionality. This is something I hope to look into further and potentially implement in a future build of the application.

Taking programmatic screenshots was also another idea that was spawned after discussions with layout artists. By taking screenshots, it allows layout artists the ability to save out their camera blocking and develop simple storyboards. While yes, you can take a screenshot on the iPad by holding the Power & Home buttons simultaneously, I wished to enable is programmatically so I would be able to map it to a button on the Playstation Navigation Controller. Again, gaining access to saving from the app to the iPad proved challenging and something that could be achieved using Prime[31]’s Etcetera plugin.

One of the key reasons that gaining access to hardware functionality of the iPad proved difficult is because it requires some expertise in XCode. XCode handles the compiling of the Unity project in order to build it to iOS devices. Over the course of attempting to develop these experiments, I learned about how to go about accessing different core iOS functions such as notifications, in-app purchases, access to other running apps, etc. Apple, by default, locks this functionality to ensure that apps do not tamper with core functions to enable a smooth, streamlined user experience. While I did not have success with these experiments, I was able to enable saving video to the Photos app in a later experiment (read ahead).

Record & Playback of Camera Takes

With the success I had with EZReplay Manager in the past, I modified the scripts to work again with this latest build of the app. While not as easy as before, I had to ensure that the Replay Manager would be able to record a dynamically moving camera controlled by the Move Framework. This required a lot of cross-talk between the Move and Recording scripts. With this functionality, the Manager was able to playback recorded camera motions from the Move-controlled camera.

Playing back a recording did not seem sufficient enough - I wanted to be able to save a recording for future review. While I am still figuring out a method to output the camera path to another program for further tweaking, I was able to implement the iVidCap script for recording directly to the iOS camera roll in the iPad Photos app. This would enable layout artists the ability to save out and share takes for review with the director. iVidCap, while simple in concept, required some script modification to getting it to work with the dynamically moving Move-controlled camera. In addition, it required some XCode modification, inclusion of additional frameworks, as well as writing a custom script to call iVidCap outside of Unity when recording to the actual hardware device.

With all that overcome, both iVidCap and EZReplay Manager work seamlessly with recording and playing back the Playstation Move camera. While recording & playback works as fast as it did in my September build of the app, recording to the iOS camera roll takes about 3-4 minutes for a 30 second camera move because it is rendering each frame in real-time to the device using the iPad’s on-board processing. Like with the real-time depth of field, the performance of this recording function could become faster with the iPad4 (currently I am using the earlier iPad3) with double the processing power of its predecessor.

PlaystationEye Overlay

Another experiment I toyed around with is the ability to see an overlay of my 3D world over the output of the PlaystationEye camera. Thanks to the latest updates in the PSMoveWrapper, I now have access to the real-time stream from the PlaystationEye camera, the device that is tracking the position of my Playstation Move and streaming that position information to the iPad in real-time.

With access to the 640x480 IP camera stream, I was able to set up a simple mini-camera setup on the corner of my iPad display where I could see the output of the Move camera appear as a video texture below my 3D objects which appear augmented in the space. While a nifty feature, it allows me to create a simple augmented reality setup where my real-world motions with the Playstation Move-controlled camera can be seen side-by-side from both the birds-eye perspective as well as in first-person perspective. I am still brainstorming on practical applications of such a tool within a previsualization setup, but the functionality is there!

——————————————

That is the progress thus far. Stay tuned in the coming week as I release my conference demonstration video that overviews the core functionality of the system including true freespace motion on the iPad as well as a practical demonstration of the recording/playback functionality I developed this week. More to come in the coming month so stay tuned :]