In preparation for submitting some of my thesis work to conferences globally, I put together this overview demonstration video showcasing some of the features of the MobileVCS system I have developed.

Some of the core functionality available in the system thus far include:

  • Native iPad app connecting directly to the Playstation Move.Me server
  • Ability to map the Playstation Move and Navigation Controller buttons to functions within the application
  • 1-to-1 Freespace Realtime control of the Virtual Camera position and orientation through the Playstation Move controller.
  • Panning, Dollying, and Focal Length Modification functionality available through analog and trigger input on the Navigation Controller.
  • On-Screen controls for modifying the Virtual Camera’s aspect ratio.
  • Record and Replay a Virtual Camera shot. Replay controls include standard playback functions as well as the ability to scrub across a time slider to pin-point a specific replay position.
  • Capture the recorded shot and save it directly to the iOS Camera Roll for playback and sharing. 

Looking forward to continuing to iterate and build upon this previsualization tool over the coming months!

Thesis: Virtual Cinematography - February 2012

Another month of progress developing the Mobile Virtual Camera System. With the hardware side of the system developed for testing, it was important to focus on what I can achieve on the software end that would make this form factor functional. What I hope to achieve, at the beginning, is to rebuild the core functionality of freespace motion and record/playback which I achieved separately. The following includes some of my experiments and observations as I begin to build software functionality to the MobileVCS:

Failed Experiments

Building software for the system did prove challenging, especially as I begin converting my projects to the latest Unity4 update. Experimentations began on how I could leverage some of the core iPad functionality such as real-time depth of field on mobile, taking screenshots and saving it to the Photos app, and accessing raw back-facing camera to be applied as a video texture in Unity.

image

I envisioned a functionality within the application where I would be able to mimic the tap-to-focus functionality that is so popular on mobile phone cameras. First I began experimenting in Motion Builder with real-time depth of field with the traditional Virtual Camera System. Even there, the depth of field would work, but would cause significant performance hits when the camera was in motion, understandably so. I was hoping to have better luck in Unity, with my fingers crossed that it was optimized for Mobile. Unity4 included a more robust script for calculating depth of field, something that was enhanced using Direct X11. With it came a binary visualizer where you can see the depth plane viewed from the camera perspective. Building the visualizer into the iPad proved easy to implement and worked flawlessly in real-time, with the ability to dial in the focal plane on the fly. Unfortunately once enabling the full rendered view, much like Motion Builder, caused the framerate of the system to drop greatly with the camera moving in real-time with the Move. That being said, with the visualizer working on mobile in real-time, it is possible to build the two-color visualizer in the app to establish the focal plane for a camera and output the to a program such as Maya for final rendering.

image

Next up, I looked into enabling some core iPad functionality into the app such as access to the back-facing camera as well as taking programmatic screenshots with a push of a GUI button.

The purpose of enabling the back-facing camera was to potentially be able to, in real-time, overlay the 3D world over the real world. This would allow you to physically see how far you need to move in to achieve a particular camera move. While this is possible using Unity’s WebCamTexture, the camera stream proved not optimized for mobile use. I looked into and began testing with another Unity script that did potentially enable access to iOS camera as well as recording/playback functionality. This is something I hope to look into further and potentially implement in a future build of the application.

Taking programmatic screenshots was also another idea that was spawned after discussions with layout artists. By taking screenshots, it allows layout artists the ability to save out their camera blocking and develop simple storyboards. While yes, you can take a screenshot on the iPad by holding the Power & Home buttons simultaneously, I wished to enable is programmatically so I would be able to map it to a button on the Playstation Navigation Controller. Again, gaining access to saving from the app to the iPad proved challenging and something that could be achieved using Prime[31]’s Etcetera plugin.

One of the key reasons that gaining access to hardware functionality of the iPad proved difficult is because it requires some expertise in XCode. XCode handles the compiling of the Unity project in order to build it to iOS devices. Over the course of attempting to develop these experiments, I learned about how to go about accessing different core iOS functions such as notifications, in-app purchases, access to other running apps, etc. Apple, by default, locks this functionality to ensure that apps do not tamper with core functions to enable a smooth, streamlined user experience. While I did not have success with these experiments, I was able to enable saving video to the Photos app in a later experiment (read ahead).

Record & Playback of Camera Takes

With the success I had with EZReplay Manager in the past, I modified the scripts to work again with this latest build of the app. While not as easy as before, I had to ensure that the Replay Manager would be able to record a dynamically moving camera controlled by the Move Framework. This required a lot of cross-talk between the Move and Recording scripts. With this functionality, the Manager was able to playback recorded camera motions from the Move-controlled camera.

Playing back a recording did not seem sufficient enough - I wanted to be able to save a recording for future review. While I am still figuring out a method to output the camera path to another program for further tweaking, I was able to implement the iVidCap script for recording directly to the iOS camera roll in the iPad Photos app. This would enable layout artists the ability to save out and share takes for review with the director. iVidCap, while simple in concept, required some script modification to getting it to work with the dynamically moving Move-controlled camera. In addition, it required some XCode modification, inclusion of additional frameworks, as well as writing a custom script to call iVidCap outside of Unity when recording to the actual hardware device.

image

With all that overcome, both iVidCap and EZReplay Manager work seamlessly with recording and playing back the Playstation Move camera. While recording & playback works as fast as it did in my September build of the app, recording to the iOS camera roll takes about 3-4 minutes for a 30 second camera move because it is rendering each frame in real-time to the device using the iPad’s on-board processing. Like with the real-time depth of field, the performance of this recording function could become faster with the iPad4 (currently I am using the earlier iPad3) with double the processing power of its predecessor.

PlaystationEye Overlay

Another experiment I toyed around with is the ability to see an overlay of my 3D world over the output of the PlaystationEye camera. Thanks to the latest updates in the PSMoveWrapper, I now have access to the real-time stream from the PlaystationEye camera, the device that is tracking the position of my Playstation Move and streaming that position information to the iPad in real-time.

image

With access to the 640x480 IP camera stream, I was able to set up a simple mini-camera setup on the corner of my iPad display where I could see the output of the Move camera appear as a video texture below my 3D objects which appear augmented in the space. While a nifty feature, it allows me to create a simple augmented reality setup where my real-world motions with the Playstation Move-controlled camera can be seen side-by-side from both the birds-eye perspective as well as in first-person perspective. I am still brainstorming on practical applications of such a tool within a previsualization setup, but the functionality is there!

image

——————————————

That is the progress thus far. Stay tuned in the coming week as I release my conference demonstration video that overviews the core functionality of the system including true freespace motion on the iPad as well as a practical demonstration of the recording/playback functionality I developed this week. More to come in the coming month so stay tuned :]

Thesis: Virtual Cinematography - January 2012
Building the Prototype Mobile Virtual Camera System

Quite a bit of progress to update on! Over the past couple weeks, I have been focussing on developing a prototype system as a homogenous unit. This required the following challenges to be solved:

  • Establish that the iPad can directly connect with the Playstation 3 Move.me server using a native iOS app. To facilitate mobile use, include a custom field to enter an IP address as well as auto-detect 
  • A Character Controller needed to be written to allow Playstation Move Navigation Controller to speak with Unity using the edited Move.me Wrapper.
  • Camera Control Functionality needed to be implemented including the ability to both horizontally and vertically pan/dolly/rotate the camera as well as adjust the field of view of the camera lens. As an added bonus, enable the ability to adjust the aspect ratio of the camera viewfinder.

iPad to Move.me Networking

One of the key pillars in developing for this project was the ability to leverage native iOS support in order to exploit key iOS features such as multi-touch gesture support, native keyboard input, and app distribution. After setting up a custom field for IP Address entry and setting up the Unity project to be compatible with iOS development, it was a rather painless process to be able to modify the existing network code to work natively on the iPad. With this, I no longer needed to stream the data to a laptop as an intermediary and am able to have the iPad directly talking to the Move.me server running on the Playstation 3.

image

This means all the position and rotational data from the Move controller as well as the button inputs from the Navigation controller would be sent from the controller, processed on the Playstation 3, and directly network accessed via a remote connection through Move.me on a standalone iPad app. 

Custom Character Controller

Some of the initial challenges I faced was deciding whether the Navigation controller would be a viable source of input for the system. After discussing with industry members on the best construction of the system, they had suggested that the use of tactile controls, such as the controller’s analog sticks and triggers, would act as more precise and familiar mode of input for cinematographers to accurately dial in their camera parameters. With this in mind, I focussed on developing a custom character controller that would pipe in the values I was receiving from the PSMoveWrapper into a custom MoveInputController Script. Some of the challenges I faced in developing this was in passing the appropriate information accurately between Javascript and C#, due to Unity’s unique compiling order. With this new script, I was able to not only output the information received via the Navigation controller input, but also assign it to camera properties, as discussed in the next section.

image

Camera Control Functionality: Pan, Dolly, Field of View, Aspect Ratio

Here comes the real meat of the software development, thus far, and something I will be continuing to build over the coming months. With the Navigation controller script developed and with some of my previous research on iOS app development, I began implementing some of the core camera control functionality required in a Mobile Virtual Camera System. With some of my personal research into Performance Capture and traditional Virtual Camera System capture techniques, accompanied with Glenn Winters and Drexel University’s Optitrack Insight VCS on our Vicon Motion Capture Stage, I decided to establish the core camera tools I would need to navigate around the 3D world for this initial prototype including the ability to pan/dolly the camera laterally along the base surface, crane the camera vertically, and rotate the world to orient the camera in the right direction.

image

With my new control script, I was able to pull the necessary values from each of the analog sticks and controller triggers, allowing me to parse and normalize those values so that they can manipulate the position of the camera in the Unity scene. All camera transformations are stored as an offset to the Main Camera, which allows them to occur simultaneously with the real-time tracking accomplished with the Navigation controller. 

As an added functionality, I created a dialog box on the iPad screen to allow for adjustment of the camera’s crop aspect ratio, with the aid of the AspectRatioEnforcer script. This script, which is applied to the main camera, parented a black crop region to the Main Camera, which is set by default to 1.33 (the iPads default 4:3 aspect ratio). This can obviously be modified to any aspect ratio required while shooting the desired scene, whether it be  Widescreen (1.85), HD (1.77), Classic 35mm (1.5), or Cinemascope (2.35).

Constructing the Mobile VCS

Once I was able to establish the core functionality of this system, I felt that the right next step would be to build a prototype to demonstrate and test some of the camera tools I have developed within Unity. In order to mimic the familiar design of the Virtual Camera System (screen, 2 analog controllers, and tracking markers), I set out to find the necessary clamp which would act as the hub of my system. I found the iKlip Microphone Stand for iPad to be the perfect modular fit to hold the iPad. Instead of a microphone stand, I used .75” copper tubes, L- & T-connectors, duct tape, and plenty of zip ties to pull together a relatively “stable” construction of the system. For some pictures of the system in all its shallow depth-of-field’d glory, view the pictures at the beginning of this post.

While obviously the system has its limitations in construction (and is still being held by duct tape and zip ties…), it serves a solid enough design to continue prototyping and testing the software of my Mobile VCS. Check out the video demonstration, the first of many Thesis Vlogs, overviewing some of the core functionality of the system in action thus far:

In the coming weeks, I will be working to build a simple demo scene that I can begin testing some of the virtual camera principles in action. In addition, I will work to further refine the construction of my system as well as continue to add additional previsualization functionality to the software. All in a days work, until next time :]

Thesis: Virtual Cinematography - Fall 2012 Recap

An overview video highlighting some of the progress I have made thus far with my virtual cinematography tool utilizing the Playstation Move and Apple iPad. Within the past couple months, I had the incredible opportunity to present my work to members of both the film and gaming industries from companies such as Microsoft Game Studios, Halon Entertainment, Firaxis Games, and Carbine Studios.

The next couple months of development is crucial in determining the finer details of the proposed tool for layout artists. I will be working on native mobile tool development and user interface design within Unity3D specifically for the Apple iPad, implementing hardware design techniques in terms of building the entire physical previsualization system, and actually get an opportunity to travel on set of some film shoots to get a greater understanding of the filmmaking process. In addition, I plan to submit and present my research (hopefully) at a couple conferences including SIGGRAPH 2013.

A lot done so far, and lots more to take on. Excited for the road ahead and the challenges it will bring :]

A promo video I created for Drexel’s new Excite Center featuring a game developed by Tim Day and Kevin Gross for physically disabled patients.

Description:
Cerebral palsy (CP) is the most common physical disability of childhood, and children with CP often experience decreased physical activity, motor skills, and functional mobility. Rehabilitation for CP youth is moving more towards activity-based interventions to increase physical activity and fitness. Evidence suggests that virtual reality and gaming may provide opportunities for youth with CP to participate in discrete physical activity tasks and improve motor planning. However, interventions that are fun and provide an opportunity to achieve recommended levels of moderate to vigorous physical activity are not possible in current games. The primary aim of this project is to develop a prototype using the Microsoft Kinect to allow for youth with CP to access games more quickly and to play games for a variable amount of time with no lower or upper limits so that they may improve aerobic capacity, muscular endurance and overall health-related fitness.

Pound for £: An Animated VFX Short

For the past two months, I have been fortunate enough to work along side some talented graduate students, Glenn Winters, Nate Lapinski, and Yujie Zhu, on an ambitious visual effects piece set to be released in December.

The piece is set in late 19th century London in the midst of a financial crisis which led to a revolt on one of the city’s central banks. Due to a new legislation, the bank president, regrettably, has to make the toughest decision in his life of choosing between his career or the people his bank serves. The under-two minute piece with both visually and musically driven and will contain some technical challenges that we are excited to take on including: 1000+ crowd Massive simulation, Particle Effects/Explosion Dynamics in Houdini, Advanced Shading Techniques in Pixar’s Renderman, Character & Crowd Animation captured in Drexel’s new Motion Capture studio, and Camera Shots captured using a Virtual Camera System.

I encourage you to drop by our production blog for the latest updates, screenshots, and videos as we embark on this exploration :]

Fall 2012: Thesis Recap & Moving Forward

Over the past several months I have been hard at works in making significant technical strides on my thesis in attempting to solve the challenge of Virtual Cinematography beyond big studio production environments. Through my exploration, I spent the summer working primarily on the iPad development, exploring the capabilities of the tablet within the game engine Unity. 

Take a look at my presentation overviewing the progress I made in User Interface Design and building a Networking Infrastructure.


User Interface Development

Leveraging third-party packages and tools built within the Unity Mobile infrastructure, I have developed a functional proof-of-concept demonstrating some of the skills I have developed running natively on an iPad. The demo below consists me demonstrating features of the mobile Virtual Camera System including touch regions, touch-drag, pinch-to-zoom, mini-map + tap-to-warp, basic help system, and camera record/playback functionality.


Networking Infrastructure

A key exploration was to demonstrate that it was possible to stream data in real-time from an iPad and a desktop station. I wanted to prove that position information of objects in the environment can be manipulated & transferred across a local network. While there are some limitations on the mobile network transfer data cap, I was able to demonstrate a simple demonstration of object manipulations also running natively on both my iPad and laptop across a private network.


To Infinity and Beyond

The core focus of the upcoming months is to flesh out my thesis proposal for approval by my committee. I am working closely with Professor Paul Diefenbach, Ph.D, head of the Graduate Digital Media & Game Design program here at Drexel University, as my primary thesis advisor. Professor Diefenbach will be a great aspect in helping me push my system and will be a great asset due to his professional expertise in game development and interface design. I look forward to priming my research proposal while also making strides on finalizing & focussing my project design.


Stay tuned to the PRODUCTION tab of my blog (or search for the “thesis” tag) for the latest updates on my thesis development.

Keep Calm and Thesis On :]

Summer 2012: Thesis Update - Week 9: User Interface

Success! I have made significant headway in prototyping the user interface for the iPad virtual camera system. These past couple weeks after SIGGRAPH I have locked down combining certain features available within the Unity Mobile and FigureGestures frameworks (as well as some of my own custom code hackery) to create a workable user interface for a virtual camera. I have successfully built a standalone app to an iPad (seen above) and implemented the following UI features:

Multi-Finger Gesture Input and Visual Feedback

The app will now display a visual cursor to mark when it recognizes a touch input from the user. This can later be changed to a joystick, slider, etc. but is a visual confirmation that the app is definitely reading input not only on the touch pads, but anywhere on the screen. 

Read More

Summer 2012: Thesis Update - Week 5

I have begun experimenting with both touch interface design and building a basic networking example. 

Touch Interface:

There are some great open-source resources available for iOS developers through the Unity game engine. I have begun experimenting with Unity’s built in Standard Assets for Mobile as well as two gesture packages, TUIO and FingerGestures. Within Unity’s standard assets, I was able to set up a simple touch region for the FPS controller as well as parenting the camera to an object for movement. This separates the camera attributes from the player control attributes. In addition, I used the free iOS app Unity Remote 3 to quickly prototype and iterate on touch screen designs. 

The TUIO and FingerGestures frameworks are excellent resources to get off the ground running with enabling touch to objects in your Unity scene. TUIO has some great resources for one-finger actions and Finger Gestures enables multi-touch swiping and gesture commands. I was able to utilize some of the built in scripts to create a simple touch-drag function onto a spherical object within the scene in an XY plane. I still need to tweak some of the code to enable touch-drag on all axes. It was also exciting to see that these packages have support for some camera control scripts (orbit, zoom, pan, etc), which will be of great help in the development of my thesis.

Over the coming week, I want to explore developing touch menu systems within Unity as well as exploring more touch script functionality for camera control.

Networking Example:

I have spent some time looking into Unity’s networking architecture for both desktop and mobile. Before building for mobile, I wanted to run through the Unity Multiplayer Tutorial which has some great example code to build a basic networked game. Once again the goal of this is to be able to stream content directly to the iPad so that it can run on it’s native processor as opposed to using a desktop streaming app. I was able to get up an running a standalone desktop build running on two computers where blocks are able to be spawned and streamed between the two computers.

The next step is to begin testing this for mobile. I will be attempting to provision and build this example using Xcode and seeing if I can spawn these cubes wirelessly on the tablet. Some challenges that I expect to face are some bugs when I begin to implement the touch-class into this example. In addition, if I am able to get a standalone build on the iPad, I plan to implement the touch-drag script onto each spawned cube so that I can manipulate their position in real-time. Hopefully going through more of the iPhone multiplayer tutorial will aid in this exploration as well as checking out some of the backend code on some of Unity’s freely available multiplayer open-source games. 

Summer 2012: Thesis Update - Advanced Interaction - Week 3

Over the past week I was tasked with collecting resources that facilitated in the production of my Unity-iPad networking infrastructure. Here is a list of links featuring tutorials, videos, and documents that will help in developing this technique:

iPhone Multiplayer Demo (Executable + Documentation) - 
http://unity3d.com/support/resources/tutorials/iphone-multiplayer-tutorial

Unity Networking Multiplayer Example (Executable + Documentation) - 
http://unity3d.com/support/resources/example-projects/networking-example

Unity Networked Multiplayer Resource (Documentation) - 
http://docs.unity3d.com/Documentation/Manual/NetworkedMultiplayer.html

Optimizing for Mobile with Unity (Sample + Video+ Documentation) - 
http://blogs.unity3d.com/2012/03/23/shadowgun-optimizing-for-mobile-sample-level/
http://video.unity3d.com/video/3709506/unite-11-shadowgun-rendering
http://forum.unity3d.com/threads/130163-ShadowGun-Optimising-for-Mobile
http://www.quickfingers.net/optimizing-unity-games-for-the-ipad/
http://maniacdev.com/2010/02/unity-iphone-developer-first-game/ 
http://docs.unity3d.com/Documentation/Manual/iphone-basic.html

Input for iOS (Documentation) -
http://docs.unity3d.com/Documentation/Manual/Input.html 
 

Misc Unity iPad/iPhone Development (Videos) -
http://www.youtube.com/watch?v=GUQChcWVFt8 (Networking)
http://www.youtube.com/watch?v=BDwiR29ODA0 (iPhone Game)
http://www.youtube.com/watch?v=O6kqi9uWef0 (iPad Game) 

Over the coming weeks I plan on utilizing these tutorials asa well ass the TUIO and FingerGestures touch-input frameworks to begin development in my Unity-iPad networking bridge.

Summer 2012: Thesis Update - Advanced Interaction

Through the summer course in Advanced Interaction, I plan to explore methods of piping in one software or hardware technology into another using some scripting tools (and a little black magic). Some of the projects I initially proposed for this project include a dynamic Unity to Maya connection, Unity to iPad networking solution, and Playstation Move to Maya coding framework. Here is my assessment on the feasibility of the aforementioned projects:

Unity to Maya

Goal: Develop a connection between Unity Player and Maya to transfer animated assets “live”.

Rationale: Maya is the industry standard software for animation and since this VCS is designed as a film tool, it would be ideal to be able to transfer assets recorded and animated in Unity with my VCS setup to Maya with ease. In addition, this cross communication will enable assets created in Maya to be auto-updated within Unity prior to using the VCS.

References:
Unity-Kinect MoCap Data to Maya - https://www.youtube.com/watch?v=p8N20UmDHpI
Complete Maya-Unity Workflow - https://vimeo.com/10968593

Unity to iPad

Goal: Develop a network bridge between a Unity standalone app and an native iPad app built from Unity to stream “live” visuals to the tablet.

Rationale: The latest iPad hardware is capable of significant processing power and graphic intensive applications. That being said, it is best to develop a standalone desktop app that can stream visuals to the tablet to lower the processing burden on the device. This will allow even large and computationally intensive scenes to be displayed on the iPad without sacrificing performance.

References:
iPhone Multiplayer Demo - http://unity3d.com/support/resources/tutorials/iphone-multiplayer-tutorial
Unity Networking Multiplayer Example - http://unity3d.com/support/resources/example-projects/networking-example
Unity Networked Multiplayer Resource - http://docs.unity3d.com/Documentation/Manual/NetworkedMultiplayer.html

Playstation Move to Maya

Goal: Develop a connection between Playstation Move and Maya to allow for information to be streamed and read with the desktop software.

Rationale: As of now, I am using Unity 3D for all of my development because it has a strong community, ample production capability, and CMU has developed a solid SDK for the Playstation Move for me to build off of. That being said, my end goal is to have this VCS work with Maya and a direct connection to the animation application through the Move.Me server would be ideal for future development.

Resources:
Lightwave with Move Integration - http://www.youtube.com/watch?v=pE-9LwZz7Ww&t=3m54s
PSMove 3D tracked in Maya - http://community.us.playstation.com/t5/Move-me/Is-there-a-way-to-bring-PSMove-s-3D-tracked-information-into/m-p/37209735/message-uid/37209735#U37209735

Verdict:

After discussion realistic feasibility of the three projects described, I have decided to pursue the Unity 3D to iPad Networking solution for the following reasons -

  • Networking is absolution essential to the development of the VCS
  • This project would work well in tandem of my development in the VCS User Interface (which is being developed for the New Media Project course)
  • Lots of official and community resources in iPad networking solutions with Unity

Presentation:

Summer 2012: Thesis Recap

Virtual Cinematography: Beyond Big Studio Production

Between my New Media Project course and my Advanced Interaction course this summer, I plan to make significant headway on my thesis development and further build upon the progress I have made in the past 6 months.

Abstract:
With inexpensive and accessible hardware such as multi-touch tablets and new video game controllers, filmmakers no longer require large motion capture studios to visually plan their 3D worlds. Utilizing the game engine Unity, the Playstation 3 Move controller, and an Apple iPad, I propose to design a pre-visualization solution for a 3D virtual camera system that enables full touch and analog freedom for the filmmaker to control a digital camera and assets within the virtual world. 

Where I am at in development:
Utilizing the Playstation Move-Unity3D framework provided by Xun Zhang from Carnegie Melon University coupled with Sony’s Move.Me server application, I have developed a prototype of the Virtual Camera System allowing users to freely pan, rotate, and tilt the virtual camera in 3D space.

Demo - version 01:

Thesis project featured in “RePlay Lab: Game Design at Drexel” promotional video at 0:58

Stay tuned to the PRODUCTION tab of my blog (or search for the “thesis” tag) for the latest updates on my thesis development.

Keep Calm and Thesis On :]

Check out this Drexel University RePlay Lab: Game Design at Drexel promo video featuring a glimpse of what I am hard at work on for my Master’s thesis :]

http://www.youtube.com/watch?v=6Tx_lEZ5VsM @ 1:10ish

Source: youtube.com

Top