Ideation goes on...

I wasn't very satisfied with the "incarnations" of the physical games like Charades and Taboo using 3DI technology, so I started looking towards a different path. I was still interested in games and consequently tried to come up of ways to enhance conventional gameplay. After some thought, I imagined it would be fun to be able to control your point of view (in first-person games) or your avatar's movement (in third-person view) using body gestures. There are so many games that could be used as testbeds, especially the ones you control a virtual character and are involved ins ome kind of quest (e.g. Warcraft, Diablo, etc.). During my research I "stumbled" on FAAST (Flexible Action and Articulated Skeleton Toolkit), developed in the Institute for Creative Technologies at the University of South California. Their toolkit seems to do pretty well what I was thinking of (an example video is included below; more videos on their website), plus provide an easy way to manipulate skeleton points in Vizard using the VRPN software. Eventually, every cloud has a sliver lining and this implementation relieves me from depending on Microsoft's drivers to use Kinect, a fact that restricts implementation on computers running under Windows 7.

What about some learning?

I then decided to slightly change route and think of something with more educational dimensions, just in case this would prove to be a better source of inspiration. My thoughts were focused on how to make learning more fun and engaging by involving the whole body (besides the mind) and not just the hands using a conventional interaction device like a mouse or even a PS/Wii remote. And the idea emerged! Two users would be able to participate in the learning experience (I am keeping the collaborative aspect of my design): a teacher/parent and a student/child. They would both have to stand in front of a large display, although they could be either co-located or in remote spaces. The teaching/learning experience would involve the teacher throwing things on the student and the student trying to grab them!

Let's assume, for the shake of providing a practical example, that a mother was trying to teach her 5 year-old son the meaning of "skyscraper". She would have pictures or physical models of various artifacts, like buildings, cars, furniture, toys that she would have to move in front of a camera, and the camera would recognize each item and create a virtual object on the fly. Then, by making the gesture of throwing the physical artifact the virtual object would be catapulted towards her son's avatar in the virtual world; her child would have to either move and avoid the item if it didn't describe the projected word (e.g. a car model) or would have to try and reach the item if this was the right one (e.g. a miniature skyscraper). This could also involve the construction of combined words, so in our example the mother could throw -among other miscellaneous items- the picture of a sky and a scraper (see sketch below).

Of course there is a major challenge in this implementation, and it has to do with how to transform a physical object into a digital one. Maybe the simplest way would be to capture an image of the object and apply it as a texture to a generic shape, but the aesthetic result wouldn't be much appealing. Another idea is to have those items already rendered as models in the virtual world, and just use computer vision to compare the object in front of the camera with the virtual model. However this implementation is limited to the number of items that can be used in the educational activities. Finally, I could use the KinectFusion algorithm for 3D local mapping of the objects, but this would make the whole process much more complicated and less of a fun experience for the teacher/parent; additionally, it wouldn't probably work in the case of pictures.

Eureka?

One of the ideas that were spinning in my head (mainly in my dreams... or should I say nightmares?) but I could not put into context, was to use the human silhouette as a portal to another virtual world! I eventually decided to give this idea some more thought while I was awake, since it was too recurring to be ignored. Many great game ideas sparked from this initial one (one of them involving Pong!), but I will describe the prevailing one which derived from distilling those former ones. Although I don't have a partner in this project, thankfully my wife participated willingly in my loud brainstorming sessions during our last night dinner for her birthday! And the innovative design idea which came up and I initially baptize visceral gaming is the following...

There are two players playing in front of different screens; they could be remotely located or for greater social interaction, and eventually enjoyment, they could stand side by side. The world view of the first player will be augmented by a portal into a second world view provided through the figure of the second player and vice versa! The players would have to participate in a quest (e.g. find all the pieces to complete the image of an ancient temple) and could play either collaboratively or competitively. The challenge, and novelty, in the game stems from the fact that they can interact meaningfully only through the portal that is created from their partner's (or opponent's) body figure. This means that they will have to "plant" artifacts in the virtual world that they inhabit, which will entice the other user to approach (i.e., they would be essential for their quest) in order to "enter" or have a view of the virtual world (inhabited by the other person) in which they can actually interact/play to complete their own quest. Sounds complicated, eh?

I believe that the above sketch which illustrates this initial idea might clarify things. A lot of refinement should be made in order to come up with a meaningful game scenario that would allow both players to be immersed in the experience and truly enjoy it. The core challenge in this design is how to implement the portal that would provide an adequate view through someone's silhouette, for the player to interact with this environment-through-keyhole. Maybe the figure should be reshaped into a regular shape (e.g., a circle with a diagonal equal to the distance between the other player's head and toes) or be adapted depending on the other player's body posture. Another concern has to do with the actual interaction with both virtual worlds; should body gesturing be used throughout, or should I use different devices for different world views (e.g., body gestures for the inhabited world and Wii remote for the visceral world)? 

No matter what, I truly believe that this idea has the potential to evolve into a challenging game experience and, eventually, into an impressive demo video with a high coolness* factor!

* The quest-style game definitely contributes to a sense of accomplishment, and even more if you manage to complete the game before your opponent! The social character of the game, especially if the players are co-located, and the fact that you play through some else's body will definitely foster intense connection. Gamers will undoubtedly find the whole experience as providing them with identity: "I am an avid gamer!", and the sensation deriving from the physical movements and actions with immediate impact on both environments will hopefully be high.