For the record, I have no official knowledge of what the Xbox team is doing with Natal. I have no communicated advance/inside information about Microsoft’s plans in the Entertainment & Devices space. Now – that being said, this is what I personally believe is the secret killer app of Xbox 360’s Project Natal… and for whatever reason, few people in the media seem to have put two and two together about this stuff. Maybe it’s all conjecture however…
UPDATE June 17th, 2010:
WHOOPS! Microsoft General Manager Wil Mozell confirmed what I called Project Natal’s Kinect’s Secret Killer App.
Wil Mozell, a Microsoft GM who oversees many of the companies designing Kinect’s important launch titles:
"Kinect isn’t going to replace the controllers that have worked for those types of games for the last decade—that’s not what we’re trying to do. Kinect will work alongside those controllers for hardcore games. For throwing a grenade, for vocal commands, for…""For head tracking??"
"Yes, head tracking! Exactly."
He gets a big smile. He wants to say more. Bound by Microsoft confidentiality agreements, he can’t.
[taken from Gizmodo’s article, “Xbox 360 Kinect Puts ‘Play’ Back in ‘Gameplay’”
————————–
THE KILLER APP IS…
Take a look at this video by Carnegie Mellon University PhD Researcher, Johnny Chung Lee. Some of you may have seen this before because it’s frankly 2 years old and leverages a Windows laptop-controlled Nintendo Wii controller & sensor.
In case it wasn’t clear, the killer app is 3D experiences using head tracking – without headgear. There’s a few things to note about Johnny Chung Lee’s demonstration:
- NO SPECIAL TV… NO SPECIAL GLASSES
No new equipment –like a 3D TV or HDMI 1.4 middlware – needs to be purchased in the way of A/V consumer electronics. Today’s projection screen, LCS flat screens & plasma displays are all capable of providing this great 3D-like experience. More importantly, no special 3D glasses need to be worn to get this experience. This has vast implications because 3D glasses can cause eye fatigue in people that can wear them, and requires GOGGLES for people that wear eyeglasses, something that’s usually more expensive.
- VIEW BY HEAD POSITION… NOT BY GAME CONTROLLER
No interaction is necessary with a game controller to obtain different views in 3D. All that is necessary is for the game console to be able to tell the distance (Z-axis) and position (X & Y-axis) of the user’s head relative to the TV screen to provide a video image that is relative to these variables. (i.e. provide a different “angle”)
- CONTROLLER AND HEAD TRACKING SIMULTANEOUS USAGE
Just because your view/perspective in 3D is controlled by your head position, doesn’t mean you can’t ALSO use the controller to control your aim & movement as it normally does when gaming. The ability for head tracking to change your relative position provides the ability to ‘duck & weave’ interactively making the 3D experience truly immersive.
So with that being said, here’s a few things to know.
- JOHNNY CHUNG LEE WORKS FOR MICROSOFT
Johnny Lee is the guy you saw in the video. We hired him as a researcher for Microsoft Applied Sciences. Excitingly, he’s been working for Microsoft for a while. After the interface work he did with the Wii, I think he proved that he’s got a unique & valuable insight into how concepts like head tracking and other natural methods of interfacing with computers can be useful & possibly revolutionary.http://johnnylee.net/
http://research.microsoft.com/apps/video/default.aspx?id=103806 - LEE IS WORKING ON XBOX 360’s PROJECT NATAL
Not surprisingly, Johnny Chung Lee’s been working on Project Natal. Anyone paying attention to his past work as well as his comments on his blog about the work he’s currently doing should be able to spot where Natal & his VR head tracking research have commonalities. In the post below, he talks about the power of both the hardware & the software that underpins Natal. Here’s a quick except of what he has to say:”The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.
But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I’ve seen in academic research, never mind a consumer product. At times, working on this project has felt like a miniature “Manhattan project” with developers and researchers from around the world coming together to make this happen.”
http://news.cnet.com/8301-13860_3-10254466-56.html?tag=rtcol;pop
http://procrastineering.blogspot.com/2009/06/project-natal.html
http://www.bruceongames.com/2009/06/05/johnny-chung-lee-on-xbox-natal/ - PROJECT NATAL ACCURATELY RECOGNIZES HEAD POSITION & DISTANCE IN REAL TIME
First of all, let’s clear up a common misconception of the media: "Project Natal” is NOTHING LIKE the Playstation Eye Toy. The media continues to use them both in the same breath despite the fact that Eye toy doesn’t have half the technology within Natal. Sure they’re both video cameras… but that’s where the similarity ENDS.One of the many core functions* of “Project Natal” is the ability to recognize objects – such as a player’s head – in real time, and do this using it’s own built-in CPU/processor. It does not depend on the Xbox 360 for the computation of this information, allowing the primary CPU cores to be used for gaming software. By providing this as user input, another form of gamer control is added in addition to the game controller.
This has been demonstrated over and over again. What makes this amazing is that Natal itself not just recognizes a person’s head, but does it is real time and can tell the relative position of your head relative to the TV.
http://www.newscientist.com/article/mg20527426.800-microsofts-bodysensing-buttonbusting-controller.html
http://archive.videogamesdaily.com/features/microsoft-natal-xbox-360-why-its-a-big-deal-p1.asp
http://www.popsci.com/gadgets/article/2010-01/exclusive-inside-microsofts-project-natal
I’ve just got on last thought to leave you with:
Imagine playing Halo: Reach. A group of Elites & Brutes are guarding an area just around the corner. You move your Spartan up to the corner and lean with your head to see around the corner. You spot the enemies and fire off a few well placed shots using your controller right trigger before Elites start returning fire. You lean back with your head to stay out of the line of sight and wait for the gunfire to die down… but now you can’t actually see what’s going on around the corner – just as you would in a real firefight. You notice the ricochets have diminished so you again lean around the corner and throw a grenade using your left trigger and lean back. Then to change things up, you tippy-toe over the wall to see a view of the enemy that they don’t expect and aren’t watching for. You see that only one Brute remains. This should be easy to clean up…
—————————————————-
* Project Natal is very much more than just a camera. It has software & a CPU built into it that enables it to do a variety of things without the assistance of the Xbox console’s CPU. It can judge distances between objects & the display, recognize speech for voice commands, distinguish between player’s faces, visually recognize player’s movements, identify people in the dark through infrared, interpret player gestures, etc. For details around what Natal does outside of just “provide an image of the gamer”, take a look at these links:
kurtsh
Kurt's Thoughts 