Xbox Kinect Applications To Health And Medicine

Chris Niehaus (Washington, DC) —

It’s not quite the technology from Minority Report, but in some ways it’s even better (no gloves!): Some time ago, Microsoft has launched a new entertainment product named Kinect for the popular Xbox 360 gaming platform. The launch involved an unusual amount of fanfare, including a massive Times Square event with dancers, gatherings across the country with actress Minka Kelly (Esquire’s sexiest woman alive, no less), and fanboys blogging about the pros and cons of the new product.

But technologies like Xbox and Kinect, while mainly advertised as fun and games, will also have serious government, health, education, and other “public good” applications in the not-so-distant future.

Natural User Interfaces as a Bridge to the Future

Kinect is amazing by itself,but it is just the most prominent example of an area that Microsoft and other companies are heavily investing in. This topic area,called Natural User Interfaces (NUIs), includes everything from speech recognition to multi-touch surfaces (i.e., more than one person can touch a surface at the same time), and include products like Windows 7 and Surface (the latter of which has been hilariously parodied).

NUI can sound a little vague, or ill-defined, perhaps, spanning many boundaries. It is perhaps best described by a Microsoft researcher named Bill Buxton in a video called “Making User Interfaces Natural” as follows:

It’s not about speech, it’s not about gesture, it’s not even about the phone, and it’s not about human-to-human communication. How these things work together in a natural and seamless way that reduces complexity for the users – that’s what we’re about. Getting these things right opens up another dimension in how we have technology integrated into our lives.

It is with this sentiment that we start to see a vision for devices like Surface and Kinect that goes beyond the trivial, and beyond the purely entertaining, into realms like healthcare and education, those heavily influencing public good and forming the foundation of a robust public life.

Three Intriguing Features of Microsoft Kinect

For better or worse, I’ve been an avid gamer for quite some time (Xbox Live handle = CorbinHood), and I had a Kinect long before it was in public view. Besides seriously impressing my wife, through experimenting with the product I also discovered three nonobvious technological breakthroughs (while unproductively racing cars, spiking volleyballs, or flying in zero gravity). In no particular order:

(1) Kinect gets a lot of attention because of its motion tracking abilities. But its speech recognition, while less discussed, is just as powerful. It can do things like filter out background noises from what’s really important – the user’s voice. That means, for example, that you can blast Kanye West or Taylor Swift in the living room, and Kinect still picks up you saying “pause” to it – without screaming. The secret here is that there’s an incredible amount of “audio engineering” behind the scenes that effectively builds a listening cone around your body, even while you’re moving.

(2) On top of motion detection and speech recognition, Kinect has pretty good facial recognition too. That means, for example, that it can tell different users apart based on facial characteristics, whether as a gaming feature or as an added security feature. The idea that security can be performed initially with an ID and password, and then later using biometrics offers more control over access for both consumers (i.e., keeping kids off adult-rated games on the console) and business/government (i.e., hospital access restricted to certain specific doctors with surgical or other privileges).

(3) Kinect is quite inexpensive, coming in at $150 in the U.S., and as such this highly advanced piece of technology will be immediately available to a wide swath of society. Of course, one can debate whether the price is too high for a gaming console “add on,” and some are, but few would debate that $150 is too high to bring innovative healthcare or educational features into your home.

Healthcare Applications of Kinect Technology

One of the most compelling applications of Kinect technology for public good is in the area of healthcare. Just as an initial example, the game Your Shape: Fitness Evolved (by game publisher Ubisoft) does an excellent job of demonstrating how Kinect can power “home rehabilitation” via facial recognition validating patient identity, and the cameras grading aspects like range-of-motion of a body part. In future instantiations of products like this, doctors and patients could connect through the Xbox Live social network, with rehab courses prescribed, graded, and assessed according to a schedule.

In discussions with researchers, developers, and customers, we see at least four broad areas of health/medicine within which technologies like Kinect can be applied:

(1) Physical therapy and rehabilitation: As described above, your device can verify who you are, measure what you’re doing, and connect you to your doctor(s) in various ways. There are probably positive effects of this technology on medical conditions that we can’t quite envision yet; One early “success story” in this regard involves Kinect and child autism.

(2) Telemedicine: There are many ways in which medicine can be performed remotely using something like Kinect. Features like the ability to interface with other video chat platforms like Windows Live Messenger allow general hands-free communications while walking around a room – or even the outdoors.

(3) Medical training and education: Everyone knows that doctors go through lots of instruction before they become “doctors.” Technology like Kinect can be used not just for remote learning, but also to display virtual human patients that are interacted with through motion-sensing. Virtual teachers can take students through a digital gross anatomy course (no formaldahyde, either).

(4) Neurocogitive and psychological practices: Whether it’s the ability to visualize and analyze brain images taken through Functional Magnetic Resonance Imaging (fMRI), or to perform group psychological therapy with some people with the counselor in person and some remote, there are immense possibilities here.

When the top six health insurers have 100 million members, and when over 120 million people between ages 20 and 65 are overweight, there is a huge opportunity to engage audiences on medical and health issues in a proactive way. Kinect may be an affordable tool for everything from virtual doctors’ visits to injury rehabilitation to general preventative medicine and healthy lifestyle habits. Commerical off-the-shelf software (COTS) for medicine and related fields is suddenly a very real and exciting possibility.

Beyond Requirements: U.S. Army Health and Medicine

One of the biggest researchers on and users of technology in the realm of NUIs is the military. In one project, Microsoft’s Office of Civic Innovation (which I manage) is working with the U.S. Army’s Communications-Electronics Research, Development, and Engineering Center (CERDEC) to bring new and useful interfaces to the warfighter – whether in forward combat areas or in medical facilities like the Walter Reed Army Medical Center.

One of these projects is called COMET, which is a Microsoft Surface-powered multi-touch “command and control” applications, but the possibilities go far beyond that. Our discussions with Army medical personnel and technologies about the four areas above – rehabilitation, telemedicine, training and education, and neurocognitive and psychological treatment – are of great interest for many reasons. Consider that the Army is deployed around the world, trains people under heavy stress for very difficult missions, and deals with an inordinate amount of conditions like post-traumatic stress disorder (PTSD) and traumatic brain injury (TBI).

While it’s very fun to play Kinect games and imagine the possibilities, for public sector organizations like the Army this technology is very serious business. We are working with them to go “beyond requirements” to push the limits of how emerging technologies like NUIs can help them with their very difficult, very human problems.

A Glimpse Ahead: Computers Will Never Be the Same

Wired ran a bold story explaining how technology like Kinect shows where the future of Microsoft – and computing – may lie, once that technology “merges” with what we now know as the PC. It’s smart speculation, but with the recent acquisition of a 3D-gesture company named Canesta, there’s no question that Microsoft (and surely other companies) will be interested in this emerging area of NUIs for some time to come. To paraphrase Bill Buxton, making technology work in a natural and seamless way for users is the ultimate goal.

And this goes beyond health and medicine into other areas of public good like education, where “gaming” is getting a fresh look. Unlike the somewhat lackluster “edutainment” of decades past, NUIs put new approaches in a higher class. For example, the Smithsonian is using NUIs on Surface devices to engage children in learning.

Eric Kloper, Scot Osterweil, and Katie Salen, in their MIT research paper Moving Learning Games Forward: Obstacles, Opportunities, and Openness, write:

They [commerical games] are changing the perception of the nature of video games, making them more accepted in a greater diversity of places. For example, gaming is becoming part of activities now a a regular activities in senior centers, in libraries and museums, as well as within the workplace.”

Technologies like Surface and Kinect that use NUIs are lowering the barriers for engagement among people both young and old, and this will free up new gaming and interaction scenarios that can be fun and productive at the same time.

An interesting blog post called “The Future of Kinect: How Microsoft Plans To Put A Video Game Controller In Everything” digs deeper into where Microsoft’s Kinect team sees the technology going, and how it is but one piece of a broader portfolio of Microsoft innovations. Kinect creative director Kudo Tsunoda comments in the article, “Think about a world where machines understand what people want from them. You can see that extrapolate out to a host of other devices.” The post also highlights theoretical ways in which Microsoft might work with partners like AT&T and Ford to bring these technologies into new spaces in creative ways.

Kinect in your car? The pace of technological advances might be accelerating faster than you think. I leave you with a new video from Microsoft strategy and research lead Craig Mundie about the cutting edge of Kinect and health care.

Chris Niehaus is the Director of the Microsoft Office of Civic Innovation, based in Washington, DC.

Credits: Photo of Kinect health from popculturegeek.com, photo of Microsoft Surface from the album Images for the Future, photo of facial recognition from Moggs Oceanlane, photo of fMRI from ChrisDag, photo of military rehab from the U.S. Army, photo of Canesta poster at SXSW by Nan Palmero. All used under Creative Commons.

Original post