Devs Hack iPhone API for True Augmented Reality
An international team of computer scientists has created software that lets anyone perform on-the-fly analysis of live streaming video on the iPhone. Used alongside existing methods of displaying data on top of the camera's view, this new functionality signals a fundamental change in the kinds of Augmented Reality (AR) that iPhone developers can create. Existing AR apps, like Yelp, Layar, Wikitude and others display data on top of a camera's view but don't actually analyze what the camera sees. This new development changes that.
The iPhone has a private API for analysis of live-streaming video but developers' requests that it be made accessible haven't been granted by Apple. The new software opening up access to that API was made freely available to anyone this morning by the team that built it.
The Visual Media Lab at Ben Gurion University in collaboration with HIT Lab NZ wrote the code in question and unveiled it along with video demonstrations at the AR-specialist blog Games Alfresco today. The unveiling comes just days before the International Symposium on Mixed and Augmented Reality in Orlando, Florida.
In a demonstration video the team showed how software built on top of the now-exposed API could look at a 2D image drawn on paper and render the image in 3D. Then the 3D rendering is subjected to a physics simulation.
This is of course just one use-case. Video AR-enabled software could do almost anything in direct response to the actual images seen through the iPhone's camera view, in real time. Image processing locally will be easier and faster than comparison with a large number of related images, something that would likely require some connection to the cloud, but these are early days.
GamesAlfresco author Ori Inbar calls this the dawn of an era of "user-generated Augmented Reality."
For the first time ever, the core code necessary for real augmented reality ("real" here means precise alignment of graphics overlaid on real life objects) on iPhone 3.0 is available to the public
How will Apple respond? That's a big question; the company has had an ambivalent relationship with the emerging field of Augmented Reality so far and exercises infamously obtuse control over applications distributed through its app store.
For now the code is being distributed for its creators by Ori Inbar, whose email address to request it is available at the conclusion of his coverage on GamesAlfresco.
The possibilities here are huge. While location-based AR is clumsy at best so far, due to the imprecise nature of GPS and mapping data, these kinds of object-centric AR tied to the actual viewed world open up a whole new world of potential developments. Let's see what you've got, AR devs of the world!
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
This is incredible! This is a HUGE step forward in making AR more mainstream! I am also impressed with the coverage that RWW is conducting with regards to AR. Keep up the great work everyone!
Hopefully it'll lead to this. "ARhrrrr" looks WAY cool.
http://www.youtube.com/watch?v=cNu4CluFOcw
I wrote an article (part of one) defining what is AR. In it I talk about how AR MUST learn to look through the lens and understand what it sees to be truly successful. After seeing this post, I decided to share it out.
http://www.perivision.net/wordpress/?p=2154
I need to edit it a bit more, but all comments welcome.
How will Apple respond?
http://www.youtube.com/watch?v=nY2dBfYqyaY
Marshall, this is a really exciting development. Way cool. (I signed in with Facebook this time. What's the URL field for?)
Why does it flip the analysis/image?
Exciting stuff. I too was wondering about the image being flipped (rotated about x and y it seems). Why?
I'm a bit confused by this article, since it seems to be conflating two things (ori's article was a bit confused too).
The video you link does not appear to have been done on an iPhone (clues: the video quality is too good, and you can see it running on the laptop beside the paper during the later half of the video). The project is neat, but not as technically ground breaking as you all seem to imply, if it's being done with a stationary camera on a desktop (moving cameras are much harder: when we had sketch recognition in Art of Defense, on the N95, it was very hard to deal with). Look at the old Sony tech demos, and the videos for EyePet.
As for the video API, I'd like to see how it compares to the one we're using; I wonder if they've actually managed to hack into the camera API directly (like a bunch of us did with 2.0) or if they're just doing the slow, painful screen scraping (like we and others are doing with 3.0).
They'll probably respond with more and more PC vs Mac commercials. Is anyone else finding that they're actually starting to work and that you're questioning your sanity for owning a PC?
In all seriousness, I really look forward to all the AR stuff. My gf's pink IPhone will surely appreciate it.
Frank
Real Estate Investing FAQ
The first thing I thought of when reading the article was an app that overlays costumes, masks, hats, etc over people and even changes the background whenever you're taking a picture or video of them. I think that kind of thing would really appeal to iPhone users. Anything to add a little fun to all the pictures people take and post to Facebook.
The first I had with AR was that Best Buy ad where you had to use your webcam. Hopefully this not only pushes the technology forward but makes it more mainstream.
This looks amazing, I'm sure after seeing this, Apple will have to make something similar :D
Nice one Marshall!
Some more building blocks get in place for AR. Sweet. TIm
what is he writing i can't see
apple is going down the hill...
I think apple has a really bright future. They are about to launch new products.