IBM's New Image Recognition-Based Search
We've all seen photos of ourselves in locations we can't quite remember. Often they're from exotic travels or from days long past. Regardless of the reason for your memory loss, IBM is working on a tool that can help. In collaboration with the European Union consortium, the company is testing SAPIR (Search in Audio-Visual Content Using Peer-to-peer Information Retrieval). The image matching search technology allows users to pull results from large collections of audio-visual content without using tags for search. Instead, users can upload images and match them to similar ones - perhaps even ones with signage and labels. The system analyzes everything from digital photographs, to sound files to video. From here it automatically indexes and ranks the media for retrieval.
A recent IDC white paper reports, "The digital universe is messy...95% of the data in the digital universe is unstructured, meaning its intrinsic meaning cannot be easily divined by simple computer programs. There are ways to imply meaning to unstructured data, and the semantic web project is promising to develop the tools to help us do that in the future."
Two such "divining" projects include CoPhIR (Content-based Photo Image Retrieval) Test-Collection and IBM's MUFIN (Multi-Feature Indexing Network). These projects tie into SAPIR's back end by extracting data from the Flickr archive and indexing features such as scalable color, color structure, color layout, shape edges and texture.
As shown in the video of Madrid's Plaza de España, SAPIR identifies matching media in the same way that humans derive intrinsic value from visual and sensory clues. Users can also choose to combine search terms with additional text to further drill down in search results. As is the case with regular search, if you already know the city where your image was taken, you're one step closer to finding your result. Additionally, SAPIR also has the ability to index sound and video files.
While the catalogue of media is still very limited, theoretically we may one day be able to search for almost anything using this technique. If Ashton Kutcher wears a pair of sunglasses we like, we can scan the image and search for the storefronts stocking them. If we're looking for the name of a town square, we can find it in the tags of similar images. And finally, if we're looking to self-diagnose we can compare photos of ourselves against jaundice or malaria patients.
The advantage of this tool is that we may one day have a chance to collect up the disparate bits in the digital ether and identify them as useful points of information. To test SAPIR in its early research stage, visit the homepage. You may also want to test out MUFIN to compare results.
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
Now just integrate with Google Street View...technology is getting scary...
Im impressed so early in the process, I did a test search for "tree" and selected a image that i liked and clicked "visually similar" and it showed some amazing similar photos with same lighting conditions, angle and even branch/leaf type.
While these are great they also need to be effective at the ability to tag false positives easily. This type of search is very helpful for media/agency/students who may need to sort through thousands of images to find the best one.
looking forward to playing with this type of search in the future.
Posted by: thelostagency.wordpress.com
|
September 10, 2009 5:16 PM
On a somewhat related note:
You might want to take a look at the work of LTU Technologies. They've been around for years.
http://www.ltutech.com/
and their Corbis visual search demo.
http://www.ltutech.com/en/demos/corbis-ltu-visual-search
Here's an interview with the CEO about the company:
http://www.digitalmediabuzz.com/2009/07/image-search-technology/
From the Article:
“Our technology examines the pixel content of images, the different shapes, the structure, the texture, the colors, the arrangements,” Winter says. “We encode that into a bit of binary code that we call the image DNA. That image DNA is sort of a mid-level description of the image. We use that data to compare images and classify them and track them. We can actually compare image DNA pretty easily.”
You can also compare with http://mipai.esuli.it/ it works on the same amount of images using only one desktop computer.
Saw this one on CeBIT this year and spoke to the team behind it.
As far as I understand, the main point of the research is not image matching but the peer-to-peer distributed search index.
If I remember correct one of the examples they've showed was to do a music search, for example.
Great tool. With SAPIR you can find out where you are, without GPS! And it based on MPEG-7.
Take a look at your ReadWriteWeb article about "MOBVIS", too.
"The MOBVIS technology could be used to inform visitors about the objects in an area be them buildings or landmarks."
http://www.readwriteweb.com/archives/hyperlinking_the_real_world.php
“Our technology examines the pixel content of images, the different shapes, the structure, the texture, the colors, the arrangements,” Winter says. “We encode that into a bit of binary code that we call the image DNA. That image DNA is sort of a mid-level description of the image. We use that data to compare images and classify them and track them. We can actually compare image DNA pretty easily.”