This morning someone emailed me and asked me to send them a headshot for my bio. I went to Google image search and typed in alex iskold. The results that I got were a mix of photos of me, my diagrams, as well as photos and pictures of people and objects I had never seen before.
It was not surprising that Google Image search did not know what I look like, but it is curious how and why images got labeled with my name. It was tempting to experiment. This post is the result of my short image ego-surfing adventure, around the image search engines.
Most image search engines today classify images based on text - not on the content of the image. This means they do not run pattern recognition algorithms over images, but instead rely on the text that is part of the image label; i.e. tags as well as image surroundings.
So it is not reasonable to expect that an image search engine would be able to understand that I am searching for a picture of a person. But still, our expectations are that search results are correct and also consistent.
This is a actually a big problem with the text-based approach to searching for rich media. There are ways to improve the matching by being more conservative, by presenting results differently, and by letting the users tag the images - or even vote with their clicks! Let's first take a look at my image ego-searching to see how various engines do. Then we'll discuss ways that text-based image search can evolve.
Here are some top results when searching for alex iskold on Google Image Search:
So there are certainly matches, but there are also a lot of mismatches. The two headshots that came up of different people (the two in the left column) is confusing. Someone searching for me would have no idea which one is the correct one - now that is a bad false positive! Another problem, admittedly a much smaller one, is that it pulled out the picture of Mike Arrington at the TechCrunch NY party. But I suppose that's flattering, so I can live with that :-).
The first problem that the Google algorithm ran into was came from Amazon profile pages. In the picture below, you can see that a picture of an individual called Alexander Arvei and my name appear together - on a profile page of a third Amazon user. Curiously, there are multiple images of Alexander, multiple mentions of my name and also the string Alex is found inside the string Alexander. So it seems that a combination of all these things threw Google off.
The second match also came from an Amazon profile, confusing me with a person named Allan Heydon. The reason for this was much less clear, as there was only a single occurrence of his photo and my name on this Amazon profile page.
Next I investigated why I got confused with Michael Arrington. The culprit was this post on bub.licio.us about the recent TechCrunch party in New York. In this post there are two pictures - one above the other - with my name in between. So the results are wrong because the pictures are not clearly labeled.
Google Image Search generated much worse results when I tried to tweak the search query and control the images selected. For example, when I selected the Small Images Only option, none of the results returned had my picture. There were two RSS buttons, what appeared to be a naked body part, two book covers from my Amazon lists, as well as two pictures of Bijan Sabet - who simply commented in his blog on one of my R/WW posts. The same thing happened when I searched for "alex iskold" in quotes - all my pictures went away! Desperate, I decided to try other search engines.
In a nutshell, other search engines I tested had the same mixed results - although less of them. Microsoft Live Search returned only 22 results:
Yahoo! Search returned only 9 results. To its credit, the first one was right. But it also returned a picture of a high school girl and a soccer player.
But the search results that really made my day came from Snap:
Alex Ferguson, the manager of Manchester United!
A lot of image searches are likely to be people searches. So a natural question is: can a text-based search engine do a better job than we get today? It seems like it has a long way to go before we get good results. So what can be done to make this type of image search better? To start with, it would be helpful to tell the engine we are looking for a person - this can be easily done with a simple checkbox. Next, it would be really helpful if the results were ordered by proximity to the picture. Just because my name is on the page, does not mean that all pictures on that page should be displayed in the results!
And since we live in the social web era, it would make sense to let people correct the results. It seems that Google is already doing this with clicks. But it would be great if it would let us tag the images right there as well. Feeding peoples corrections back into the search engine ranking would make a big impact on the overall results. And people would do it, because it's likely you want your picture to come up when someone searches for your name.
Finally, there is a question of whether it even makes sense to search for peoples pictures on Google. Even if it does not, people will do it anyway - so it better supply decent results. And if it can't, there will be an opportunity for specialized vertical search engines for people pictures.
In the meantime, please head over to your favorite image search engine and try searching for your name. Of course share your findings with us!
Listed below are links to blogs that reference this entry: What Does Google Think You Look Like?.
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/2129
Below is what happened in search today, as reported on Search Engine Land and from other places across the web:... Read More
Here is a summary of the week's Web Tech action on Read/WriteWeb. Note that due to recent travel and the start of the Web 2.0 Expo, this week's Wrapup is later and shorter than usual. Top Web News The big... Read More
Disclosure: the writer of this article, Emre Sokullu, recently joined Hakia as a Search Evangelist. The following article in no way represents Hakia's views - it is Emre's personal opinions only. Google is like a young mammoth, already very strong... Read More
Comments
Subscribe to comments for this post OR Subscribe to comments for all Read/WriteWeb posts
Alex, that was a great read indeed :) I would like to see Google apply some face recognition technologies over their results, and perhaps power of masses also. So that you pick the best result, after it reaches some consensus threshold, Google compares it with others and come up with relevant results all the time. They have Neven Vision for this, so they're certainly working on something like that - startups like Riya don't have chance to access all these images though, I would like to see them joining this challenge - competition is always good. But all in all, as you point out, there's absolutely a gap here.
Posted by: Emre Sokullu | April 12, 2007 9:37 PMThanks Emre!
I agree that if they know that it is a picture of a person, they could probably do fairly fast on the fly classification to at least group the images on the same page.
In the mean time a twist on a quote from Seinfeld comes to mind: Why do not just let me tell you which one of these is me!
Alex
Posted by: Alex Iskold | April 12, 2007 9:47 PMWhat a great post, thanks! I have been wondering about this issue for a while and at one point I concluded: any page that has my first name AND last name scattered around on ONE single page turns up as a result. And so far this seems to be true. When ego-googling myself for pictures without brackets the first five pictures are actually relevant. Three of them are pictures I made, one is a picture on my homepage and one is a group picture including me. The next result is Carrie Anne Moss though :) This picture is posted on a webpage that has two links to a Helmond weblog. Again the scattering of names seems to hold. This is my personal Google picture search problem, my last name is a city! And anything with Anne or the name of the city in it seems to be a result. Ouch!
Using brackets results in the same top five and all the pictures on the first page are somewhat relevant. Somewhat is the keyword here because none of the pictures depicts me (no I don't look like a gun.) But the interesting part is that I can locate all the pictures: several concert pictures I made, some pictures (not made by me) that are located on a friend's website, illustrations (not made by me) on a blog I commented on, some pictures I took for various media outlets and a t-shirt design I made.
Interesting find: The pictures seem to depict my (social) network instead of me!
Posted by: Anne Helmond | April 12, 2007 11:58 PMHi,
Great post.
I tried my name in Google Image search and it's clear what's going on. My name appears in an "authors" sidebar on a blog and the upshot is that *any* image in *any* post by *any* author on that blog gets matched; Google is looking at the entire page as rendered and is unable to make any distinction between the context in which my name appears and the main context (ie the post) on the page.
I also got a huge number of results of images on another blog where I have been mentioned a few times (in unrelated posts).
But, in fairness, the first image of a person returned, and second result overall, is of me, so I guess that's not too bad.
Posted by: Paddy Byers | April 13, 2007 2:37 AMThe other obvious problem is when you have a common name...
I share a name with a famous jockey, orchestra conductor, footballer, film writer and politician. So I don't a look-in - even when using other keywords alongside my name :)
Posted by: Andrew Davies | April 13, 2007 5:26 AM@6 Andrew - so true :(
Posted by: Alex Iskold | April 13, 2007 6:17 AMAlex - ask me the same question a few days back - thanks for that :-)
Posted by: Sascha Neck | April 13, 2007 6:58 AMThat was fun: http://tinyurl.com/2tq6sp
Google Images got me on the fourth image, not bad even though they consistently refer to an old blog of mine as being my #1 in serps versus my domain (which is even the same domain that that old blog sprung from!)
Alas, their first image depicting me would be even cooler... but then I would have to be dead!
Posted by: BillyG | April 13, 2007 7:05 AMAnother nice article from Alex..Kudos.
Is it a google week or something.. More articles on the famous subject 'Google'? Probably lot of Diggs :)!
Google image search (even though they dont say it.. it is still in 'beta') for many reasons and i'm sure they are improving the A.I (Artifical Intelligence) on it!
I have hardly came across any person in web with my last name 'Bhakthan'. I tried it in google image search and it shows 'some' 6 images in which couple of them are cartoon and there is no relevance to the name you searched for.
Went one step further and started researching on the the advanced options..They have some neat options to do complex search on image, but that doesnt return my image either.
Image search should be more of a People search if it is a name of a person and not sure the 'core logic' in google is able to differentiate the same. Spock is trying on the same lines.
If you search on things like 'Smart'.. check out the results, it tries to relate the term to the Car and most of the images returned are really funny!
'Hillary Clinton' returns a scary picture of her..
Long road to go, but with all its massive search power (69.4 percent of the overall search in USA!) Google will catch up one day soon..
Posted by: Srikanth Bhakthan | April 13, 2007 12:00 PMThanks for a nice article.
Trivia: The Snap-result is due to a Danish tabloid, Ekstra Bladet. "Iskold" means "Cold (or cool) as ice" in Danish. The caption on the picture says, that the air is cold as ice between Ferguson and Beckham.
Posted by: morten | April 15, 2007 2:32 AMYep. Current image searches are lacking. But there's hope. I ran across some recent research and software in MIT Technology Review and added a post in my PuterGnome blog today.
Posted by: CBrulee | April 17, 2007 4:53 PMInteresting article!
Polar Rose from Sweden promises: "Polar Rose makes photos searchable by analyzing their content and recognizing the people in them."
In fact, they believe in the power of masses. You can enter the name of a person you recognise in a photo. Read more in the upload blog:
http://www.upload2007.com/index.php?tag=polar-rose
(Sorry, this article is only available in German)
Posted by: Anna | April 18, 2007 4:02 AMPerhaps there is space for some kind of ascii image submission / search service.
Posted by: Nick | April 18, 2007 3:25 PMHi,
Posted by: Nicolas Steegmann | April 20, 2007 2:31 AMOn the subject of image people search you might want to try our new image search on www.exalead.com : we recently increased our image index up to 1 billion images, plus we have now a "face" refinement which I believe is a world premiere.
In fact I did your test search for you : "alex iskold" gives you back the right image, and when you refine on "Face" (in the right column) you're left with only the image of your portrait, as expected....
You wanna try "paris" and then refine on "Face" to see how it works...
blondhair blueeyes 8yearsold ingr:2
Posted by: sarahmckinnon | April 28, 2007 9:33 AMblondhair blueeyes 8yearsold ingr:2 freckles
Posted by: sarahmckinnon | April 28, 2007 9:34 AMdark brown hair, with blonde, red, and black highlights. Blue eyes 14;little bit freckles
Posted by: Tiffany | April 29, 2007 2:03 PMwhat you need is a user correlated search engine such as blogsaic.com from http://www.blogsaic.com/
Posted by: dazza | May 12, 2007 5:09 PM