Like to hack on Twitter feeds, streams and APIs? Then there's good news for you this morning. Twitter has acquired a small startup company called Bagcheck, but the real score in the deal was bringing co-founder Sam Pullara onto the team. Pullara was previously the Chief Technologist at Yahoo where he lead many of the best programs at that beleaguered but technically awesome company.

Yahoo Pipes, Yahoo Query Language, Yahoo Search BOSS and other inspiring technologies that enabled hackers all around the web to roll out sophisticated mashups powered by Yahoo's backend were championed for years by Pullara. Not everyone liked him, but people who love to experiment with data have got to be excited about his coming to Twitter, the world's most promising stream of publicly available, semi-structured, real-time social data. Twitter's relationship with developers has been troubled at times, but Pullara's joining the team is the latest step the company has taken to make amends with its developer ecosystem.

What About Bagcheck?

The company Pullara co-founded may see its technology integrated into Twitter as well. Thomas Vander Wal coined the word "folksonomy" (a user-driven system of popular classification) and is a big fan of Bagcheck.

"What Bagcheck has done that is brilliant is make it easy to add things from around the web into the Bagcheck site through their 'BagIt' bookmarklet, which captures 'social objects' and keeps them well structured.

"Twitter is good (not great) for discussion of things, but is really lacking a good view of the social object, in the 'socially mediated object' sense that Karin Knorr Cetina used - where people having clear view of the object they were discussing had much better (accurate, robust, and constructive) conversations than those discussing things in the abstract.

"Bagcheck could provide that hook into social objects to bring things within clear view. This will help with resolving things like 'Which Planet of the Apes movie are you talking about?' and other ambiguous conversations."

If Sam Pullara can advance the kind of paradigm represented by Yahoo Pipes and YQL into the Twitter ecosystem, that would drop the already low technical cost of entry and greatly increase the power available to Twitter's ecosystem of developers.

Pullara can also work on big partnerships. He lead Project Hummingbird, which was Yahoo's search partnership with Twitter, but we hear he got even further with a Yahoo/Facebook partnership. Those are powerful additions to a hacker/executive resume.

You may remember when a megahyped search startup called Cuil, with tens of millions of dollars in backing, launched then faded away a few years ago? The developer that built the same interface with better performance in four hours, using Yahoo's Build Your Own Search Service (BOSS), was Sam Pullara.

This Spring, Twitter brought on Jason Costa, former developer relations manager at both Google and Facebook, to be the new Twitter developer relations manager.

Twitter is going to need to stock up on hot developer types, because Google Plus will open an API soon and the competition for developer attention is going to be massive. That's going to be great news for developers and the users who love them.

Some people say it's too late for Twitter; that it has mistreated its developer ecosystem too much to win against any viable challenger. Time will tell, but it seems clear that Twitter isn't going to just let all the hacker attention flow to the little upstart Google Plus without putting up a fight.

As you seen from the video, a simple black-and-white marker is used to trigger a 3D experience with animated objects at which the user can "shoot." Nothing new there. But notice as the objects fly out of the wall and the camera turns, the phone continues to track the objects in their relative virtual position even after the 2D marker has left the camera's vision. That's pretty impressive.

"Sensor fusion" refers to the software's ability to simultaneously use camera and accelerometer data to create better spatial awareness. Since the phone can tell it has been turned a certain way thanks to its sensors, the software can translate that motion and continue to project 3D objects in their relative positions.

The other interesting thing about this demo is it is a streaming session between two people in a multiplayer game experience. Two people can "shoot" at the same set of 3D models and play together to fight virtual foes. Layar says it is the first online multiplayer AR game, but I seem to remember a very early "virtual tennis" promotion where two people could bat a tennis ball back and forth in virtual space.

Either way, the demo is very fascinating and points to a bright future for mobile augmented reality. "Sensor fusion" that includes GPS chips and gyroscopes is going to be an important feature to interactive AR experiences in the near future. Being able to continue an AR experience even when the camera moves away from a marker will open the door for some amazing experiences.

In a post on the Yahoo Developer Network blog, VP Social Platforms at Yahoo Neal Sample broke the news.

It's hard not to be disappointed by news like this, but perhaps some innovative engineers will be set free to work on other things. And perhaps some unfulfilled dreams will be allowed to die, so that they might be reborn to try again elsewhere.

Paul Graham's essay about what happened to Yahoo is worth reading, as well. (As is this counter argument from Yahoo evangelist Tom Hughes-Croucher.)

There are other cool projects in the works at Yahoo. I hope they find more success than these ones have, with the exception of YQL.

What's worse, for marketers hoping to tap into a diverse and savvy audience of shoppers, diners, and other local consumers, the details on audience make-up are disappointing. According to Forrester, LBS users are 80% male and 70% are aged 19-35.

So should marketers stay away for now until these services mature? Absolutely not. And here's why.

Yes, the audience for Foursquare, Gowalla, MyTown, Brightkite, Loopt and the other smaller players in the location-based services game is young, tech-savvy, educated (70% have a 4-year degree) and predominately male.

But so is the average early adopter of new technology, historically speaking. As Foursquare's co-founder Dennis Crowley explains, we're still in the early days of location-based services. "Go back to Facebook's stats in 2005 or Twitter's stats in 2006, and you'll probably find data that tells a similar story."

And yet, despite the relatively small size of these apps' user bases (Foursquare and Brightkite have around 2 million users, Loopt has 4 million, Booyah's MyTown has 2.5 million, and Gowalla 340,000), these apps attract the ideal customer for marketers to target with mobile coupons, ads and other location-based campaigns. These users are the "influentials," which, in marketing speak, means they're more likely to spread the word about a business, share their opinions and encourage others to join in this new-fangled check-in game.

In fact, the report finds that 38% of these users are more likely than others to have friends and family ask their opinions prior to purchase. They're 14-20% more likely to use their phones for comparison shopping and, in general, they're more likely to research a product or service before purchasing.

In other words, when these folks get a good deal, they know they're getting a good deal. And they'll probably tell someone about it.

Challenges for LBS: Fragmentation & Potential for Big Players to Crush Small Ones

Although LBS apps have the potential to connect marketers to key customers, there are still a number of issues beyond the demographics of the current audience and its size. For one thing, the market is heavily fragmented. Not only are there half a dozen or so "major" players in the check-in business, there are also other location-based apps that focus on different types of experiences. For example, the object-based social network Stickybits allows anyone to tag real-world items with barcoded stickers, while other niche apps like SCVNGR turn checking in into a puzzle-solving game. Although not mentioned in the report, we could add Miso to this list of niche apps as well, as it lets you "check in" to TV and movies you're watching.

Another potential problem for today's LBS applications is the potential for a big player to come in and dominate the market. Google already has location-based services Latitude and Buzz, both which have the potential to increase as the Android market grows. Yahoo has partnered with Nokia on mapping and has acquired various location-based services, too. We also have to point out the elephant in the room, even though Forrester did not: Facebook. If the social network with now half-a-billion users ever formally launches an LBS app or service, it could be game over for everyone else.

Is Forrester Warning Marketers Away from LBS?

The knee-jerk reaction to Forrester's report (as evidenced here in the comments of this AdAge article) is that the research is essentially downplaying the potential of the LBS market. That's not necessarily the case.

Instead, Forrester not only highlights some high-profile partnerships between LBS apps and major brands (Starbucks/Foursquare & Brightkite, NatGeo, USA Today & The Washington Post/Gowalla, Gap/Loopt), it also recommends particular companies take special notice of the potential here.

Companies engaged in event marketing, those with retail storefronts and those whose online tools translate to offline interaction (Craigslist and ZipCar, for example) could all benefit from using this technology, the report recommends.

And while other businesses may not be able to generate a critical mass of volume with LBS-based programs, now is the time to experiment, says Forrester. Male-targeted companies may have better luck than others, but only for the moment. Other marketers may want to wait for demographic and adoption rates to increase and for the "vendor dust to settle," though. While yes, Forrester concludes that, in some cases, "the potential for LBS doesn't match the hype," that statement ends with an important qualifier: "....YET."

However, in stark contrast to the 140-character wunder-app, Meme has proven in the months since its release to be a much better platform for multimedia sharing and cross-platform content curation. Now, the Tumblr/Twitter/Posterous hybrid is offering an API built on top of YQL, Yahoo!'s query language that we covered back in May, when we were impressed with its power, versatility, and uniqueness. The Yahoo team has already used the API to develop a version of Meme for smartphones.

As an example of what YQL allows developers to do, Atlassian CEO Mike Cannon-Brooks told us in May, "YQL... allows you to build tables of data from other sources online, using Javascript as a programming language and run it on Yahoo!'s servers, so the infrastructure needs are very small." Also from our May coverage:

According to Yahoo Chief Technologist Sam Pullara, the idea behind YQL (launched in October 2008) was to create an agnostic query language similar to SQL, a language familiar to most developers, and let developers use that language to use the Internet as a huge database. "If you make it universally and simply accessible so every application developer doesn't have to learn every API, it's be easier for developers to create apps from the data users have taken so much time to make available on the Internet."

Although YQL looks a lot like SQL, it treats the info on the web as a virtual table that developers can manipulate in a standardized way, regardless of the API that data came from. Developers only had to know how to use YQL to quickly create simple mashups.

Interested developers can check out the Meme documentation. The API, the site says, "is intended for developers who are familiar with RESTful Web services." In addition to offering superior support for multimedia content and simple access through YQL, Meme also has an excellent built-in repost function, an asymmetrical friendship model, and OAuth compliance.

For Users: Flexible Queries

One of the most interesting aspects of this library is that it gives users a lot of flexibility when they structure their queries. Ident Engine will happily accept full URLs like http://www.twitter.com/rww or just simply 'twitter.com rww' or the Webfinger email structure ('rww@twitter.com').

Not a Developer? Try these Demos

While Ident Engine is mainly aimed at developers, the team has made a number of cool demos available on its server. The Combined Profile Demo, for example, looks at a user's profiles on multiple services and then builds an aggregate profile for the user, based on commonalities it finds in these profiles. The Lifestream demo goes a step further and discovers a user's profiles on a variety of services and pulls in the latest updates. The project's homepage also features a number of additional demos.

Glenn Jones also recently released a Firefox plugin, Identify (our review), which builds on the same premises and allows users to quickly bring up the aggregate profile of any user while looking at one of their regular social network profiles.

The Read/Write Web

While the earlier incarnations of YQL were mainly meant to read data, with the addition of these three new SQL verbs, the focus has now shifted towards writing data back to the net as well. Developers can now use YQL to write and modify data on web services and applications.

To explain how useful this can be, the Yahoo team used a few different examples. A developer can now easily use YQL to update a Twitter account (even authentication with OAuth is possible), for example, or add a new comment to a blog post, or insert any data into a remote database. Basically, developers can now use YQL to write data back to any web site that uses forms for data entry and to any API, including authenticated APIs.

To try this, here is an example from Yahoo (you will have to log in to the YQL console):

Try creating a new tweet from the YQL console, follow this link <a href="https://developer.yahoo.com/yql/console?q=use%20%27http%3A%2F%2Fwww.yqlblog.net%2Fsamples%2Ftwitter.status.xml%27%3B%20insert%20into%20twitter.status%20(status%2Cusername%2Cpassword)%20values%20(%22Playing%20with%20INSERT%20UPDATE%20and%20DELETE%20in%20YQL%22%2C%20%22twitterusername%22%2C%22twitterpassword%22)">to run this</a>:

use 'http://www.yqlblog.net/samples/twitter.status.xml';

insert into twitter.status (status,username,password) values ("Playing with INSERT, UPDATE and DELETE in YQL", "twitterusername","twitterpassword")

Pullara and Trevor also stressed that because Yahoo runs YQL on five datacenters spread over three continents (three in the US, one in Europe, and another one in Asia), executing commands through YQL is generally very fast. Yahoo also set some relatively generous rate limits for the service. Developers who use the service and who identify themselves with an access key can make up to 100,000 calls per day, while anonymous users are restricted to 1000 calls per hour, which is still a pretty good number.

There's a great amount of data available on the Web in APIs or even straight HTML. It's all there for the parsing - and parsed data from social media in particular is held to be a goldmine. But traditionally, it's the heavy lifting (the broad variety of programming languages used in APIs, the challenges presented by complicated authentications, the occasional need for massive pipes) that has made accessing and sorting data into useful applications a laborious process.

Yahoo!, chiefly to serve the needs of its own engineers, has been developing a sophisticated solution that is agnostic across all Internet platforms and that lowers both the burden of labor and the barriers to entry for social and other web application developers, many of whom are already singing the praises of the newly released YQL Execute.

"YQL Execute allows you to build tables of data from other sources online, using Javascript as a programming language and run it on Yahoo's servers, so the infrastructure needs are very small."

In the slightly more technical language presented on the Yahoo! Developer Network Blog, "The Execute element can contain arbitrary developer code that the YQL data engine runs during the processing of a YQL statement."

It also handles authentication for third party sites.

Is there anything like it currently on the market?

"Nothing... It's pretty awesome," said Cannon-Brooks.

Yahoo! Query Language

Although YQL looks a lot like SQL, it treats the info on the web as a virtual table that developers can manipulate in a standardized way, regardless of the API that data came from. Developers only had to know how to use YQL to quickly create simple mashups.

Open Data Tables

YQL was used to support a broad range of APIs, almost anything publicly available online, from FriendFeed and Google Reader to the Guardian newspaper. "No one has yet pointed out an API they can't figure out how to map," said Pullara.

However, some data could not be accessed without authentication, such as Google Calendar or Netflix. Those APIs were very often very sophisticated and even complicated for the end developer. For these APIs, Yahoo! rolled out YQL Execute on April 29.

YQL Execute

YQL Execute also allows developers to access multiple services and get a single result back. For example, an app developer could call up New York Times articles with specific tags AND Flickr photos with related tags; YQL Execute would return a combined result with both articles and related photographs. Another benefit for developers is the use of the massive Yahoo! infrastructure, as all the heavy lifting of data is done on Yahoo! servers.

And because of the speed, simplicity, and scope of these tools, implications now range much farther than simple mashups. With access to authenticated and private data, more sophisticated applications can be written quickly and easily.

The Dark Side

The above quotation is from Scott D. Reinhart, who has been eyeball-deep in application development longer than many "social media gurus" have been out of high school.

Right alongside the generally held social media dictum that a rich data stream is inherently bankable, there is the hotly debated issue of data ownership. Especially when data is made more valuable by having been parsed, organized, and compared, and most especially when someone creates a revenue stream from previously unmonetizable data, questions of ownership and copyright flare up around the social web.

"Public APIs allow you to easily develop using mature platforms," said Reinhart, "but they [large IT and social media companies] usually have a hidden intention. In this case they advocate putting your database layer onto their systems... So let's say I use the Yahoo! data layer, I use BizSpark to get my development tools, and I am making MySpace (Open Social) and Facebook apps using jQuery - who owns my code? Technically, they own everything. They can claim I just made a mashup.

"I would, as someone approaching these systems, stop drinking the Kool-Aid and read the terms of use. Check what it says about ownership."

Yahoo! Servers for YQL Developers

"If you create an open data table, there's no requirement to upload it to Yahoo! We do cache data that we pull from APIs and the web to make it faster, but we don't store that data. It passes through without being collected for permanent storage."

By contrast, with other services such as Google or Amazon Web Services, developers are required to upload their data, which is stored and executed on the company's systems. In using Yahoo! YQL, a developer's data has "a very transient experience and expires from the cache," said Pullara. "It's a convenience, not a requirement in any way."

The Price of Free

"We want to enable rather than discourage more useage ," said Pullara. "And while people don't want to pay, they do want to know they're a customer and have a relationship with Yahoo!"

Currently, Yahoo! has set certain limits on use of their infrastructure. App developers are limited to 100,000 calls per day, per IP address. If the application runs in a browser (hence, on many different IPs), it's a non-issue. Pullara said, "The limit targets those who would abuse the platform... people who might spin up DoS attacks. You have to have controls in place to make sure that doesn't happen."

Many developers are enthusiastic about the legitimate and value-adding implementations of the technologies. "The YQL improvements are just sex on legs," said Cannon-Brooks via Twitter. "The most exciting, least talked about 'tech of now' is YQL."