Developers: Never Mind the APIs, Here's YQL Execute
"I Tried YQL Execute and All I Got Was an Authenticated Javascript API Processing Layer in the Cloud"
There's a great amount of data available on the Web in APIs or even straight HTML. It's all there for the parsing - and parsed data from social media in particular is held to be a goldmine. But traditionally, it's the heavy lifting (the broad variety of programming languages used in APIs, the challenges presented by complicated authentications, the occasional need for massive pipes) that has made accessing and sorting data into useful applications a laborious process.
Yahoo!, chiefly to serve the needs of its own engineers, has been developing a sophisticated solution that is agnostic across all Internet platforms and that lowers both the burden of labor and the barriers to entry for social and other web application developers, many of whom are already singing the praises of the newly released YQL Execute.
"It adds a lot of power," said Mike Cannon-Brooks, co-founder of Atlassian, an Australian collaboration and development software company widely recognized as one of the biggest stars in the Enterprise 2.0 world.
"YQL Execute allows you to build tables of data from other sources online, using Javascript as a programming language and run it on Yahoo's servers, so the infrastructure needs are very small."
In the slightly more technical language presented on the Yahoo! Developer Network Blog, "The Execute element can contain arbitrary developer code that the YQL data engine runs during the processing of a YQL statement."
It also handles authentication for third party sites.
Is there anything like it currently on the market?
"Nothing... It's pretty awesome," said Cannon-Brooks.
Yahoo! Query Language
According to Yahoo! Chief Technologist Sam Pullara, the idea behind YQL (launched in October 2008) was to create an agnostic query language similar to SQL, a language familiar to most developers, and let developers use that language to use the Internet as a huge database. "If you make it universally and simply accessible so every application developer doesn't have to learn every API, it's be easier for developers to create apps from the data users have taken so much time to make available on the Internet."Although YQL looks a lot like SQL, it treats the info on the web as a virtual table that developers can manipulate in a standardized way, regardless of the API that data came from. Developers only had to know how to use YQL to quickly create simple mashups.
Open Data Tables
Then, this February, Yahoo! launched open data tables. "Initially," said Pullara, "we had a lot of default tables in the system, mostly Yahoo! API, things like Flickr, local search, Yahoo! weather. For accessing the rest of the Internet, we created dynamic tables that understood things like XML, Atom, RSS, comma-separated value tables such as spreadsheets, etc. Dynamic tables let you access them but not abstract them. Open data tables let you map a 3rd party site, making the data accessible with YQL."YQL was used to support a broad range of APIs, almost anything publicly available online, from FriendFeed and Google Reader to the Guardian newspaper. "No one has yet pointed out an API they can't figure out how to map," said Pullara.
However, some data could not be accessed without authentication, such as Google Calendar or Netflix. Those APIs were very often very sophisticated and even complicated for the end developer. For these APIs, Yahoo! rolled out YQL Execute on April 29.
YQL Execute
"With Execute," said Pullara, "the code only needs to be written once, and not necessarily by the app developer. The authentication is all covered by the Yahoo cloud."YQL Execute also allows developers to access multiple services and get a single result back. For example, an app developer could call up New York Times articles with specific tags AND Flickr photos with related tags; YQL Execute would return a combined result with both articles and related photographs. Another benefit for developers is the use of the massive Yahoo! infrastructure, as all the heavy lifting of data is done on Yahoo! servers.
And because of the speed, simplicity, and scope of these tools, implications now range much farther than simple mashups. With access to authenticated and private data, more sophisticated applications can be written quickly and easily.
The Dark Side
"The fact is this: If you do not patent, if you do not copyright, if you do not privatize, and if you do not own, you will be ripped off by someone; and you asked for it."The above quotation is from Scott D. Reinhart, who has been eyeball-deep in application development longer than many "social media gurus" have been out of high school.
Right alongside the generally held social media dictum that a rich data stream is inherently bankable, there is the hotly debated issue of data ownership. Especially when data is made more valuable by having been parsed, organized, and compared, and most especially when someone creates a revenue stream from previously unmonetizable data, questions of ownership and copyright flare up around the social web.
"Public APIs allow you to easily develop using mature platforms," said Reinhart, "but they [large IT and social media companies] usually have a hidden intention. In this case they advocate putting your database layer onto their systems... So let's say I use the Yahoo! data layer, I use BizSpark to get my development tools, and I am making MySpace (Open Social) and Facebook apps using jQuery - who owns my code? Technically, they own everything. They can claim I just made a mashup.
"I would, as someone approaching these systems, stop drinking the Kool-Aid and read the terms of use. Check what it says about ownership."
Yahoo! Servers for YQL Developers
However, Pullara said of Yahoo!'s claim to developers' IP, "We don't own anything."If you create an open data table, there's no requirement to upload it to Yahoo! We do cache data that we pull from APIs and the web to make it faster, but we don't store that data. It passes through without being collected for permanent storage."
By contrast, with other services such as Google or Amazon Web Services, developers are required to upload their data, which is stored and executed on the company's systems. In using Yahoo! YQL, a developer's data has "a very transient experience and expires from the cache," said Pullara. "It's a convenience, not a requirement in any way."
The Price of Free
Yahoo! has begun investigating potential commercialization of YQL technologies."We want to enable rather than discourage more useage ," said Pullara. "And while people don't want to pay, they do want to know they're a customer and have a relationship with Yahoo!"
Currently, Yahoo! has set certain limits on use of their infrastructure. App developers are limited to 100,000 calls per day, per IP address. If the application runs in a browser (hence, on many different IPs), it's a non-issue. Pullara said, "The limit targets those who would abuse the platform... people who might spin up DoS attacks. You have to have controls in place to make sure that doesn't happen."
Many developers are enthusiastic about the legitimate and value-adding implementations of the technologies. "The YQL improvements are just sex on legs," said Cannon-Brooks via Twitter. "The most exciting, least talked about 'tech of now' is YQL."
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
Cool... lot's a data, but another tool and language to learn.
@homepage - That was kind of the point of YQL, to be similar enough to SQL that devs wouldn't have to learn a new language.
Cool, what about writing to a api provider ?
These examples show only some read, but there is no PUT/write requests..
This sounds like an incredible service that could further democratize online product development. More and more, data is becoming an unlimited resource. I think data 'ownership' will be come less important and less valuable. Just knowing something about a consumer is no longer enough to provide a competitive advantage, you need to know what to do with that knowledge as well.
Evan
Consider therapy or counseling. A divorce is never easy. Your attorney is not the most qualified person to assist in dealing with your emotional pain/recovery (that is no more so than a good friend lending an ear). Although a significant amount of personal information is shared during the divorce process between an attorney and client, and often times a certain amount of hand holding occurs, an attorney is unable, and should not become a therapist or counselor. You'll get more per fee dollar if you keep distinct the roles an attorney and therapist play. No one will ever argue that the dissolution process is easy. There are however various ways to make this period of time less painful and stressful for everyone involved.
-------------
mukesh251
real estate - real estate
Consider therapy or counseling. A divorce is never easy. Your attorney is not the most qualified person to assist in dealing with your emotional pain/recovery (that is no more so than a good friend lending an ear). Although a significant amount of personal information is shared during the divorce process between an attorney and client, and often times a certain amount of hand holding occurs, an attorney is unable, and should not become a therapist or counselor. You'll get more per fee dollar if you keep distinct the roles an attorney and therapist play. No one will ever argue that the dissolution process is easy. There are however various ways to make this period of time less painful and stressful for everyone involved.
-------------
mukesh251
real estate - real estate
I've actually been working on a similar open source project called Axpress for a while now. I really like the idea of a query language which provides access to multiple databases, and databases which don't yet exist - or require computation to create on the fly. A query language like YQL or Axpress is a potential next step to providing an interface which understands natural language.
Posted by: http://openid.aol.com/zdwiel
|
May 2, 2009 1:27 PM
One correction Jolie. It's "Processing" not "Proccesing".
With Execute, developers now have full control of how the data is fetched into YQL and how it’s presented back to the user. With Open Data Tables, developers can build tables that manipulate, change, and sign the URLs to access almost any protected content, allowing YQL access and combining data across a variety of different authenticated services such as Netflix or Twitter.
They won't even consider adding an Access-Control-Allow-Origin: * header to their XML or JSON output. It's either give Yahoo full control of your website or do it server-side, making secure standalone webapps impossible.
We are actually going to add Access-Control-Allow-Origin in the next push after seeing the suggestion on Hacker News. Seems like a great thing, wish more browsers supported it.
The latest version of YQL has been deployed and supports Access-Control-Allow-Origin.
@Sam That was me on Hacker news :P "Sephr"
RWW's @jolieodell dares to tackle the powerful beast that is the new YQL Execute http://bit.ly/J1gxO and so far has lived to tell the tale [from http://twitter.com/marshallk/statuses/1680054262]
Posted by: Marshall Kirkpatrick
|
May 24, 2009 10:39 AM
Your attorney söve is not the most qualified person to assist in dealing with your söve emotional pain/recovery (that is no more so söve than a good friend lending an ear). Although a significant amount söve of personal information is shared during the divorce process between an attorney and client, söve and often times a certain amount of hand holding occurs, an attorney is unable, and should not become a therapist söve or counselor. You'll get more per söve fee dollar if you keep söve distinct the roles an attorney and therapist play. No one will ever argue that söve the dissolution process is easy.
thanks for comments
I'm with you on this Evan. Anything that helps to further democratise online product development is a step in the right direction and I must say long overdue as well.
Andrew. Australia
Cartoon DVD
TV series DVD
dvd decrypter
Welcome to asics gt 2140!There are best running shoes at www.touchasics.com,our asics under armour shoes are anthentic and one hundred percent originals package,good running shoes here are just stocked from container order of asics abroad like orders cooperated for wal-mart and so on.So there are various styles and enough size that you may interest here.And due to certain stocks from our own factory and non tax fees home and abroad,we develop our business enlarging cooperation broadly via sample offering and free shipping here.So price here shown to you are too cheap based at the reason above.Just come on and catch a chance to enforce our development here.
One word: Tough. Introducing the all-new Danner Nike Air Force Ones 2009 Hot boot
Cheap Nike Air Force Ones shoes is very popular in the market,
New Nike Air Force Ones also is my favorite.
http://www.kickaf1.comCool, what about writing to a api provider ?
These examples show only some read, but there is no PUT/write requests..