CONFIRMED: Facebook Gets Faster, Debuts Homegrown PHP Compiler [Update]
The rumors have been flying over what's going on over at Facebook headquarters. The word has been that a PHP team was brought in and made to sign non-disclosure agreements before discussing a PHP project that has been in development for the past two years. Alex Handy, senior editor of the Software Development Times Blog, predicted last Saturday that Facebook "has rewritten the PHP runtime from scratch," and several sources have confirmed for us tonight that Facebook has indeed been making some changes to the basic PHP runtime environment.
According to our sources, Facebook has been working on a PHP compiler that will
increase speed by around 80% and offer a just-in-time (JIT) compilation engine that will offer a number of advantages. The project is very similar to Google's Unladen Swallow project, which rebuilt the Python compiler, boosting the speed fivefold and opening the door for multi-language integration.
Update: Facebook officially announced the project, titled HipHop, this morning and confirmed that it would be released as open source this evening. Facebook will be hosting a talk at 7:30 PST with PHP developers to get into the specifics. The talk will be streamed live on Ustream. Developers are encouraged to comment on RWW's open thread discussing the implications of the PHP runtime rewrite.
Richard Crowley, an engineer at OpenDNS who is familiar with the project, told us that David Recordon, an engineer at Facebook, invited him to come to Facebook's headquarters Tuesday morning but wouldn't give a reason. Crowley mused to us that it was likely his PHP skills and personal connections that got him through the doors and clued us in on what he thinks will be going on behind them.
PHP is normally an interpreted language, which means that every time a user accesses a PHP page, the server needs to take the code and interpret it to produce the final product. A compiler, however, makes this process much quicker, as the code is interpreted before the user ever asks for the page. The problem here is that any time that page needs to be changed, the code needs to be recompiled.
Crowley explained to us that the JIT compiler Facebook is introducing occupies a middle ground that not only retains the flexibility of PHP as an interpreted language, but offers the speed of compiled languages like C.
"Compiling PHP to code a CPU can directly execute certainly has performance implications. It would be silly to alter the workflow to be more like C or C++ by doing all parsing and compilation ahead of time. At the other end of the spectrum, it's slow and out-of-fashion to interpret every statement within the runtime. A JIT (Just In Time) compiler compiles frequently-executed portions of the program to machine code for speed while maintaining the flexibility of interpreted code."
Essentially, the closer a coding language comes to bare metal, the faster the program will run. Facebook's programs are reported to run around 80% faster than before now that the runtime - the code-compiler-program sequence - has been restructured and rewritten.
Crowley continued, saying that he expected Facebook to announce a JIT compiler based on the Low Level Virtual Machine, which lays at the heart of Google's Unladen Swallow project. Crowley's suppositions were confirmed by anonymous sources tonight.
When we asked what he thought of this style of release - years of secretive development by two lonely coders, likely locked in a deep, dark vault - Crowley said it was typical.
"[Facebook] tends to do giant code dumps," he said. "Facebook tends to build something big, use it, and open source it."
We wish Facebook had made this project open, as we're sure many PHP developers would concur that the efforts would have been much swifter and more beneficial to the public had more folks and a larger team been involved from the outset. And we do wonder about possible duplication of effort from others who may have been working on the same issue.
There are a number of us out here running PHP (ahem!) that could certainly have benefited from the speed boost.
Nonetheless, we're looking forward to seeing what this does for our PHP browsing experience and how the open-source community reacts to the news. Will the community rally around the new compiler and push PHP into a new realm? Let us know your thoughts in the comments.
Update: In fact, it turns out that the PHP source code is getting translated first into C++ and then compiled with g++. While the details of the process are different, the end result is the same as are many of the implications. Jolie O'Dell wrote an update earlier today, explaining exactly how Facebook has altered PHP and we'll be taking a look at some of the implications shortly.
Additional reporting for this story was also done by Jolie O'Dell.
Share
Facebook
Comments
Subscribe to comments for this post OR Subscribe to comments for all ReadWriteWeb posts
Can someone tell me the difference between this and say memcache/eaccelerator?
Can anyone tell me how this is different from a simple opcode cache like APC? (which will be included in PHP6) - I assume the performance increase versus APC and this will be much lower.
"Facebook tends to build something big, use it, and open source it."
What is the big thing that Facebook has built and "opened sourced".....Is there something that they have "open sourced" that does not directly benefit them rather than leveling the playing field for everyone ?
Facebook should "Open" up the data that is owned and created by its members so that they have the ability to take it where ever they want.
Who do we cheer for in this game of closing the internet
Do no evil illusionist vs Ayn Rands' spawn
@Bruce Wayne - FB open sourced Thrift and Cassandra
I'm also interested in how this differs (in implementation difficulty and performance) from an opcode cache. I only need to allocate ~14MB of RAM to APC to have a 100% cache hit rate, meaning no PHP file ever needs to be interpreted from source to serve a page unless I modify the code.
if this is real, i really appreciate facebook effeorts and congratulate them, but with no hope of "opensourcing" it, stop kidding ... cant expect from FB!
and to the memcache/eacclereator argument, well you never know, FB coders might have done only the things they require to fast on.... which might be better than memcache/eacclereator or at least diff... give credit where its due!
Aren't most of the other accelerators also chaching JIT compilers? Wonder why FB decided to create their own in-house.
That's good that Facebook is moving to PHP. Right my browser takes forever to load Facebook pages, which gets pretty annoying. Hope to try it soon.
@Jolie O'Dell: Thrift was already there in form of protobuf (developed at gooogle) but still I agree they opensourced it, forgot that.
havent heard of Cassandra, more light....
Facebook reported several times that they do use APC, there's lots of presentations all over the web of them talking how to optimize it. So, this new stuff ought to be better than APC.
@Jennifer: what the heck was it a joke?
For those who are interested, Facebook actually has developed or significantly contributed to a ton of open-source projects: some big, some small. The official list is here: http://developers.facebook.com/opensource.php
This is brilliant news. My first reaction was "WTF" but I can really see a benefit to the PHP community here.
Zend has been working on PHP for a long time now and with PHP 5.3 they have done some amazing things, but there is nothing much for us to look forward to with PHP 6. Most of the features intended for PHP 6 have made it into PHP 5.3 and it seems that Zend aren't really bringing us anything awesome.
Facebook have a fairly unique position to do this as they are the largest PHP web-site and get an insane amount of bandwidth. If they can improve PHP to be useful to them, the benefits will trickle down to us little folk and we'll have quicker websites too.
I look forward to seeing what else this HPHP will involve.
if facebook gets PHP compiler i have now the answer that php is better and faster than other.
This is good news, also their open source.
@Bruce Wayne: In addition to Cassandra and Thrift, they also open sourced Hive DB, the SQL-like implementation for Hadoop.
I guess that's the end of the "Unresponsive Script Message" i often got when using facebook .... Just kidding of course. I will be one of the million beta testers.
Faster sounds good, but I would be happy if Facebook was simply stable. I may only be about 10% of the time that Facebook sputters and fails on me, but it's still annoying.
This all sounds incredibly important and I truly wish I understood it…because I know I need to be worrying about all these things on my blog too….HELP!
English? What does this mean in layman's language?
Wesley- Memcached is caching in memory as the name suggests. APC/eAccelerator are PHP Opcode caches which is similar to what compiled PHP would bring us, though I suspect not as fast.
Ari- Please, bro. Not everything is meant for laymans.
agrees with Jolie & Warren, FB has done a couple of Things, expcialy compared with other Companys alike - but the only thing we really care about, will it be more then an opcode cache like APC or not.... well, i guess we just have to wait and see
I think facebook will open-source it. They (and all of us) would want PHP to be the language of choice for web apps. Other wise they would have gone for J2EE or .net rather than a building new PHP the compiler.
They are doing what they can to gain a competitive advantage (perhaps why it was kept quiet until now). I find my self wondering how this plays into their monetization strategy. Perhaps it will also speed up the experience and provide advantages for advertisers as well.
-Travis
Speed and performance on the web between Facebook and Google is like the nuclear arms race during the Cold War between the U.S. and the former Soviet Union.
I'm no programmer, but I'm savvy enough to understand some of the lingo... Is JIT used in conjunction with an opcode cache such as XCache or APC?
For projects like Care2x, where most of the code do not change a lot after installation but output is almost always different, something more than a cache mecanism would be nice.
How does this new jit-compiler compare to the zend engine shipped with php5 / 6?
Amazing. I have always been a fan of MySPace, but FaceBook is starting to look mighty appealing.
Jess
www.web-privacy.cz.tc
They built a "compiler", not a simple "opcode cache".
That means they can now declare static vars and optimize the function calls, arguments and return values to hell. method calls and inheritance lookup also.
Thats why they get 80% and not just 10%. Like the roadsend php compiler (written in bigloo scheme, a lisp language).
I believe they only enhanced roadsend with some private language enhancements to be able to optimize stupid run-time dynamics away.
JIT'ting is nice to further optimize dynamic loops.
In reality they should just use a better language which supports expressiveness and is easier compilable than
simple PHP. But they are obviously locked into that mess.
Just to correct a misstatement from the article. Unladen Swallow's ( http://code.google.com/p/unladen-swallow/ ) goal is for a 5x speed up, but they are still in the initial phase and far from reaching that goal.
I think everyone who's asking about opcode caches like APC and eAccelerator are forgetting how big FaceBook is. They are undoubtedly already using technologies like these to their maximum potential. Companies don't just do things like this for no reason. I can guarantee that whatever they created will produce performance gains above and beyond APC and the like.
It is fun to watch good ideas spread.
http://www.mindmeld.ws/blog/2010/02/02/java-changed-the-world/
And this is why I use ASP.NET. No need to rewrite the runtime when your site gets too big.
@Edgar hmmm a site as big as FB or MySpace! Can you name any similar site in ASP.NET?
Very excited about this possibility. Facebook has pioneered some great open source tools already and they have enough skin in the game to be the right entity to build something like this. I'm ready to beta test it.
I remember when MySpace was jokingly referred to as CrawlSpace. Facebook took on a similar reputation until now--apparently.
@abcphp.com
MySpace is written is ASP.NET. But to further answer your question - all the microsoft properties are asp.net and are some of the most highly trafficed sites on the internet.
@Nathan sorry about that, thought myspace was running ColdFusion. If you look at the top 20 (http://www.alexa.com/topsites) how many of them are PHP and asp.net?
Someone should have done this a long time ago.
There seems to be a lot on confusion about what this post says.
This post says Facebook is using a new engine under PHP. PHP currently uses the Zend Engine. They will be replacing that with LLVM.
Op-code caches only store the op-codes that are feed to the engine. So it is not significant that the code isn't parse every time, like the post says it does, as most are using an op-code cache that stops that. What is happening is that in place of Zend Engine op-codes generated from the PHP source code it will now generate whatever the LLVM engine requires.
This is a big effort as there is not a clean separation of PHP from the Zend Engine. So to add a new Engine would take a lot of work.
It is unlikely that it would change much about the language, as much is alluded to in the post. Also lets not forget PHP is just some glue code around a ton of C libraries so making sure that all the extensions work would also be a very important aspect of a project like this.
I hear comments like "they should have used a better language." or the one that made me cringe, "this is why I used .NET".
Let's see the choices when FB was being written:
1. ASP
2. PHP
3. Java
4. ColdFusion
5. Python
6. Perl / CGI
7. ??
You could make the argument that ruby was around, but lets not. It was nearly a twinkle at that point in time without a real "test" behind it.
Given those eliminate all the commercial products (ASP, ColdFusion) from the list,
leaving Java, PHP, Python and Perl. Noting that ASP (at the time) was nothing more than a scripting language around COM objects -- a complete disaster with NO security model and terrible performance through IIS (how many sites out there do you remember being hacked...or worse, simply crashing) ColdFusion is something that I run into a lot now, I run into this because people want to get rid of it and replace it...
Looking at those choices, one could pretty quickly point out that Java would (at that point in time) required some serious iron to scale, it was also not a popular server platform for Web development. This leaves PHP, Perl and Python. Let's just say right now that Perl is a piece of junk for anything other than processing text files and really needs to be taken out back and shot. Why aren't people doing Perl anymore? Because PHP exists... plain and simple.
So it leaves the argument that Python, which by all accounts is a formidable language, good enough for Google, and it should be good enough for most things, right? Why didn't they use Python -- I think I can answer that question. At the time FB was being built, Python simply wasn't all that popular. PHP had grown into a behemoth that was being sold as the "scripting" language of the Web, you could find cheap hosting with anyone, and usually it supported PHP -- with Python, it was a different story.
With that said, people who crap on PHP need to understand, there's always something "better", but its usually about the best tool for the job. There's nothing wrong with PHP at all, its a good language, and its very scalable. Making jabs at PHP is really dopey because we could sit here and make the same dopey comments about Ruby or Python...
I just recently had a conversation with someone about why I didn't use Ruby for a large project -- I said simply: Because this application needs to run on commercial hosting systems easily. I can't be sure that Ruby is going to be running on those boxes, but I certainly can assume that PHP is...
The same goes for .NET, is it bad? No its not, but would I write a Web app like Facebook on it? Hell no. But would I write a business application thats used by thousands of employees that have to integrate with SitePoint, Outlook,. Exchange? Sure would...
Its good to know that FB is doing something with its slow loading pages and script failed messages. Its annoying sometimes when your messages aren't posted due to script fail.
With this project on the go, will FB begin to charge its users? I hope not.
@Jolie O'Dell
Your right. They have released Thrift and Cassandra.....but they have released members data and cotent from the fb silo.....and they do not share the hundreds of millions in revenue that they generate off of the member owned and created content....
It's about time. Other solutions like this have been attempted, but none of them have been viable.
It takes a project with the volume of Facebook to provide real statistics that lead to art-expanding moves like this one. Anyone else who might have tried something like this would be guessing, and anyone who doesn't use real data when forming their opinions is likely wrong (as cool as they sound).
Thank you Facebook for providing this advancement to human technology.
Cool,
I guess the JVM and CLR are years ahead of the game. Microsoft should now put some effort into Iron PHP running on the DLR.
It is always good to hear about huge performance improvements. The energy savings will once save the planet.
Klaus
@Derrick I agree with you
http://puppy-accessories.blogspot.com
It is always good to hear about huge performance improvements. The energy savings will once save the planet.
i like it i man brebes IN
It's a good thing that Facebook is releasing their project Open Source.
It'll be interesting to see what becomes of this. Remember some years ago Yahoo! tried to do the same thing with PHP but failed at it.
If this works, it'll be great for PHP a lot more developers will develop Apps for PHP.
1 2 Next