As we close down 2011, we can reflect on (and learn from) the numerous, high-profile outages that occurred: Bank of America in March; Amazon EC2, Verizon LTE and Yahoo! Mail and Microsoft in April; and then Apple and Microsoft in August. In analyzing these disasters, I've come up with four lessons to be learned - they'll help protect your company's reputation, technical integrity and customer satisfaction during technical crises.

Lesson #1: Your company's brand is on the line

IT systems are not just internal systems anymore. Most companies experienced their first painful lessons with the advent of web sites and ecommerce. But today, it seems that every company has a growing amount of exposure to potential service outages that result in many unhappy customers. This said, it's critical that IT and line of business executives continue to get more aligned.

We must also realize that systems have a tendency to crash at inopportune times. Look at the Verizon LTE network outage in April. The company's fastest network, the LTE network was unavailable for customers and LTE devices were unable to be activated. The crash happened just 24 hours before the latest 4G-LTE smartphone, the Samsung Droid Charge, was scheduled to launch. The outage delayed the launch by two weeks and no doubt had a significant impact on its sales and reputation.

Lesson #2: Be proactive

Given the potential losses of network service outages, one might think that IT execs are totally focused on preventing major outages. But in my experience, they're not. The key issue that "prevents preventing" outages is the infrastructure and application monitoring systems in use today. Many were architected when a company's IT environment could still be visualized on a couple of PowerPoint slides. Their designs were based on the idea that IT experts would define the performance thresholds, rules and exceptions necessary to identify unacceptable behavior. But today, the typical enterprise application infrastructure is so complex that it defies an IT organization's ability to fully understand. The result, unforeseen outages that often take days to resolve.

These monitoring systems are still great for generating the data required to understand the systems behavior - just ask the operations center that receives tens of thousand of alerts a day. But the real challenge lies in making sense of the alerts, and taking the right action to resolve the inevitable issues quickly.

Lesson #3: When crisis strikes, communicate early and often

Face it - we live in a 24/7 world and your customers know when there is a problem. It's best not to ignore it and hope they don't notice.

When the Microsoft cloud crashed in September, they kept customers in the loop, promising updates at precise time increments. The official Windows Live Status site read, "We're aware of a problem with Hotmail that's affecting some people. We're investigating and will provide an update by Sept. 9 11:30 p.m..."

RIM failed twice over - responding several hours into the crisis and providing little details to ease customer angst. "We understand the frustrations our customers are experiencing through the delays with the messaging an browsing...I'd like to take this opportunity to apologize unreservedly to all those people affected by this situation. We're taking this situation extremely seriously and we're doing everything we can to restore normal operation to our service, " said CTO David Yach.

Lesson #4: Make amends

RIM eventually offered users $100 in premium applications and in some cases, free technical support for a month. While the costs to support the offers are likely high, it is likely worth it... It's also important to remember that if you don't provide compensation for customer inconvenience, your competitor's will. The day that Yahoo! Mail crashed in Aprill 2011, Microsoft wasted no time offering annoyed customers something to make them feel better. The official Hotmail account tweeted, "First 1k #ymail users to testdrivethenew@hotmail.com and send feedback today get HM+ free for 1yr. SwitchToHotmail.com."

Don't let your competitors be your customer's solution to your outage crisis.

With major companies like RIM, Amazon, Microsoft, Yahoo!, Bank of America, and many others all experiencing major network outages this past year, it's time to realize that it isn't a matter of if your IT department will someday face a crisis, but rather when your IT department will face a network crisis. Be prepared and have a plan for how your company will react and respond from both a technical and public relations perspective to minimize the aftermath.

As we look forward into 2012, we wish you smooth running networks and fast resolutions to the challenges coming your way.

Update 11:45 a.m.: Many users are reporting that service has returned (and we're back online at RWW). Others still having trouble.

Update 11:52 a.m.: @Gmail just tweeted the all clear but quickly deleted the post.

Update 11:56 a.m.: Okay, now we're clear.

We've put together Twitter reports confirming the outage using Storify, and we'll update as we learn more about what went wrong.

The Docs team pushed a change that was "designed to improve real time collaboration within the document list." That sounds like a good idea. Unfortunately, it revealed a big old memory management bug that they couldn't detect until it was exposed to the full force of Google Docs users. Basically, the machines that check for updates didn't clear their memory properly, so they filled up and crashed, shifting the load onto other machines, causing them to crash, and away we go. The team caught the problem within half an hour. It's worth reading the blog post to see exactly how.

Downtime is the bugaboo, the monster under our bed at the dawn of the cloud era. No service is 100% reliable, but cloud services are becoming more and more vital to keep our businesses running and our sites up. A cloud service provider's handling of an outage is absolutely crucial to keeping its customers happy and earning their forgiveness. But since outages usually require detailed technical explanations, they are often left to engineers whose tone might not be as gentle or apologetic as can be. When Amazon's EBS hosting services went down in April, bringing some of the Web's most important sites with them, the explanation was long-winded and dense, and the fallout was not handled well. Warren's post today couldn't be more different.

How have cloud outages affected you? Tell us your stories in the comments.

Twitter's support team confirmed the issue at 11:42am EST, but have not updated since, leaving many users frustrated and unsure of why they can't see replies. Among the flustered is American actress Felicia Day, whose tweet about the outage garnered a number of replies that she, ironically enough, cannot see.

A direct message sent to Twitter Support asking for an update was not responded to by the time we published this post. We'll update this post as we learn more.

While this was obviously a major problem for the company to solve, I wonder: did you notice? I can't say that I did. I haven't scrobbled in months on end. I can't remember the last time I even streamed music from Last.fm. I've moved on to bigger and better things...have you?

According to the Last.fm blog post, the issue was caused by a hardware failure. Yesterday afternoon, a fault in a blade chassis in one Last.fm's datacenters broke, and took down the power supply for its rack with it. The onsite teams couldn't resolve the issue with the chassis, but managed to restore power to the rest of the rack. However, the chassis had contained several critical components of the top-level load balancing systems, which are used to evenly distribute traffic across Last.fm's data centers.

Because the remaining data centers were then running under a higher than usual load, outages began to occur. There were problems not only with the radio service, but also with scrobbling and the website itself.

Now, it's a matter of waiting for fresh DNS information to propagate around the Internet. When that occurs, the service will be up-and-running for everyone. Last.fm users should know that scrobbles are safe in the meantime, thanks to client caching.

So, Did You Notice?

Obviously, plenty of regular Last.fm users did notice the outage, and were tweeting about it over the past day. But for me, the outage only served to remind me that the service existed at all. I no longer care for streaming radio services like Last.fm - or Pandora, for that matter - I'm using a subscription based music service instead, where I can access (nearly) any song I want on demand, create my own playlists and browse through playlist suggestions from others. You want "cloud iTunes," you say? Well, you can have it now.

In my case, I'm using MOG, a subscription-based streaming music service preferred by a couple of us here at ReadWriteWeb. However, Rdio, a similar service from Skype, KaZaA and Joost creators Niklas Zennstrom and Janus Friis, is the better choice, according to many whom I've spoken to.

It comes down to what features you really need. Rdio, for example, will match your iTunes catalog with its own, providing easy access to your favorite tunes. It also offers better social networking features for sharing recommendations via Twitter and Facebook. And its mobile applications are well-designed and easy to use.

MOG suffers a bit in the design aspect of its apps, and according to some reports, their stability too, but its catalog is currently larger: 10 million tracks to Rdio's 7 million. However, these numbers change regularly, as record deals are brokered.  If you start to feel like you're missing the serendipity that services like Pandora and Last.fm offered, both MOG and Rdio provide a "radio" option that lets you stream music, but MOG's more configurable, much to the disappointment of some Rdio fans.

Although neither of these services are free, like Last.fm is, they're both definitely worth the money. And frankly, since the time I started using subscription music, I haven't been back to Last.fm at all.

But I wonder where the rest of the early adopters are with this. What are you using these days for streaming music? Are you still a Last.fm fan? Or have you also moved on to MOG, Rdio, or - if you're fortunate enough - Spotify? Or maybe you still prefer iTunes?

As we now roll into day two of the "great social media outage of 2009," you may be surprised to learn that it's not over yet. Although Facebook and LJ have recovered, Twitter is still having issues. Not only was the site down once again early this morning, Twitter developers using the API are complaining that the company is sending mixed messages by reporting that they're "back up" - when in reality many Twitter applications are still unusable.

Twitter Developers Want Communication, Too

In a recent discussion thread in the Twitter Development Group, Twitter developer Jesse Stay of the popular SocialToo application asked:

Other developers quickly chimed in agreeing, noting that Twitter has yet to communicate the API status to developers, leaving them in the dark as to when their apps will work again. It seems that some developers have been experiencing issues with everything from oAuth sign-ins to timeouts to applications being completely down. Paul Kinlan, developer of the Twollo application, an app that helps you find followers with similar interests, even said that he had to refund a paying customer because of the situation. No doubt he is not alone.

While the developers are generally sympathetic to the situation and understand that fighting off the attack is priority number one, what they're finding hard to deal with is the lack of communication. Throughout the attack, Twitter has updated their Status Blog with notes about the service itself, but nothing about the API. Developers are frustrated and unsure of how to address the situation with their users given that Twitter has not provided any official information to them either through the blog or their own Twitter account.

We know that Twitter's architecture has made it more vulnerable to this type of attack than Google or even Facebook, but in situations like this, communication is key. Hopefully today Twitter will do one of two things: either (preferably) stabilize its service and API or (at the very least) let developers know the status.

Update: Looks like Twitter listened: http://status.twitter.com/post/157979213/restoring-api-and-sms.

Update 2: See also our coverage of Cyxymu, the apparent target for these attacks.

To make matters worse, Facebook is also experiencing issues this morning, as many rebuffed Twitter users are now finding out when they go to post their status on the social networking site instead. The Facebook outage appears to be intermittent, though, and isn't affecting everyone. If you notice anything at all, it may be only that posting status updates and comments displays an error message. But simply clicking the post button again may force them through (at least that was my experience). The site also loads slowly at times and displays occasional "transport" errors.

Ironically, you can see several complaints on Twitter about this issue right now when doing a Twitter search with the keyword Facebook.

As a last resort, some users may turn to their LiveJournal blogs and attempt to update their mood to "sad" to reflect their feelings about this odd cyber-outage. Unfortunately, they'll be foiled there as well since LJ is down, too.

Some users are already spreading a rumor that this is some sort of social media attack by hackers, but it's more likely just a bizarre coincidence. Maybe the universe just wants us to actually get some work done today.

Typically, these outages have never lasted for long, but once again, this outage shows how dependent we have become on Google for so many of our daily tasks.

As of now, all of our attempts to contact any Google property are timing out, but we'll keep you posted once we find out more about this.

Update: Judging from the comments here and the conversations we are tracking on Twitter and OneRiot, it looks like these outages are somewhat localized, though we haven't seen any patterns evolve yet. It does seem, though, like the outages are mostly in the U.S., though we are also seeing some reports from European users while Google seems to be working just fine in most Asian countries.

Currently, this looks like a networking issue, as users on some ISPs are able to access all of Google's services while their neighbors are unable to connect to Google's servers.

There were some reports that this issue was specific to AT&T customers, but this author uses Verizon FiOS and experienced the same issues. There is a chance, though, that any traffic that was routed over AT&T's network was delayed or dropped, but we are still looking into this.

Update 2 (9:10am PST): Looks like our collective nightmare is coming to an end - more and more users now report that their services are back up and running again. Here in Portland, OR, the outage lasted for just over an hour, though Gmail is still only working intermittently.

Update 3 (9:50am PST): Google just contacted us with this statement, which, even as these statements go, says very little: "We're aware some users are having trouble accessing some Google services. We're looking into it, and we'll update everyone soon."

Update 4 (10:20am PST): According to Google's official Twitter account, the issue has now been resolved.

As many other sites use Google Analytics and/or Google's APIs, this outage does not just affect Google, but has also brought down a substantial number of third-party sites that rely on Google's services.

If Google is working for you, or if you are also experiencing outages, feel free to vent in our comments section, or use this as an opportunity to check out some alternative services like Yahoo, Live Search, or DuckDuckGo.

According to the company, during a routine upgrade on Friday night, they ran into some problems with photo storage. The issue appears to have stemmed from several drives failing at once. Because simultaneous hardware failure such as this is rare, Facebook says they're still trying to figure out what happened.

In the meantime, though, the photos are being copied to new drives - a process that can take some time due to the large amount of data that was affected. The company asked users not to worry because they store photos in a way that maintains multiple copies of the data in case of hardware failures such as this. By early this week, everything should be back to normal.

The Cloud is Not Perfect

This recent issue with Facebook photos is just one of many cloud-based outages and issues we've seen recently - a trend that moved some to question the level of trust we should have for these online services. Over the past few weeks, we've seen the social bookmarking site Ma.gnolia completely fry and lose all their customers' data when the service's database crashed and took with it a half a terabyte of information. Unfortunately, no reliable backups were to be had.

But while Ma.gnolia may have represented the dangers of trusting a small startup with your data, Google has proved that even companies as large as themselves are not immune from problems. A recent four-hour Gmail outage from routine maintenance "gone wrong," caused a cascading failure in Google's European data centers.

And on the heels of the Gmail outage, Google Groups also went down for some time. All groups were affected for a short while.

These incidents led Google to follow in the footsteps of other cloud storage companies like SalesForce.com and Amazon in the launch of a Google Apps status dashboard so they could better communicate with customers whether their online applications were up and running.

In Google's case, repairs were made and customers were credited as necessary, but even so many businesses and individuals were affected in ways that can't necessarily be quantified so easily. Although some quickly rallied to Google's defense, reminding that Gmail's uptime is often much better than on-site hosted email systems, the point many folks are missing is that unlike in "the old days," not everyone keeps copies of their data on their computers anymore - so when the cloud shuts down, that data is just gone, albeit only temporarily in most cases.

Be Safe, Use More than One Service!

Still, even if you've switched over to cloud services for storing and accessing the majority of your data, you can prevent outages from affecting you. The trick is to store your data in more than one online service or use a hybrid cloud/desktop solution. For example, if your email is mission critical, use an IMAP-enabled desktop or mobile client. Those doing so during Gmail's outage were able to access their inbox to retrieve old emails - they just couldn't send and receive.

For photos, like the ones that recently went missing from Facebook, there are a number of online services where they could have been stored. Today, there's really no reason to only keep your photos in one spot. An easy way to upload photos to multiple sites is to use a tool like Pixelpipe which shoots photos, videos, and audio files to over 60 social networks, photo/video sites, and blogs.

Even documents can be stored in more than one location. Google Docs and Zoho may be the best known of the web office services, but you could also keep critical files saved to your computer then backed up using another third party backup service like Mozy. Or you could upload files to storage sites like Google's own Google Sites or Microsoft's "Sharepoint Lite" Office Live.

No matter what you do, there's always a chance of losing data, even if you only save files on your computer, completely eschewing the cloud altogether. But that's probably not as safe as keeping files in multiple online services. With the number of services available, an outage shouldn't mean we have to lose access to our files. Use hybrid solutions or spread your data across multiple services instead and you'll almost always be okay, outage or not.

What do you do when you're left waiting for a meeting or a lunch? If you're like us, you probably check your email. What do you do if your email keeps you waiting? Something else! What better excuse could hundreds of thousands, if not millions of people, have to do something else with a few precious hours than to say "all of Gmail is down?" We presume that books were read, "to read" lists were plowed through and who knows - maybe a few babies were conceived!

The fact is though that GMail is just so good as a product that even with hours of downtime, even with our concerns that Google is Big Brother waiting to happen, even with the company's silence about regular hours-long delivery delays - we just can't quit Gmail.

Now that Gmail is back, though, maybe it's a good time to go install this Greasemonkey script to replace the Gmail logo in your email inbox. It's a little complicated but the next time the service goes down at least you'll have been trained to remember what's really important in life. Not Gmail.