In 1975, Frederick Brooks wrote a classic book on software project management called
The Mythical Man-Month.
In the book, he famously argued that adding more people to a development project will hinder rather than help to get things done faster.
The reason is that having more people working on the project introduces a non-linear overhead in communication.
Five years before Brooks' book, a software development methodology called the Waterfall Model was coined. This approach applied the insights from mature engineering disciplines (mechanical, civil, etc.) to software. The idea was to construct systems by first gathering requirements, then doing the design, then implementing it, then testing, and finally getting it out the door in one linear sequence.
We have come a long way since then and learned a lot about making software. The Waterfall Model is now considered a flawed method because it is so rigid and unrealistic. In the real world, software projects have ill-defined and constantly evolving requirements, making it impossible to think everything through at once. Instead, the best software today is created and evolved using agile methods. These techniques allow engineers to continuously re-align software with business and customer needs.
With the advent of modern programming languages (Java, PHP, Python and Ruby), rich libraries, and unprecedented infrastructure services like those from Amazon, we are arriving at yet another evolutionary step. Digg, del.icio.us, YouTube and other poster children of the new web era were developed by just a handful of programmers. To build software today all you need is a few good men (or women!). In this post we trace how we got here and where we are heading next.
Non-technical people tend to think that software is soft or easily changeable. Since there are no visible nuts and bolts and no hood to open people think that software can be tweaked and re-wired on a whim. Of course, this is not the case. Software, like any mechanical system, has a design and the structure; it is not as soft as it seems.
Yet the accelerating pace of business requires constant changes to software. Older development methods completely fail to address business needs. Using the Waterfall Model, these changes were impossible, the development cycle was too long, systems were over engineered and ended up costing a fortune, and often did not work right.
The problem was that the Waterfall Model was arrogant. The arrogance came from the fact that we believed that we could always engineer the perfect system on the first try. The second problem with it was that in nature, dynamic systems are not engineered, they evolve. It is the evolutionary idea that lead to the development of agile methods.
In the early nineties a number of agile software development methods emerged. While they differed in details, they agreed at large that software development needed a major rethinking. First, software has to embrace change. Today's assumptions and requirements may change tomorrow, and software needs to respond to changes quickly. To meet the challenge, agile approaches advocate focusing on simplicity. Make the simplest possible system that satisfies today's requirements and when tomorrow comes, be ready to adapt.
Two techniques pioneered by agile methods are worth particular attention - refactoring and developer testing. Refactoring, elegantly described by Martin Fowler in his classic book is the idea of improving the design of the existing code without changing how it works.
Refactoring is what allows agile systems to embrace change, while remaining elegant and free from rot. Like an interior decorator continuously changes and improves the layout of your furniture, agile developers move code around to improve the product as a whole. Code is constantly changed to make sure we end up with the simplest, and best possible system that reflects our current needs.
To make sure that changes do not break the code, agile methods introduced unit tests. As each agile project unfolds, it grows the base of unit tests. Each test is focused on a single component of the system and acts as an insurance that the component works as expected. Typically, the tests are run continuously against the code and require immediate fixes in case of a failure.
The software systems created using agile methods are much more successful because they are evolved and adapted to the problem. Like living organisms, these systems are continuously reshaped to fit the dynamic landscape of changing requirements. Without a doubt, agile methods made a major impact on how we think about building software today - dynamically and continuously.
While we discovered better way of making software, we also discovered better programming languages. C was replaced with C++, then came Java. Perl was great, but PHP and Python took its lessons further. More recently came Ruby, which has become very popular because of its natural way of expressing code. Because of this evolution, today we have a number of excellent, and virtually equivalent programming languages.
While the choice of programming language is typically a sensitive subject, the truth is that it is not the language, but the libraries that come with it that make a difference. C++ never had the standard libraries that Java has. Yes, Java is the simpler language, but people used C++ for a decade just fine. What gives Java the edge is its rich set of reusable libraries. The story is similar with PHP. It has been the language of choice for web developers precisely because it comes with such rich support for web and database processing.
In addition to the libraries that come with modern languages, the open source movement has also contributed a wealth of code towards global software infrastructure. Notably, just the Apache foundation on its own has created a huge amount of high quality reusable code. We have now arrived at an age where we have a strong foundation for building complex software systems. We know the methods and we have the tools, so what does that mean?
Since early days of software development people struggled to build good systems. More and more people where thrown at the problem, making matters worse. But with the recent explosion of social web we've witnessed a new and interesting phenomenon: a handful of developers are now able to build systems that are used by millions of people. How can this be?
The secret is that as with any good endeavor it only takes a few good men (and/or women!). With a bit of discipline and a ton of passion, high quality engineers are able to put together systems of great complexity on their own.
Equipped with a modern programming language, great libraries, and agile methods, a couple of smart guys in the garage can get things done much better and faster than an army of mediocre developers.
We are likely to see a few changes over the coming years:
Ironically, we are coming full circle with the mythical man-month. What was true twenty years ago is true of course today, but for a whole new set of reasons. An awesome array of programming languages and infrastructure libraries combined with agile methods has allowed us to break free of old software development dogmas. Just a handful of great engineers can now successfully build systems of great complexity. Craftsmanship has finally come to software engineering!
Listed below are links to blogs that reference this entry: The Future of Software Development.
TrackBack URL for this entry: http://www.readwriteweb.com/cgi-bin/mt/mt-tb.cgi/1704
I don’t generally comment on this blog about other blog entries but this was really good. Alex talks about some history of software development and the Waterfall Model vs. Agile Methods. Alex did a great job so go read it at Read/WriteWeb. ... Read More
Alex Iskold, founder of AdaptiveBlue, consistently churns out articles that provide interesting and thorough analysis of trends in technology. His latest, on the future of software development, continues that pattern. Software engineering is a complex... Read More
I found a cool article through digg that supports a lot of the way Rob feels about software development. An excerpt from the article: Ironically, we are coming full circle with the mythical man-month. What was true twenty years ago is true of course to... Read More
WebAlive Systems Descriptions & Rules Confluence Jira Agile Discussion Agile I Some Background Mike Cohn on Agile Estimating Part I Read More
Sponsor: Here is a summary of the week's Web Tech action on Read/WriteWeb. Note that you can subscribe to the Weekly Wrapups, either via the special RSS feed or by email. Note: I've just arrived back home after my latest... Read More
Alex Iskold wrote this nice entry about the "future" of software development. The future is now! He wrote about agile web development and its roll in software development. You can check out the article here. The article goes into detail... Read More
Comments
Subscribe to comments for this post OR Subscribe to comments for all Read/WriteWeb posts
Google is a great example of this, especially using advanced tools to make it vastly easier for developers to accomplish complex tasks.
Posted by: Steve Severance | October 16, 2007 2:47 PMThanks for the great post.
I've always been a big fan of the Agile approach. I've seen others say preach about it and when times get tough or projects need to be finished in a hurry (always.. right?) they fall back to the compfort zone of the Waterfall method.
Posted by: Mike Austin | October 16, 2007 3:06 PMGreat post! We have customers that develop software, work with many software companies whose products we sell and have developed some internal applications ourselves. Some days I curse software and technology and some days it just makes me giddy like a kid.
Cheers,
Michael Rowles
Posted by: CopiaTECH SMB Security | October 16, 2007 3:30 PMCopiaTECH
Great post on Agile Development Alex! With regards to the waterfall method, another major shortcoming of it is that it was applied (as you said) from established engineering disciplines. In a major civil engineering project, it is essential to nail the design on the first run. The physical and financial constraints make a design revision extremely impractical. Software, however, CAN be modified on the fly with little to no cost (relative to modifying concrete designs of physical structures.) Because of this difference, software design necessarily requires the team to adapt to changing constraints. If your team doesn't adapt, somebody else's will. This simple fact has had a major influence in forcing the software industry to adopt evolving design methodologies, especially with the pace of innovation that the web has enabled.
Posted by: Benjamin DiGregorio | October 16, 2007 4:27 PMgreat read, one problem I have with it was no mention of any Microsoft tech... I can get a quality web app up and running in a short time with user management, caching, curd etc with writing little code thanks to the .net framework and a default install. I'll be honest I see a lot tech bloggers not getting how much power .net 2.0 has. Sure a lot of developers were not that impressed with .net 1.1 but with 2.0 and beyond the features make using it a much smarter option. People need to get beyond their apple/google tinted glasses and actually speak to the people that build with these tools. Anyone who says .net 2.0 is not up to standard has not used it for any length of time and is most likely just a day coder.
Posted by: Darren | October 16, 2007 5:04 PMhey , "if you rebuild they will come" - concept is going to be the next wave of s/w commissioning..innovating the innovations process is going to real exciting when you factor in the quick and dirty development to suite the community .. the biggest hurdle I see is the rub between the legacy thinkers vs the new and dirty s/w methods for adoption at the Enterprise level !!
great post ..
Posted by: /pd | October 16, 2007 5:07 PMDarren: i read you loud and clear with .net framework. MSFT is has come a long way to get to where they are now .. and IMO are tightening the grip in this space of agile. With Project Orca /MSFT eSCRUMv.10 and net3 in the making, it surely makes agile a much more suitable and smoother process for the enterprise to engage into.. Iwon't be surpised if MSFT just manage to once pull it off.. yeah the reboot effect !! :)-
Posted by: /pd | October 16, 2007 5:20 PMAlex, why do you think software offshoring will make less economic sense?
Posted by: Peter | October 16, 2007 6:12 PM@Peter
When off shoring started the costs were 1:10. Nowdays its more like 1:3. Given that off shoring has a lot of issues, communication being a major one and that with great tools like I discussed in this post, it just will make more sense to hire a few good developers locally instead of many over seas.
Alex
Posted by: Alex Iskold | October 16, 2007 6:22 PMAlex : a good e.g on TOC of offshoring ..is this
Posted by: /pd | October 16, 2007 6:56 PMThere are lots of ways to look at this phenomenon. Is agility part of the (undocumented) requirements for the successful examples in this article? Certainly, the ability to prioritise project elements has been key to successful software development from the outset ... but the pioneers of software engineering did not recognise it as such.
Posted by: Bob Buckley | October 16, 2007 7:08 PMThe waterfall model does not show that some requirements and design decisions can be deferred ... another aspect of agile development.
Software reuse is not an aspect of the waterfall model. The best agile examples use large chunks of existing software ... and the development of those existing software libraries were not necessarily as agile as the products they are used in. The existence of such code helps agile programming of user interfaces for useful combinations of reusable modules.
Yes. The libraries are significant. The agile projects of today are tiny compared to projects that depended on serious software engineering in the past. Perhaps that's the real difference :-)
opps...
http://munjal.typepad.com/recognizing_deven/2007/04/episode_26_indi.html
Posted by: /pd | October 16, 2007 7:22 PMGreat link, pd! Everyone should take a look. Pay attention to the costs part.
Alex
Posted by: Alex Iskold | October 16, 2007 7:27 PMHi Alex - slow day at the office? ;-)
Here's a quote from Alan Kay (it's in my talk slides BTW) that I think you'll like. I guess everyone knows the first part:
He went on...
As you point out, what was true in the sleepy old 20th C, is even more true today.
Terry
Posted by: Terry Jones | October 16, 2007 8:00 PMGreat post, Alex. Sorry if I missed them - but I'm interested in seeing some real-world examples of software systems produced using agile methods - did you, or any commentators, include any references to shining examples (all I ever read is 'agile' + '37Signals' + 'Getting Real')?
Posted by: Neil | October 17, 2007 12:51 AMNice article Alex,
IMHO Thanks to use of libraries we start to play more of the role of integrators. This is great as highlights the fact: integration is necessary and important. It is a great growing field, where new applications and ideas of how to put them together are growing. But what is even more important, by employing specialized libraries, we can focus on the core of the problem rather then build necessary tooling, information pipe-lining from scratch.
BTW: We use agile methodologies in our team and with success;)
Posted by: KBac | October 17, 2007 1:47 AMYes, and going further, according to Charles Simonyi, one day software might even write itself.
Posted by: The Business of Software | October 17, 2007 3:15 AMGreat stuff!
I think somehow we all are following the Agile programming approach if not fully the of course partially. Or may be some of us already following is completely but still don't know it is called as Agile programming. So, nice introduction for them who don't about it. Even I read about it yesterday in some different book (Agile programming with Springs). I remembered the time when I had studied waterfall model during college days but never followed seriously during work. The things kept on changing from client (even today things are same, client need more and more changes) and gosh we had to change the code/design again and again. However, thanks in the end as we always finished the software on time and with sufficient satisfaction of the client through this fruitful approach.
Posted by: Rock | October 17, 2007 4:24 AMI can't see anything you've described not being possible by following the waterfall method. Somehow people have turned the phrase agile development into a mythical beast capable of achieving miracles.
Nearly all of the things you describe as characteristics of agile programming are also core to the waterfall, it's all in level of detail you go to within and across each phase. Skilled practitioners will adapt their execution of a waterfall delivered project according to the reality of the environment. The reverse seems to be the case for agile, where the focus is more on executing the lifecycle in a fixed way, irrespective of the project or conditions.
You can be agile following the waterfall approach by breaking your project up into manageable releases, based upon functional scope or depth. Even large waterfall projects are generally preceded or followed up by a range of short cycle releases.
Agile to me seems to be a creation of the lazy analyst or developer whom resents having to operate in a disciplined way and truly believes it's not worth trying to define as much detail up front as possible. I would love to see some real quantitative statistics comparing like for like with these approaches.
Most of the evangelists I've come across have left their roles before they've ever witnessed the outcomes of their influence on the project execution. I'm sure it seemed cool to them at the time but everyone else is left to clean up the wounds as stakeholders realise core functionality is missing, has been misinterpreted or is way over schedule because too much time has been spent on the beautification of code instead of delivering what was required.
Posted by: Waterfall | October 17, 2007 4:51 AMOh - and who said that just because you're following the waterfall approach you can re-use existing code, frameworks or 3rd party components? That's nonsense.
Posted by: Waterfall | October 17, 2007 4:56 AM"Agile development" is a cool term, like "extreme programming", "web 2.0" and "mashup", but its applicability depends on the scale and nature of the project.
Case in point, I've worked for five years on a project for a financial institution. The 'agile' approach has done more harm than good, with severe performance issues and delays in implementation because there was no proper planning.
You may know this cartoon: http://ravenyoung.spaces.live.com/blog/cns!17376F4C11A91E0E!1574.entry
Sad but very true.
Posted by: Gaspy | October 17, 2007 5:11 AMWhile I think your option makes sense when it comes to the youtube/facebook/twitter set I don't think that it really applies to software development as a whole. The reason for this is the mismatch of those who embrace agile development and those who see it as an unstructured, unmanageable and unplannable approach to software development. Most of those people are in management and at lot of them do not understand never mind support development which is not easily mapped into MS project. but give it a few more years and this will change...
Posted by: Noelo | October 17, 2007 5:13 AMGreat post! I agree with all you said except this one:
"The developers who do not have great programming skills are going to have to look for jobs elsewhere."
I am project manager for several years now and I strongly believe that there is room for all kind of developers. Very often the attitude is the one that makes a difference, not the skills. But if you want to build brilliant software you need the skills.
Posted by: Adrian | October 17, 2007 5:15 AMCome on! Is really the "The Future of Software Development" some bullshits about frameworks/libraries or about software developement methods ? You're still thinking software development like on "old programmer", to write a software you need to write some lines of code in that way or that way or whatever method you choose... What about the fact that the more we develop software, the more we use blackboxes, we assemble them, we handle them. What about the future of software development something where lines of code would be useless, just connect the boxes... Your post talk about the past and the present not the futur.
Posted by: Enough | October 17, 2007 5:18 AMThis is a great post and it is true that software development has evolved a great deal. I just want to point out that in addition to good programming skills a successful software developer has good communication skills. Regardless of how small or large your team is I believe this is what separates good developers from the rest. I have expounded about this a bit more here.
Posted by: Nikhil | October 17, 2007 5:26 AMGreat post - I am a fan of Agile methods, but the one problem I have with it is how do you accurately provide cost estimates for an Agile-based project?
With the Waterfall approach of assuming you know all requirements from the start, at least you have something to estimate against, even if you have to revise that during the project.
I can see how Agile is great for internal projects, but unfortunately clients always want to know things like cost, delivery date etc up front...
I'd love to know your thoughts on this?
Posted by: Gareth | October 17, 2007 5:41 AMAlex, your best post yet! Puts it very well in perspective. Mythical Man Month was the first book I read on the industry in 1980 and it is still the best. In my experience all good software PRODUCTS were built using agile methods (even before the term was coined). True story. In 1983 I was leading a "bet the business" product development for a UK trading systems product shop. Advanced stuff for the times - in-memory data caching, real time event driven updates, publish/subscribe. The company allocated budget for 10 developers. In reality 1 guy hacked up prototypes to show stakeholders and 1 guy wrote great code that is still in use today - not sure what the other 8 did!
Posted by: bernard lunn | October 17, 2007 6:12 AMWhat is truly amazing is that 32 years after that booked decried the Waterfall Method it is still the primary use by hundreds of thousands of developers building and maintaining those old enterprise systems.
Nice post Alex! But... Made me remember an OOOOOLLLLDDDD article on Byte Mag about OO techniques with the title "There is no silver bullet"... My point is that Digg, Google and other projects done with "a handful of talented programmers" might be not a good example for those people working with in-board avionics software, or CAD/CAM simulation appls or even with some highly transactional business applications. Agile methods, code libraries and talented people will ALWAYS be IMPORTANT, but still NOT ENOUGH to most projects. Don't you think?
Very best regards!
Posted by: Clayton Costa | October 17, 2007 7:10 AMSoftware developers have always had huge problems when working in large teams. That hasn't changed at all. In many ways Agile is just dodging the real issues.
The development of many of the systems you mention comes from a significant increase in the amount of code that is available to developers. The languages are actually more convoluted than they used to be, and the technology is definitely messier and less trustworthy. Users expectations have dropped as well, as they have gotten used to dealing with more problems from their increasingly broken software. The foundations have gotten worse.
It is accessing tonnes of code and the information available on the web has made it possible for small teams to write larger systems, but that is not necessarily a good thing. The side effect is that it looks easier to write systems, so salaries are coming down and with a glut of available code, it is quickly becoming a commodity. The future of software development seems to be in quickly gluing together lots of small broken pieces. The days of actually making it work correctly are rapidly disappearing.
Posted by: Paul W Homer | October 17, 2007 7:59 AMYou are partly right. It is true that good libraries create a revolution in software engineering. Software engineering is incrementel and therefore we can stand on the shoulders of giant. At least at API's are interoperable and run anywhere (java).
Ok, some good software is written by only a few people. But that doesnt mean that this is a correlating factor. I think good software libraries and good interface (software interface APIs, docs) enable better systems.
Posted by: Tjerk Wolterink | October 17, 2007 8:18 AMI'm glad .NET is ignored here. Microsoft is a great, evil, subversive empire of cash which will not necessarily embrace agile methods because it's good for Software Engineering, but will more likely squeeze, strangle, suppress, smother, extinguish and then clone from the carcass agile methods because it's good for their market share. I have a very hard time reading about structures in .NET that are copied from Java and "improved on". I'm only happy to discover later that the stealing only led to real problems. read: link from ACM Queue. Makes me laugh. There is a moral problem when you can't generate structure on your own, and so you steal it from others. I won't align myself with this, and neither should you.
Recently, my company was approached by Microsoft to "renegotiate" our license fees for SQLServer. The rate was going to double, approaching $70,000 a year for a small web application used by less that 500 clients. I opted to port the app to PostgreSQL, and we used the Microsoft license fees to hire a new engineer. We picked up an Open Source guy. And so it goes at my company.
Posted by: Jason Michael | October 17, 2007 8:33 AM"High-quality, passionate software engineers will be in very high demand and will make substantially more money." Can this statement be backed up with reliable data? What is the threshold that turns coders "do not have great programming skills" into a "high-quality, passionate software engineer" and what number of people constitutes "very high demand" and how is it different from the situation now? "Software off shoring will make less and less economical sense." I would like to see this expanded, particularly in reference to the first point which said there would be "very high" demand for "substantially more money". The two seem opposites. "Computer science is going to remain a highly competitive and prestigious field." When has computer science ever been prestigious? These generalizations are not based on much, except an observation that people can now use slow, interpreted languages like Ruby instead of C. :)
Posted by: Scott | October 17, 2007 8:48 AMMy (limited) experience with Agile methodology has shown that it can be effective on smaller efforts (say a simple web application), or as the initial release of something will end up as a medium enterprise app.
But anything complex or large appears to be undermined by Agile as no documentation is typically produced. Even with small projects our customers have been disappointed because they felt they didn't get a lot of bang for their buck when they looked back on all the time they had to invest - as opposed to a more formal methodology where they are tapped for information at the start of the project, but someone else, such as a Business Analyst, does the lion's share of gathering, documenting, and managing changes to the requirements.
It also seems to over-tax Developers when they have to assume the other aspects of a project that are usually assumed by Project Managers, Technical Writers, and Configuration Managers.
Posted by: Yahmdallah | October 17, 2007 9:13 AMThe waterfall model was rapidly supersceded by the spiral model which is essentially the model you describe as being "revolutionary" today. Its been around for years. On another point, the latest sofware does allow for rapid development but for complex applications that require specific business purposes the work is still in the programming and requires larger teams to accomplish. AI systems, robotics, complex financial software, automated business softwares, automated factory softwares. For example, Siemens spends more on sofware development than microsoft and employ 27000 people to write software globally (http://w4.siemens.de/FuI/en/archiv/zeitschrift/heft2_98/artikel11/index.html). Theres much more to software than web 2.0, search engines and social networking sites. love_long_time@hotmail.com
Posted by: Jimbo | October 17, 2007 9:14 AMInteresting read! I have experienced the situation where more programmers are thrown in to try and save a project, it does not always work but I guess there are exceptions. good article!!!!!
Posted by: Ronald | October 17, 2007 9:28 AMWhile the article concludes that software development is best with just "a few good men", it misses out on the genesis of why that is the case.
Frequently, change is given as the reason why Agile programming works, just as it is here. However, change all by itself does not necessarily imply only a few people can effectively work together. There are lots of cases where we can break down change in a way that a hoard can work on it.
The real genesis for the ideas behind a lot of this came from Bell Labs' James Coplien, and have to do with understanding how development groups communicate. See my blog for more, as well as ideas for facilitating this to improve productivity:
http://smoothspan.wordpress.com/2007/10/17/to-build-better-software-you-need-fewer-people/
Posted by: Bob Warfield | October 17, 2007 11:15 AMI'd argue that people still struggle to build good systems. You just don't get to see the system, you see the product of the system. And the agile development methodologies have been around for a while. They were what good engineers eventually stumbled upon as a way to be better engineers. They are more accessible now because they have become mainstream.
But my biggest grip is the nearsightedness of the article, but i can not blame the author or contributers, it can be hard to see depending on what code you have looked at and more importantly - who you have worked with. Craftsmanship has been in software from the get go, period. Those who fail to grasp that fact simply did not realize it because they were not taught it. Why didn't they realize it? Probably because of the way people are generally taught. I didn't realize it myself until i was taught as a craftsman. By this i mean working with a senior software engineer (or a few) close enough and paying attentions to see the how and what of their craft. Once you start to see it, you can find code that has the telltale signature of a craftsman.
As for those people who do create good systems. They either learned by apprenticeship and hard work, and/or by being to impatient to find a book or someone to tell them how to do what they wanted to do and just did it over and over until they figured it out for themselves.
I would not say there is nothing new under the sun, but there is definitely fewer new things than most people want to admit. Many are morphed ideas of the past with a "modern" touch/language. This is important though, because it is a natural evolution, and at the end of the day it leads to progress.
Posted by: Tim | October 17, 2007 11:38 AMI agree, agile is the way to go!
Posted by: Chris May | October 17, 2007 12:45 PMThrowing more programmers at a problem is why mega companies like Microsoft have such a difficult time creating reliable, secure, usable code in an economical and timely manner.
Posted by: CVOS man | October 17, 2007 2:18 PMLydor Wyssocky from "the Mindset" once said: "If we would just stop using the names of one methodology or another in our discussions, most of our disagreements will disappear."
I've had the interesting opportunity to work on a high security software product. When I first arrived, we followed an extremely rigid development methodology: changes had to go through multiple levels of peer review (which is a very valuable practice) including a special review with focus on security.
Unfortunately, at the time, government procurement requirements for products such as this resist the notion of an evolving product and expected products to be fully designed up front, down to the pseudocode level (see the TCSEC, or Orange Book). To date, this situation hasn't much improved. Look into the Common Criteria Evaluation Scheme (CCEVS) to see the modern-day equivalent. Consider: by the time a product completes it's procurement certification and is deployed, if any substantial features have been designed or bugs have been found, the certification must in practice be done over again in order for the new version to be deployed. This refers to a process which can take half a year or longer!
This illustrates an important point in the methodology debate: ANY methodology ("it doesn't matter what you call it"), when applied with too much strictness, is going to resist the nature of software. When too much value is placed on any phase in the lifecycle, the evolution of the product is certain to suffer.
It's clear that the software industry has been trying to convince itself of an unfortunate delusion: that flaws can be designed, inspected, or tested away. In the public sector, these problems are exacerbated by bureaucracy and politics. While agilism has been an important step in "waking up" from this dream, the lessons learned from it will take time to propagate. Agilism is another step towards developing software better, and it can be better than the "dark ages" brought upon by misapplication of the waterfall model; but it is just another step in the evolution of software development. Developers will need to be flexible as deficiencies in the existing generation of methodology is discovered.
Posted by: PostAgilist | October 17, 2007 5:32 PM"Agile Dev" and all the OO stuff is just a bunch of mumbo-jumbo wordage meant to confuse non-techie management (and other techies even) and give fodder for dissertations.
Only the brightest on a team (usually 1 out of 10) can manage to produce well designed and implemented code sets. The others only produce mangled spaghetti in 11 dimensions with 7 conflicting sauces.
I recently worked on a large project for a large corp and it took 20 people to do the work of 2 on similar projects i've worked on a decade ago.
Software dev is in a state of crisis and reaching critical mass. Look for some new paradigm to fix the mess. Processes and flows are just that... processes and flows.
"Agile" is just an admission that the tools being worked with are teetering top heavy disasters waiting to happen.
Every new framework claims to ease the horrendous burden of the previous framework, which itself claimed to simplified the horrific complexities of the previous.. on and on.. ad nauseum.
Software dev currently sux and is in a state of crisis.
Posted by: Martfuncher | October 17, 2007 10:26 PMI don't see the irony, nor how we have come full circle to the Mythical Man-Month.
Small teams of competent people have been eating everyone else's lunch since before Fred Brooks wrote his excellent book.
It's arguable that we really haven't learned much about how to build software in the last three decades. What worked then, works now. We have a lot to learn on our journey to the Future of Software Development.
Posted by: Austin King | October 17, 2007 11:02 PMWithout a strong foundation, how can one even start to be agile in software development? You'd be changing every so often without even knowing what the big picture is.
The balance between agile and convention method is important. I don't see agile being advantageous in the beginning, but rather in the later part of the development process when the foundation is solid.
Posted by: KwangErn | October 18, 2007 2:02 AMI wrote a full response to this article here.
However I think the ommision of .NET in an article on the future of software development is an oversight. Even if you don't use .NET for windows development you will see its influence on other languages becoming apparent over time, moving query to a first tier programming methodology with the advent of LINQ as well as the process oriented development of Windows Workflow.
Still its an interesting read and a good conversation stimulator.
-David Murphy . Buzzsort
Posted by: David Murphy | October 18, 2007 12:41 PMHi, I'm always amazed and really annoyed when people criticize the waterfall model. The Waterfall model was proposed by Winston Royce and within the very same paper he explained how flawed the model is. He only ever intended the model to be an example of what not to do. Unfortunately, like when the paper was released, that criticism has been ignored. People really need to understand things properly before criticizing! http://en.wikipedia.org/wiki/Waterfall_model#History_of_the_waterfall_model
Posted by: James Hollingworth | October 18, 2007 2:24 PMThis gets me excited to write some kick-ass software!
Posted by: Casey Watson | October 18, 2007 10:46 PMWhoa there Rockafella
Posted by: maseonek | October 21, 2007 1:59 AMI think there is one more thing to consider on this post, it is the support of the Upper Management. Like for example, you are already using agile methods on developing software but then the management doesn't want to give you a better internet connection or a better machine and your developers are using a lousy chair and the working area is not not fit for deep thinking. If you are going to spend 12 to 18 hours of your day in front of a computer monitor wouldn't it be nice if it's 19 inches or even 30 inches? These things needs to be addressed first before considering what method and programming language to use. I say that whichever method you use in your software development it has to start with the support from the management.
Now I just want to make it clear that I do love Agile methods and Getting Real approach on software development, it's just that some things were overlooked during the process.
Before you start developing software here are the things that you should consider first:
1. If you are serious in this business then give your 3 to 4 developers the best machine they can have and then give them a chair that's ergonomically designed, a big table not a cubicle, a big computer screen 19" is the minimum, don't give your developers a crappy mouse and keyboard, a fast internet connection and an endless free coffee and a pizza from time to time.
2. Don't treat your developers as slaves because they are the one who determines if your business will make it or break it. Respect.
3. Give them the option to telecommute 4 times a month or anywhere they want because programmers needs a new environment to make the creative juice flowing.
4. After the project give your developers 1 to 2 weeks vacation with pay. They are also humans who gets really tired.
5. They should be happy with their pay checks.
This is not complete but this is all what I can think of right now, other readers can add. Thanks
Frederic Tubale
Posted by: Frederic Tubale | October 22, 2007 2:36 AMSimply great !
Posted by: vtortola | October 22, 2007 7:56 AMYou analysis of the waterfall model is flawed. That's to be expected from an article of this length that tries to answer such a complicated problem.
The waterfall model was applied to software development by the engineers and companies that build computers (electronics). It's incorrect to assume it was just copied from any odd engineering discipline.
The waterfall model works very well from manufacturing computers. You first design a couple of prototypes which is your iterative part. Then you set up your production lines and produce a million computers. It is nowhere said that defining requirements is separate from prototyping. In fact, out of prototyping a computer can emerge new understanding and new requirements.
Software of course only gets produced once. So if you would apply the waterfall model to software development correctly you would iterate over several prototypes and then go home.
Posted by: Steven Devijver | October 23, 2007 2:02 AMGreat post!!!
Posted by: Akarsh MG | October 24, 2007 12:36 AMI don't see the reason why you conclude that "Software off shoring will make less and less economical sense." from what you write in the article.
Posted by: danielek | October 28, 2007 2:24 PMDoes anyone have an idea on where the future of software development is going... or what would be technically possible in lets say 10, 20, or even 30 years in the realms of software development? I would be interested to hear comments. Thanks
Posted by: BISStudent | October 29, 2007 11:30 AMI think agile methods work just because the software is soft. It can be tweaked on a whim. If one designs an airplane (and probably uses the waterfall method) once it’s out of the door you don’t even think you can change a jumbo jet to a fighter later if needs will change. You’ll build other kind of plane right from the start, not modify the old one. But with software you can, at least in theory, change the nature of a program after it is out of the door.
We all have probably always used more or less agile methods, even when we thought we used waterfall. Requirement documents have been changed to match the actual implementation afterwards and internal "test versions" have been built during the early steps of design just to see if those things boldly written in design document can really happen. In my opinion Agile is just more honest name to methods that have been there all the time. Maybe the next big thing is "chaotic development" because that’s how it usually is – dynamic systems will evolve out of chaos.
Posted by: tok | November 1, 2007 2:45 AMInternet providers release information about their users at the first official request. Your ISP, your boss, or someone can be monitoring all your internet activities at this very moment.
Posted by: TieseNobliefe | November 2, 2007 12:53 PMSome users might want to enjoy more privacy than that.
That is why [URL=https://www.anonymouschannel.com/?pin=vjytjq]you need AnonymousChannel[/URL].
Interesting article, as far as it goes, but I have to agree with Waterfall, Martfuncher and others about the gap between the hype and reality of Agile methods. True, the waterfall approach was too rigid and unrealistic to deal with the inevitable changes. But most self-declared "Agile" projects I've seen so far have simply been grabbing at a trendy buzzword to justify poor quality software development from start to finish, especially on massive government IT projects where you really cannot afford to keep building/throwing away while your users spend years trying to work out what they really want.
Now, I'm happy to accept that these were "bad" Agile projects, and that there are some "good Agile projects out there as well. But the reality is that people are dumb and they will screw up every way they can, given the chance to do so. The choice of a design/development methodology is partly about finding ways to prevent/trap those screw-ups early on. So yes, let's learn from some of the helpful insights of "Agile", just as we already learned from spiral/iterative/prototyping etc (I like DSDM myself, but that's just me I guess). But ever since Java and the OO New Wave of the early 90s, the IT industry has been drowning in endless waves of hype over the Next Great Thing. "Agile" is mostly used as a buzzword by people who all too often can't be bothered to stop and think about what they're doing, and why, before they ever write a line of code. You really can design away a lot of problems, if you apply some brainpower before you start, even if it's simply a case of designing in the capability to re-factor later on.
As for offshoring, I'd like to put in a plug for Chad Fowler's great book "My Job Went To India (and all I got was this lousy book)", which contains his own recommendations for staying in work as the industry shifts work offshore. And don't believe for a minute that slash-and-burn capitalism will pause for a moment when India becomes too expensive, before offshoring from India to China and whichever country is next on the list of cheap-labour/high-volume IT suppliers. Get used to it, and try to stay ahead of the outsourcing/offshoring curve.
Posted by: Chris | December 18, 2007 6:11 PM