While I don't see it as a formal methodology, I actually believe that the most success production model in use today for so many kinds of IP is the Studio Model. GETTY
Sometimes as a writer, you touch a nerve. I touched one in my article The End of Agile, which has gone viral with roughly 300,000 hits over the last few days. I'm still digging out from the deluge of email, Tweets and Linked In messages. Many, many people agreed with me, which I think should be a wake-up call to anyone in the Agile space. A number of others, mostly those who bill themselves as Agile experts, proceeded to tell me, in polite terms, where I could stuff it - that what I was describing was Scrum, not true Agile, that I evidently had no idea what I was talking about, that I was not at a key founding event, and, well, you get the idea. The joy of being a journalist is that there will always be people who disagree with you. Sometimes they are right. Sometimes.
I had a good conversation with several people out of this, and because I think that there are problems with Agile (and not just Scrum), I also believe it makes sense to air some of these. My first is from Scott Heffield, VP of Innovation at Veracity Solutions:
Michele Einhorn, in her blog article, 12 Principles of Agile Methodology, laid out a quick summary of the Agile principles, something worth going back to in this kind of discussion (she includes more details under this, but in the interest of brevity, I'm just giving the bullet points themselves.
- Satisfy the Customer
- Welcome Change
- Deliver Frequently
- Work Together
- Build Projects
- Face-To-Face Time
- Measure of Progress
- Sustainable Development
- Continuous Attention
- Keep It Simple
- Organized Teams
- Reflect for Effectiveness
Steve Denning, a Forbes colleague, also wrote a rebuttal to my article, one that I think is worth reading in its own right, even if it gets a little too close to the Agile is Great because Agile is Great category.
On the other hand, there have been multiple people who have also noted the parallels between Agile and Religion, with one of the best (thanks to Darius Blasbland, CEO of RainCode) linking to the following:
Oh, and one more on the other side of the fence:
Finally, because I cannot resist the opportunity to get the final word, I thought it would be worth laying down where I think Agile (or to be honest, whatever comes beyond Agile) is going.
The Studio Model
A bit of background: I have been programming for forty years. I saw the rise of the personal computer, the battles between Banyan and Ethernet networks, the emergence of object-oriented programming, the explosive growth of the Internet, the steady evolution of computer games, the transition from binary to text-based services (REST over SOAP) the rise (and arguably fall) of XML, the growth of JSON, mobile application development, the rise (and arguably fall) of Big Data and Data Science, and the evolution of artificial intelligence in several incarnations. I have a resume that goes on for about six pages, which I have to periodically prune because no one in their right mind will read six pages of resume, have worked as a consultant or employee to several Fortune 50 companies and US and European agencies over the years, and have authored twenty three books (and have a couple more in the works).
In that period, I have seen several projects succeed. I've seen a lot more fail. Ironically, the ones that succeeded best predated Agile by at least a decade. They were almost all games. I spent a few years working for a company called Manley & Associates, back in the early 1990s. Most of the projects we worked on had a software development life cycle of about six months, from the time that a project was commissioned to the time we delivered a completed cartridge or CD.
We used a home-grown methodology which I've since dubbed the Studio Model, because it reflected the way that you create movies, television programs, orchestrated concerts, video games, and to be honest, most intellectual property. It is not Waterfall nor is it Agile (though like any methodology it incorporates some of both by necessity).
If I had to articulate it as a manifesto, it'd run something like this:
- Vision Is Critical. Intellectual property (e.g., software) ultimately is the reflection of one person's vision. That person is the one who determines the boundary of the work. It may be a writer, a director, a designer, an architect, or a conductor, but that particular person ultimately is responsible for both the initial direction and design of the project and in many respects is also responsible for ensuring that the resulting product reflects the best potential product it can. This also means that while creativity is important, creativity that does not contribute to the overall vision is ultimately counterproductive.
- Good Design Is An Absolute Requirement. The Agile Manifesto gives short shrift to design, yet the design of the product is the most important thing that will happen during the project. All too often projects fail because the urge to code - to show demonstrable output - overwhelms common sense. Projects are complex, have a great number of interdependencies and all too often the people who are coding have little to no guidance about what they are coding, which usually translates to integration errors. Effective design early on goes a long way towards mitigating these. Yes, designs change over time - they should, because good design must account for discovery - but too often not knowing what you're building until you get there means that what you end up with an inchoate mess.
- Flexibility, not Failure. Fail fast is an oxymoron. Any organization that fails fast will fail fast. The studio model is built around the idea that any group or team is creating alternatives - different takes of a scene, for instance - for the director to evaluate and choose to build from. Some of these takes are failures (though they often make really good blooper reels) but for the most part, such takes are simply different implementations of the same idea that emphasize different functionality. This is VERY reflective of IP creation, by the way - an author may write the same chapter four or five times, not because the chapter failed, but because it fits into the overall narrative in different ways that had implications for what follows.
- Change Is Exponentially Expensive. At the beginning of a movie's production cycle, that movie can be anything, and it will always be the greatest movie never produced. Every decision that is made creates interdependencies, and single interdependency "hardens" the product by establishing constraints on what it can be - and on the costs of changing the product. It is necessary to acknowledge that change will happen, but change should always be weighed against whether the improvement from the change is worth the cost to make that change. As a thumbnail metric, costs double every two weeks for change.
- Redundancy Matters. Most studios employ the idea of understudies at all levels, not just at the upper echelons of management. Your typical programmer carries a huge amount of information about a project in their head, and despite much exhortation, will very seldom commit that information to paper - physical or electronic. Homicidal buses are a very real thing - people jump ship or are fired, get promoted, get sick, go on vacation, have kids, and each of these things takes that person out of circulation either permanently or at least for long enough to potentially derail a project. Pair programming and similar manifestations of this don't exist because face-time is better than chat time - it exists because crap happens and you need to make sure you have a backup when it does.
- Acknowledge Cycles. In a typical movie, certain activities happen at certain times. The beginning of a movie involves a great deal of design, but very little actual execution. Your writers are busy, but you don't even need your actors on set until you have enough of a story to be able to start fitting actors to parts. As the storyline gels, you can begin early activities with those actors, but the only filming will be in studio to evaluate. As the product becomes more clearly fleshed out, the activity changes for different roles - you determine locations, start pulling together costumes from designs, begin to do a detailed rendering of your VFX, begin composing scores, and so on. This is a rolling effort, and it means that at specific stages some teams are going to be very busy, some are going to be sitting idle. This is as true in software as it is in movies, and acknowledging those cycles can go a long way to managing burn out.
- The Customer Is Not The Visionary. In general, the person paying the bills has specific business objectives in mind, but often has no clue about how the specific IP product needs to be implemented. This means that it is the responsibility of the visionary, not the customer, not the implementation teams, to establish when and where to demo what functionality. I've seen far too many projects killed because of pointless demos that killed productivity because no one was granted (or reached out to claim) the authority necessary to succeed.
- Minimal Viable Products Aren't. In a similar vein, the concept of the minimal viable product is something that should never have entered the development lexicon. Most intellectual products are systems, and systems are successful because the sum is more than the parts. Put another way, they are noted for the quality of emergence. All too often, in most software projects, the magic only happens when about 85% of the project is done, when the pieces begin to fall into place, and you can determine the points of interaction. This leads to the next point:
- Complexity Lives At the Edges. Part of the reason you need a visionary is that you need a narrative - a story that seamlessly knits the various disparate pieces together. Agile's approach, which is ultimately a reductionist one, falls down when you need to integrate the components, because all too often teams, left to their own devices, solve their problems without factoring in the impact of integration with others. Integration comes from understanding early what the protocols for communication are, the models for interactivity, and an agreement about what is promised vs. what is delivered. This is another reason why a single director or architect is so important - that person should be spending their time ensuring that these protocols exist and are consistent for both that particular project and for that organization.
- Stewardship Plays a Part. One significant flaw in Agile is that it is programmer-centric and task-oriented. However, a significant portion of any project relies upon existing resources and SMEs who serve to curate, annotate and document the institutional information that is necessary for projects to succeed. They are the ones that end up documenting code and make information consistent, findable and describable within a common metadata framework. These are ongoing processes, which IMHO have always fallen outside of most Agile frameworks, yet as enterprise consistency becomes an ever-larger requirement, the role of librarians and data stewards become much more pressing.
- Self-Organizing Groups Don't. Programmers are, perforce, generalists within their fairly narrow fields of specialization. This is not atypical among creatives, who are often very comfortable switching up roles, but the reality is that you would not want your programmer to do matte paintings for movie backgrounds (even digital matte paintings), nor do you want your costumers taking over camera operations, save in those rare cases where the focus is on the costumes. Most self-organizing groups that work do so because someone establishes themselves as the leader, someone else chooses to be their second and their ops person, then the rest largely wait for that leader to start making assignments, precisely because human nature is to specialize. We note the number of self-organizing groups that succeed primarily because the number where they don't is embarrassing. Additionally, once a project gets underway, structures begin to harden, as organizations might recognize roles, but people recognize the authority of people in those roles.
- Recognize, Reward and Train Up Expertise. You want to know where some of the best hardware and software developers are on the planet? Industrial Light and Magic. They work there because they want to create magic. They are compensated well for it, but that's not the (only) reason they are there. They want to let their parents, their friends and their kids know that they helped bring magic to life, and they were the best of the best. This is also a place where people with talent are continuously brought in to receive additional training as understudies, to become the next generation of stellar designers, programmers, artists and writers.
Studio vs. Agile
All too often, proponents of certain methodologies want to claim that their methodologies are the reason for success, when in reality, the deciding factor was the skill and tenaciousness of the people involved, the presence of a clearly articulated vision that could be changed as needed but that was not written in Jell-O, and on recognizing the distinction between providing flexibility and fueling failures.
Agile is not, by itself, a methodology. The Agile Manifesto is a wish list, written primarily by programmers, in response to the incessant micro-management by non-technical managers who were in general too incompetent to learn about the technology that they managed. I cheered when I first read it (when it was published), nearly twenty years ago, even though I'd had enough experience in the field to question some of it. That's because I worked in the game industry, where a lot of the agile manifesto was self-evident. It was only later, as I worked with ever-larger organizations, that I began to see that, where agile was implemented in practice that it failed to take into account human nature in critical ways.
Agile proponents who contacted me after my most recent article frequently made the argument that Scrum, the most widely used Agile framework and one of the earliest, was not representative of true Agile. That may have been the case, though it's a weak argument that if anything lends more weight to the view that Agile is almost a religion. What I've seen in general is that a lack of a consistent vision, poor design early on, going for bargain-basement talent and the tendency to want to work towards a minimum viable product throughout the process has done more harm to projects than whether Scrum, Kanban, Spiral, or even Waterfall was used.
This is not to say that Agile is worthless. Agile legitimized the idea that all stakeholders must be involved in the process of shaping the product's constraints and parameters (something that even now is still more preached than practiced). It gave a voice to developers and (some) others in the production process who up until then often had little say, and its message to managers in particular about the need to trust in the competence of the people they manage is one that cannot be stressed loudly enough. Its emphasis on change management has spurred a lot of thought about the nature of change, experimentation and development costs in the field. And for all that, I think that certain Agile tools are a bit on the cheesy side, the idea of formalizing the process of development in such a way as to give creatives both the opportunities and the tools to shape and push back on design decisions is invaluable.
Yet, there are two key sets of problems that the Agile community faces. The first, and foremost, is that it decentralizes responsibility too much - it essentially punts on the whole issue of governance or editorial guidance. This is that whole vision thing all over again. Most people see editors as quality control people, but editors are stewards - they are people who serve to establish, preserve and modify a particular vision, and their role in that regard is roughly analogous to the movie director or software architect as given above.
Creatives, in general, have a love/hate relationship with editors, because editors by their very nature limit what a creative wants to create, but a good editor is necessary to make sure that multiple creatives are working towards a broader vision. In the media world, this means telling the broader story, in software, this means thinking about integration and protocols of communication. Agile empowers autonomous teams, but those teams still need to be able to pull together towards a common set of goals, and this means that sacrificing some autonomy for cohesiveness.
Agile also does not (ironically) distribute very well for precisely that same reason. Finding the balancing point between centralization and distribution is a difficult process. Organizations are becoming increasingly virtual, both at the individual and at the team level, and this is a trend that is only going to increase with time.
This is not a failing of the Agile Manifesto. It was very much a product of its time, fairly early in the history of the Internet, and after nearly twenty years, many of its expectations are beginning to seem ... quaint. Instead, it is simply a reflection of how much our way of working has changed in twenty years.
For instance, one of the key aspects of software development in the first place is in the realm of service application programming instructions (APIs). Documentation in the context of the manifesto was primarily focused on explaining intent - why was a certain function written, what was the developer attempting to do and so forth. Increasingly, however, such APIs are now written using the concept of self-documenting features, and the next generation will likely be considerably more discoverable than they are today. This opacity of how the application works today may seem to be a detriment, but in reality, the kind of deep-level documentation was done so seldom (and done so poorly) in the past that it didn't matter that the function was transparent. The next generation of APIs, likely built around knowledge bases, will be even easier to document automatically.
Similarly, git repositories and semantic knowledge bases also serve as a way to provide a capsule of the state of a project at any time, and not surprisingly, project management tools are now chasing after those repositories to better provide a more comprehensive snapshot of how a project is progressing. Additionally, these tools are extending out beyond code and into digital assets to such a degree that a natural form of integration seems to be taking hold in most organizations. Moreover, you're seeing the increased use of Machine Language-based classifiers and summarizers to better ascertain, for a given piece of content, how best to classify that content and provide some form of human-understandable description. This shifts the role of stewards to one of quality control and clarification of edge cases.
Finally, you're beginning to see the rise of context-free data, where the data contains its own descriptive metadata, which in turn translates into the ability for data to build its own UX and UI dynamically. To put this into perspective, when the Agile Manifesto was written, Google was a year old, Facebook, Twitter and YouTube were still several years out, and "the Cloud" didn't even exist as a concept yet. Cell phones existed, but were very slow and primitive, with the first SmartPhone (the iPhone) still ten years in the future.
This means that you're going to see the number of big integration projects peak as data hubs and knowledge bases become more commonplace within the next two to three years, and then a fairly dramatic drop off as this process becomes one of setting up a configuration file (likely with a context-free tool) then letting the databases do their magic.
This doesn't mean that this will wipe away all need for programmers, but it does mean that increasingly those programmers will be creating applications that are more studio-like than they are corporate.
This was actually the point I wanted to make about the end of Agile in my previous post. It's time to move on. Agile may be everywhere, as several readers suggested, but scratch the surface a bit and you'll find that most of those successful agile projects were ones where you had a strong architect or steward, a culture that was already primed to work in a more Studio-Model like manner, a strong design in the first place as a foundation, and exceptional team-members that used agile in the way it should be used - as a scaffold, rather than a crutch. There are good things to take out of the last twenty years of Agile, but this is not 2000, and it's well past time to acknowledge what's worked with Agile ... and what hasn't.
Kurt Cagle is Managing Editor for Cognitive World, and is a contributing writer for Forbes, focusing on future technologies, science, enterprise data management, and technology ethics. He also runs his own consulting company, Semantical LLC, specializing on Smart Data, and is the author off more than twenty books on web technologies, search and data. He lives in Issaquah, WA with his wife, Cognitive World Editor Anne Cagle, daughters and cat (Bright Eyes).