Partly it isn’t because creators haven’t adjusted their methods to match the new technology – we’re still wedded to a single start-to-finish process, with one outcome at the end. Partly it isn’t because feedback can be hard to gather and digest, and even hard to act upon.

An iterative method has become one of the defining characteristics of how I like to write, organise, and present text on the internet. At least, beyond this domain. But until now, I’ve struggled to apply it to internet-based video.

This article introduces internet-based iterative design, and uses YouTube’s “Hot Spot” analysis to show how we can start to apply an iterative approach to video and movie-making.

Iterative Product Development

The author of a published paper book generally gets one shot. One chance to have their works committed to paper. To be read by milllions. Or tens. A huge amount of effort goes into “getting it right”: Construction of text and story, editing, proof-checking. And in spite of this, book publishing remains a high-risk activity: For every top-selling author, there are others whose work ends up as pulp.

In contrast, the cost of making corrections or changes on the internet can be minor. At the extreme, the author simply types some new words. An update that might have taken a book publisher months or years, can be committed in minutes.

The ability to make rapid changes in response to rapidly gathered feedback makes the internet interesting: The most basic server access logs can be analysed to reveal that chapter 2 is generated much more interest than chapter 1, yet chapter 4 is hardly getting read. With enough readers, those patterns can be seen in days, or even hours. So perhaps the content in chapter 2 should be expanded, and we should re-write chapter 3 to better maintain interest?

Now we’ve expanded chapter 2, and noticed it has become even more popular. Obviously there’s a greater demand for the writing or information in chapter 2 than the author thought there was. Gradually the content evolves and gravitates towards (in the language of entrepreneurs) the nearest unserved market. Iterative product development isn’t just about “making it better”. It’s a way of finding an audience, customers, earnings.

The written word is an easy example to understand, but maybe all good design iterates in response to user feedback?

Probably always has. Stone wheels? Computers sold as recipe books? Especially where a cool technology is looking for a problem: An inventor that doesn’t start by trying to address a problem, but merely discovers a method of doing something. Post-It notes are a popular example, but this pattern is common from the Victorian era onward. For example, it took half a century of different people trying to use pressurized air for land-based transport propulsion, before a market niche was established.

The internet allows this process to happen a lot faster, but only if the presence of the internet becomes integral to the design process.

Personally, this methodology has turned a few highly technical pages on the mechanics of fishing, into a fishing guide read by millions (see Appendix: Iterative Writing in Practice at the bottom).

Limitations

Iterative product development isn’t a panacea. Or a free ride to perfection and untold riches:

• While internet-based products and services are comparatively (to manufactured goods) cheap and easy to change, constantly making changes can become very time-consuming. Designing with the expectation of change, helps. But ultimately you will reach a point where further changes don’t generate enough extra audience interest to (financially) justify the time spent making the changes. This is when to stop.
• There is no guarantee that your product will find the absolute biggest unserved market, merely the biggest such market near to the topic/interest area you started with. If you started developing an idea in an obscure niche, it’s not realistic to expect to grow outside of that niche.
• Iterative development is not an excuse to design garbage. Quality remains important: The first attempt has to be sufficiently “good” for enough people to use/read it to generate feedback.
• The process of analysing feedback, and developing new content, requires 2 distinct skill-sets. Someone that is good at the second, may be unable to do the first. An instinctively good designer might still produce a better product, although (I would argue) their method leaves more to chance.
• It’s not clear that this method could be applied to an entirely physical product – Zara’s version of fast fashion is a good example, yet customer feedback still takes weeks to filter through into new clothing.

Video

Most internet-based text content is easy. Changes can be made and distributed in seconds. Reasonably good feedback is available using tools like Google Analytics.

Video content poses a few problems:

• Minor changes require re-rendering and uploading, which (for anyone without Hollywood-scale production facilities) can take several hours, even for just a few minutes of video footage.
• Major changes mean re-filming, editing, sound design, and similar alterations that could take days. May not even be possible, if showing specific events or people.
• Detailed feedback is hard to get. At best you’ll get a reaction to the whole video – typically a number of people that watched, and the rating or comments of a tiny proportion of them.

The first 2 problems aren’t going away anytime soon. The best defense is to save all the footage you shot, including materials that didn’t make the final edit. Recording at full 1280×720 pixel resolution, 30 frames per second, I find that for each minute of the final movie:

• I shot about 20GB of footage,
• take about 10GB to the editing stage, and
• use about 5GB in the final version.

So I’ve already created a lot of redundancy – until I run out of hard disk space. That redundancy helps make minor changes, such as altering the length of a scene, but it won’t let you re-write the story or change the location.

However, small edit tweaks can make the difference between “good and great”, so some iteration is possible within video. In theory. The problem is that without detailed user feedback, how do we know what to improve?

A friend who worked in “new media” when it was new (in the mid-1990s), used say, “the skill was to know when to stop”. To misquote Damien Hirst, “a painting is finished after a long period of looking at it, during which nothing is added”. Personally, video editing involves a lot the later: Constantly replaying a rough version and making changes, until I start making adjustments that seem to make it worse again.

Unfortunately, the creator is not the audience. It’s easy for them to produce things that simply don’t appeal to anyone apart from themselves, don’t solve whatever problem their audience were having, or don’t appeal to viewers’ emotions.

Hot Spots

Which is why I find YouTube’s Hot Spots fascinating.

As often, it started by accident. I couldn’t upload the video below to the host I normally use for embedded video. So the YouTube version of the video became the primary version. Almost all of the video’s 10,000 daily views were hosted on YouTube. This meant that within a day, YouTube’s “Insight” analytics were displaying some representative data about how users were viewing the movie.

The video is primarily a tutorial, intended to introduce the new World of Warcraft fishing contest. All the footage is captured in-game – some of it “live” during the contest, some recorded afterwards. The whole movie was conceived, scripted, filmed, edited, and rendered over the course of 2 days.

Fishing is a good test, because it isn’t a terribly interesting thing to watch. It’s hard to make a “good” fishing video, especially for an audience that aren’t all hardcore anglers.

The video has been favourably rated, comments are complementary and (critically) not (yet) asking questions that the video was intended to answer. Plus a few other sites have embedded it. Good start, but could it be better?

Frame-Based Feedback

Below is the “Hot Spot” graph generated from the first 20,000 views. YouTube’s explanation of the measures:

The ups-and-downs of viewership at each moment in your video, compared to videos of similar length. The higher the graph, the hotter your video: fewer viewers are leaving your video and they may also be rewinding to watch that point in the video again. Audience attention is an overall measure of your video’s ability to retain its audience.

The base (x) axis shows the time the video has been running – it lasts 2 minutes and 2 seconds.

Let’s try and analyse what the graph shows. Numbers refer to points marked on the graph:

1. I added 12 seconds of introduction and title, primarily to give the viewer time focus, adjust the volume or screen size, or let any navigation/control widgets fade away. There are no credits – this is a 2 minute tutorial, not a feature film. Unfortunately there are several ways to read the initial downward decline:
   • Viewers think they missed something at the very start, so are restarting the video (the first sound is triggered while the screen is still black).
   • Viewers have observed the water, net, and title, and didn’t want to still be observing it 5 seconds later – they’re getting bored and leaving.
   • Some viewers started the video by accident, and never intended to watch, however good or otherwise.

   I compared this video to the graph for an earlier YouTube-hosted video. It has the same structure of introduction, but a longer, less focused (a wide, familiar city-scape) initial image. That second video was much colder over the first 10 seconds. So perhaps introductory shots need to be shorter, and move to “the action” faster?

2. The titles are gone, and the video moves to a slow-paced tutorial, with gentle text and rather repetitive scenes. If you watch, the video, you’ll notice a lot of views of the same Walrus-like character, with a lot of gnomes (the pink-haired creature) casting a line or catching a fish. The graph says that’s “ok”, but remains far from hot. In contrast, the comparison video performs better at this stage. The main difference is that the comparision video moves between topics faster, with less repetition of similar-looking scenes. There are a few reasons for repetition: The gradual building of momentum (ever faster scene changes – see next point) was intended to create the sense of excitement that these “first player wins” competitions create, but I don’t think it works. I also wanted to show that the shark (the aim of the contest) could be caught in lots of different places. Past videos have lead players to conclude that only the one precise place shown in the video was valid. Overall, this stage should drag a lot less than it does, and if possible, be made less repetitive.
3. The video gradually builds momentum, until by point 3, the scenes are changing at the rate of around 2 per second (slightly more by the end of the sequence). Audience engagement warms. Possibly this is “exciting”. Possibly too exciting, forcing some viewers to re-wind because they cannot digest the scenes fast enough?
4. The top of this second rise in temperature is marked by the 3 bangs and flashes, cutting to blurred, greyscale, slow-motion sequences. If those don’t make you look, nothing will! In the video’s narrative this is the first time something happens: The gnome caught the shark, and is now running home, desperate to get back first. It’s one of those heart-stopping moments (and in the original storyboard, was intended to use heartbeats). Cool. That is, hot. But worth noting that special effects alone can provide a negative distraction. For example, the comparison video’s coldest moment is when a sequence of quotes and images of their who said them, are merged together into beautiful blue water. Looks great artistically, but doesn’t engage the audience.
5. The heat is maintained while the first prize is displayed. This may be because the tempo of the video doesn’t slow down enough to let viewers digest everything (I had to win quickly, so making sure I had enough footage wasn’t a priority…). It may also reflect greater interest in one of the prizes (the ring). Either way, this section should have been longer.
6. Contest won, interest is dropping. The 6th point occurs when the runner-up prize is displayed. Sadly, the high-point of the story is “the win”, yet the tutorial aspect of the video has to cover not winning. And chronologically, not winning happens after someone has won!
7. We’re ending on a low, which can’t be good. I suspect this is because there isn’t much interest in “the boots” among many players. Perhaps I shouldn’t have dedicated 10 seconds to showing them being used, when the main aim of the video (how to win the contest) was clearly complete? The final giggle was an attempt to liven this section up a little, but comes too late.

The Hot Spots graph doesn’t show everything. Doesn’t reflect any variation between different people viewing it. It may not even be desirable to keep a movie “hot” throughout. There are almost certainly other ways of analyzing viewer behavior.

But areas for improvement emerged that were not seen when making the video. Even if I don’t re-make this particular video, some of improvements will hopefully filter down into new videos.

Yes, But

It’s easy to dismiss this as an expensive training exercise: Wouldn’t it be better just to ensure movie-makers were experienced before they started? If all of them turned out Casablanca, I’d agree. In reality, expertise does not mean infallibly. While YouTube is almost infamous for showing how apparently (to my eyes) terrible videos can be highly popular.

It would be great to think that Saturday night’s cinema audience might see a slightly better version of a film than Friday night’s audience, based on what the first audience enjoyed most. But not terribly practical. Similarly, television news might be history before the second iteration.

But down at “YouTube level” iterative production methods start to become more viable. Still tricky, but something that only took 2 days to initially create, can probably be remade daily, if required.

This isn’t just about rapid re-production. More important is the ability to start to read the minds of an audience the creator can never see. Try to assess what aspects of the video should be expanded. What the audience want, but are only partly getting. And to do that analytically, without the movie-maker ever meeting their audience.

At the extreme, it’s the introduction of almost scientific methodology into an artistic process, traditionally based around the artist’s opinion of their own work, and their experiences to date.

Most intriguing is that “the next” Steven Spielberg (or similar) probably isn’t learning their art with an amateur 8mm camera. They’ll be uploading camera-phone videos, animating Lego, or “machinimating” goblins. And there’s a chance they will start learning to use the analytical feedback available to them, in a way older generations never could…

Learn More

• The long lost formula for start-up success. No, really – Nigel Eccles (an Edinburgh acquaintance, although I don’t think we’ve ever discussed this) wrote something similar about iterative development. That we’re thinking alike isn’t entirely unexpected, since we’re both rather analytically-minded.
• Incremental versus iterative development – Useful clarification of 2 often-confused terms, by Alistair Cockburn.
• Wikipedia – Introduces various similar software-orientated methods.

Appendix: Iterative Writing in Practice

How did one small gnome attract so many readers, in spite of initially writing about the wrong thing? A healthy chunk of curiosity helps: Search and you may find things that work even better. But since you asked, consider this:

• Split text into separate pages (not like this article): It is far easier to trace and monitor page views, than to work out where on a page a reader is reading.
• Write around the edges of your topic: Both broader introductions and more specific detail than your core starting material. If the introductory material becomes more popular than the core, expand that introduction, and so on.
• Understand who is trying to read: Specifically their education, age, time-pressure – and write to a style and length that this audience can read.
• Watch what users do: Extensive forum discussions or comments are subtle indicators of what you need to offer. See what users do in the absence of anything you’ve written. A 200-post forum discussion about something you thought was trivial, clearly isn’t.
• Personalise it: The internet is a scary place, and you don’t help ease that fear by presenting words as a robot.

That isn’t the whole story. And there are many techniques within that. Remember that all the basic design guidelines on things like the structuring of text still apply.

In my opinion, it’s an on-going exercise in the discovery of the fact that most people aren’t like you, and have different problems and needs. Logical, really: If you write for yourself, you’ve optimized the text for people like you. Yet people like you write… and so have the least need of someone else’s writing!

But having listened to many of their senior developers talk during the recent Paris “WorldWide Invitational“, I suspect actually, they just don’t know yet.

Increasingly publisher-driven games tend to be heavily pre-produced, then implemented by programmers who work for hire: The details are known a long time before release, and the only reason not to talk about them is competitive. But if you don’t have such a precise battle-plan, you can’t release information with any real certainty. So you either get a reputation for saying little, or get a reputation for producing games that ultimately exclude many “expected” features.

Blizzard are one of the most successful game developers, so they must be doing something right. It is interesting to try and understand how they develop games. On this page:

• Redoing Half of It
• Evolving Design
• Even Death Knights Evolve
• Limiting Evolution
• Scale
• Transferability
• Technology and Casual Markets
• Questions, Questions

Redoing Half of It

Start with Samwise Didier’s (art director) comments:

“Art isn’t finished until the game ships. [...] Every time we have a finished race, we end up redoing half of it [...] and even then we’ll patch it for 10 years.”

Why redo “half” of all your game assets, when you could save time and money by getting them right first time?

Art is first completed to match the original design intention, but then the purpose of the thing may be changed radically by the design team. That requires new art or animations. Redesign of art or animations typically occurs when the design team are “80-90% sure” they have the design right. For example, Starcraft 2’s “Thor” unit has evolved radically over the game’s development, such that many of the original animations are now used for the “wrong thing”. Once the design purpose of the unit settles down, those animations will be redone.

Design generally leads art. This creates “a better game” (Samwise again). But sometimes artists can create “something cool”, which designers then make work.

The design teams are continually evolving content. In this Gamasutra interview (which is an excellent companion read to this article), Rob Pardo denies Blizzard use an entirely iterative approach:

“You can’t really iterate until you have some stuff built. Unless you have enough art and gameplay infrastructure in the game, you can’t tell if you’re going in the right direction.”

Evolving Design

While there is some evidence of quite detailed initial prototyping (from the first demonstration of Diablo 3), there is considerable scope for change as a new part of a game world is developed. The box below provides some examples.

The third stage is critical, because it allows the playability of the zone to be tested, without (potentially) wasting time perfecting the zone visually. The very rough nature of the 3D model allows quick changes to be made, tested, changed again, and so on, until the result feels right. There’s a very strong parallel to David Law’s paper mock-ups – the most basic level of prototype that allows designers and potential users to get a feel for whether the design is right.

What the box above does not state is the time-line: Lake Wintergrasp, for example, was announced last year. But it is not yet close to finished: The art and environment is partial, with certain mechanics and balancing within the zone still unknown. Yet this is content that we can expect to see by the end of 2008. If the entire game were built like this, almost nothing would be known for sure until quite close to the game’s release.

In practice, some phasing of development is evident. In the first 2 WoW expansions the early (lower-level) content tended to be completed before the later content. So in Wrath of the Lich King, the Howling Fjord was somewhat playable by Summer 2007, when it was first previewed, while areas like Lake Wintergrasp were still a hole in the map. By Summer 2008, the Howling Fjord was almost complete, and Lake Wintergrasp had developed into a semi-textured zone, with the design summarised, but not detailed.

Even Death Knights Evolve

The box below describes the design process behind the new class in WoW’s Wrath of the Lich King expansion, the Death Knight.

Yet even the WoW Death Knight class example, which appears to be quite linear and structured, evolved considerably during development.

For example, the Death Knight’s runes were originally conceived as pretty free-floating icons within the User Interface. But they transpired to be hard to see. What really matters to players is “when does the ability become available”, so any button or icon has to clearly show “on” or “off” states. The second iteration, a class-specific blade-style border round the character’s portrait solved the first problem. However, then a new rune power mechanic was added (which charges up as the player uses other abilities), the redesigned blade-style border forced the runic power bar to appear where players were not expecting to see it. So players tended not to see it at all, forcing a third design to be developed.

Limiting Evolution

Core concepts within the game are unlikely to change, but specific details that are not yet “right” will continue to evolve, as this example from Tom Chilton (WoW’s lead designer) shows:

“We don’t plan to change the core role of classes. Instead we keep working on those [classes] that don’t work well enough.”

The semi-iterative design approach still has limits.

Sometimes design ideas evolve many times. Rob Pardo (vice president of game design) on Starcraft 2’s “Merc Haven”: “We love the look of the building, but haven’t figured out how to use it yet.” They have tried to figure it out 4 or 5 times.

The overall approach also starts to explain why Blizzard develops new games quite extensively before announcing or cancelling them (Gamasutra’s information suggests they cancel more than they release). Jay Wilson, lead designer of Diablo 3, responding to a question about why Blizzard had been silent on Diablo since 2001, thus:

“Development of Blizzard products is a long affair. [...] It has to play and look awesome.”

Scale

Does this approach scale?

In a relatively few years, Blizzard have expanded from around 50 people, to over 2000 employees. They have 600 employees in their Paris offices alone. The human resource implications of that are terrifying, given the highly specialised knowledge and skill-set required: No surprise that Blizzard actively try and recruit the game’s players, who already have considerable background knowledge and enthusiasm.

While most of these people are customer-facing (community or support), and not developers, it still raises the question, who is in control of development?

Using WoWWiki’s figures for US WoW patch release dates, we can see that the typical time gap between patches for has gradually increased from earlier to more recent (higher numbered) patches. The dotted line shows the rough trend. Crudely, it is now taking at least twice as long to get a patch out than it did when WoW was first released:

World of Warcraft Patch Waits

While each patch is unique, so cannot be compared directly, recent patches tend to contain a similar volume of content to older patches.

One possibility is that Blizzard have a “Man Month” problem. This concept was popularised by Fred Brooks’ book, The Mythical Man Month. It suggests that adding team members to a software project over time actually slows down the project, because the new hires spend so much time learning existing code, and everyone spends more time communicating. Logically MMOGs add a whole new dimension to the problem, because they are continually developed for many years: Inevitably developers will leave, and be replaced, even if the team size remains the same.

A related possibility is that managing or communicating through the “chaos” of continually evolving game design, simply takes longer as the project becomes bigger. Or at the extreme, most of the individual developers have no clue how their work fits into the rest of the game, because maintaining an overview becomes a job in itself.

Or it could reflect other unseen factors.

Shortly after WoW’s release, considerable effort was put into the “back-end”, to make game servers more stable. This work wasn’t clearly visible to players as new content, but probably sucked up a lot of development time. It is possible that now a lot (perhaps even most) development time is spent dealing with hacks, exploits and other cheats. Blizzard stated that dealing with cheats was a key priority (the comment related to Battle.Net, but we can assume applies generally). This work would also be largely unseen, but will slow down development of other content. (I can only assume that the decision to take legal action against the makers of Glider (a ‘bot that automates mundane aspects of play) was a rational long-term financial decision, where legal action is cheaper than developing coded solutions.)

Intriguing.

Transferability

Nobody can remember the last bad game Blizzard released. In fact, nobody can remember the last game that wasn’t released to popular acclaim followed by huge sales.

I suspect it is this legacy that dictates Blizzard’s design approach. Financial backers are prepared to risk pumping millions of dollars into the void, because the odds of producing a top-selling game are exceptionally good.

But if almost any other developer were to attempt the same approach, they’d be viewed a little like Derek Smart (who took almost 10 years to write a game that was essentially unplayable when released) – far too much risk, spread over far too many years. Which is why almost nobody except Blizzard can do what Blizzard do.

The big exceptions are in neighbouring markets, notably casual games, where the costs of producing games are relatively small. Perhaps it is another reason why the “next WoW” is more likely to come out of the casual gaming or social networking arena, than from the traditional “boxed” video game market.

Unless Blizzard do decide to go ahead with World of Starcraft, which Samwise Didier “announced” and then immediately cancelled during L70ETC L80ETC’s closing concert on Sunday evening (the band “levelled-up” during the show, from 70 to 80).

Technology and Casual Markets

Blizzard are already known for not always following prevailing industry trends in their use of technology. Almost 10 years after most of their peers moved from isometric style [edited - see comment by Itsnoteasy below] to full 3D environments, Blizzard’s latest game, Diablo 3, is still using an isometric style. Curiously, many of the most popular “games” use relatively primitive graphics – particularly those played heavily by children (from Habbo Hotel to many of the games for hand-helds/portables). Probably not the news the manufacturers of graphics cards want to hear.

Rob Pardo noted that, “Development on Mac’ historically was a great strategy for keeping system requirements low.” For World of Warcraft, they are expecting a lot of future growth to come from more casual audiences, simply because more people will have access to computers meeting the specifications for the game.

WoW was designed around the philosophy of “easy to learn, hard to master” from the start: Content becomes progressively harder, until only a tiny proportion of players can complete the final stages. Blizzard are aware that there is already a barrier to entry into the easiest dungeon content – it is simply too hard for some players. Themes of accessibility and approachability were constantly reiterated.

It remains unclear whether there will be any change of focus away from the high-end content, towards much more casual content. There is a clear desire to promote competitive e-sports, which require extremely challenging content or play styles.

It was revealed that WoW currently has “a very small team” that works on non-loot generating [I think those were the words used] aspects of gameplay, such as holiday events. So they “haven’t been able to do as much as they would like”. I presume “very small” is a euphemism for “we have someone who does that, sometimes”.

It will be interesting to see if they can broaden the player-base for their games, without loosing their e-sport stars to other games.

Questions, Questions

This article has tried to explain the semi-iterative approach Blizzard appear to use when designing their games. It shows how this approaches makes it almost impossible for them to release precise preview information until a game is about to be released. It also helps explain why a lot of answers read like this (with apologies to Bioware):

“[Question] What resources will Inscription (the new profession in WoW’s Wrath of the Lich King) use? [Statement] Well, we had this idea that it should use herbs [Conjecture] because ever since we eased the complexity of Alchemy last year, herbs aren’t being used so much; [Exploration] but we’ve only written 3 lines of code so far, [Contradiction] and the last one reads like the first two, so: [Weary resignation] I don’t know for sure, ask me again in December…”

Makes me wonder why they provide any pre-release detail at all. Do they really need the publicity?

As Tobold neatly illustrated (while I was preparing to post this article), understanding how and why Blizzard’s developers see their worlds is far more revealing that being told about detailed features.