Characterisation of projects: part 5 – so where did all that get us?

When last seen, I was trying to organise my thoughts around a way of discriminating between projects of different types and how they should be managed – see this post and those it refers to. I finally got that done  – it’s being published in the International Journal of Project Management next May (DOI link here). In a vastly more comprehensible form than the draft chunks here, I might add: it’s no fun getting ripped to shreds by a reviewer, but it does force you to think about what you’re trying to say. Just like product development really: negative feedback is the most useful sort, so get out there and seek it.

To summarise:

• The selection factors for using an emergent approach (Agile if you’re from software, probably something homebuilt and/or lean-derived if you’re not) are orthogonal to those for using a plan-driven approach (call it BDUF if you will, or just standard PM good practice). This contrasts with prior vews which tend to see them as opposed. It also implies an area where neither work well: unfortunately this is the area where a lot of big nfrastructure projects live.
• The factors in question summarise to uncertainty (high uncertainty is bad for plan-driven) and impact, or consequences of uncertainty (high impact is bad for emergent).
• Uncertainty – the probability of unexpected events occurring -  may be due to technical unknowns, lack of definition, “soft” factors, or because things are too complex to understand fully, or you simply don’t have time to understand them.
• Consequences – how much it matters if unexpected events do occur – may be due to what’s at stake, or to how well the team is able to handle the unexpected: inexperience, team size, dispersal, or organisational/cultural boundaries all increase the impact of unexpected events.
• Its possible to compare projects using these axes, and see which has higher overall risk (risk = probability x impact), and which is better suited to which PM approach. Project A below is best suited to plan driven, while B could use either plan driven or emergent. Project A is riskier than any of the stges of project C. Project D doesn’t suit either, is high sisk, and is probably going to get ugly. But in its earlier stages a problem structuring approach (e.g. soft systems) might be appropriate.
• It’s also possible too look at a single project over time and see how its risk profile changes. Generally projects move towards lower uncertainty and higher impact as they progress: this suggests a move from an emergent approach towards a more plan-driven one (e.g. project C below).

I’m playing at work with whether this framework is of any actual practical use, but in the mean time, that’s enough talking about it.

Framework

Characterisation of projects: part 4 – implications of the framework

I’ve previously described a characterisation framework for projects, in which I differentiated projects based upon the level of uncertainty involved, and the potential consequences of that uncertainty. I’ve also referred to some prior frameworks, and discussed how some project management process models fitted the framework.

In this post, I talk about what use this framework (which I’m now calling the UC Framework, with capital letters even) might be for project managers.

In its simplest application, placing a project on the UC Framework allows it to be compared with the comfort zones of various process models, assisting in the selection of a process model. For example, an NPD project involving a small team and known markets and technology ( a “line extension” project) would typically have low scores on both U and C axes, suggesting that plan-driven, Agile, and possibly even ad-hoc approaches would be effective. Selection of a process model then comes down to individual project requirements. In this example the characteristics of the product might point away from Agile techniques and a compromise between plan-driven and ad-hoc – effectively a bare-bones implementation of a plan-driven model – might be used.

The model can also be used to identify leverage points for adjusting the project’s parameters to provide a better fit to a preferred process model. For example in major infrastructure projects, the major driver of project uncertainty is complexity, and this can be sufficient as to raise project uncertainty outside the plan-driven model’s comfort zone [75]. If a plan-driven process model is to be used, reducing the uncertainty effects of this complexity by improving the team’s understanding of it could improve project management effectiveness [76].

Given a system for quantifying the axes, different projects could also be plotted against each other on the UC Framework, allowing a comparison of the relative risk (UxC value, by analogy to other risk management techniques) of significantly different projects to be made. The framework also highlights opportunities to trade off different factors against each other to minimise project risk. For example does the increase in CU gained by shortening the project duration and thus improving environmental stability? Alternatively is the criticality of the project such that the extra costs of collocating the team are justified? caused by using a large geographically dispersed team outweigh the reduction in

Such assessments are necessarily only snapshots, and projects change over time. Over the course of a project, the level of uncertainty associated with it will typically drop, as information is acquired and decisions made – indeed, a project can be viewed as an information search [77]. Similarly, over a project’s course the consequences will typically rise: team size and distribution is more likely to increase than decrease during the project, “sunk costs” increase throughout, and commitments which raise the stakes are typically made in the later stages. This can lead to the need for adjustment of the process model, and even the need to switch models entirely – this is particularly true for projects which commence in a problem-structuring mode, as this mode is likely to be less helpful in the latter stages of a project unless implementation proves to be trivial. Such changes over time can be highlighted by periodic reassessment of the project.

The left and upwards trend is broadly similar to the shape of a “constant risk” curve. This is consistent with the tendency of people to adjust their behaviour to maintain constant “target risk” levels (Risk Homeostasis Theory [78]) – project managers and team members will tend where possible to defer the more critical decisions until they are made more “comfortable” by the availability of better information. Real options theory [79] suggests that this is economically sensible behaviour, and that committing early to important (C increasing) decisions may be counterproductive. However the UC Framework shows that the risk associated with such decisions may be reduced by minimising other C-increasing factors, or by pulling forward U-reducing activities in proportion.

Characterisation of projects: part 3 – fitting methodologies to the framework

A while back I wrote about a characterization framework for projects, in which I differentiate projects based upon the level of uncertainty involved, and the potential consequences of that uncertainty. I’m hardly the first person to try and differentiate projects in some systematic way – I highlighted a few other efforts here.

In this post, I look at how some of the major project management methodologies fit on this framework. I use “project management” here in a much broader sense than conventional usage: any method for organizing and undertaking project tasks carries implications ofr how that process can or should be managed, so even methods which are normally thought of as being about execution rather than managing, are in a wider sense PM methods.

In a plan-driven methodology such as those advocated by PMI IPMA etc, project goals are identified and the necessary steps to achieve this goal determined. The steps are organized into an optimal sequence given resource and other constraints to form a project plan. This plan is then followed, with the objective of management being to administer the activities and to deal with deviations from the plan.

Considering this methodology on our framework, its effectiveness can be seen in the ideal case to be independent of the consequences axis. As the objective of planning is to anticipate and avoid surprises, their consequences are irrelevant. However the effectiveness of the plan driven methodology will vary along the uncertainty axis. The plan is inevitably based upon assumptions about the goal, methods, required effort and resource constraints among other things. As uncertainty increases these assumptions will be less valid. This can be compensated for to some extent by increased planning effort, however in the limiting case developing the plan itself becomes a dominant project task. The upshot is that as uncertainty increases, deviations from plan will increase, and the project will become less plan driven, and more driven by ad-hoc responses to surprises.

The plan-driven methodology is an example of an uncertainty-limited methodology, as shown in Figure 2.

Another such example is that of the problem structuring methodology. Problem structuring methods such as the Soft Systems Method (SSM) focus upon the definition of the objectives and environment of the project. Their effectiveness is again independent of the consequences axis, as implementation – and therefore the opportunity for consequences to occur – is assumed to be trivial. But effectiveness again varies along the uncertainty axis. As noted above, the relative importance of implementation and validating assumptions – or problem definition – changes. At low uncertainty levels, where the situation is relatively clear, “the process of inquiry into the problematic situation” [67 p S50] becomes a trivial one, and problem structuring methods become irrelevant. The problem structuring methodology is thus also uncertainty limited, but the limit is at the lower bound.

Not all methodologies are limited by uncertainty. The simplest project methodology is the ad hoc one, also known as “winging it” or “hacking”. In this methodology, no planning or preparation is undertaken, and the project team simply follow their noses and deal with problems as they arise. This methodology’s effectiveness is in the ideal case independent of uncertainty, as the lack of anticipation means that everything comes as a surprise – uncertainty is effectively maximum in all cases. However its effectiveness is highly dependent upon consequences: lack of any control system means that any event which the team is unable to handle is likely to quickly run away to disaster, and that it will not necessarily be known whether the event has been handled satisfactorily until too late. This methodology is thus restricted to situations where the teams ability to handle unexpected events is high (meaning extremely small and expert teams), and “disasters” cannot be overly serious: the methodology can be considered to be consequence limited.

A similar argument can be applied to the Agile methodology. This is similar to the ad hoc methodology in that uncertainty is dealt with by adaptation rather than anticipation. The technique is again limited in the consequence dimension by the team’s ability to cope with the unexpected (which is dictated by factors such team size, distribution, empowerment and competence), and by the magnitude of the worst case disaster. However techniques within the Agile methodology are largely aimed at providing control systems to mitigate these limitations, by maximizing team responsiveness and providing rapid feedback as to the effectiveness of responses. The upper consequence limit under which Agile techniques are effective is thus higher than for the ad hoc method, again as shown in Figure 2.

The limiting values shown in Figure 2 are arbitrary, and in fact a tapering off of effectiveness rather than a hard limit would be more realistic. However the graph shows clearly the respective “comfort zones” of the methodologies discussed, and areas of overlap and of lack of coverage.

In areas of overlap, the issue for methodology selection is not so much one of effectiveness as efficiency. For example in the area of relatively low consequences and uncertainty, both Agile and plan driven methodologies are effective. However the cost drivers of each are different. Plan driven methods impose significant overhead costs in documentation and control, although these an be minimized by “lightweight” implementations (see for example [68]).  On the other hand, as a “learning” approach Agile has a lower ideal efficiency than the “prescriptive” approach [69, 70], imposing risks of wasted work and suboptimal sequencing.  Which cost dominates will depend on the particular circumstances of the project in question.

Although projects with significant uncertainty and high consequences have been managed successfully, the graph shows a gap in this area between the methodologies considered. The high incidence of failure in large capital projects, particularly in the IT and construction fields [71-74] suggests that this is a combination which is not easily manageable using existing project design methodologies.

Characterisation of projects: part 2 – some existing characterisation schemes

There are a lot of project characterisation schemes out there. Most companies have one, whether formal or informal. There are a few published ones however which lend themselves to thinking about how different projects require different ways of managing them. Here are some of the ones I think are most useful.

A couple of early characterisation systems come from the NPD field. Wheelwright & Clark suggested back in 1992 that projects be differentiated according to their impact on the product portfolio, while Turner & Cochrane used a goals and methods matrix to break projects into 4 types based on uncertainty of each.

More recently, Aaron Shenhar & Dov Dvir have evolved a four factor framework (the Diamond Framework) for adapting management style to the nature of the project. They’ve evolved this over a decade or so, and the current variation on it is published in Chapter 50 of The Wiley Guide to Managing Projects, as well as their recent book Reinventing Project Management. The four dimensions are Novelty, Complexity, Technology, and Pace, and they advocate adjusting the project scope (as opposed to product scope) and management approach based on variations in each of these. This framework is primarily aimed at engineering-based projects (NPD, software, construction) – the framework doesn’t look to me to be terribly useful with respect to the “soft” aspects.

Lynn Crawford & Julien Pollack’s framework for assessing the softness of projects is a good guide to this last factor. Although they don’t explicitly say so, it’s basically a way of selecting between controlling, learning, and problem structuring approaches to a project or problem. They assess projects according to 7 factors: Goal Clarity, Goal Tangibility, Success Measures (quantitative v qualitative), Project Permeability (how subject to outside influences), Number of Solution Options (or method clarity), Practitioner Role (expert v facilitator) and Stakeholder Expectations (technical vs social).

Alastair Cockburn ’s Methodology Grid breaks projects down by the number of people (roles) involved and criticality, and then by project priorities (productivity, cost, traceability etc). It’s strictly a categorization rather than classification system, as project priorities isn’t a graduated scale. In the web page linked to above (this grid shows up in any number of other Cockburn and Crystal-related places) he discusses in some depth the issues and drivers surrounding selecting and designing a methodology, and gives some principles:

• A larger methodology is needed when more people are involved
• More publicly visible correctness (greater density) is called for on a project with greater criticality, where more damage can result from an undetected defect.
• “Weight is cost”: a relatively small increase in methodology size or specific density adds a relatively large amount to the cost of the project.
• The most effective form of communication (for the purpose of transmitting ideas), is interactive, face-to-face

Cockburn is an (agile) software guy, but there’s little in this page (beyond the examples) that links it to software – the arguments are universal.

Ole Jepsen’s Project Analyzer uses a dozen or so criteria to assess the necessary rigor level of a project – pretty much equivalent to my “uncertainty tolerance” dimension. It breaks this down into 4 categories (quality management, requirements management, planning, and documentation), and illustrates nicely the cost of increasing rigor in terms of overhead vs productive work. It comes from the software industry, but looks useful across engineering-related projects.

Todd Little’s Houston Matrix uses 10 criteria to characterise projects by their associated level of uncertainty and of complexity. Todd then categorises projects as Dogs (simple projects with low uncertainty) Colts (simple projects with high uncertainty), Cows (complex with low uncertainty) and Bulls (complex with high uncertainty), and advocates adjusting the management mix as appropriate. This isn’t a million miles from my typology – several of the factors he uses for uncertainty are also tolerance drivers. Plugging the projects from this paper into my schema would mainly result in a rightward skew of the upper regions – some of his dogs would become bulls. As he notes, this has a tendency to happen over time anyway.

Characterisation of projects: part 1 – an outline typology

Project management has tended to be a one-size-fits–all discipline. This attitude is reinforced by the profession’s Bodies of Knowledge. The PMI, for example, claims that the “knowledge and practices described are applicable to most projects most of the time,” [48], while with Prince2 “you can apply the principles ….. and the associated training to any type of project” . However there is plenty of evidence to suggest that current best practice in project management can give less than ideal results [49-52].

One possible factor driving this is the possibility that one size does not in fact fit all, and that standard methods are inappropriate for some projects. Although mainstream PM literature is starting to recognise that projects are heterogeneous, and that this has some management implications, it is still not very accepting of alternative approaches. The NPD management field has recognised for a while the difference in project characteristics but, made no great attempt to derive implications for management methodologies. Most attention to the implications of characteristics for methodologies has come from the software, infrastructure, and organisational change fields. These fields have widely disparate approaches & experiences, but little attempt has been made to integrate the thinking. I think there’s scope to pull the various characterisation systems together and see if something can be derived which tells us something about the applicability of the various management methodologies.

Approaches to date have mostly focussed on uncertainty [53-59] and complexity [53, 55, 56, 59-61]. However complexity is to a large extent a driver of uncertainty, via inability (for resource or systemicity reasons) to fully comprehend the situation [53]. Time pressure is also a driver of this, through reduced ability to take the time for comprehension and increased propensity to turn up the control inputs.

The software industry focuses also on criticality [62-66]. This is independent from uncertainty, and is more about the consequences of uncertainty. Criticality is not recognised as a factor by other fields as the range is smaller – generally high in infrastructure due to costs, generally low in organisational change.

There are other factors which impact upon the consequences of project uncertainty.  When the team’s ability to disseminate and integrate information is low, its ability to react to events in an unplanned fashion is reduced.  It is therefore likely that  unexpected events will have a larger impact than they might otherwise have before the team brings them under control  – the consequences of uncertainty are magnified. In these circumstances the gap between perceived and actual uncertainty may also be higher, so an additional buffer is required. Drivers for poor communications are large or geographically dispersed teams. Other drivers which leverage consequences are team capability (coherence, experience, and proportion of Cockburn level 2 & 3 members) and “hardness” of stakeholder expectations.

Consequences  of uncertainty are independent of the magnitude of the uncertainty itself, thus we have a 2-dimensional typology (see Figure 1).  The orthogonal relationship is similar to that seen in risk management approaches such as FMEA, where the probability of an event multiplied by its consequences give the risk factor. This typology lets us draw a number of conclusions – or at least discuss a number of factors – relating to project behaviour and management tools. I’ll discuss these at a later date.