Leading vs lagging metrics

Two of the objectives I was just referring to provide a very clear example of the leading vs lagging thing, and highlight a few of the issues with leading indicators.

Both product cost and sales are lagging metrics: you’re not going to know what they actually are until well after its too late for a new product development team to do anything about it.

Confidence in hitting targets, on the other hand, is a leading metric: you can measure it a long way before the actual target is hit (or missed) – maybe even in time to do something about it.

The tradeoff, however, is accuracy. Is the confidence misplaced? It’s a pretty safe assumption that the further away from the target date you are, the less accurate people will be at predicting the outcome, but how far ahead can you go and still get any useful correlation?

And how do you find that out? If I get the project team to estimate future sales, am I then going to be brave enough to change the product or the targets based on that estimate? Probably only if I have some historical data saying they’re right more often than not. And given that as noted sales is a lagging metric (in my employer’s case very lagging – the rampup to big sales volume is longer than the product development project itself in most cases) it’s going to take me quite a while to gain that data.

If on the other hand I’m feeling brave and I make changes based on what the team estimates, how will I ever find uot whether they were right? What would have happened if I hadn’t made those changes?

Slow

Is it just me, or are wordpress.com’s dashboard and editing functions a lot slower than they were 18 months or so back?

Project sentiment survey – first steps

I’ll put aside for a bit the justification for why using a sentiment survey as a management tool seems like a good idea, and talk about the actual survey I’m trying out.

Sentiment surveys in the financial world seem to be mostly variations on “do you think things are going to get better or worse?”. While you could ask the same question of a project team, I don’t think it’d be all that helpful. It’d likely just tell you what you already knew – that particular stages of your project are riskier than others. A more useful metric is “how likely do you think it is that we’re going to hit the project targets?“

Existing PM methods use basically this same question, they just measure it with more concrete metrics. If your metrics – be they Earned Value, burndown rate, milestones passed, or just the manager’s gut feel – say that you’re not going to hit the targets, the project manager takes the appropriate action. This may involve deploying extra labour or money, revising the task structure, or changing their targets themselves.

Overlaid on this at a higher level, phased development projects like Stage-Gate do the same thing: evaluate the probability of hitting the targets, and if it seems unlikely, take appropriate action. The difference is that the evaluation   happens less frequently, and focuses more on business targets – return on investment, strategic fit etc – rather than more tactical ones like schedule & cost.

“How likely do you think it is that we’re going to hit the project targets“, however, is probably too vague a question to be useful. Apart from the fact that it  assumes the respondent knows what all the targets are (which is a riskier assumption than we’d like to think), it doesn’t tell you enough to let you form a response  if the answer is “unlikely”. A more useful question would not only define the targets, but provide some guidance as to which ones are at risk.

Which brings us to the question of which targets we define. A good start would be our favourite Iron Triangle goals of project cost, schedule, and scope. However in NPD those aren’t the whole story – a late, over budget project can be deemed a success if it results in a good enough product. Some measure of product merit is also necessary.

However all of those targets are complex to define and measure. “Scope” involves a lot more than just designing something that meets the technical spec, “schedule” is hard to define in NPD, where a project often doesn’t end with a big milestone but morphs into ongoing product support operations, and “cost” is often largely invisible to the grass-roots project workers. And lets not even get into how you define product merit. But complex is not what we’re looking for here. One of the major advantages of metrics like this is that they’re quick and easy to gather: if it’s complex enough that people find it a burden, it won’t work.  More than that, the target needs  to be phrased in a way that allows no doubt over its achievement: “how likely are we to do X” is useless unless “have we done X” allows for a simple yes/no answer. So we need simple proxies for all of these objectives.

Here are the ones I’m trialling, which I think capture these 4 major goals of schedule, scope, cost, and product merit, though I’ve taken 5 questions to do it. The survey questions will be of the form “how likely is it that this specific objective will be achieved?

A) Delivery objective: Complete AAA production units by ZZZ date.

“Project completion” is amorphous concept where I work. Our organisation allocates ongoing product support to the same group that developed the product, so there’s no clean “handover”. Often standards approvals for various markets, long term durability testing, incremental improvements and product customisation will mean that the “project” will still be burning a good chunk of resource for the first few years of a product’s life. “Launch”  is another amorphous concept. We have a small number of large customers, most of whom will have seen product samples before production starts, and all of whom will want to do extensive testing of production items before committing to volume sales.

So I’ve picked as a proxy for project completion, and therefore for schedule, completion of the major physical deliverable: the first full-scale production run.

B) Performance  objective: Demonstrate compliance to specification doc#BBB issue C by YYY date (YYY being the “last responsible” date to meet objective (a) above).

The scope of an NPD project typically boils down to “come up with a product that does X and costs less than Y” – there’s a lot implicit and explicit add-ons, and things that need to be achieved along the way, but that’s the bones of it. And generally speaking those two requirements are mutually exclusive – if you’re in trouble achieving one the easiest way out is to compromise the other. So I’m asking about them separately. In our company, “does X” is covered partly by a bunch of implicit assumptions about the common things that all our products need to do, and partly by a technical requirements spec – the latter details both functional and compliance requirements. So I’m using demonstration of compliance to spec as a proxy. Of course I also need to put a date against that – “will we achieve spec before we commit to production” is by a long way not the same question as “will we ever achieve spec”.

C) Product cost target: Achieve target cost US$DDD by XXX date (XXX being 3-6 months after ZZZ above)

The other half of the scope as described above is product cost. Product cost is a nice clean numerical target (give or take arguments about how meaningful most companies’ costings are). However it’s in the nature of production ramp-up that target costs normally aren’t hit straight out of the box. While in industries with short product life cycles that matters, in ours it doesn’t particularly, as our ramp up is usually pretty slow due to customer testing as described above. So a target date well after initial production is appropriate.

d) Labour objective: Achieve the objectives above with only EEE person-weeks more work.

Most of our projects have two main cost items: internal labour and external tooling cost. Other cost items pale into insignificance beside those two. And tooling is pretty well controlled by the capex approvals process, and in any case is only relevant or visible to the mechanical and manufacturing engineers. Labour, however, is what it is, and tends to blow out badly. Plus it’s visible to everyone. So labour’s the proxy I’m using for cost:

e) Sales objective: Sell FFF units by YYY date.

The simplest metric of product merit, and ultimately the only one that counts, is “do people want it?” And more specifically, do lots of people want to buy lots of it for lots more than it cost us to make. That’s probably capturable – sales x margin – but guessing the margin miles in advance seems to me to be a bit of an ask for the Sales team, never mind the rest of the project team. So I’m simplifying by just using sales volume as a target. There’s usually a first-year sales target in the business case – probability of hitting that target is a proxy for how the team feels about how good the product is (and about how good they think the sales force is, but that’s another story). So that gives me my final metric.

Sentiment surveys as a management tool.

I’m looking now at the use of the sentiment survey, aka confidence index, as a tool for managing NPD projects. The basic thesis goes like this:

• Management is basically a feedback control system
• Control theory shows us that one thing you really don’t want with these systems is feedback delay.
• This is not news: shortening feedback loops is a good chunk of what both lean manufacturing and agile software development philosophies are about.
• One way to shorten the loop is by the use of leading rather than trailing indicators in your progress measurement system
• To make a system work in an SME it needs to be quick and simple
• Business and consumer sentiment surveys are a quick simple leading indicator in economics: one question, small sample, simple analysis and they correlate reasonably well with more formal indicators like GDP and consumer spending
• In my experience most NPD team members have a reasonable “gut feel” for how a project’s going, even if they can’t quantify why
• This isn’t currently taken advantage of, except that the project manager will tend to fudge the numbers to suit their own gut feel
• A cross-functional NPD team meet most of the criteria to be a “wise crowd”: the collective gut feel is probably therefore better on average than any individual’s.
• Therefore a “how’s the project going” sentiment survey may be a useful feedback mechanism for management

I’ll elaborate more on this in weeks to come

Characterisation of projects: part 5 – so where did all that get us?

When last seen, I was trying to organise my thoughts around a way of discriminating between projects of different types and how they should be managed – see this post and those it refers to. I finally got that done  – it’s being published in the International Journal of Project Management next May (DOI link here). In a vastly more comprehensible form than the draft chunks here, I might add: it’s no fun getting ripped to shreds by a reviewer, but it does force you to think about what you’re trying to say. Just like product development really: negative feedback is the most useful sort, so get out there and seek it.

To summarise:

• The selection factors for using an emergent approach (Agile if you’re from software, probably something homebuilt and/or lean-derived if you’re not) are orthogonal to those for using a plan-driven approach (call it BDUF if you will, or just standard PM good practice). This contrasts with prior vews which tend to see them as opposed. It also implies an area where neither work well: unfortunately this is the area where a lot of big nfrastructure projects live.
• The factors in question summarise to uncertainty (high uncertainty is bad for plan-driven) and impact, or consequences of uncertainty (high impact is bad for emergent).
• Uncertainty – the probability of unexpected events occurring -  may be due to technical unknowns, lack of definition, “soft” factors, or because things are too complex to understand fully, or you simply don’t have time to understand them.
• Consequences – how much it matters if unexpected events do occur – may be due to what’s at stake, or to how well the team is able to handle the unexpected: inexperience, team size, dispersal, or organisational/cultural boundaries all increase the impact of unexpected events.
• Its possible to compare projects using these axes, and see which has higher overall risk (risk = probability x impact), and which is better suited to which PM approach. Project A below is best suited to plan driven, while B could use either plan driven or emergent. Project A is riskier than any of the stges of project C. Project D doesn’t suit either, is high sisk, and is probably going to get ugly. But in its earlier stages a problem structuring approach (e.g. soft systems) might be appropriate.
• It’s also possible too look at a single project over time and see how its risk profile changes. Generally projects move towards lower uncertainty and higher impact as they progress: this suggests a move from an emergent approach towards a more plan-driven one (e.g. project C below).

I’m playing at work with whether this framework is of any actual practical use, but in the mean time, that’s enough talking about it.

Framework

Sweating the Small Stuff

What’s stimulated this return to the intarwebs is Tony Pantello’s comment on a Metacool post, which reminded me of another 20×20 presentation I made to the Designers Institute of New Zealand a few months back, under the auspices of the group I work with at UoA. Last time round with DINZ, I covered a lot of ground: this time I decided to talk in a bit more depth about one subject, and the subject was how low level standardisation gives high level freedom, which I releted to the idea that constraints breed creativity. As I’ve said before, the 20×20 format is itself a good example of this very subject.

As you might imagine, talking to a bunch of designers about standardisation went down like a lead balloon, particularly once I started giving examples from the software industry, but they did me the courtesy of listening, and who knows, maybe it even gave some of them food for thought.

The text is below. As per last time, I think I’m within the bounds on copyright, but if one of them is yours and you’re offended by my helping myself, let me know.

———————————————————-

Tonight’s theme is all about doom and gloom.  If the downturn turns out to be severe and prolonged, a lot of us are going to have time on our hands.  That makes is a good time to be working on our businesses not in them – thinking about how we can do things better.  And businesses aren’t going to be looking for same old same old – they’re going to be wanting out of the box solutions to get them out of the box.

.

.

So tonight I’m going to be talking about one idea that might help.  It’s an area I’m sure isn’t even slightly dear to your hearts:  standardisation.  More specifically, about the idea that standardisation at a low level not only makes for a more efficient design process, but actually liberates creativity at a high level.

.

.

I’ll start with an example:  Language.  Written English is totally standardised at the lowest level.  You work with the letters you’re given, and those haven’t changed since the 17^th century.   And I’d point out that the latin alphabet is a lousy standard:  the letters don’t match the phonemes, and several of the letters and numbers look the same.

.

.

At the next level up, words, you get a little more latitude.  Maybe not as much as Lewis Carroll took, but words do change over time:  if you need it enough, you can invent a new one.  Go up a level again and the standard relaxes:  there are quite a few ways to construct a sentence.  One step further, and standardisation disappears:  at the paragraph level, there’s good practice but no real  rules.

.

.

As a consequence of this, at the highest level, the designer – or author – is free to focus upon the play of ideas and concepts, rather than the administrative details.  Shakespeare is famous for playing with both words and grammar, but their very presence and rigidity gave him this opportunity.

.

.

Another example is electronics.  Electronic circuits are built entirely from standard basic devices.  What’s on the screen won’t mean much to most of you:  it’s the symbol for an IGCT, or insulated gate commutated thyristor.  The interesting thing about the IGCT  is it’s the most recent  device to be invented:  in 1968.  Modern electronics are built on a foundation that hasn’t changed since the 60s.

.

.

Again, move up a level and the standard relaxes.  Most electronics designers work entirely with standard packages, but new ones with variations on the basic devices are coming out all the time.  And if you need a custom package badly enough, there’s nothing except time and money stopping you designing one:  at Wellington we’re doing it right now.

.

.

This standardisation means, once again, that electronics designers are free to focus on what they should be doing:  coming up with circuits that do cool stuff in new and better ways. The electronics industry is developing at a massive rate, and that’s entirely down to low level standardisation.

.

.

.

For a third example, look at Toyota.  Toyota are one of the most successful and innovative manufacturing companies in the world:  the history of manufacturing innovation over the last 20 years has been pretty much the auto industry chasing Toyota, and everyone else chasing the auto industry.

.

.

.

And Toyota base this entirely on low level standardisation:  both of products and processes.  They even standardise their reports.  The A3 report, which comes in about 5 flavours, accounts for most internal communications at Toyota, and it means that the author can focus entirely on what people need to know, rather than how they’re going to say it.

.

.

For their products themselves, they standardise as much of the detail as possible.  Details like shutlines, swages, hinge mounts and location features are standard across all Toyotas, from a Corolla to a Lexus.  And as a consequence, designers, both of the product and of the part, get to focus their attention on the stuff that needs to be different.

.

.

At Wellington Drive, we’re an IP based company, and we put a lot of effort into design:  probably 1/3 of our staff work in NPD, and about a quarter of those are engineering designers.  The design we do is almost entirely about function rather than form:  if we believe Steve Jobs that “design is the fundamental soul of a man-made creation”, well if an electric motor has a soul, it’s in what it does and how it does it.

.

.

One of the issues we have at Wellington though is one I’m sure that you’re all familiar with:  the problem with creative people is they can’t resist creating.  But creative people creating cause work for other people, and the creative people only have so much time themselves:  if they’re creating at a low level, they’re not adding value at a high level.

.

.

Take this thing for example.  This is a wire termination block.  The metal part traps the wires in the plastic housing and cuts through the insulation to make a connection.  We’ve built this into about half a dozen products by now, and each time we’ve redesigned it a bit.  But the thing is, it worked fine the first time.  And we’ve got dozens of little bits like that, whether they’re mechanical features, bits of circuits, or blocks of code.  We’re going to stop redesigning them.

.

.

At a more obscure level, we’re also taking the idea of low-level standardisation and seeing how far up we can push it.  Software is like language but even more so:  at the fundamental level, there are only two letters.  Above that, there are levels of code which as you get further from the basic binary, offer more and more freedom, but less and less control over the precise detail.

.

.

Historically, we’ve tended to work in the lower level languages:  C and assembler.  These give our software designers a high level of control, but tend to focus their attention at a low level.  The question we’re asking ourselves now is:  how can we focus their attention on the highest level, the algorithm?

.

.

.

And it turns out the answer is by standardisation.  By standardising the relationship between high and low level code, you can automate the generation of the low level stuff.  We’re doing a project with the University’s computer science people to find ways of describing software with something called function blocks, which are even more abstracted than the high-level languages. This will free our software designers from the drudgery of detail.

.

.

So why are we doing all this?  Well, up to a point it’s about  efficiency:  doing more with less.  But for me it’s also – in fact primarily -  about creativity, and the fact that necessity is the mother of invention.  These are two of the icons of British car design.  One of them was subject to a lot less constraints than the other, and that one is undoubtedly the more beautiful.  But it’s the other one that changed the way we thought about cars and about class.  Which is the better piece of design?

.

And how does this relate back to design in a wider sense?  Through the idea that design depends on constraints. That less constraints isn’t necessarily better.  And that constraints don’t just come from God and the client:  you get to choose some of your own as well.  And by picking the right ones, you not only make yourself more effective, you channel your creativity into the places it can do the most good.

.

.

I close by returning to my first two examples.  If you wanted to pick disciplines that have changed the world, you could do worse than these two.  And you could do worse than follow their example.

As I was saying…

I’ve been inactive here for over a year. In that time, life has changed a bit. I’ve rejoined the rat race, as CTO of Wellington Drive Technologies, and dropped my PhD down to part time. Actually, for all practical purposes I’ve dropped it down to “no time” as I got myself sorted out at Wellington. Hence the silence here. But I’m starting to work on it again, so perhaps its timely to start blogging again. We’ll see if it sticks.

Characterisation of projects: part 4 – implications of the framework

I’ve previously described a characterisation framework for projects, in which I differentiated projects based upon the level of uncertainty involved, and the potential consequences of that uncertainty. I’ve also referred to some prior frameworks, and discussed how some project management process models fitted the framework.

In this post, I talk about what use this framework (which I’m now calling the UC Framework, with capital letters even) might be for project managers.

In its simplest application, placing a project on the UC Framework allows it to be compared with the comfort zones of various process models, assisting in the selection of a process model. For example, an NPD project involving a small team and known markets and technology ( a “line extension” project) would typically have low scores on both U and C axes, suggesting that plan-driven, Agile, and possibly even ad-hoc approaches would be effective. Selection of a process model then comes down to individual project requirements. In this example the characteristics of the product might point away from Agile techniques and a compromise between plan-driven and ad-hoc – effectively a bare-bones implementation of a plan-driven model – might be used.

The model can also be used to identify leverage points for adjusting the project’s parameters to provide a better fit to a preferred process model. For example in major infrastructure projects, the major driver of project uncertainty is complexity, and this can be sufficient as to raise project uncertainty outside the plan-driven model’s comfort zone [75]. If a plan-driven process model is to be used, reducing the uncertainty effects of this complexity by improving the team’s understanding of it could improve project management effectiveness [76].

Given a system for quantifying the axes, different projects could also be plotted against each other on the UC Framework, allowing a comparison of the relative risk (UxC value, by analogy to other risk management techniques) of significantly different projects to be made. The framework also highlights opportunities to trade off different factors against each other to minimise project risk. For example does the increase in CU gained by shortening the project duration and thus improving environmental stability? Alternatively is the criticality of the project such that the extra costs of collocating the team are justified? caused by using a large geographically dispersed team outweigh the reduction in

Such assessments are necessarily only snapshots, and projects change over time. Over the course of a project, the level of uncertainty associated with it will typically drop, as information is acquired and decisions made – indeed, a project can be viewed as an information search [77]. Similarly, over a project’s course the consequences will typically rise: team size and distribution is more likely to increase than decrease during the project, “sunk costs” increase throughout, and commitments which raise the stakes are typically made in the later stages. This can lead to the need for adjustment of the process model, and even the need to switch models entirely – this is particularly true for projects which commence in a problem-structuring mode, as this mode is likely to be less helpful in the latter stages of a project unless implementation proves to be trivial. Such changes over time can be highlighted by periodic reassessment of the project.

The left and upwards trend is broadly similar to the shape of a “constant risk” curve. This is consistent with the tendency of people to adjust their behaviour to maintain constant “target risk” levels (Risk Homeostasis Theory [78]) – project managers and team members will tend where possible to defer the more critical decisions until they are made more “comfortable” by the availability of better information. Real options theory [79] suggests that this is economically sensible behaviour, and that committing early to important (C increasing) decisions may be counterproductive. However the UC Framework shows that the risk associated with such decisions may be reduced by minimising other C-increasing factors, or by pulling forward U-reducing activities in proportion.

Fundamental principles: Show me the money

“An engineer is a man who can do for a dime what any fool can do for a dollar” – Anonymous

The “a man” bit’s inaccurate nowadays, but the rest of it hits the nail right on the head.  At its basic level, the profession is all about tradeoffs, and the biggest one of them is performance vs cost.  Maximising bang for buck is what it’s all about.  Awareness of the short term and long term cost implications of everything you do and every decision you make is one of the core abilities of a top-level engineer.

This is one of an occasional series of posts on the fundamental principles of engineering.

ZED Forum

I attended AUT’s ZED Innovation and New Product Development forum yesterday. As with most such things, patchy but some of it was pretty good. Oddly enough, Chris Quinn from Gen-i, who spoke mainly about maintaining a culture of innovation in a large organization, was the standout for me in terms of food for thought. Both Hans Van der Voorn of Australo and Paul Adams of EveridgeIP had interesting stuff to say – Hans about the need for speed and product usability in startup companies, and Paul about leveraging unused IP.  Australo are doing cool stuff – this sort of company that finds a tiny, high-margin niche and aims to own it strikes me as the way of the future for NZ. Bill Day of Seaworks and Geoff Ross of 42 Below were, as expected, highly entertaining speakers though Bill’s theme, how entrepreneurs think, was some distance off topic for the forum – enterpreneurialism and innovation aren’t the same beast, though there’s a strong trend within forums of this type in NZ to conflate them.

Couple of interesting quotes:

From Chris Quinn:  “Culture is the sum of the conversations of the senior people“.  So true – leadership can’t kid themselves that people won’t notice if they talk the talk but don’t walk the walk.

From Mark Cowsill (MD of Frucor):  “Every company has a process, they’re all pretty similar, and I don’t believe they’re a competitive advantage at all“.  An interesting perspective in this context, particularly given that the general theme of his presentation seemed to be “we’re really good at this stuff, but buggered if I know why”.  Having said that, I tend to agree with him about the competitive advantage bit – the only sustainable competitive advantages are culture, recruitment, and ability to learn (which collectively add up to getting, keeping, and leveraging the best people).  A screwed up process can be a pretty powerful competitive disadvantage though.