I’ll put aside for a bit the justification for why using a sentiment survey as a management tool seems like a good idea, and talk about the actual survey I’m trying out.
Sentiment surveys in the financial world seem to be mostly variations on “do you think things are going to get better or worse?”. While you could ask the same question of a project team, I don’t think it’d be all that helpful. It’d likely just tell you what you already knew – that particular stages of your project are riskier than others. A more useful metric is “how likely do you think it is that we’re going to hit the project targets?“
Existing PM methods use basically this same question, they just measure it with more concrete metrics. If your metrics – be they Earned Value, burndown rate, milestones passed, or just the manager’s gut feel – say that you’re not going to hit the targets, the project manager takes the appropriate action. This may involve deploying extra labour or money, revising the task structure, or changing their targets themselves.
Overlaid on this at a higher level, phased development projects like Stage-Gate do the same thing: evaluate the probability of hitting the targets, and if it seems unlikely, take appropriate action. The difference is that the evaluation happens less frequently, and focuses more on business targets – return on investment, strategic fit etc – rather than more tactical ones like schedule & cost.
“How likely do you think it is that we’re going to hit the project targets“, however, is probably too vague a question to be useful. Apart from the fact that it assumes the respondent knows what all the targets are (which is a riskier assumption than we’d like to think), it doesn’t tell you enough to let you form a response if the answer is “unlikely”. A more useful question would not only define the targets, but provide some guidance as to which ones are at risk.
Which brings us to the question of which targets we define. A good start would be our favourite Iron Triangle goals of project cost, schedule, and scope. However in NPD those aren’t the whole story – a late, over budget project can be deemed a success if it results in a good enough product. Some measure of product merit is also necessary.
However all of those targets are complex to define and measure. “Scope” involves a lot more than just designing something that meets the technical spec, “schedule” is hard to define in NPD, where a project often doesn’t end with a big milestone but morphs into ongoing product support operations, and “cost” is often largely invisible to the grass-roots project workers. And lets not even get into how you define product merit. But complex is not what we’re looking for here. One of the major advantages of metrics like this is that they’re quick and easy to gather: if it’s complex enough that people find it a burden, it won’t work. More than that, the target needs to be phrased in a way that allows no doubt over its achievement: “how likely are we to do X” is useless unless “have we done X” allows for a simple yes/no answer. So we need simple proxies for all of these objectives.
Here are the ones I’m trialling, which I think capture these 4 major goals of schedule, scope, cost, and product merit, though I’ve taken 5 questions to do it. The survey questions will be of the form “how likely is it that this specific objective will be achieved?
A) Delivery objective: Complete AAA production units by ZZZ date.
“Project completion” is amorphous concept where I work. Our organisation allocates ongoing product support to the same group that developed the product, so there’s no clean “handover”. Often standards approvals for various markets, long term durability testing, incremental improvements and product customisation will mean that the “project” will still be burning a good chunk of resource for the first few years of a product’s life. “Launch” is another amorphous concept. We have a small number of large customers, most of whom will have seen product samples before production starts, and all of whom will want to do extensive testing of production items before committing to volume sales.
So I’ve picked as a proxy for project completion, and therefore for schedule, completion of the major physical deliverable: the first full-scale production run.
B) Performance objective: Demonstrate compliance to specification doc#BBB issue C by YYY date (YYY being the “last responsible” date to meet objective (a) above).
The scope of an NPD project typically boils down to “come up with a product that does X and costs less than Y” – there’s a lot implicit and explicit add-ons, and things that need to be achieved along the way, but that’s the bones of it. And generally speaking those two requirements are mutually exclusive – if you’re in trouble achieving one the easiest way out is to compromise the other. So I’m asking about them separately. In our company, “does X” is covered partly by a bunch of implicit assumptions about the common things that all our products need to do, and partly by a technical requirements spec – the latter details both functional and compliance requirements. So I’m using demonstration of compliance to spec as a proxy. Of course I also need to put a date against that – “will we achieve spec before we commit to production” is by a long way not the same question as “will we ever achieve spec”.
C) Product cost target: Achieve target cost US$DDD by XXX date (XXX being 3-6 months after ZZZ above)
The other half of the scope as described above is product cost. Product cost is a nice clean numerical target (give or take arguments about how meaningful most companies’ costings are). However it’s in the nature of production ramp-up that target costs normally aren’t hit straight out of the box. While in industries with short product life cycles that matters, in ours it doesn’t particularly, as our ramp up is usually pretty slow due to customer testing as described above. So a target date well after initial production is appropriate.
d) Labour objective: Achieve the objectives above with only EEE person-weeks more work.
Most of our projects have two main cost items: internal labour and external tooling cost. Other cost items pale into insignificance beside those two. And tooling is pretty well controlled by the capex approvals process, and in any case is only relevant or visible to the mechanical and manufacturing engineers. Labour, however, is what it is, and tends to blow out badly. Plus it’s visible to everyone. So labour’s the proxy I’m using for cost:
e) Sales objective: Sell FFF units by YYY date.
The simplest metric of product merit, and ultimately the only one that counts, is “do people want it?” And more specifically, do lots of people want to buy lots of it for lots more than it cost us to make. That’s probably capturable – sales x margin – but guessing the margin miles in advance seems to me to be a bit of an ask for the Sales team, never mind the rest of the project team. So I’m simplifying by just using sales volume as a target. There’s usually a first-year sales target in the business case – probability of hitting that target is a proxy for how the team feels about how good the product is (and about how good they think the sales force is, but that’s another story). So that gives me my final metric.
