Keil, Mark; Mann, Joan; Rai, Arun: Why Software Projects Escalate – An Empirical Analysis and Test of Four Theoretical Models; in: MIS Quarterly, Vol. 24 (2000), No. 4, pp. 631-664.
The authors describe that: "Software projects can often spiral out of control to become ‚runaway systems’…". Keil et al. define escalation behaviour as constantly adding resources to the project. Thus escalated projects typically overrun their schedule and budget. The authors describe the case of the Statewide Automated Child Support System (SACSS) of California’s Dept. of Social Services. The project was started in 1992 with a projected budget of USD 75.5m and a go-live date in 1995. The project escalated and did cost an estimated USD 345m and was finally terminated in 1997 without any deliverables in place.
Based on a large scale survey of IS-Auditors Keil et al. found that 30-40% of all software development projects show some degree of escalation. Then the authors analyse four different theoretical models how to explain escalation – (1) Self-Justification Theory, (2) Prospect Theory, (3) Agency Theory, (4) Approach Avoidance Theory.
Self-Justification Theory – SJT is grounded in Feistinger’s cognitive dissonance, most easily self-justification can be characterised as a retrospectively rationalising behaviour which is found to violate internal or external beliefs, attitudes, or norms. As the wikipedia article on SJT describes it self-justification typically manifests in two forms internal SJT or external SJT. Internal SJT strategies are changing the violated attitude, downplaying or denying the negative consequences; whilst external SJT strategies are all sorts of external excuses from bad luck, to lack of competencies. Keil et al. argue that two effects are relevant for escalation behaviour – social and psychological self-justification. Whilst psychological self-justification is a strategy to overcome dissonance, social pressures increase the need for self-justification, e.g., saving your face.
Prospect Theory – Kahneman & Tversky’s Prospect Theory and their Cumulative Prospect Theory describes decision-making under uncertainty and risk. Keil et al. argue that Prospect Theory explains escalation behaviour, because the theory postulates that when choosing between two adverse events, deciders are seeking greater risks. Commonly this is also referred to as ’sunk cost effect‘.
Agency Theory – Jensen & Meckling’s concept of principal-agent relationships covers many examples of principals delegating decision competencies or execution of tasks to agents. A principal-agent relationships turns sour (aka principal-agent problem) when goal incongruence and information asymmetry create a constellation of imperfect contracting and monitoring. In general terms – the agents maximise their self-interest at the expense of the principal. Simplest real-world example – the software integrator who you hired to do most of your work never wants your project to finish.
Approach Avoidance Theory – Rubin & Brockner’s concept of approach avoidance which is described that every approach vs. avoidance decisions is driven by the iconic little angel vs. little devil. In the case of escalated projects, these forces either encourage persistence or abandonment of the project. Three factors explain why projects are not terminated – (1) size of reward for goal attainment, (2) withdrawal costs, and (3) goal proximity. Keil et al. argue that especially the third factor ‚goal proximity‘ creates a completion effect which is explained by the need for task closure. They argue that this is a better conceptualisation of escalation symptoms than the sunk cost effect, aka throw good money after bad money. They describe that completion effects pull an individual towards the goal whilst sunk cost effects push an individual further. A beautiful real life example is the 90%-completion syndrome:
This syndrome refers to the tendency for estimates of work completed to increase steadily until a plateau of 90% is
reached. Thereafter, programmer estimates of the fraction of work completed increase very slowly. In some cases, inaccurate estimation leads to situations in which software projects are reported to be 90% complete for half of the entire duration of the project, an obvious impossibility (Brooks 1975).
Keil et al. test 6 constructs taken from these 4 theories which were previously connected to escalation behaviour – psychological self-justification, social self-justification, sunk cost effect, goal incongruence, information asymmetry, and completion effect.
With a series of pairwise logistic regression models between the groups of escalated vs. non-escalated projects – all 6 constructs and therefore all 4 theories can empirically be proven. However the best classifier for escalation vs. non-escalation is the completion effect.