Projects + Management = Project-Management

Kronmal, Richard A. (1993) „Spurious Correlation and the Fallacy of the Ratio Standard Revisited“, Journal of the Royal Statistical Society. Series A (Statistics in Society), 156 (3), pp. 379-392.

This has been on my mind for a while. A lot of our research uses looks at cost overruns as the variable to measure the project performance. More precisely we most often use Actual/Estimated Cost – 1 to derive a figure for the cost overrun. A project that was budgeted for 100 and comes in at 120 thus has +20% cost overrun. If the scale needs to be transformed, which in most cases it does, the simple Actual/Estimated ratio offers some advantages, i.e., figures being non-negative.

Most criticism for this comes from the corner of Atkinson (1999)*, i.e., that the holding the project accountable for its initial Cost-Benefit-Analysis (+Time) is an unfairly narrow view that ignores the value of building stuff itself, the wider and possibly non-quantitative benefits for the organisation and the wider and most likely non-quantitative benefits for the stakeholder community.

However, a second corner of critics has also a powerful argument. Ratios cause all sorts of statistical headaches. First, dividing a normally distributed variable by another normally distributed variable creates a log-normal distributed variable, i.e., it creates outliers that are solely an artefact of the ratios.

More importantly than distributional concerns are spurious correlations. This is an example from the article

„… a fictitious friend of Neyman (1952), in an empirical attempt to verify the theory that storks bring babies, computed the correlation of the number of storks per 10000 women to the number of babies per 10000 women in a sample of counties. He found a highly statistically significant correlation and cautiously concluded that ‚. . . although there is no evidence of storks actually bringing babies, there is overwhelminge videncet hat, by some mysteriousp rocess, they influencet he birth rate‘!“ (Kronmal 1993:379)

What happened in that example. The regression should have been the test of the number of storks and the number of babies in a county. The argument for the ratio is that it will control for the number of women in the county. The argument against it is that that creates a spurious correlation. Better would be an ANCOVA type structure. Or as the article puts it

„This example exemplifies the problem encountered when the dependent variable is a ratio. Even though Y, the numerator of the ratio, is uncorrelated with X, the independent variable, conditional on Z, the ratio is significantly correlated to X through its relationship to Z, the denominator of the ratio.“ (Kronmal 1993:386)

Three more observations are made in the article

1. Using the two variables and their interactions instead of a ratio commonly makes for a worse model than using the ratio, particularly in stepwise regression models.
2. Ratios are an interaction and can only be adequately interpreted in an equation that includes both of these variables (the main effects)
3. Use a full regression model with interactions, then include the ratio if it adds to it

The final advice is

But what if the ratio is the ’natural quantity of interest‘, just like in our performance measurement?

The division of the outcome by the estimate is to remove its effect from the numerator variable. Kronmal questions whether „this is the optimal way to accomplish this“. He goes on further „…even when such rates are used, there is no reason not to include the reciprocal of the population size as a covariate. For other ratios, the purpose of the denominator is usually to adjust for it. In these instances, there is little to commend the use of this method of adjustment.“ (Kronmal 1993:391)

I will think about this a while, get in touch if you want to share thoughts on this.

* Atkinson, Roger (1999) „Project management: cost, time and quality, two best guesses and a phenomenon, its time to accept other success criteria“, International Journal of Project Management, 17 (6), pp. 337-342.

Yesterday at lunch I had a discussion with two of our MSc students on how to write. We started of on how to write a good thesis and ended up talking about how to write a good essay. This morning I got an e-mail from the Chair of the Examiners, who is the person running a committee that decides the marks for student work.

N.B. marking in Oxford is its own case study of accountability, transparency, and power. I don’t understand how such an intricate system has evolved that relies on double-blind processes combined with committee decisions and multiple-levels of hierarchy to quality control all to derive ‚objective‘ marks while the revelation that facts are constructed came to this institution as a big surprise.

The email I got this morning asked me to give some students feedback on one of their essays. I have to admit switching from communication by powerpoint to communication via unformatted, double-spaced, prose was one of the greatest challenges of starting with this DPhil. I also just read Dan Ariely’s brilliant blog post and the subsequent op-ed in the LA Times on this topic.

Drum roll. Here is my list on ‚How not to write you essay‚

1. Answer a different question. Well, why wouldn’t you. Time is short, the deadline looms. Luckily, in this other course there was a required reading, which you still remember and which could shine some new light on the question. Brilliant idea! Of course there are bonus points to be earned for bringing in new literature. This is perfect murder of two birds with one stone. Unfortunately the execution often falls through. The argument, already a basket case full of apples and oranges, doesn’t get the cream and chocolate sprinkles on top, which it deserves but rather gets a completely new addition, which looks more like a block of cheese with a smell of old socks rather than a fresh idea.
2. Look up the etymology of the key concepts. No argument has ever been advanced by looking up the etymology, well outside the realm etymologists anyways. It is always good to know that the word project can be traced back to 1450. Always good way to use space.
3. Give good solid definitions for all concepts. A good essay ought to start with a long laundry list of working definitions for key concepts. Let’s define risk, organisations, bias, projects, and my favourite major programmes. Once that is out of the way we can actually start looking at the question. Again a great way to use the space.
4. Write up the lecture slides. Just on the off-chance that the marker hasn’t read the slides, just copy them and expand the text a little. Did you make a recording of the lecture. Even better. Easy peasy lemon squeezy.
5. Cover everything that has been touched upon in class. Decision-making is hard, to decide what concepts to use and which ones to ignore is risky. Avoid cutting something out whenever possible. On the flip side if you cut something out you should not talk about why you took a specific lens.
6. Make shit up. Drop names. I do have 10 years of experience in this, so let me tell you what I think. I think that the following 8 factors are the key to success in the field. Also, since it is my own opinion I don’t need to add references. Time saved! Damn, they want a reference. Let’s just put an article here whose title sounds as if they would agree with my thinking. Done!
7. Be Malcom Gladwell „A cursory reading of 5 journal articles has brought me here today to tell you…“

My list for a good essay

1. 1 idea per paragraph, first sentence explains how this is important to answer the question, last sentence gives the so what? and answers the question. Sounds simple, then go on and do it!

My background is in Computer Science and my old prof Eric Schoop introduced me to information mapping most essays I have to read would certainly benefit from bringing stronger principles to writing.

It was in the air for some time and then I read it in wired and last week in the Economist, the face of the web is changing once again. WWW becoming the less dominant force. Well, I used gopher:// when I was little and the university’s library still offers telnet access.Times are changing and I now write mostly at: http://www.facebook.com/pages/Oxfords-BT-Centre-for-Major-Programme-Management/122182117824942 Looking forward to seeing you there! -Alex

This is beautiful, and done by some very clever colleagues of mine.  It is thought provoking and you since a lot of people out there run wild for ‚2.0‘ postfixes to whatever they do; this is the ultimate checklist.  It is in essence what Jeff Jarvis wrote in What would Google do? although only few people like the book and I have not even started reading it.

Everyone, please vote which approach you think is right.  Let me outline that for you.

Yesterday I read the Black Swan by Nassim Nicholas Taleb.  It is a great book.  In short it is about our (as in the human mankind) inability to predict rare events.  He details a lot of psychological reasons, e.g., tunneling, narrative fallacy; for us not being able to predict these Black Swans and he also shows what we can do about it.  Great book – highly recommended.  Anyway, yesterday I was reading page 159 (for the ones who have a copy handy); and there he makes the hypothetical argument that we think about project deadlines as if they were probabilistically the same as our life expectancy – partly because that’s how we evolved.  So what do you think – is that really true?  But let’s understand that distinction in detail first.

1) Living People

Life expectancy figures are the very centre of an actuary’s daily life, at least the ones who insure health and life &c.  When you meet one at a party you’ll understand how exciting this topic can be.  What life expectancy figures do is to look at the dying age distribution with in a population; ages are ordered neatly by age and then the actuaries compute the probability of dying before your next birthday.  That also gives you then an expected age which is the year by which 50% of your fellow birthday-boys and girls will be dead.  The expected age is per definition an average. 

If you look up the tables and they chart quite nicely as well (cf. the graph below) you’ll see that in 2004 in the US the expected age for a newborn is 77.8 years.  Some of them die (a saddening 680 that is) before they reach their first birthday. So if you made it there then you can expect to live for another 77.4 years which allows you to expect your death shortly after your 78th birthday.  When you turn 30 then you can expect to live for 49.3 more years (or 79.3 in total), when you reach 50 you could expect another 30.9 years (80.9 in total) and so on. 

Source: CDC Life table for the total population:  United States, 2004

What if project delays would be like a living population?

In the book Taleb argues that we typically think of project delays having the same probabilistic properties as life expectancy curves.  That is on average all projects are delayed by 3 weeks, and when a project is delayed by 2 weeks it will be delayed by an additional 1.5 weeks and so on.  Since I don’t have nice raw data and my own data pool is not yet big enough for nice analysis like that, once again I plundered the cost overrun figures from the Standish Group report.  When computed and charted it looks like this:

What does it say?  Well, when the project is still immaculate with 0% cost overrun, you would expect it to overrun its budget by +98%.  If your project already shows a budget overrun of 100% it will need another +63% (so in total it will be +163%), if it has overrun its budget by +300% you would put aside another +27% (totaling it at +327%) and so on.

2) The Black Swan

So what are these Black Swans all about?  An excerpt from the McKinsey Quarterly (No. 1, 2009) where the author summarises his central thesis beautifully (cf. the full article here):

Before Europeans discovered Australia, we had no reason to believe that swans could be any other color but white. But they discovered Australia, saw black swans, and revised their beliefs. My idea in The Black Swan is to make people think of the unknown and of the potency of the unknown, particularly a certain class of events that you can’t imagine but can cost you a lot: rare but high-impact events.

So my black swan doesn’t have feathers. My black swan is an event with three properties. Number one, its probability is low, based on past knowledge. Two, although its probability is low, when it happens it has a massive impact. And three, people don’t see it coming before the fact, but after the fact, everybody saw it coming. So it’s prospectively unpredictable but retrospectively predictable.

Now that we’re in this financial crisis, for example, everybody saw it coming. But did they own bank stocks? Yes, they did. In other words, they say that they saw it coming because they had some thoughts in the shower about this possibility—not because they truly took measures to protect themselves from it.

Now, a black swan can be a negative event like a banking crisis. It also can be positive: inventing new technology, making new discoveries, meeting your mate, writing a best seller, or developing a cure for cancer, baldness, or bad breath. In The Black Swan, I say that in the historical and socioeconomic domain, black swans are everything. If you ignore black swans, you’ve got nothing. And I showed that the computer, the Internet, and the laser—three recent technological black swans—came out of nowhere. We didn’t know what they were, and when we had them right before our eyes we didn’t know what to do with them. The Internet was not built as something to help people communicate in chat rooms; it was a military application and it evolved.

So these things have a life of their own. You cannot predict a black swan. We also have some psychological blindness to black swans. We don’t understand them, because, genetically, we did not evolve in an environment where there were a lot of black swans. It’s not part of our intuition.

In the book The Black Swan he argues on page 159 that when we make predictions of project schedules we tend to make them without looking for external events.  In his example of a publishing deadline – it may be the sick grandmother, sudden financial troubles which force the author to take on a night shift job, or a terrorist attack that troubles your mind for some months.  These things happen, yet we never acknowledge them in the first place.  So he argues a project that is late by 3 months should be expected to be late by another 5 weeks, if it then is not ready after 5 months you would expect another 6 weeks, at a year delay you would rather expect it to be delayed another 5 years than expecting it to be ready within the next 2 weeks – he argues that in reality the marginal expected project delay increases and does not decrease.

So, if we go ahead and compute the same Standish Figures with these probabilistic assumptions then we get the following picture:

So what do these numbers tell us?  Well, if the project is in budget, we better expect +98% budget overrun.  If it however is +100% over budget you better expect +226% more (that is a +326% total budget overrun); and when you get there at +326% you would expect +441% additional costs adding up to a whooping +767%.  You get the idea.

So if we chart that as "expectation of total budget at completion at cost overrun of"-diagram the curves look like that:

3) Your turn – the vote

What do you think is true for Projects – do cost overruns of projects show probabilistic features of living people or are they Black Swans?

Thanks. 

Cunningham, Scott W.; van der Lei, Telli E.: Decision-Making for New Technology: A Multi-Actor, Multi-Objective Method; in: Management of Engineering and Technology, 2007, No. 5-9, pp. 1176-1185.

Cunningham & Lei present a method that does not aggregate the individuals‘ preferences but instead considers strategic and economic factors in the assessment of multi-criteria decision making (MCDA).

They explicitly model exogenous and intrinsic values into their criteria. The exogenous values are based on MCDA (value at risk & trade-offs), and game theory (strategies & values).  The intrinsic values are based on cooperative game theory (negotiations), and preferences (revealed preferences from historic choices & value elicitation).

Using weighted linear value functions they model the system of a decision-maker on new technology.  Then the authors expand their system model to include alliances and outside options.  Their results show are somehow unexpected, the system does not agree on an equilibrium price (aka exchange rate), because individual companies in the alliance profit from raising prices locally within the network.  Thus they ask: "Given scarce resources, where must alliances trades be made to maximally enhance profitability?"