Skip to content

Benchmarking: The only way

December 16, 2012

To control something,you have to measure it

Control Engineering 101, will teach you this thumbrule over and over again. If you have to ‘control’ something, you have to measure it first. However, we need to understand that ‘controlling’ doesnt mean to restrain something or the usual literary way we understand this verb. In this proverb you can safely replace the word improve wherever you find the word control.

Philip Tetlock has an interesting observation to make. During the closing years of the 80’s decade, the liberals as well as conservatives had something or the other to predict about the action of Soviets and thus guide the posture apropos of United States.But as is wont no one could really predict the fall of USSR. However what followed was startling because each one of the expert pundits added a rider to defend their core of the belief system. Each of them found out something which they believed  played an important role in the subsequent events, and about which they added an appropriate rider. Tetlock calls it “outcome irrelevant learning”.

If we look around ourselves we will find the a huge chunk of decisions which we are called upon to make has considerable amount of vagueness,arbitrariness or plain incompleteness about the facts surrounding it. Pursuits of successful speculation, political decision making, strategic decision making(to lead the company into a high risk, high reward pathway or a low risk, mediocre reward path), hiring people etc, to speak just a few of. Additionally to add to the milieu, we often claim to have intuition or knowledge to aid us in the same. Or in other words a skill which sets us apart. Stock pickers talk about hunch,political experts talk about  irrelevant superfluous things like the college they went to(to incite halo effect) or quote the instances where they have got right(survivorship bias) to improve respectability,leaders claim again hunch etc. But each time if we genuinely make a note of success vs failures we will come to a jarring conclusion- people are extremely confident about themselves and their skills, they know a lot less than they believe they know and worse the guy on top probably knows fewer things than the intern who joined last month in making decisions which matter.

Decision making is a relatively newly acquired skill when we talk from an evolutionary point of view and hence, we have very fat tails of outperformance among the morasses of mediocrity.Very few people have really been able to do better than a chimp or random chances in the long run.

However consequent to the Red Queen Effect in life we need just a bit of edge to be successful. We need to make slightly better decisions than others,slightly better investments than the others and hire slightly better people than others.However since the vast majority wallow in the dark labyrinth of human misjudgements, we just need slight improvement. Which implies the marginal value of improvement at this stage is extremely high. The possible rewards are seemingly vast. Warren Buffett is an extremely pertinent example of the same, Nate Silver similarly. What they have done is, they have spent quite a lot of time thinking about how to navigate in the best way through huge forests of data and observation.

Though it will be an easy exhortation to ask an average person on the street to make effort to improve his decision making, it will not be an easy project at all. However the project is very simple(not easy). Daniel Kahnemann’s advice is to invest in a very cheap notebook. It automatically implies that people should actively be 1) aware of themselves 2) really motivated to improve their decision making 3) take the effort to spare 5 minutes making a decision, writing the inputs in that cheap notebook,ruminating over the different aspects and formulating  an action course to resolve the situation.

It is difficult exercise from a psychological perspective. We are given to believe that we canWe can make split second decisions, we should have faith in ourselves etc etc. The deep overconfidence is ingrained in us to such a degree that it is difficult to challenge it. A second and even more important idea is, people are very very reluctant thinkers. Thoreau once said-

People would rather die than think. Most do

And this is true for all of us.

So whats the easy low hanging fruit to begin with. Keep records. Measure your decision making, measure your cognitive hunches. Because to control something you have to measure it first.

And this brings us to the idea of benchmarking. I have so very often seen this that it is not even funny. The best performing portfolios belong to those who benchmark, keep record of everything involved in the decision making. Coversely, if you want to lose money, make decisions on the basis of hunch, never keep records of the same, never keep records of the variables that you took help of while making the decision etc. In other words lack of benchmarking. Buffett has the best performing portfolio over the long run because he constantly subjected his decisions to measures of logic and business sense(my own observation). He has encouraged very few biases (look at his comment on the worst investment he has ever made).

To round it off, Tetlock said in one of his talks:

The Nate Silver episode illustrates in a small way what I hope will happen over and over again over the next several decades, which is, there are ways of benchmarking the accuracy of pundits. If pundits feel that their accuracy is benchmarked they will be more careful about what they say, they’ll be more thoughtful about what they say, and it will elevate the quality of public debate.

It would behoove for us to believe someone is benchmarking us.

One Comment leave one →


  1. Click Jump

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: