The holy grail of insight

It’s strange how the prolonged use of language can diminish the value of words and leave us reaching for ever-higher superlatives. People are no longer ‘great’ they are ‘awesome’; ‘genius’ is a word that is now routinely used to describe people of no discernible merit.

When I started my first ‘proper’ job in 1987 I was tasked with producing something called Management Information. There was a sense of being at the cutting edge of business practice back then since the term Management Information had just replaced the less-exciting but perfectly adequate phrase Monthly Statistics. As a producer of Management Information I was trained to look beyond the numbers and establish useful information to help company leaders make sound strategic decisions and grow the business; as the victim of a mass-redundancy programme four years later, I guess I failed in that endeavour.

The era of Management Information lasted until around the turn of the millennium when Business Intelligence arrived as the new kid on the scene. Intelligence is smart and richer than Information so Business Intelligence just has to be better, right? We don’t just want our leaders to be informed, we want them to be intelligent because that is a higher path.

BI has been the language of corporate data analysis for nearly two decades now; but there is a new word emerging in the lexicon. Intelligence is all very well, but now we want Insight. Insight is deep and meaningful; it takes us to a higher plane, a higher state of consciousness. Insight is wise and all-knowing.

It’s tempting to write this continual inflation of language off as pure hyperbole – but I think there is a rational distinction between these things if we think of how they relate to the data lifecycle.

The world is a complex and dynamic place and data exists to describe that world in systems. Before data happens, we need to establish an understanding of the world and create a data model that represents that world. That data model defines the entities in the world, the relationship between those entities and the attributes of those entities. If we get this wrong, the data can never truly describe the world.

Having built our model, our processes (systems, people, whatever…) interact with the world and start to translate observations into data, using the language of the data model. If these processes fail to observe and translate correctly then the data will not be a true description of the world.

If you perform a statistical analysis on the data you generate statistics. As simple numbers, statistics don’t tell us much. If you step back through the data lifecycle to understand how the data was generated you can contextualise and interpret the statistics to generate information. If you go further back and understand the data model and its relationship with the world you can understand the information and achieve intelligence.

Genuine insight can only be achieved if you go back to the start of the lifecycle and have a rich understanding of the world that the data describes. The more you are removed from that world, the less likely you are to achieve true insight.