Goodbye Data Swamp, Hello Data Lake
Data lakes have been around for eight years, and while the original concept was promising, many of the lakes have unfortunately turned into data swamps. This has prompted criticism of the concept itself.
Here’s the theory: if we take all the data we possess about customers, we could deduce a great deal and draw powerful insights. We could use this information to enhance customer experience better look after our customers and sell better products and services, tailored to their needs.
It’s the Sherlock Holmes School of Marketing: stunning insights and conclusions drawn from the observation of disparate data. The first time Holmes and Watson meet, the detective deduces from looking at Watson, that his new acquaintance is a convalescent army doctor freshly returned from the fighting in Afghanistan.
He explains: “Here is a gentleman of a medical type, but with the air of a military man. Clearly an army doctor, then. He has just come from the tropics, for his face is dark, and this is not the natural tint of his skin, for his wrists are fair. He has undergone hardship and sickness, as his haggard face says clearly. His left arm has been injured. He holds it in a stiff and unnatural manner. Where in the tropics could an English army doctor have seen much hardship and got his arm wounded? Clearly in Afghanistan.’”
Clearly?
Well maybe in those simpler Victorian days. But in our modern world, Sherlock might not be able to make sense of all of the data and may very well get it wrong:Here is a gentleman of a medical type, but with the air of a military man [“I just like the look and the style. I’m not a medic – I work as a barista.”]
Clearly an army doctor, then. He has just come from the tropics, for his face is dark [“I just got back from a week in Ibiza”] and this is not the natural tint of his skin, for his wrists are fair[“Where I was wearing a wristband and a watch”]. He has undergone hardship and sickness, as his haggard face says clearly[“It was a very heavy week, I am dehydrated and I seriously need to detox.”] His left arm has been injured. He holds it in a stiff and unnatural manner[“I fell off a table during a drinking game.”].
So much for the famous detective’s showy parlour games. But there is more to Holmes’s view of the world than his logical pyrotechnics. His creator, Arthur Conan Doyle, knew a thing or two about information – as he shows in his books.
“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”
– Sherlock Holmes and The Adventure of the Copper Beeches
He knew the vital importance of possessing all the information, and not aggregating or cutting out any data and being able to understand the patterns and connections.
“It is of the highest importance in the art of detection to be able to recognise, out of a number of facts, which are incidental and which vital. Otherwise your energy and attention must be dissipated instead of being concentrated.”
– The Adventure of the Reigate Squire
Moving from the nineteenth to the twenty-first century, we have radically changed the way we gather, analyse and interpret data. Modern police detection uses DNA profiling rather than Sherlock’s analysis of different cigar ashes, but twenty-first century marketing has failed to grasp the challenge and the opportunities.
Telcos, in particular have not kept pace with the expectations of their consumers to provide them with a deeply personalised experience. There is plenty of effort, but much of it is wasted.
Most telcos still operate a marketing and customer experience culture embodied by the question: “What can we do next for our customers?”: Campaign managers fight for the next 100,000 customers to put in the audience for their next offer. Contact centre colleagues are asked to sell the next best offer to broad segments of customers who simply aren’t interested. Customer journeys are mapped out at great expense to describe the customer experience for 10 to 20 different customer types.
Telcos do this because the data they have about their customers is limited. And most of the time it is also aggregated. They have CRM data, loyalty scheme data, third-party socio-demographic data, customer care contact data. This data tells the telco teams how much a customer has spent, what products they have, what they last complained about, or their propensity to churn. Laughably, some telcos call this their 360-degree customer profile.
This level of data used to be ok. But the world has changed. Customers’ expectations have massively increased; pushed up by the FANGs (Facebook, Amazon, Netflix and Google) of the world.
Let’s for a moment remove ourselves from big corporation mentality and consider how we think about and interact with our friends, family and neighbours. Humans interacting with humans. Let’s be more Sherlock.
Sherlock doesn’t think about Jane next door as a “Young Professional Mum”. He doesn’t think of her as a part of a segment. He thinks about Jane as a unique human, someone who works in the City, has a young daughter, enjoys running, she’s a fanatical cook and however much she tries, she can’t get on with the latest technology.
A Sherlock who gets his deductions right, would look like a man of authority and intelligence – someone you could trust. A Bad Sherlock who ignored the data or didn’t understand the data would look like an idiot – a figure of ridicule.
Telcos face the same problem. They can deeply understand their customers and deliver a real customer focus that customers engage with, but they can (and have) alienated them by making broad assumptions based on data aggregation rather than personalised knowledge. Like a poor detective who keeps deducing wrong outcomes based on misinterpreted information. Customers lose confidence in the Telco, become frustrated and even start to mistrust them.
It’s Paradoxical my dear Watson.
“I had,” he said, “come to an entirely erroneous conclusion, my dear Watson. How dangerous it always is to reason from insufficient data.”
– Sherlock Holmes and The Adventure of the Speckled Band