Last week Ayasdi came out of stealth mode and told the world it had a new wayto analyze big data, and I think the implications for CRM and social are very largeindeed. The new way is called “topological data analysis” (TDA) and hearing about it has the feelof hearing about relativity for the first time (or Salesforce.com) and learning thatspace is curved.
Who would have thought it, but Big Data is not some amorphousmass but something with topology — an entity with curves and folds and shapes?
Why is that important? Well, understanding the shape of data turns out to be,mathematically, a shortcut to understanding it or to extracting meaning fromit. Shapes include clusters, and they can tell us where the interesting bits are.
Consider the implications. No longer does one have to be inspired to ask goodquestions of data so as to write queries that deliver information. With topologicaldata analysis, you can first identify the interesting clusters of data and then askwhat’s so interesting about that?
I’ll Ask the Questions, Dave
It’s a big shift in perspective and maybe philosophy. Certainly, it takes the human race down a notch in its own esteem. Now we don’t rack our brains to ask piercing questions of our data — we have machines that do it better, so we have to stand back and watch.
This may seem odd, but what if there’s a bombshell lurkingin your data that you were never inspired to ask about? Would the data hold itssecrets forever? Well not any more.
Right now, topological data analysis is a very geeky mathematical concept — just acouple of years removed from Stanford and a DARPA lab — but the potential it holdsis big.
The Next New Age
I believe that the Information Age is winding down, just like the Age of Steam didand just as all “Ages” do. That’s not to be feared — it’s something to be embraced.What will take the place of information as the major disruptor and economicdriver? Whatever it is, it will have to stand on the shoulders of the InformationAge and use the latest and greatest tools.
Part of that means topologicaldata analysis for the simple reason that our ability to exploit discoveries inboth pharmaceuticals and oil and gas — to take two for the moment — is maxing out.
It costs upwards of US$100 million to drill an oil well in the Gulf of Mexico; it takesa team of people a few billion dollars and a decade to bring a new drug to market.It hardly gets said, but these investments cost the same whether or not the oil wellhas oil at the bottom of it, and it’s the same story if the pharmaceutical comes acropper.
Those numbers are big — so big that they represent ceilings to furtherdiscovery unless we find breakthroughs that will reduce the costs and the risks ofgetting it all wrong.
All Roads Lead to Discovery
Already we’re seeing topological data analysis crack some amazingly hardnuts, not only in the aforementioned pharmaceuticals, oil and gas, but also in financialservices and government. Anywhere there’s big data there is an opportunity fortopological analysis, and that means the mass of social data we generate too.
People at Ayasdi tell me that when they apply topological data analysis to 20-year-old data from pharmaceutical research, they find new and interesting information. So far, I don’t think they’ve come up with any new drugs, but it’searly days.
The market has other entrants too, and while Ayasdi might be taking thehighest road to the biggest customers and perhaps the hardest problems,other companies using machine learning are implementing roughly the sameidea.
‘CustomerDNA’ by Any Other Name
Consider Mintigo for example. This company focuses on identifying salesprospects, which is not the same as generating leads, but it’s a cool and importantidea nonetheless and essential in many industries.
Mintigo analyzes existing customers to build a sophisticated data model of whata successful customer looks like for your organization. This is to say that Mintigolooks at the data surrounding those customers and identifies the clusters ofrelevant data that qualify them as a match for your company and its products.
From there, it’s a simple matter of targeting the machine’s model on the generalmarketplace to see what it drags in. They call it identifying your “CustomerDNA.”
Call it “CustomerDNA” or “TDA” or more broadly, “machine learning.” Whatever youcall it, we’re on the cusp of another revolution that simplifies a major headacheand reduces the cost of important business processes to manageable levels again.With these as catalysts, can new discoveries and economic growth be far behind?
Loading ...
Hi Denis. Great article as always. I think you’re right that the Age of Information is coming to a close, and something else is on the way. It makes me think more than anything of the Age of Discovery–I think we’re discovering the world of information that we ourselves have created. Just like the Age of Discovery did for Europeans, it changes how we think of ourselves and our place in the world. We’re starting to understand how to relate to ourselves and others as information and not just discrete, closed-off brains. Maybe CRM was an early step, but I think it extends far beyond that.
-John-Paul Narowski
founder, karmaCRM