“What a piece of work is a man, how noble in reason, how infinite in faculties, in form and moving how express and admirable, in action how like an angel, in apprehension how like a god!” Hamlet, Act II, Scene II.
I believe, among many other things, Hamlet was expressing the Earth was merely a sterile piece of rock, and the magic was with the humans on it. When examining analytics, I find the magic will rest with the people and the analytics packages, in keeping with earlier in the discussed soliloquy, “seems to me a sterile promontory.”
Analytics packages are often promoted by over-exuberant sales efforts as “the solution” to bring what is considered to be the Holy Grail of business closer to executives, a business that: has no operating costs, no significant competition and requires no managerial attention. You will find themes underlying like “reduce costs,” “more efficient,” “reduce staff,” “quicker than competition,” “better than competition,” “reduce threats,” and “reduced attention required” buried somewhere in every analytics pitch. Every new technology makes similar claims. Look at CRM, ERM etc.
I recall some significant technology changes that occurred very early in my career. The transition from punch cards to keyboards/screens and typing to word processing. Computer input at that time was by key punch. Since engineer time was expensive, the engineers marked up listings and sent card decks to “key punch departments,” which consisted of whole rooms full of machine operators using key punch machines. Typing was done in typing pools with ink markups of the typed page.
When new technology was released, these key punch and typing pools were eliminated. First, data entry was done more or less the same; with the key punch departments using Cathode Ray Tubes (CRTs) and keyboards, and word processing pools using the same CRTs and keyboards for data entry. Input to these departments were a text marking on a page.
The technology was passed down to the originating employees rather quickly. Business changed, and changed for the better. Huge pools of data entry and document distribution employees were eliminated. Communications increased. Competitive advantage was gained by early adopters. The productivity evolution was staggering as measured in a simple dimension of reducing cost per page.
However, there were dark sides. As communication methods became less expensive, more communication occurred, and it was not always better communication. Look at the evolution from scribes, to typewriters, to word processing departments and the impact and benefits technological improvements initially had on bottom line expense and overall employee productivity. With the advent of the internet and smart phones, nowadays it is possible for an employee to spend previously productive time blathering his or her views to the entire connected planet through social media sites and e-mail.
Let’s examine the Holy Grail score for electronic communications:
- Reduced Costs: Decreased costs of creating the printed word, but the created words used have increased drastically. Entire departments focused on networking, computing, web presence, etc. have been created with employees costing significantly more than data entry employees.
- Reduced competition: Competitive advantage is fleeting. Competing companies catch up or do not survive.
- Reduced managerial attention: Management is required now to delve into security, monitoring of employee systems for illegal activities, and maintain significant web presence.
The transition was inevitable because sooner or later, every company had to adapt by adopting the new technology or otherwise perish. Early adaptors gained competitive advantage, but it was fleeting. Line item costs were reduced, but organizational costs increased as the companies communicated more. Presumably, increases in sales balanced the increases in costs. For the employees, complete careers ended or quickly morphed as data entry skills were no longer in great demand. Using modern text recognition software, data entry is all but dead now. New careers became available and they were better positions with higher pay. Old labor practices could not fit well into these new positions, therefore increasing education and training costs.
Applying this to Big Data analytics with its claims of providing simplicity, cost reduction, and significant insight gained with few people, we see similar issues arising. Considering Hamlet again, it is often stated that monkeys in a room with typewriters will eventually type Hamlet. The thought is not that the monkeys will gain any insight, but the random key strikes and infinity will statistically yield Hamlet.
Much mathematical consideration has been put into this problem and there is even a Wiki page devoted to it http://en.wikipedia.org/wiki/Infinite_monkey_theorem.
Big Data Analytics can be very similar to the monkey theorem. If you have three variables, there are nine binary correlations between the variables, and three of them are worthless because the correlation is a variable with itself. Now let’s say we have a simple analytics package that just looks for correlations and a business with 1,000 different variables (a relatively simple business). There are one million less one thousand possible binary correlations. There may be forty thousand flagged by the analytics package as significant ones. Who is to say which are important and which are unimportant? Is there a causation scenario? A high correlation does not mean that there is a cause effect in the data. Are there hidden correlations buried in subsets of the data such as “June-only” that could be missed by a yearly screening?
Examining forty thousand correlations and looking for data correlations in subsets could span careers for whole rooms of analysts, not causing cost reduction layoffs. A similar feature exists with the monkey theorem. With the monkeys, mathematically speaking, the universe will almost certainly cease to exist before a world filled with monkeys and typewriters randomly type “Hamlet.” Look at the forecasted need for Data Scientists and Data Engineers, a huge shortage is coming. However, just as data entry died, classic business analysis will die out and shift to new forms of analysis and insight.
What of the claims by vendors with respect to “Our platforms have all the analytics built in.” There is no free lunch, and there is no substitute for business insight and the gut feel of experienced people. Using built in analytics without insight is substituting company employee insight for vendor insight. Decisions of whose insight is to be used should be a conscious effort by executives, not default surrender to a dark urge to push the button and forget about your business.
No software package could generate such a succinct phrase that cuts to the heart of the existential angst faced by Hamlet: “To be, or not to be, that is the question: Whether ’tis nobler in the mind to suffer The slings and arrows of outrageous fortune, Or to take arms against a sea of troubles And by opposing end them.” In a great sense it is the question faced by all when deciding about future analytics in our organizations.
