The Next Steps in Big Data Analysis & Its Impact on Recruiting

Big data has become a mainstream topic and more and more organizations are hiring data scientists and learning how to gather, collate and extract useful information from vast amounts of data.
At this point in the evolution of data science, we can collect data and examine it for patterns. We can look for recurring events, skills, transactions, or actions that have led to some specific results.  For example, a credit card company can watch an individual’s spending patterns, the locations where they typically use the card, and the amounts they normally charge to the card. Whenever they notice a significant change in one or more of those patterns, they can notify the user to see if someone has stolen the card.
Recruiters can use pattern recognition to look for the skills, interests, and previous experiences that their best employees have and compare those to the ones of potential hires to make better decisions. Doctors can collate numerous pieces of information about patients with a particular disease to look for trends, patterns, or common habits that may cause or cure the disease.
IBM has developed Watson, the Jeopardy winning computer that can make inferences and assign probabilities to all manner of things. It has examined millions of cancer cases and sought out which treatments have worked best based on genetic information, gender, and age.  With Watson’s ability to assign probabilities to various treatments, doctors can choose the best treatment scheme for a patient.
Big data helps us find these patterns; using it we can assign probabilities and generally make better decisions. All of this is powerful and amazing, but it is only the beginning of what big data and analytics can do.
 

Hierarchy of Information

 
A former IBM researcher named Steve Haeckelm developed this graphic representation of information as a hierarchy. At the bottom is raw data and at the top wisdom.

 
 

Intelligence

 
We have moved up this hierarchy fairly quickly and today are at the edges of intelligence. Intelligence is often defined as being able to solve problems, deal with complex ideas, and learn from experience – things computers are just beginning to be able to do. This is where Watson is as it learns from its mistakes and seeks more or better information on its own. Other programs such as Apple’s Siri learn from corrective inputs provided by users. Apple says this about Siri: “The more you use Siri, the better it will understand you. It does this by learning about your accent and other characteristics of your voice.” The Nest thermostat learns our living patters and the temperatures we like during the day and at night and adjusts the furnace and air conditioning accordingly without our help. We will soon see many other applications that will learn from our behavior and make changes to accommodate it.
Using this ability to gather data, seek out patterns and learn, computers can help organizations make much better decisions about supply chains, shipping efficiencies, inventory management, pricing, hiring, promoting people, and workforce planning. Many organizations already rely on these somewhat intelligent machines to help predict seasonal buying patterns, the demand for new products, and which styles and colors are emerging as popular.
But there are more steps in the hierarchy to climb and that’s where research and experimentation are heading. The next and most promising step is to cross the line in Haeckelm’s diagram from intelligence to knowledge.
 

Knowledge

 
Knowledge is the sum of all the data, information, and intelligence gathered and the ability to generate new information from that to make strategic decisions.  We can define a knowledgeable computer as one that examines a data set, applies all the information it can find about that data, looks for patterns, assigns probabilities, and then makes a strategic decision. The U.S. army has moved in this direction by deploying sensors around a theatre of operation, collecting data from them, combining this with knowledge of the people in the area, the terrain, their previous actions and behavior patterns and using this to help make strategic decisions about how to proceed. Today these decisions are vetted and approved or modified by a commander in the field, but could conceivably be automated to some degree soon.
 

Future of Recruiting

I am sure computers will disrupt the recruiting process as well. Smart computers will be anticipating the people needs of an organization continuously by scanning the strategy of the firm, its business plans, sales growth, development and R&D focus areas, turnover, and salary costs versus sales. They will at the same time compare past hiring behavior under different conditions (e.g., during growth spurts or recessions). These patterns will provide the information it needs to source candidates at the right time who have the right skills and who have a high probability of success. They will be able to assign a probability to how likely a person is to be open to a new opportunity based on that person’s past behavior, tenure, and location. These potential candidates can be selectively given to a hiring manager for further conversation and engagement. The computer could even supply the hiring manager with suggested ways to contact each person and offer a strategy for successfully getting them interested.
Real disintermediation is possible in recruiting, as the best possible relationship is the one between the potential candidate and the hiring manager. Reducing the number of unsuitable candidates and finding only the ones most likely to be hired by the manager can remove costs and time from the hiring process.
 

Predicting Future Trends & Directions

 
These intelligent and knowledgeable computers may also change how we predict future trends. A computer can be tasked with searching across publications, newspapers, and social profiles and many other sources to look for emerging trends and patterns. They can even decide on their own where to look for data based on their success at predictions in the past. The data they find can be compared to data generated by surveys and that gathered from government statistics about demographics, employment, and other quantitative information. From this they will see emerging trends early and even suggest ways governments and organizations could take advantage of them or prevent them.
We are at the dawn of a very different era in computing. While perhaps not yet at the level of wisdom, certainly flirting with the edges of intelligence, knowledge, and the ability to make strategic decisions unaided by humans.
 

You can hear more of  Kevin’s take on the future at this annual ‘Fearless Forecast: Workforce 2015 and beyond’ session at #ATC2015 in Sydney and Auckland this June.
Kevin will also be participating in a FREE webinar with Simon Townsend from Allegis Global Solutions Thursday the 23rd of April. They will be discussing the changing way people are bridging divides with knowledge sharing, and peer to peer mentoring. Sign up for the webinar here.