If you’ve ever applied for a job, you’ll have had at least some exposure to words and phrases you rarely see in day-to-day life: on-boarding, talent pipeline and application tracking system (ATS), to name but a few.

Yet whatever terms you use, the fight for talent is all encompassing: never more so than today. And if today’s recruiters find it difficult, if not almost impossible, to fill vacancies, the skills gap is likely to get wider in coming years. Global consultancy Korn Ferry estimates that by the end of the decade, as many as 85 million skilled positions could remain unfilled globally, with all of the world’s leading economic powers facing the same challenge.

“Left unchecked,” the report warned, “in 2030 that talent shortage could result in approximately $8.5trn in unrealised annual revenues.”

The danger of a bad hire

In a 2021 survey conducted by global human capital services provider CareerBuilder, nearly two thirds (74%) of US companies said they have hired the wrong candidate, costing them almost $15,000 per failed recruitment. Nor do the implications of hiring the wrong person just stop at the process itself. Lost productivity, missed business opportunities and the effects ill-hired candidates have on work colleagues cannot be underestimated – though they are more difficult to quantify.

Professor Peter Cappelli (pictured) says that companies don’t yet have the tools or big data to benefit from new recruitment technology.

Such difficulties make it more imperative than ever for businesses to ensure they are sourcing the right people, for the right roles, at the right time – and that they manage that talent once they’ve secured it.

It should come as no surprise, in other words, that data science is playing an ever-more important role in recruitment. But what exactly are big data and data science – and how do the two combine in the realm of data-based recruitment?

In a nutshell, explains Professor Peter Cappelli, director of Wharton’s Centre for Human Resources, big data comprises the ‘three Vs’: hugely diverse data (variety), coming in increasing volume and with growing velocity. “Big data typically refers to alternative sources of data that we haven’t collected before,” continues Cappelli, an industry veteran with more than three decades of experience in human resources, employment policy and talent management.

Data science, for its part, combines a mix of specialities including artificial intelligence (AI), data analysis, statistics and scientific methods, to obtain value from all that information. Otherwise data would be just that – data with no meaningful purpose. Data science-based recruitment, or recruitment analytics, helps recruiters evaluate their strategies and adjust them in line with their objectives, as well as identify challenges in their talent management pipeline and determine which recruitment channels are having the greatest impact. Practically speaking, it can help explain where candidates come from, provide insight about the calibre of candidate being drawn to the specific roles, and help address issues within the recruitment process. If, for instance, candidates drop out, what part of the process led to it?

At the same time, this data-based approach can provide insight after the candidate takes up a new position. As an example, if a new employee decides a role isn’t right for them after starting, analytics can be used to determine the catalyst for their decision, perhaps highlighting discrepancies between the advertised role and what it involves in practice. According to the CareerBuilder survey, after all, two thirds of candidates took up roles only to find the position was not right for them, leading half to leave within six months.

Predicting the frontline

In short, having access to data and knowing how to read it is becoming a critical component for businesses everywhere. That’s similarly true in helping to smooth the process of integrating candidates into a company and its culture. So-called predictive hiring is a way of determining how appropriate a candidate is before they’ve even stepped through the door – potentially saving time, money and even preventing ill-feeling.


Almost all Fortune 500 companies use an applicant tracking system (ATS) to monitor recruitment.


$43.4 bn

The size of the online global recruitment market by 2027, up from $28.8bn in 2019. The growing popularity of social media as an avenue for recruitment is driving the market’s growth.

Fortune Business Insights

It combines candidate-specific information with current employee data, essentially representing a beefed-up screening process that not only looks at CVs, but also trends among would-be co-workers. Among other things, that can be achieved through candidate surveys. More to the point, there’s evidence that CEOs and hiring managers are deeply aware of the importance of successful hiring. As far back as 2013, for example, a study by Harvard Business Review found that 43% of business leaders said investing in people was a priority, while 71% claimed human capital was the leading factor in ensuring sustainable economic value.

Today, with businesses’ on-going skills shortage, those figures will likely be even higher, which is why companies as varied as Microsoft and Shell are exploiting analytics in recruitment and talent management to improve their enterprises.

Examine the specifics, and the benefits of analytics are just as clear. At Microsoft, for example, managers use data to profile employees with the greatest potential of leaving, allowing them to intercede early and encourage retention through career development, remuneration or other measures to promote satisfaction.

Shell, for its part, uses analyses to the betterment of the company in even more sophisticated ways. A good example is when the oil giant asked employees to play video games – and then conscripted psychologists, neuroscientists and data scientists to identify which in-game ideas had the best chance of success.

Despite these examples, however, Cappelli argues that many companies don’t have the tools or the big data to really benefit from the technology. That’s especially true for smaller ones, he stresses, which may not have access to the special software to parse vast information sets.

At the same time, recruitment analytics clearly has some downsides too. Privacy and cybersecurity concerns are perhaps most obvious here. But from a business perspective, more serious is the risk that an employer misses talent by leaning too heavily on AI – or even uses a platform that unfairly discriminates on the basis of race of sex. It goes without saying that these theoreticals could quickly become a legal nightmare if not kept in check. That is shadowed by broader challenges, for instance the fact that some candidates may feel uncomfortable applying or working for a company that uses AI and data.

The ball is in their court

Despite these worries, however, the impact HR analytics is having on the recruitment market and people management is obvious. A 2019 study by the Corporate Research Forum found that 69% of mid-sized organisations now had a team responsible for people analytics. That’s even as skill and labour shortages continue to ramp up recruitment pressures. To put it another way, the ball is increasingly sitting in the court of candidates themselves – with potential employees able to be hyper-selective about who they work for and what they do.

“The biggest change in a tighter labour market is to identify and contact candidates directly, so-called ‘passive candidates’ who might be interested if we contact them.”


The typical number of days it takes to fill a vacancy in financial services in the US.

LinkedIn Workforce Insights

“That is why there is this focus on the candidate experience,” Cappelli stresses. “The biggest change in a tighter labour market is to identify and contact candidates directly, so-called ‘passive candidates’ who might be interested if we contact them.”

To do that, Cappelli says, recruiters need to be clear on what sort of candidate they actually want as well as how they plan to find them. They then need to figure out an approach that would make them apply or accept a position. Conversely though, casting the net too wide is almost as dangerous. As Cappelli says, having too many inappropriate candidates can be a hindrance too.

The search for talent has changed significantly throughout the past few years – and so too have the tools to do it. “If your goal is to hire good people, using data is the only way to get better,” Cappelli summarises. “If you do this poorly – typically relying on criteria that you like but for whom there is no evidence that it can predict – you will spend time and money getting candidates who quit early, who cannot do the job well and who will otherwise fail.”