AI Frontiers Series: Human Resources
Reflecting on my first ‘multi-industry brainstorming session' nearly three years ago, I'm astounded by how machine learning concepts, once deemed ambitious, are now achievable for HR departments. The rapid pace of AI has been transforming industries, from manufacturing to healthcare. Yet, the Human Resources sector has been slow in embracing this digital revolution.
While the mix of qualitative and quantitative tasks in HR may suggest it's an unlikely candidate for AI adoption, this assumption overlooks the potential AI brings to the HR landscape. In this article, we aim to explore this untapped potential. We will focus on the key challenges in the HR industry, significant performance indicators, and how AI technologies can help overcome these challenges. This stands in stark contrast to the limitations of traditional People Analytics solutions.
Unlike the supply chain data problem discussed in the previous article, the primary concern here isn't just the availability of high-quality data. The processing of this data and adherence to data regulation norms are equally crucial. In the following sections, we will discuss how these factors can be addressed to unlock the transformative power of AI in HR.
However, before diving into these use cases, it is important to note that your organisation should first assess its readiness to implement these strategies, as recommended by Chowdhury, S., et al., (2023). Plus, as I keep emphasising, checking some broader requirements will surely be of great help (for that you can refer to this article).
Challenges in HR Management
An easy way to generate valuable ideas is by first understanding the key performance indicators (KPIs) relevant to a specific industry, area, company, etc. Once we have answers for all of them we can start thinking about other "nice to have" solutions. In this regard, all HR departments face the same array of challenges, including talent acquisition and retention, employee engagement, performance management, diversity and inclusion, and regulatory compliance (Pereira, V., et al., 2023).
Some typical HR KPIs include:
- time-to-hire
- cost-per-hire
- employee turnover rate
- employee engagement score
- diversity ratios
It is important to highlight that the processes that these metrics are measuring require a delicate balance of quantitative analysis and human understanding. Then, to fully comprehend the potential of AI in HR, we must first ask some questions regarding the current state of the processes that are measured by each KPI. Note that this is a standard procedure you should follow while working on any machine learning project. In this sense, we'll go over each step of the process by using some trigger questions.
1. The Talent Acquisition Challenge
While hiring new talent, HR professionals need to post a job position, sort through a massive pile of resumes, conduct initial screenings, schedule interviews, and finally, make hiring decisions based on an amalgamation of hard data and soft skills. But do they have the tools to carry out this tasks in an efficient and transparent manner? How can we assert there is no bias in certain processes? Let's analyse the whole story.
1.1. Job Position Creation and Posting
Does your company still write job positions without AI assistance? __ Following the identification of a hiring need, the initial step of any selection process involves creating and posting a job description on either an owned or a third-party platform such as LinkedIn. Crafting these descriptions can be quite laborious; however, it's a crucial step to attract top talent and project an image of professionalism in today's rapidly evolving job market. Furthermore, given the solutions that we'll be discussing throughout this article, a vague definition of a job position can pose some limitations on the quality of the results we obtain in some use cases (I'll come back to this soon).
Job descriptions that are poorly worded can deter qualified candidates, attract unsuitable ones, and complicate the recruitment process. Moreover, they can tarnish a company's public image by suggesting a lack of professionalism. Vague descriptions might lead to legal complications due to ambiguous job expectations and potentially discriminatory language, infringing equal employment opportunity laws. In contrast, well-drafted job descriptions attract premier talent, expedite hiring, endorse diversity and inclusion, and bolster the company's reputation. Therefore, organisations must invest sufficient resources in creating thorough, accurate, and inclusive job descriptions, thereby ensuring effective recruitment, legal compliance, and an enhanced public image.
One straightforward solution is to harness the power of AI text generation. By leveraging cutting-edge tools Large Language Models (LLMs) you can optimise your job description templates and achieve impressive results.
As an illustration, consider a job position for a senior data scientist specialising in HR with proficiency in AWS. By utilising AI, the prompt could be drafted in less than 10 seconds, proving not just efficient but also highly effective.
Note that we can provide a more accurate prompt to reflect the values of the organisation, and the topics that should be evaluated throughout the rest of the recruiting process.
Following the posting of a comprehensive and engaging job description, it's natural to see a sizeable pool of candidates and, consequently, a surge in applications. This progression leads us seamlessly into the next phase of the recruitment process: the meticulous and arduous task of resume analysis.
1.2. Resume Analysis
Are you still manually sifting through every single resume that lands in your inbox? Is your HR Talent Acquisition Specialist dedicating precious time to analyse hundreds, if not thousands, of applications, even after finding a suitable candidate? Do you ever wonder about the specific variables your recruiters are weighing in their decisions, or how they're using the resumes left in your database to match candidates with future job postings?
If these questions have crossed your mind, it's clear you recognise that the conventional method of manually reviewing resumes and analysing a high volume of applications no longer cuts it in our data-driven age.
Automated resume parsing solutions have indeed been a significant leap forward, allowing us to extract relevant information from resumes with relative ease. But the buck doesn't stop here. After the parsing stage, we're faced with a mountain of data that can feel overwhelming and unwieldy. It's no longer enough to simply sort this data into tables and run a basic analysis.
Additionally, while we might assume that leading HR platforms would have mastered the art of resume parsing, the reality is often disappointing. Their performance can be subpar, especially when presented with a resume that deviates from the conventional format, leading to a loss of critical candidate information.
One solution to overcome these obstacles is semantic matching. In machine learning, this concept refers to the process of understanding and comparing the meaning or semantics of different pieces of text or data. It involves assessing the similarity or relatedness between words, phrases, sentences, or even entire documents. The goal is to determine how well different pieces of text align in terms of their underlying meaning, rather than just relying on surface-level patterns or exact word matching. Semantic matching plays a crucial role in various Natural Language Processing (NLP) tasks, such as information retrieval, question answering, sentiment analysis, and text classification. It helps to bridge the gap between the human understanding of language and the computational capabilities of machine learning models.
How can we use it in this case? By summarising the job profile based on the parsed data and performing a semantic match with the job position, a score can be generated. This score reflects how well each attribute of the profile aligns with the requirements of the job. Through semantic matching, you can efficiently prioritise and evaluate candidates based on their suitability for the role they applied to (and other roles as well, all in an automated fashion). Note that the quality of the results can be lacking if the job positions are not well-defined.
If semantic matching does not sound convincing to you, there are still other alternatives you can explore to address this issue to a significant extent.
- Automated pre-screening: implement mechanisms that filter out candidates who do not meet specific criteria or minimum qualifications. This can be achieved through rule-based systems or machine learning models trained on historical hiring data. Of course, to do this you should have a parser solution in place.
- ML-based ranking: develop a ranking system that automatically scores and ranks candidates based on various attributes, such as skills, experience, and qualifications. This approach enables data-driven decision-making and eliminates bias in the evaluation process.
Note that investing in the implementation of these changes can turn out to be more impactful than you expected. These techniques can benefit your HR operations in several ways:
- Scalability and efficiency: As businesses scale and receive a higher volume of applications, it becomes increasingly challenging to handle the workload manually. The techniques mentioned can provide scalability by efficiently processing and evaluating a large number of resumes, ensuring a consistent and thorough analysis across all applications. Plus, by automating the initial stages of resume analysis, companies can significantly reduce the time and resources spent on reviewing unqualified candidates. This allows HR teams to focus their efforts on more strategic tasks, such as interviewing and assessing the most promising candidates.
- Reduction of bias: the implementation of the solutions describe has the much needed potential to reduce bias in the candidate evaluation process. By focusing on the semantic alignment between the job requirements and candidate attributes, the evaluation becomes more objective and less susceptible to unconscious biases that may influence manual resume screening.
- Continuous learning: Machine learning-based approaches, including can continuously learn and improve over time. By leveraging historical data and feedback from recruiters, these systems can adapt to the organisation's specific needs, refine their matching algorithms, and become more accurate and effective in identifying the most suitable candidates. Thus, setting the foundations now will certainly yield tenfold benefits in the future.
Having analysed the resumes, let's move on to the crucial stage of candidate interviews.
1.3. Candidate interview 2.0
During interviews, recruiters often seek assistance to streamline the process and ensure effective communication with candidates. To address this need, our team is developing an innovative app that employs speech-to-text models and advanced algorithms. This app aims to provide real-time support to recruiters during interviews.
By leveraging the app, recruiters can enhance the structure of interviews and ensure that crucial questions are addressed. The app utilises various elements, including job position descriptions, a list of standard interview questions, candidate resumes, and real-time transcripts of conversations.
Through the use of LLMs, the app analyses the interview content and generate relevant questions and feedback. Recruiters can customise the frequency at which they receive this assistance, tailoring it to their preferences and needs.
By offering this app, we aim to optimise the interview process, saving time for both recruiters and candidates while promoting efficient and insightful conversations.
1.4. End-to-End Automation?
Are you now planning on a full automation of the recruitment process? __ Note that I never suggested using AI chatbots or other method to automate the whole recruiting process. Why? Well, I do not think we are at that point yet, and we can benefit much more from keeping the human factor. While automation and advanced techniques can greatly enhance the talent acquisition process, it's important to recognise the value of human judgment and expertise. Employing a human-in-the-loop approach, where the results of the automated techniques are reviewed and validated by human recruiters, ensures a balanced and accurate evaluation that combines the strengths of both human intelligence and Machine Learning algorithms.
As a nice closure, I leave you something to meditate on: if we automate the process using tools such as AI chatbots, the candidates can also use AI to learn how to convince these chatbots in a kind of reinforcement learning process…
2. The Talent Retention Challenge
Once you've managed to hire a candidate, next comes the most important piece of the puzzle, retaining the talent to avoid wasting all the effort put on the hiring process, and avoid the damage that the employee exit could cause to the organisation. Regarding this problem there are several things you could do.
2.1. Employee Attrition Prediction
Is your organisation predicting the probability of each of its employees leaving the company in the next quarter/semester/year? __ Successful employee retention often relies on proactive measures taken before the issue arises. How many times has your organisation made counter offers way too late for an employee to change his/her mind? Probably too many to count. To address this challenge then, it is crucial to detect employees at a high risk of leaving the organisation before they reach that point.
In this context, glass-box machine learning models can be developed to predict the likelihood of an employee leaving within the next n months and identify the factors contributing to this probability. Unlike black-box models, glass-box models offer interpretability, enabling a clear understanding of the reasons behind the predictions. This distinction is crucial because we are dealing with sensitive employee data, and the actions recommended by the model's inferences could significantly impact an employee's career trajectory. Therefore, complete transparency and understanding of the model's workings are paramount.
With this in mind, my colleagues and I have developed a model that addresses the shortcomings commonly observed in mainstream "solutions" of this issue. To go into the technical details of how to approach this problem following a correct framework, I recommend referring to this article.
2.2. Personalised Employee Development
Is your organisation offering a custom professional path? __ Employees often seek a balance between competitive compensation and opportunities for professional growth. While both aspects are desirable, younger professionals, for instance, may prioritise the chance to develop their skills and advance in their chosen field. By recognising this, organisations can leverage personalised learning and development plans to enhance employee retention by fulfilling their part of the unspoken bargain.
AI-powered recommendation systems can play a significant role in this regard. These systems leverage employee performance data and analyse individual strengths, weaknesses, and career goals. They also consider information from "similar" employees who have achieved success in similar roles or career paths. By integrating these insights, organisations can create tailored recommendations for training and "upskilling" opportunities that align with each employee's specific needs and aspirations.
Additionally, on the mainstream side, AI-powered recommendation systems can provide ongoing guidance by suggesting relevant learning materials, courses, mentorship opportunities, or networking events that align with each employee's development plan. By adapting to individual preferences and evolving career paths, these systems ensure that employees have access to relevant and engaging learning experiences throughout their journey with the organisation.
Next, tightly linked to the retention issue we have the engagement problem.
3. The Employee Engagement Challenge
Employee engagement, a measure of an employee's emotional and physical commitment to their work environment, is critical to an organisation's success. It contributes to higher performance, job satisfaction, and retention rates. Currently, engagement is typically gauged alongside job satisfaction through periodic surveys, but this approach falls short in two ways:
- Frequency: The infrequent nature of these surveys results in outdated snapshots of engagement.
- Complexity: These assessments often require multi-dimensional analyses that are challenging for humans to perform accurately.
To tackle these issues, we propose an approach that considers the organisation as a network of interacting nodes, providing a holistic perspective.
3.1. Organisational Network Analysis
Organisational Network Analysis (ONA) uses techniques from graph theory to systematically explore and understand an organisation's network, including management structures, interpersonal relationships, and information flow (Barabási, A. L., 2013).
To implement ONA, the network is constructed either through nomination surveys or using digital footprints from workspace platforms (mindful of data protection considerations). After establishing the network, traditional graph algorithms can be employed to determine the centrality of each member and their influence within the organisation. These metrics, combined with employee characteristics and past survey insights, can then be used to develop a model predicting employee engagement.
By leveraging ONA, organisations can gain valuable insights into their internal dynamics, identify key influencers, assess each individual's role, and predict employee engagement levels. However, it is imperative to ensure data privacy and protection throughout the process. Strict compliance with data protection agreements and ethical considerations should be maintained during data collection and analysis.
Note that ONA is not AI… but a first step to gather relevant data in a functional way that will allow us to create or add to our current model developments. For example, we could extract valuable data about the interaction or connectivity levels within the organisation and use this information as features for the attrition model or and engagement model that aims to predict the level of engagement within an organisation at any given time. For those interested in getting a more detailed explanation of this topic I recommend you to refer to this article.
3.2. AI Engagement Assistants
Have you considered extracting both quantitative and qualitative insights from written satisfaction forms? AI can play a pivotal role here. Rather than creating AI assistants to monitor employees – which risks devaluing the human factor – we propose using them to solicit anonymous feedback on company improvement opportunities. This feedback can be analysed and structured for the HR department to extract actionable insights, akin to the function of the earlier proposed interview assistant.
This approach can also be valuable for more specific tasks. For instance, when seeking feedback on workshops, instead of using traditional 1 to 5 scale surveys, attendees could be encouraged to provide detailed written feedback. This qualitative data can be analysed using LLMs to uncover deeper, more meaningful insights.
Although written feedback might sometimes lack quality and specificity, the rich context it provides makes it a worthy avenue for exploration.
Next, we explore the interconnected challenges of engagement, retention, and performance management.
4. The Performance Assessment Challenge
Traditional performance assessment processes often face issues related to uneven quality, incompleteness, and time consumption.
Uneven quality: arises from varied writing skills and communication abilities among managers when providing feedback. Poorly constructed or unclear feedback can fail to impact employees positively, failing to address areas for improvement or recognise achievements effectively.
Incompleteness stems from overlooked achievements or areas for improvement during the assessment period. This lack of comprehensive feedback may result in missed opportunities for guiding employees towards better performance or acknowledging their successes.
The process of crafting thorough performance reviews is time-consuming, which can divert managers' valuable time away from more productive activities, such as coaching or mentoring employees.
These challenges necessitate an efficient, streamlined approach to performance assessments that optimises resource utilisation, ensures fairness, and enhances the quality and completeness of feedback.
4.1. Assessment Assistant
For those grappling with remembering key details while conducting multiple employee reviews, a solution might be in leveraging summarisation models. In fact we've built a solution for this problem.
Imagine we have an organisation named "TheBestCompany". In TheBestCompany, they utilise LLMs models to process multiple data sources and generate employee reviews. Various data sources like feedback, participation in initiatives, and customer feedback are compiled and fed into the LLM, which then distinguishes between achievements and improvement areas and consolidates this feedback into a performance review template that can then be modified by the current human reviewer.
This automated review generation process saves time and resources, minimises bias and errors, and generates detailed and accurate reviews. The flexibility of LLMs allows customisation based on organisational needs, enhancing the accuracy, efficiency, and fairness of performance assessments. This solution enables organisations to focus on other important tasks and objectives.
5. Tackling the Diversity and Inclusion Challenge
Diversity and inclusion pose enduring challenges within many organisations, extending beyond the hiring process to include promotions and performance appraisals. Subjectivity in these areas can unintentionally obscure the merits of one employee and the shortcomings of another (Rodgers, W., et al., 2023).
Two strategies can help mitigate this. The first involves developing algorithms that dismiss sensitive variables such as age, gender, ethnicity, and religion. These algorithms, focusing on qualifications, skills, and performance, encourage objective evaluations that can reduce biases and ensure fair decisions.
The second strategy utilises models that account for these sensitive variables to understand their impact on certain outcomes. Analysing historical data, these models can reveal potential biases associated with specific attributes, helping organisations to proactively identify and rectify systemic issues, fostering equal opportunities and inclusivity.
However, it's crucial to remember that algorithms and models should supplement human judgement and expertise, not replace them. Moreover, these tools should be regularly audited to ensure they don't unintentionally perpetuate biases.
6. The Data Quality Challenge
Data quality is a frequent obstacle in HR, with manual inputs leading to potential errors, inconsistent formatting, or incomplete records in HR platforms like Workday or SAP HR. These issues can compromise the accuracy and reliability of machine learning algorithms, encapsulated by the phrase, "Garbage In, Garbage Out". Implementing a meticulous ETL (Extract, Transform, Load) process is, therefore, vital for data integrity, bearing in mind ethical considerations and the impending data protection challenge. In fact, almost 90% of the time destined to the development of all the use cases mentioned through out this article should be spent on the ETL process.
7. The Data Protection Challenge
HR data often includes sensitive personal information, requiring robust data protection measures. To safeguard this data, organisations must employ strong data governance practices, including stringent security measures, access controls, and data encryption techniques (Hamilton, R. H., & Davison, H. K., 2022).
It is essential to establish clear policies for data handling, storage, and sharing, granting access only to authorised personnel. By prioritising data protection and adhering to legal regulations, organisations can build trust with employees while preserving the privacy and confidentiality of HR data.
Working closely with legal and compliance teams is also necessary to meet all applicable laws and regulations, especially specific data protection laws like the GDPR in the EU. In all these processes, it's important to seek legal counsel and advice from data protection professionals to ensure compliance with local laws and regulations.
Concluding Remarks
All in all, we have seen that the intersection of AI and HR opens a realm of possibilities that are waiting to be explored. In this scenario, the challenge for data scientists and business professionals is to translate these possibilities into tangible solutions that streamline HR processes while preserving the valuable human element. It involves both harnessing AI's power and understanding the subtleties of HR management. Without a doubt this balance will drive the future, creating a more effective, inclusive, and empathetic HR industry.
Stay tuned for more insightful articles in this series as we endeavour together to improve the world through the democratisation of innovative solutions.
References
[1] Chowdhury, S., Dey, P., Joel-Edgar, S., Bhattacharya, S., Rodriguez-Espindola, O., Abadie, A., & Truong, L. (2023). Unlocking the value of artificial intelligence in human resource management through AI capability framework. Human Resource Management Review, 33(1), 100899.
[2] Pereira, V., Hadjielias, E., Christofi, M., & Vrontis, D. (2023). A systematic literature review on the impact of artificial intelligence on workplace outcomes: A multi-process perspective. Human Resource Management Review, 33(1), 100857.
[3] Barabási, A. L. 2013. Network science. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1987), 20120375.
[4] Rodgers, W., Murray, J. M., Stefanidis, A., Degbey, W. Y., & Tarba, S. Y. (2023). An artificial intelligence algorithmic approach to ethical decision-making in human resource management processes. Human Resource Management Review, 33(1), 100925.
[5] Hamilton, R. H., & Davison, H. K. (2022). Legal and ethical challenges for HR in machine learning. Employee Responsibilities and Rights Journal, 34(1), 19–39.
