Psychometric testing in the workplace can be a thorny subject. 

Unless I’m in a particularly feisty mood, I’ve learned it’s a topic best avoided at social gatherings. But in a professional setting, I know it’s safe to fill my boots!  That’s because among psychologists, views about assessment tools are generally positive and most have no hesitation in using them as part of their commitment to best practice and professional standards

But there’s a ‘but’:  

Why? Because there’s a myriad more ways workplace testing can risk being “incorrect” and they occur with more prevalence in workplace settings than health settings.  

So where can it go wrong?

For the purposes of this article, let’s view “incorrect” workplace testing as falling into 3 categories: 

1. Right test… used in the wrong way

Example:  

A large financial services organisation is using a work-based personality questionnaire as part of a new leader development program aimed at improving self-awareness. The instrument has good reliability and validity (psychometric properties ✅), however, the results are not debriefed in a way that is supportive of the reasons for using the test. Moreover, individuals’ test results are shared with line managers and/or peers and sometimes this happens without transparency. Sometimes the personality assessment results are accessed and used for other reasons without the individual’s knowledge.  

In this scenario, a good test is being implemented poorly with the result that the ROI for the company, the program’s integrity, the opportunity for participant development and the psychological safety climate are all being compromised. The organisation is also failing to uphold ethical standards for data privacy and security and is quite possibly in breach of its legal obligations. 

2. Wrong test… used in the right way

Example: 

A medium sized professional services firm cares for its employees and has a sincere desire to protect and improve their wellbeing.  The firm has a solid employee wellbeing strategy and accepts the importance of conducting assessments to identify the wellbeing risks/needs experienced by their employees. They used a reputable off the shelf employee burnout test to conduct a baseline measure of employees’ wellbeing (and to track subsequent improvement), and all employees were offered confidential feedback on their individual results. Team workshops were held to debrief the collect results.

Sounds good, yes?  So, what’s the problem?  

In this scenario, while the test being used has strong psychometric properties as a reliable and valid tool – those properties only relate to its efficacy as a measure of burnout whereas the organisation’s presenting need and intention was to measure employee wellbeing.  Burnout and employee wellbeing may be used interchangeably in everyday work conversations but in practical and scientific terms they are not synonymous. While they share some characteristics (namely self-regulation and resilience) their underlying constructs are different; wellbeing is a broader phenomenon.  After eighteen months of baseline and repeat testing using only the burnout test, the organisation realised its view of employees’ wellbeing was skewed and incomplete.  

While the organisation’s wellbeing strategy and implementation was generally well developed and thoughtfully deployed, including the use of a ‘good’ test, unfortunately in this case it was also the ‘wrong test’.  Wrong – not because the test had poor reliability or validity credentials – but simply because it was not the most appropriate test for the job at hand. It did not provide an accurate or complete measure of employee wellbeing which was the issue of primary concern to the organisation.  Just goes to show, even a ‘good test’ can be a ‘wrong test’.  

3. Wrong test… used in the wrong way

Example: 

This is the follow-up chapter to the scenario outlined in Example 2 above. After the organisation realised using a burnout test (even a reliable and valid one) wasn’t providing them with a sufficiently complete, accurate or comprehensive measure of employee wellbeing they decided to augment the burnout measure. Sensible idea. They did this by adding some questions about wellbeing, happiness, mental-health and psychological safety into their employee engagement survey. Less sensible!   

In this not uncommon scenario, the issue of most concern to workplace psychologists is whether the extra questions/tests which have the appearance of being the ‘right’ questions (nobody quibbles about why they are being asked) may nevertheless be ‘wrong’ because their reliability and validity as an accurate measure for the area under investigation is unknown or does not meet the statistical or conceptual standards required by psychologists (more on those later).  

The psychometric qualities of a test can be a real worry.

The main risk in this example is that there may be so much error or white noise in the data collected, that the conclusions, actions or interventions which result will be misleading and unhelpful or possibly indefensible and/or negligent. ☹️ 

There are other risks too. Where employee wellbeing is the primary focus area as per this example, the risks of multiple missteps occurring are so significant they would provide enough material for a whole other article. For today’s purposes however, I’ll give a prize to whoever gives the best summation of why this example is illustrative of an organisation using the ‘wrong test/s’ in the ‘wrong way/s’.

If you’re stumped, don’t feel bad – it is a challenging professional and practical problem. So much so that at the EAWOP Congress in Poland May 2023 there was an entire panel of experts drawn from lead professional bodies (European Federation of Psychologists’ Associations and International Test Commission) and global test publishers who were brought together to address these very issues.  

How can incorrect practices be stopped?

To decrease incorrect testing practices within workplaces, the panel debated whether workplace assessments should be subject to tighter controls and greater governance structures, and they addressed the perceived need for increased regulations as the primary mechanism to increase the standards of testing practices related to assessment in work and organizational settings. Across most geographical jurisdictions, workplace psychologists already have clearly articulated standards of practice with explicit criteria to determine the quality of tests and their usage – guidelines which are developed and/or promoted by their national psychological associations.  

Mercifully (imo!), research has shown that simply producing more guidelines and regulations focussed on improving assessment standards is a double-edged sword which does not usually translate to improvements in workplace testing practices.  

The absence of robust guidance is not the issue; it’s the application of the testing standards and the evaluation criteria which is both more contentious and where the opportunity lies – mainly because upholding governance standards is currently contingent upon on self-regulation (either that or public shaming).

Since most of the tests and questionnaires which are developed and available for use in workplaces are produced and controlled by the large test publishing houses, it is their actions which tend to have the most significant ripple effect on practices observed. However, beyond professionals in the psychological community and the most diligent of human resource experts, who else would really know or care about the lead bodies’ psychometric testing guidelines or how well standards for assessment are being upheld in practice? Honestly? 

Arguably, psychologists’ regulations and guidelines offer some scope to ‘call out’ less than thorough assessment practices. But I find myself wondering what mechanism or process would be practical and acceptable for doing so? In practice, there is little in the way of deterrence or accountability which prevents organisations or individuals from simply decreeing that their favourite tests, surveys, questionnaires or other assessment methodologies are ‘good’.  

So who is best placed to stamp it out? 

In a world infamously subjected to fake news and social media algorithms, getting to the ‘truth’ about standards and qualities of workplace tests and the ways in which they are being used in organisations is difficult unless you (1) know exactly what you are looking for, (2) know the right questions to ask, and (3) have an expert guiding you through the process. Even for many industrial/organisational psychologists, it’s a topic which sees their eyes glaze over.  And psychometric testing is fallible to the same follower fads as any other industry, product or practice.     

So…

If decision-makers within the consultancies and organisations who are using tests, questionnaires and surveys:

1. Perceive there are compelling professional, ethical, commercial and legal imperatives which necessitate adherence to best practice guidelines; and

2. If they have the depth of technical understanding needed to correctly evaluate whether a test/assessment methodology is truly fit for purpose; and

3. If they have the mandate and motivation to ensure that not only are the right tests chosen but also deployed and evaluated appropriately …

Then “incorrect” testing practices could soon be stamped out.   The conundrum facing workplace psychologists, employers and employees is… how do we elevate the changes required to achieve these conditions?  

A POV from a test publisher  

As a specialist in leader wellbeing assessment/development and as the developer and publisher of the GLWS measure of leader wellbeing, we can attest to the array of challenges associated with upholding best practice test development and usage standards, and we can say for sure that adhering to best practice doesn’t come cheaply.  

• During the ten years GLWS has been under development, there have been times we regarded conventional best practice test development standards as a nuisance, as stifling our innovation, as a barrier to practical progress and a stumbling block for gaining traction in the marketplace.  
• While we too are self-regulated and free without impost or impunity to decree our test as being a ‘good test’ we have undertaken the painstaking work required to provide attestations of reliability and validity. 

Many of the concerns relating to the misunderstanding, misleading or misuse of workplace psychometric assessment tools appear worryingly commonplace. Widespread “incorrect” assessment practices are an almost everyday business challenge.

In the case of our business and the GLWS, the high economic and human costs which occur when leader wellbeing is not properly assessed and developed are profoundly troubling.  

What needs to happen next and what can YOU do?

First of all, well done on reading this far!  Because herein lies the answer – it requires all of us to CARE.  And to invest the time and effort in caring. 

Ensuring regulations get a prominent place in the curriculum of university psychology and supervisors is important – no psychologist’s eyes should ever glaze over at the mention of psychometrics – but in some ways this is preaching to the (at least partially) converted.  Among those already committed to high professional standards, more regulations don’t strike them as the answer.

What’s needed to make a real difference, is more people paying more attention to the standards which already exist and increasing multi-level, multi-stakeholder commitment to better implementation of these guidelines.  

The divide between academic scientists, practitioner psychologists and organisation practices over what constitutes acceptable and useful workplace testing practices has indeed been a thorny issue, and it remains so today.  

If you are interested in taking this conversation further within your organisation and/or with other organisations, please contact me with a view to establishing a working group forum where the purpose is to discuss and share ideas for how to optimally support the growing demand for workplace testing and assessment by diverse professionals (not just psychologists). Our particular interest is improving applications of testing where the goal is individual, team and/or organisational development rather than assessment for selection purposes.  

I dream of the day I can profess my love of a good psychometric test without fear of being seen as a social pariah. Until then, here are some of the key resources and references which may be of help to you too.  

Professional Standards & Guidelines 

• European Federation of Psychologists Association. (2012). EFPA Standards for test use: work, education, and health and social care, levels 1, 2, and 3. 
• ITC Guidelines on Quality Control in Scoring, Test Analysis, and Reporting of Test Scores. International Journal of Testing, 2014.  

References 

• Frontiers | Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer (frontiersin.org) 
• Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in public health, 6, 1 
• MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS quarterly, 293-334.