Pseudoscience and Gobbledygook

Numerous tools exist that purport to analyse a person’s abilities, aptitudes and even their suitability for certain jobs or tasks – but are they any good?

The world is full of tools – tools for building houses, for fixing the car, for cutting your hair.  Clearly there is money to be made in tools, so it should come as no surprise that the world of employment has its tools too – tools to assess a person’s abilities, their strengths and weaknesses, their personality, there are even tools that claim to measure the suitability of a person for a particular job.

But are they any good?

Inputs and outputs

The first difficulty I have with most of the so-called “psychometric tools” is that the relationship between the inputs and the outputs is unknown.

To explain, let’s go back to physical tools.  If you have a nut you would like to tighten you would use a spanner.  If the spanner works well and you tighten the nut then the tool is clearly fit for purpose.

 

In the case of a psychometric tool, the relationship between the questions asked and the results produced is unknown.  It is hidden in the algorithm which is generally kept secret as it is the copyright of the organisation that owns the rights to the tool. 

But if you do not understand the algorithm, how do you know that the results are of value.

The providers of such tools would argue that the integrity of the algorithm is proven through the “validation” process.

Validation vs. value

I was recently asked to evaluate a psychometric tool and provide my opinion on its usefulness.  The tool, so I was told, is “one of the most validated that exists”.  Filled with a sense of enthusiasm and curiosity I wasted the next 20 minutes of my life choosing words and statements that either described me or didn’t describe me.

When I had finished I received a 48-page report.  Page 2 of the report contained the amazing statement: “Feel free to delete any statement from the report that may not apply”

Even horoscopes don’t rely on disclaimers like that!

The report then went on to make a series of statement that replayed words I had selected in the questionnaire back to me, such as: “His vision for results is one of his positive strengths.”

This, I assume is taken from responses that alluded to me being results-oriented and my desire to set out a clear vision of what we are working to achieve.  But read the statement again, it doesn’t actually say that, in fact, it doesn’t actually say anything – what is a “vision for results” anyway?

However, on the broader question as to whether the report was an accurate reflection of my behaviours and motivators, I would have to say that the answer was “yes”.  But since the report was only replaying my answers back to me, how could the answer be anything other than “yes”.

On the question as to whether it provided me with any new insights or additional information, the answer was a resounding “NO”!

My point is that just because a tool is validated does not mean that it is useful or has value.

Questions vs. answers

When I left university I applied to join IBM as a salesman, but didn’t get the job as I failed the aptitude test.  I applied again a couple of months later and again failed the aptitude test. I eventually joined another American multinational computing company Burroughs Machines (later to become Unisys) where I enjoyed a highly successful career in sales and sales management.

Clearly I did have an aptitude for sales.  So what was the IBM test measuring which meant that they felt I didn’t?

The problem is that the real question being asked and the answers being provided are often two different things.  In the case of me applying for a job with IBM, the real question being asked was: “Does this person have the potential to become an effective sales person in our organisation?”

The actual answer provided was: “The aptitude test score was not as high as that of other people.”

For the people who passed the test and succeeded, the usefulness of the tool is “proven”.  For the ones who passed but did not make great sales people, the tool was only measuring potential.  And for people like me who were rejected but went on to have successful sales careers – well they never knew what they were missing.

In other words, the tool gets away with it – being neither proven or disproven.

Useful tools

The tools I like and use myself are ones that pose questions, but DO NOT purport to give answers.

By far my favourite is the MiND tool.  It is not a psychometric tools as it does not assess a person’s personality, but rather looks at the foundations of personality, at the reasons why they are the way they are.  Moreover, the tool does not imply that there is anything that a person can or cannot do, simply that they may have preferences for approaching things in certain ways. Its value therefore is not in providing definitive answers, but in facilitating a discussion about a person’s approach, their preferences, the ways in which they would approach the job and the contribution they could make to the team.

Conclusion

There is no doubt that, when used in the right way, many employment tools can be useful.

But my advice is to view all tools with a healthy dose of sceptiscism and only use them if you are 100% certain that you know what the tool is measuring and what the results actually mean.  After all, I am sure that none of you would buy a spanner if you didn’t know what it was for.

About the Authors
Alistair Schofield is Managing Director of Extensor Limited.