From time to time I serve on peer-review panels for academic work on Knowledge Management and Information Science, and although it is somewhat arduous work, it is very rewarding and goes to the heart of how science works and how knowledge can be built through peer review and layers of quality-control. It is by no means a perfect system, but as a mechanism to reduce errors and poorly formed arguments, it functions very well.
On average I approve or promote approximately only one tenth of the papers I review and reject or return the remainder. Considering that I only receive papers that have already passed muster for basic format and construction, you might think that my ratio is overly aggressive and may be dooming many authors unnecessarily.
In this you would be right – by trying to keep false positives very low I would be making more type II errors and thus sacrifice some papers that had merit. However, progress of knowledge in science is not only incremental, but in many senses, inevitable.
Progress: Science vs Business
Even though it is tempting to think of scientific advances in terms of the “great minds” perspective, even our greatest breakthroughs were inevitable to a significant degree. If Einstein had not made the connection that Maxwell’s findings suggested, eventually somebody else would have. If Einstein had simply remained a patent clerk, physics would have been delayed by only a few decades. If Darwin had never made the historic trip on the Beagle, the theory of evolution would perhaps only be celebrating its 100 th birthday, rather than its 151 st.
One reason for the almost ineluctable progress is that results are published and available for falsification or verification, and above all – re-use. While this process works well in science in general, the same does not hold of course in microcosm because time is a constraint and re-use is constrained to a far smaller subset. For individuals or organizations where immediacy is more of concern than eventual progress, the kind of delay and high mortality rate of the peer-review process can be deadly.
In the case of a commercial firm, the tolerance for error cannot be set so low since although the innovation or solution to a business problem will almost inevitably be found, it may not be found in time to save the specific firm, or be found by that firm – it may be found by a competitor or too late to be of any use in gaining competitive advantage.
Let’s take the case of intellectual assets and recruitment.
A problem of assets
In many ways a firm has a critical scientific problem to solve, and that is to identify and acquire intellectual assets (people of course) that are at least as good as the competition, and preferably better.
It doesn’t take much reflection to conclude that if you have less intellectual firepower, competing is going to be harder and survival against competition for limited business, more difficult.
Digesting resumes is much like reviewing papers, except that while resumes are easier to read and usually much thinner than scientific papers, recruiters get more resumes than a reviewer gets papers.
.. but do they?
For every paper I receive as a reviewer, I guess that about ten times that number were screened out before the review stage, and perhaps a hundred times were aborted when the respondent read the submission criteria.
So on the whole the volumes are probably in the same realm.
Tolerance for Error
Returning to an earlier point, while science as a whole can tolerate a significant loss of papers that were actually valid, the same is not true for businesses – if you turn down a candidate who was better than you thought, you wind up with the lesser candidate on a long-term basis and your competitor probably gets the better one.
… but it gets worse.
When I get a 2500-word paper, I read the entire thing, sometimes several times to make sure I understood it, and when I reject a paper, I do so with copious notes that go back to the author, and therefore (1) they learn and their resubmission and all subsequent submissions are more successful, and (2) only fundamental flaws lead to complete rejection.
In contrast, when a resume is rejected the applicant has little clue as to the specific reasons and cannot resubmit, and rejections based on superficial or irrelevant objections are undetectable.
… but that’s not all
While I will spend several hours on reviewing a paper, recruiters are said to spend less than a few minutes.
… and finally
The peer-reviewer is selected from a pool of experts in a field and any conflict of interest is declared, but recruiters often evaluate resumes for positions in which they are inexpert and no declaration of conflict is either recorded nor required.
The peer review system deals with a similar bulk of documents, but while it has better safeguards and more rigor than recruitment processes, it can afford a far greater loss of positives than recruitment. Yet while recruitment has a similar rejection rate, it has none of the accompanying safeguards.
Perhaps recruitment would be better served by something more akin to the peer review process.
Please contribute to my self-knowledge and take this 1-minute survey that tells me what my blog tells you about me. – Completely anonymous.
Matthew Loxton is a Knowledge Management professional and holds a Master’s degree in Knowledge Management from the University of Canberra. Mr. Loxton has extensive international experience and is currently available as a Knowledge Management consultant or as a permanent employee at an organization that wishes to put knowledge to work.