An easy but painful piece

A few weeks ago I wrote a post about participating the public goods game of the academia by reviewing papers. That’s important, and in a sense irrespectively of the quality of the review.

In general, a single review can surely be influential, but within more healthy decision making processes it cannot single-handedly decide the destiny of a manuscript (often three reviews are necessary for a single paper evaluation, in computer science), so bad reviews are probably still useful for making the overall evaluation process move on. At least if they remain a minority. I’m not saying here that it’s reasonable and altruistic to write bad reviews, just that - as long as their share of all reviews remains below a certain threshold - the overall decisions should be ok. I can perfectly imagine that many of the readers will have at least an example of a situation in which they got a majority of bad and unreasonable reviews (I have my share, thank you), but the temporal dimension of the scientific process enterprise is also important, so if bad decisions were at least timely (and a desk rejection two months after the submission is not timely) we would have the possibility to pursue another occasion, maybe grasping something good in the process (even overall bad reviews sometimes include some good advice).

The work of an editor is a hard one, they should deal with situations like these, and I suspect that too many editors around do not have a full sense of the importance of their work, and maybe they accepted to take on the role without really wanting to do it. As a consequence, I really think that there are a lot of editors not doing their work properly around.

But the central part of this post is about a “deadly sin” in reviewing that I stumble upon with an alarming frequency, both as an author and as a reviewer. Let me exemplify with a fictional but pretty realistic portion of a review:

Results are described without any comparison with other techniques. Therefore, it is unclear how and to what extent the proposed technique is better than many others in the literature. Please show comparisons with different techniques, also from the point of view of computational complexity.

Of course, the review presents a bad evaluation and bad scores, with the implicit or explicit (through the recommendation) intention of having the paper rejected, giving the “please add this or that” part a grotesque pedagogic intent: “do better and avoid wasting my time with such a poor work”. Of course, reviewer’s time is so precious that she cannot waste it by mentioning even one of the “many other analogous techniques in the literature”. Of course, the reviewer is not aware (or suppresses the thought) of the Dunning-Kruger effect and she rates her confidence in reviewing the paper as very high (by the way, since we all know about Dunning-Kruger, why don’t we stop asking the reviewer to self-rate the confidence? Or why don’t we do the opposite of what some electronic conference management platforms do: give more credit to reviewers honestly reporting a low confidence?). Of course, the editor (or senior programme committee member, or whatever you call it) should detect these situations and ask for a more careful work by the reviewer, but since it took her considerable effort and time to get this report she doesn’t want to risk spending weeks finding another reviewer after irritating the previous one to dropping the ball.

Yes, it’s another rant, but it’s a cliche that people in the academia love to talk about problems and really not do anything about them… and the last time I checked I was still working in the academia instead of putting energies in my passions: teaching and doing research.

Before concluding, also a consideration about originality: as many others have more authoritatively stated before me, science is a complex process and it is simply delusional to think that every single paper advances the state of the art by adding some new technique that works better than the previous ones, at least in some situation. Papers describing roads that, for some reasons, are not promising are so important that Journals of Negative Results exist in some disciplines. Confirmatory works are important since errors (or plain frauds) do exist. Science is a collective construction, with walls and towers whose structural integrity needs to be tested with the right amount of skepticism, with structures that have to be shored up constantly. We are not all geniuses in the academia, we also need honest workers that are maybe less talented but with the integrity and intellectual honesty to keep all of us in line.