Gaming bibliometrics 101 - 1st episode

I don’t think I can really add something to the debate on research assessment. But maybe I can do something useful, at least to those that did not follow thoroughly the subject. I can provide a summary with some selected readings.

I’d start with the Declaration on Research Assessment (DORA): it is a compact, reasonable, well-written argument pointing out flaws in the ways in which researchers and the outputs of scholarly research are evaluated, and proposing several ways to improve the situation. Centrally, it states the need to assess research on its own merits rather than on the basis of the venue of publication. It also somewhat puts the emphasis on quality rather than quantity, since many current metrics and indicators are used to evaluate quality by measuring quantities.

Metrics and indicators, when used for sake of evaluation, become a target and they quickly cease to be good measures, according to the - apparently not sufficiently known or recognized - Goodhart’s law. Jerry Z. Muller, in an extremely interesting book titled Tyranny of metrics, shows examples of the effects of what he calls an obsession.

Our zeal to instill the evaluation process with scientific rigor, we’ve gone from measuring performance to fixating on measuring itself. The result is a tyranny of metrics that threatens the quality of our lives and most important institutions.

There is, of course, a debate on the topic. In particular, in Computer Science, in addition to journal rankings by private enterprises such as Clarivate and Elsevier, there is an influential ranking of scientific conferences created and maintained by an association of university departments of computer science in Australia and New Zealand, the Computing Research and Education Association of Australasia, CORE Inc. It is a local organization whose influence has far exceeded the range of its membership. Last year CORE has produced a document (CORE and the DORA Principles) presenting a set of comments to points proposed by DORA. I let the reader evaluate what is more convincing, but I have absolutely no problem saying that I am much closer to DORA that to CORE’s commentary. I’m probably even farther from CORE than DORA, if you put respective positions in a line, to tell it all.

Having said that, I want to start a series of suggestions on how to game the system of research assessment for fun and for profit (as some copywriter or scientific paper author would say). Just to make a point in how unreliable are in practice some of the widely adopted indicators of research assessment.

Let us imagine that there is a field or a discipline called sausage making that has nothing to do with an extremely trendy and popular technique called artificial positronic spicing. Let us say that artificial positronic spicing can be applied to a wide range of human activities, including sausage making, but sausage makers have never widely employed artificial positronic spicing. Of course, there are textbooks for artificial positronic spicing, but they are clearly not read by students of sausage making. Of course, there are whole journals devoted to research on sausage making, mostly read by researchers and some practitioner in sausage making. Let us also say that sausage making journals are highly reputed, due to a high sausage factor.

Consider then a far looking, ambitious researcher in sausage making that is also reasonably aware of artificial positronic spicing techniques (or that decides studying these techniques sufficiently to be able to apply them reasonably). Or even think of a powerful and rich sausage maker researcher with sufficient funds to make a couple of rich post doc positions on artificial positronic spicing for sausage making. Besides applying these techniques for sausage making research and applications, why not writing simple 101 papers based essentially on textbook artificial positronic spicing basic methodological suggestions and submitting them to sausage making journals, just putting some innocuous reference to sausage making specific data? The editor-in-chief of the sausage making journal is also pleased with the idea, since the paper could be highly cited both in the journal and in general in the field of sausage making, inflating the sausage factor of the journal, that is based on the ratio between the number of received citations (irrespectively of the semantics of the citation, or the authors) by the published articles and their number within a year. Of course, the authors are happy since a relatively simple work leads to an increase in the total received citations and the sausage index… and I guess you got the point of what that is. So, it’s a win win situation, everybody is happy, and - as long as no expert in artificial positronic spicing reads the paper - no one even spots the fact that the paper is not very original, using an extreme euphemism.

Now, if you are hungry with all this sausage talking (or disgusted, being vegetarian or vegan), I confess I am as well. If you are confused, I can sympathize and I apologize for rhetorical tricks spread all over the second part of the post. My point is that the authors, especially the young post doc expert in artificial positronic spicing but working in a sausage making group, did nothing particularly wrong, from an ethical perspective. Even the editor in chief of the sausage journal did not. The point is that the research assessment metrics, and a tendency to use mostly if not exclusively those metrics in evaluating and comparing profiles of researchers, cast a dark shadow over what could otherwise be considered a completely reasonable cultural editorial operation, and instead can be viewed as an utilitarian and exploitative near gaming behavior.

I’ll occasionally follow up this post with additional techniques for enlarging your indexes as spammers used to say some time ago…