Put it to the Board

And I thought this pillar      was solid

Just say the word “ranking” to an economist and his mouth begins to water. GDP, FDI, indebtedness, unemployment levels, income per capita, shadow economy, anything and everything lends itself to constructing a ranking out of. Even economists.

But what is the unit of measure to rank economists? Well, as Paul Samuelson once put it, the only coin of worth for economists is recognition by their colleagues. And this recognition comes in the form of citations of their insights and findings. And these, in turn, are immortalised in their respective publications.

So this is what most rankings of economists measure: number of publications and number of citations. A number of refinements have been introduced to provide more accurate ratings, such as considering only publications in the most renowned journals in the trade, trawling for citations in equally high-brow journals and so on.

But problems still abound, from the unexpectedly mundane, such as the letter with which your last name starts (see article here), to the period of time or type of publications considered. Now Bruno Frey and his colleague Katja Rost have found a much bigger, fairly terminal problem: current rankings do not measure research quality effectively. And quality, after all, is what everything is about.

This is not a moot point. Rankings determine the career fortunes of most economists, all the way from first postings through tenured professorships and, the holy grail, a Nobel prize. As the authors point out, the ranking of individual scholars is particularly far from objective. The results differ depending on whether research quantity or quality is considered. As Mr Frey puts it, "it should not so much matter how many publications have been produced, and how many citations have been accumulated, but rather what new insights have been produced and how valuable these are for society."

While acknowledging that an effort has been made to include quality aspects in rankings, the authors find fault with the practice of leaving out publications in books, handbooks or other collections of articles, or those for policy purposes or for the general public, as they may well contain important scientific information. For these reasons, they call for much more careful review of such rankings than is the rule today.

Among the many shortcomings they list, they mention the fact that citing a particular work does not mean it has been read: two researchers cited performed a careful statistical analysis and found that about 70-90 percent of scientific citations are copied from the lists of references used in other papers. In other words, 70-90 percent of the cited papers have not been read by those citing them.

After analysing many of the rankings currently in use and identifying their particular shortcomings, Mr Frey and Ms Rost explore how, and to what extent, quality can be captured by an alternative definition of scientific worth, namely membership on editorial boards.

The editorial board of a professional journal plays an important role in disseminating and evaluating information. As another researcher quoted points out, these positions are held by persons who have the confidence and trust of their colleagues, as otherwise they would fail to attract quality submissions. The journals, in turn, want to profit from the reputation of their board members. Thus, membership on boards can be taken to be a reasonable approximation to the quality of a scholar as judged by his or her peers.

Mr Frey and Ms Rost then present rankings based on editorial board membership for a sample of 115 economics journals and, bingo, when they compare them with current rankings, they find that the two kinds of ranking deviate considerably from each other. Indeed, their study suggests that rankings based on quantity are incommensurable with rankings based on quality.

Still, the authors acknowledge that the rankings based on board memberships also have their limitations. The scientific board ranking computed in this study, they stress, should not be considered a clearly superior alternative to rankings based on publications, citations or other quantity-related measures.

Therefore, instead of suggesting the board ranking as a gold standard, they propose that committees charged with deciding on promotion, hiring or funding decisions should make use of this information to find experts who can help them to evaluate research quality based on a direct evaluation of the content of the publications.

Bruno S. Frey and Katja Rost: Do Rankings Reflect Research Quality?, CESifo Working Paper No. 2443

• Other CESifo Working Papers dealing with rankings
• See our previous article on rankings of economists
• Other CESifo Working Papers by Bruno S. Frey
• CESifo Working Papers published in the past three months