Michael Burowoy describes one of the current prevailing models of the university as “the regulation model,” which aims to make the university production of knowledge “more efficient, more productive and more accountable by more direct means.” In other words, to make what universities do (supposedly, “produce knowledge”) equivalent to what industries do (“produce computers,” “produce financial instruments”), and to improve that activity according to equivalent metrics: efficiency, productivity, accountability.
This requires some means of measuring what it is that all those professors, post-docs, and grad students actually DO. And since they don’t generally make physical stuff, or much money, one option is to quantify their intellectual output somehow through publications–one of the more tangible creations expected of professional academics.
All of this is to say that how we measure academic success is an important question in the age of austerity. If budgets (and one’s employment status) depend on meeting quantitative targets, people will probably alter their behavior accordingly. Which brings me to some questions about Impact Factor!*
As I understand it, Impact Factor (IF) aims to measure the relative importance of a journal through how often its articles are cited. Bjoern Brembs summarizes some research suggesting that IF is actually a better predictor of a paper’s chance of being retracted than of its being cited. There’s some evidence this may be because journals with a high IF are more likely to publish flawed studies.
IF is also predictive of the sample size of the gene association study: the higher the IF of the journal in which the study was published, the lower the sample size. One could interpret this as evidence that high-IF journals are more likely to publish a large effect, even though it is only backed up by a small sample size, while low-IF journals require a more solid amount of data to back up the authors’ claims.
If this is the case, it provokes a few thoughts.
Citations are a bit like pageviews or linkbacks: you may get more with extreme claims and controversy, even if the quality of your work isn’t great. Simply counting citations, without any attention to the context in which work is cited, is a pretty shallow measure of importance. Studies are often cited in order to disagree with or refute their conclusions.
However, if measures like IF continue to be important, especially when it comes to budgets, we can probably expect more universities and researchers to game the system. And if all we’re measuring when it comes to “impact” is how often their work gets mentioned, we may be setting ourselves up for Huffington Post-style SEO academic journals.
Setting aside the long-term solution of reining in the regulation model and ending measurement mania, what kinds of metrics might be devised that would better measure real intellectual contributions, and that would create incentives to conduct high-quality research?