Just a brief note on something (one more thing) I don’t understand about academic publishing.
Nobody has ever explained to me in detail what a journal expects from me as a reviewer. Nobody has explained to me what a review exactly must be. I just pretend to go along with a tradition everybody seems to know fairly well although it’s clear nobody really does. This sounds pretty absurd but it’s quite common in academia.
The blurred idea we all have about peer-review is that few colleagues with a fair knowledge of your subject of interest give some unreplicable (so, statistically speaking, random) opinions about your work so that you can improve it (hopefully) and they can help the editors to decide whether your article is good enough to be published in their journal.
This last bit is the one that bugs me: I’m sorry, I can’t say whether an article is good/important enough for the standards of your journal. I don’t think it’s fair that professional careers depend at some extend on my arbitrary opinion as a reviewer (since we all care so much about individual publication records) and I won’t damage anybody (even if it’s only a little bit) based on my particular taste. And even if I’m sure I’m right, I don’t want to act as a gatekeeper: I want different perspectives to be included, not only those I like.
That doesn’t change the fact that you can be exigent: if there’s something false, you can say you think it’s false. If conclusions can’t be deduced from the results, you can also say so. But that doesn’t change the fact that 1) maybe you’re wrong, 2) authors could always change what you think it’s wrong or incomplete, 3) maybe the article is valuable for some other reason. And even if you think the article is horrible, remember it is probably going to be published elsewhere anyway.
Then, yes, controversial articles may be considered at the same level than more robust articles, but isn’t it the case already? And for that I blame the fact that the journals don’t publish the reviews along with the articles (which makes the review process less transparent and brings all the known problems opaque procedures are known for).
So, my policy as a reviewer is: accepting to review only for society journals or non-profit organizations, writing lengthy reviews that give as much information as possible to the authors and finally, unless there’s something grossly wrong (i. e. the suspicion of some dishonest behaviour), I’m accepting the article.
Ceferino Varón González 