The Role of Peer Review

We asked our editors, "As a journal or book series editor, what do you see as the role of peer review?" Here's what they told us.

There are at least two reasons that peer review is valuable. First, an editor needs to be something of a polymath, but you can't have expert knowledge in all areas. While it's easy to recognize that a proposal is on an important topic and has interesting things to say, you need an expert in that topic to help the author make it the best work possible. Second, as an editor you can make mistakes about the worth of a proposal. I can think of at least two instances where I thought a proposal was just too quirky to be viable, but the peer reviewers loved it, and in each case a really interesting book came about. Peer review gives you a reality check.

– Susan McLeod, University of California, Santa Barbara
Editor, Perspectives on Writing Book Series

Peer review, when done well, is a beautiful moment where reviewers and authors and editors collectively come together to refine knowledge. It's a time where reviewers and authors and editors bring all their varied experiences together to push knowledge into productive directions that serve the discipline, the journal, the readership, the local communities where this knowledge has been, is, and will be enacted. What does this mean for the actions I take? I look to connect authors with reviewers who understand where the author is coming from and where the author hopes to go; I look to connect authors with reviewers who understand how the knowledge we build with others impacts our work in our classrooms, colleges, communities. And when a review comes in, I see my role as a connector between the reviewer and authorâ€”making sure each feels heard and acknowledged; making sure the author sees a clear path forward with their writing, even if their writing needs to find a new home. We should not build walls with peer review. When done well, we build avenues to better knowledge for all.

– J. Michael Rifenburg, University of North Georgia
Editor, Perspectives on Writing Book Series

Good peer-review feedback is absolutely vital to the credibility of our discipline. Readers and fellow scholars need to be able to trust that our research and publications are done carefully and ethically, that our work is informed by knowledge of relevant prior research, and that we as writers are as up to date as we can be on the relevant new issues and trends in theory. Because no writer can be on top of every possibly relevant trend or have covered all the possibly relevant research, we all rely on members of our scholarly community to give us helpful advice and share with us perspectives that we may have missed in our drafts.

– Chris Thaiss, University of California, Davis
Author and editor of several books, including Writing Programs Worldwide: Profiles of Academic Writing in Many Places

"Peer review" denotes a process by which someone expert in a field of study reviews another author's work and determines whether it draws on expected conversations in the field, or draws on valued conversations in other disciplines and relates it to the field, and, by doing so through a unique lens, contributes to the current conversations and is worthy of publication. There is a little snark built into that long sentence. For as I do believe there should be a review process that helps the field's knowledge grow, a peer process also connotes collaboration as peers.

An ideal peer reviewer is a mentor, one who encourages and tells hard truths, points to excellence and problems, using language that will be heard — though, of course, one can't be sure it said well or heard as meant. That means when describing a peer reviewer, one also has to describe a peer author. A peer reviewer has an obligation to consider their own reactions to a piece — are they annoyed, impatient; can they shift from being a gatekeeper to an inquirer? Do they offer questions and show they are responding to the text as one reader who has read in the field rather than the arbiter-of-excellence? But a peer author also has to seek to understand what is being said, sometimes revise slowly and embrace being mentored; they need to understand that "revise and resubmit," is an encouragement. They also need to understand that sometimes, a piece doesn't fit what a journal or book wants, or that because they might desire/need/are desperate for publication, they may not actually be able to hear what reviewers are saying. When authors stop listening, reviewers can't be mentors. But if reviewers don't mentor, authors can't learn to listen.

– Joan Mullin, University of North Carolina at Charlotte
Editor, International Exchanges on the Study of Writing Book Series

When I was a graduate student, I did a lot of reading for the Educational Testing Service on exams like the TWE (Test of Written English), the CBEST (California Basic Educational Skills Test), and the Advanced Placement Exams for Literature and for Language/Composition. A lot can be said about the value — or lack of value — of these exams but it's worth pointing out that the process of reading and scoring these exams is structurally quite similar to the peer review process used by most academic journals and presses. Every student essay is read and scored by two experienced writing instructors using criteria specified by the exam's rubric, and though ETS keeps their statistics about inter-rater agreement pretty close to the chest, a publicly-available internal report on the GRE (Graduate Record Exam) from 2005 states that 57-60% of the time, both reviewers agree on exactly the same score, and 97-98% of the time, they are within one score point of each other. Only 2-3% of the time are they two or more score points apart (on a 6 point scale). That sounds pretty good, right? And if the reading and review processes are similar, doesn't it make sense that scholarly peer reviews should demonstrate the same kind of agreement as ETS reader scores?

Well, no.

When scoring essays for standardized writing exams, readers read thousands of essays written in response to exactly the same prompt; they work from a rubric that assumes a single response genre and that offers limited criteria for evaluation; they cannot critique or provide feedback to the writers; they are not permitted to score essays based on originality or potential; and they are not required to give essays lower scores if arguments are incomplete, superficial, or just get basic facts wrong. And since reader scores tend to "gravitate toward the middle," a phrase I remember hearing many times during scoring sessions, most essays are given scores near the midpoint of the scoring range, increasing chances that both readers will be in exact- or near-agreement.

That is certainly not the case with scholarly peer review, but even so, I could probably make a case that scholarly peer reviews have an extremely high degree of inter-reviewer agreement were I to consider "score points" alone. The standard score points for peer reviews are Accept, Accept with Minor Revisions, Revise and Resubmit, and Reject. When a piece is extraordinarily good — an exemplar of great research and cogent writing even in the first draft — reviewers tend to see it more or less the same way. (This happens maybe 5% of the time.) When a manuscript is extraordinarily bad, the same thing tends to occur. (This happens about 10-15% of the time.) The more relevant result is this: in the first rounds of review, the vast majority of submitted manuscripts will get a score of "Revise and Resubmit."

"Revise and Resubmit" is a scholarly reviewer's way of gravitating toward the middle. It means your work may be publishable, but it's not quite there yet.

Expect this result. Rely upon it. Embrace it. Your revision will improve as a result.

Now it's not uncommon for two reviewers to suggest you revise your manuscript in two completely different ways. When scholars are sent articles to review, there are no predetermined, agreed-upon assessment rubrics in play, so responses can sometimes be extremely different. (Academics often make jokes, in fact, about their experiences with "Reviewer #1" — the generous, supportive scholar who sees reviewing as an opportunity for mentorship and guidance and "Reviewer #2" — the scholar with a grudge who wonders why you are wasting your time on this research when you should have been investigating and writing about something else entirely.) Editors know this, and if they're doing their job, they will help you navigate divergent reviews and guide you in the task of revision. They are the "third readers" whose role is not just to decide which score is "correct," but to arbitrate conflicts and help you find the best path toward eventual publication.

– Michael Pemberton, Georgia Southern University
Consulting Editor, Across the Disciplines and WAC Clearinghouse Associate Publisher for Scholarly Journals

When asked to peer review a piece of writing, I try to remember the intellectual and personal effort made by the author. I also try to remember that all scholarship is an attempt to forward a conversation, often within both a small section of a discipline as well as the larger disciplinary field. Which is to say, I try to imagine how a person is attempting to help a community work on an issue. My role is not to look for errors and gaps, to try to find faults that need addressed. My role is to inhabit the vision of the author and to help the piece serve its intended purpose for our community. At best, it's a collaborative process.

When reading a piece, then, I initially try to understand what the author is attempting to add to an ongoing communal discussion. Next, I consider if the specific intended audience will recognize the framing of that discussion and find it useful. Finally, I consider whether a reader with no immediate relationship to this specific issue will find value in the piece. Throughout, I ask myself, "Is there a person underneath the terminology and citations? Can I hear the author's ethical commitment to the issues discussed?" Ultimately, I believe scholarship is produced by real people attempting to speak to real people. So any published piece should have that sense of humanity coursing through its pages.

– Stephen Parks, University of Virginia
Editor, Studies in Writing and Rhetoric Book Series

As an author, I never feel more self-conscious than when my manuscript receives a clean, straight "publish the manuscript as it stands" recommendation. Tha's just not possible. It means nobody really took a look at the manuscript. Sure, there might well be disciplinary gatekeeping, "false negatives," and misunderstandings, although much less than you would expect. Peer review gives you the chance to avoid gaps, weaknesses, and inconsistencies before it is too late. This includes painful rejections, which can help you understand the right scope and venue for your manuscript or, perhaps more accurately, the right level of revision needed to be successful. As an editor, peer review means trying to understand a manuscript on its own decisions and perspectives, assessing its rationale without the interference of what you would have personally done, and offering useful insights in a collaborative and collegial way. In sum, a peer review entails the confrontation of your textually anticipated reader with the actual expert readers before you lose track of your published manuscript.

– Federico Navarro, Universidad de Chile
Editor, International Exchanges on the Study of Writing Book Series

In emerging research specializations such as writing analytics, peer review is essential to how manuscripts are cultivated, developed, and published. The founders of The Journal of Writing Analytics had the benefit of conferences that would bring researchers together internationally and nationally to advance empirical and theoretical work. In the overwhelming majority of cases, when a manuscript is forwarded to our Board of Reviewers — specialists who are appointed because of their demonstrated commitment to the journal taxonomy and its values, as well as their disciplinary expertise — the study is one that is expressly written for the journal. The process of peer review can therefore focus on research that is presented and evaluated according to specific criteria that, in turn, serve as revision guides.

In this context, the role of peer review is far more than gatekeeping. Instead, it becomes one of cultivating a research community in its very early stages of specialization. As the editors jettisoned hegemonic aims that call for high standards (whatever that phrase can mean) and embraced, instead, values of empirical precision and consequential inferences, we saw the role of peer review as fundamental in establishing research opportunity. Peer review that is incisive, thorough, but delivered professionally and with the goal of improving the research community is what we seek at JWA.

– Susan Lang, Ohio State Univeristy, and Norbert Elliot, University of South Florida
Editors, The Journal of Writing Analytics

As a book series editor, I value peer reviews for the expertise, insights and revision suggestions the reviewers offer to authors, even on a proposal. A good peer reviewer working with a proposal will comment on the originality and soundness of the project, discuss its strengths and weaknesses, and make a recommendation about its suitability for the book series. Once a book is contracted based on reviews of the proposal and I have the full manuscript to send out for review, I'm assuming that the book will be published in our series and that, if there are problem areas, the reviewers will have pointed these out and recommended revision and/or editing. The role of the peer reviewer, then, is not only to evaluate the merits of the proposed project or manuscript, but also, drawing on their expertise, to advise me and the author(s)/editor(s) on ways it might be improved.

– Terry Myers Zawacki, George Mason Univesity
Editor, International Exchanges on the Study of Writing Book Series

As the editor of a book series, your job is a bit different than that of a journal editor. For the latter, you have a responsibility to be neutral with regard to a submission and to sent it out right away for peer review, relying on that review process for decision-making. But for a book editor, you look at submissions a bit differently, since part of your work is developmental; you want the best books possible for your series. If you have what looks like a proposal for an interesting book but it needs work, you will save everyone time and energy if you respond as the first reviewer and help the author get it into better shape before it goes out for peer review. And sometimes, if you really believe in a project but the reviewers disagree, you may want to send it out for more review or even (as Rich and I did on one occasion) just go ahead with it anyway, despite the reviewers.

– Susan McLeod, University of California, Santa Barbara
Editor, Perspectives on Writing Book Series