Top nine pitfalls to avoid when writing a journal peer review report

INTRODUCTION

Pre-publication peer review of scientific manuscripts is considered a means of achieving quality control in medical literature. Prepublication peer review can potentially protect against scientific flaws and ensure the originality of scientific material disseminated to the scholarly community. However, the peer review system has its weaknesses and failures.^[1,2] Shortcomings of the peer review system have been attributed to various factors such as disagreements regarding standardizing quality assessment measures for peer review reports,^[3] shortage of skilled reviewers,^[4,5] and peer review bias.^[6]

In general, rejection of manuscripts submitted to medical journals is considered scientifically justified if they contain inappropriately presented, baseless research questions, or serious methodological flaws that impact the validity of the conclusions, etcetera.^[7,8] Similarly, acceptance of manuscripts is considered scientifically justified if they present a novel research question, a sound methodologic design and execution, and non-biased conclusions.^[7,8] The root causes of rejecting manuscripts submitted to medical journals have received little attention in the literature.^[9-12] Studies have cited justifiable or unfixable causes for rejection as serious methodological flaws, lack of research novelty, and occasionally justifiable or potentially fixable causes.^[10-12]

Nonetheless, manuscripts could be unjustifiably rejected or accepted, that is, not based fundamentally on sound and objective academic criteria.^[5,6,9] There is insufficient evidence to suggest that journal acceptance rates directly correlate to research quality and rejection rates directly correlate to inadequate quality.^[5,6,9] This is despite the fact that high-ranking biomedical journals generally tend to have higher rejection rates.^[9] Experienced researchers pointed out numerous substandard peer review practices as focusing predominantly on negative aspects of the research, vague or ambiguous reports, criticism that is unsubstantiated by the study’s content, suggesting a different research point and misunderstanding research novelty, etcetera.^[5]

Only 2.3% of scholarly journals permit peer-reviewer reports – which may include the corresponding author’s responses – to be published.^[13] This anonymity does not allow the scientific community to comprehensively analyze and vet the quality of the peer review reports, including the degree of conformity of the reviewers’ recommendations (acceptance/rejection) to the comments provided in the review report and standard reporting guidelines.^[1,3,7] This also deprives the scientific community of ways to provide constructive feedback on deficiencies of the peer review system and propose developments.^[1,3,7] Published peer reviewer reports could serve as a how-to and how-not-to guide for critically reviewing a scientific manuscript, especially for early career researchers.^[13] We hoped to highlight the major pitfalls to avoid when writing journal peer review reports of medical manuscripts. In addition, we aimed to provide practical tips on how to overcome these pitfalls and guide junior reviewers through the peer review process.

DO NOT MISINTERPRET THE MEANING OF RESEARCH NOVELTY

A manuscript should not be labeled confirmatory and, therefore, rejected in essence because it replicates a previously reported topic. Existing literature on a specific research point/topic should not be deemed scientifically robust by default and thus taken for granted. Further, what seems to be a confirmatory study may be novel in that it researches another dimension of a previously reported topic. Further, a seemingly confirmatory study may have actually addressed deficiencies of previously reported topics. For example, a new study may validate a used scoring system for a previously reported intervention or reinvestigate retrospective research in prospective settings to overcome the inherent limitations of retrospective studies. This can yield stronger evidence and more valid conclusions. Restudying a standard surgical technique in a different patient population or even widening its range of indications in the same patient population is considered a form of research novelty. Replicating published studies with the intent of compensating for any of the studies’ methodological and/or logistic deficiencies is considered an innovation and an important contribution to science [Figure 1].^[14] Reviewers should not erroneously label such studies as simply confirmatory (replicative). Reviewers should spotlight and grade all aspects of the research novelty-research gap presented in the introduction.

Close

Figure 1:

How can replicating published interventions be innovative?

Export to PPT

STUDY’S RATED LEVEL OF EVIDENCE DOES NOT NECESSARILY CORRELATE WITH QUALITY

A manuscript should not be rejected or accepted principally because of its hierarchy or the level of evidence it represents. It is true that certain research designs or study types typically yield stronger evidence than others, such as systematic reviews of randomized control trials. However, that is not necessarily by default.^[15] Well-conducted classic pre-post interventional case series or matched comparative studies can produce stronger evidence than flawed systematic reviews of randomized control trials. Therefore, ranking high in the level-of-evidence hierarchy is not necessarily indicative of research quality and strong evidence yielding.

DO NOT FOCUS ON CLINICAL PRACTICE SKILLS AT THE EXPENSE OF ACADEMIC SKILLS

Peer reviewers may focus mainly or entirely on clinical practice and technique-related issues at the expense of academic and methodological aspects of the scientific paper. Scholarly reviewers’ primary duty entails meticulous and systematic assessment of the robustness of research design, execution, and presentation.^[6,16] Being entirely dominated by clinical practice skills or preferences can impact the quality of the peer review report. This is because clinical practice decisions and choices are usually influenced by a plethora of factors such as physician experience, availability of resources, institutional policies, cultural and socioeconomic issues, etcetera.^[6,17] A manuscript should not be rejected fundamentally because its research point or technique does not conform to the reviewer’s clinical practice preferences. It is important to make a fine distinction between clinical practice preferences/skills and academic skills required to review a research paper, which should be judged on its own academic and scientific merits as novelty, integrity of methodology, and soundness of interpretation of study results, etcetera.

DIFFERENTIATE BETWEEN FIXABLE AND NON-FIXABLE METHODOLOGICAL FLAWS

The importance of properly designed and comprehensively executed research methods cannot be overemphasized. Although this is a key to any research proposal, methodological flaws are a common reason for manuscript rejection. Reviewers should make every effort to capture how the actual and potential confounding variables were eliminated or neutralized. Failure to neutralize key confounding variables that are likely to render the validity of study conclusions invalid is considered an unfixable methodologic flaw. The use of non-validated scoring systems and assessment tools that are unsuited to measure the outcomes of an intervention or achieve study objectives represents another unfixable methodologic flaw. The above-noted flaws mandate manuscript rejection. However, non-vital missing information is potentially fixable and does not mandate rejection in essence.

BEWARE OF BIAS IN REPORTING OF RESULTS

Reviewers should ascertain that all relevant variables have been soundly correlated as per the study objective and methodology. This is irrespective of the statistical significance of the correlation. The aim is to avoid bias or selective reporting of the results, which in turn could negatively impact the study conclusions.^[18-20] Descriptive statistics could be as informative and evidence-yielding as sophisticated analytic statistics. The choice of appropriate statistics depends on the objectives and methods’ design. Technical issues related to the arrangement of figures and tables and the flow of ideas – organization – in the manuscript do not principally merit rejection.

COMMON MISCONCEPTIONS ABOUT THE DISCUSSION

Comparing results with the existing literature should be based on an accurate assessment of cited papers’ methods and clinical context. The concept that the discussion section of a manuscript is based overwhelmingly on comparing one’s results with the existing literature is contested. Although important, reviewers should also search for inferences from the results based on credible evidence and explanations. Discussing the broader implications of the results in a conceptual manner is equally important as comparing detailed numerical data with similar works in the literature. Therefore, discussing the implications of study results for practice and public health- beyond the local context in which the research was executed is encouraged. That is the implications of the results for other age groups or disease populations, etcetera. Reviewers should check if the authors have discussed possible avenues for future research [Figure 2].^[8]

Close

Figure 2:

Functions of the discussion.

Export to PPT

NUMEROUS STUDY LIMITATIONS DO NOT NECESSARILY DISCREDIT THE RESEARCH WORK

A manuscript should not be deemed flawed merely because of the presence of numerous limitations. Limitations should be judged by their influence on the study’s validity and not by their numbers. Elaborating on the study’s limitations strengthens the scientific stance of the research work because it helps readers interpret the results and conclusions within the study’s specified clinical context. Nevertheless, reviewers should make a clear distinction between inherent or inevitable study limitations and those arising from serious methodological flaws, which could compromise the validity of the results and the credibility of the evidence. Limitations due to serious methodological flaws are not acceptable.^[8]

BEWARE OF OVERSTATED OR MISINTERPRETED CONCLUSIONS

Reviewers are cautioned against misinterpretation or faulty extrapolation of both statistically insignificant and statistically significant results in submitted manuscripts.^[18,20,21] Reviewers should beware of conclusions that are not fully supported by study findings, e.g., unjustifiably generalized to a broader patient population or a longer follow-up period. Besides, reviewers should be aware of the use of causal language links cause to effect in the conclusions to describe otherwise associative-non-cause-effect-relationships between variables. A manuscript should not be undervalued/rejected solely because of its negative or statistically insignificant unattractive results. Negative results are equally important to the advancement of science as positive or significant results, provided the research question is scientifically credible, methodology is robust and results and conclusions have sufficient validity.^[8,18,19] Reviewers should ensure that there are no inconsistencies between the conclusions reported in the abstracts and full-text articles.^[22]

DO NOT UNDERESTIMATE THE VALUE OF REFERENCES

Reviewers occasionally overlook reference lists. Reviewing the literature comprehensively, meticulously, and relevantly can assist authors in preparing a novel research question, overcoming published literature’s flaws, curating contextual study correlations, delivering valid interpretations of the outcomes, and arriving at non-biased conclusions. Therefore, the reference list of a manuscript serves as an indicator of the level of the author’s familiarity with the extent and depth of their presented research problem. Failure to cite key and recent references, misplaced or misrepresented, and methodologically flawed references are regarded as a fair proxy for the scientific/academic integrity of the whole manuscript [Figure 3].^[23]

Close

Figure 3:

Graphical abstract: Toward a balanced peer review report.

Export to PPT

CONCLUSION

Journal editors rely fundamentally on peer reviewers to make informed decisions on the scientific trustfulness of submitted manuscripts and whether or not to publish them. Consequently, robust peer reviewer skills are paramount to critically appraising evidence in any scientific material. Attention to the most common pitfalls of writing a peer review report and adherence to reviewer reporting guidelines can improve the quality of evidence and inform clinical practice. It is of particular importance to resolve misconceptions regarding research novelty, level of evidence, clinical versus academic skills, handling of methodologic flaws, reporting bias, functions of the discussion, study limitations, formulating valid conclusions, and judging reference lists.