Low prevalence of spin in conclusions of interventional pediatric orthopedic studies

INTRODUCTION

The medical literature is the primary source of information for academics, clinical practice physicians, health policymakers, and the public. Several factors can erode trust and reliance on the published literature. Enhancing academic research – design, performance, and writing skills, especially for early-career researchers – was cited as a means of improving research quality and boosting trust in the medical literature.^[1,2] Control of research reporting quality by eliminating sources of bias and enhancing impartiality in research writing is important for the trustworthiness of the findings and conclusions of published medical literature.^[3] Research bias includes reporting bias, interpretation bias, and publication bias, among other sources of bias related to different stages of the research process, such as study design bias, recruitment bias etcetera.^[3] There is a general consensus that validated assessment tools of bias/spin in published interventional research fall under three main categories, namely, (a) misleading reporting, (b) misleading interpretation, and (c) inappropriate extrapolation.^[4,5]

While research reporting bias refers to selective and inaccurate reporting of study results/correlations, interpretation bias refers to misinterpretation or non-contextual extrapolation of the conclusions of a published article.^[4,5] Multiple factors can drive reporting and interpretation bias, such as authors’ own professional preferences or aspirations, conflicts of interest, and academic and societal pressures etcetera. This may occur consciously or subconsciously.^[6-8] Such biases are occasionally attributable to substandard research writing skills in drawing proper inferences and clinical contextual interpretations of the resultant statistical values.^[2,9,10] Authors may unjustifiably use causal language to interpret assumingly positive statistical correlations between intervention and effect where they actually are associative or uncertainly defined at best.^[2,9,10] Research interpretation bias, also known as spin, has been documented across various medical^[5,7] and surgical disciplines.^[5,11] Research interpretation bias has also been documented in abstracts reporting on the treatment of various orthopedic disorders such as trauma,^[12] arthroscopy and sports medicine,^[13] and Achilles tendon injuries,^[14] among others. Furthermore, a systematic review (SR) of surgical randomized controlled trials (RCTs) found that spin or interpretation bias was more likely to be reported in the abstracts than in their corresponding full-text articles.^[11] Alarmingly, interpretation bias or spin of media coverage of medicine was found to be attributable to the interpretation bias initially found in the conclusions of the published articles.^[15] Interpretation bias of research writers/producers can take various forms. For instance, overstated conclusions that exaggerate the favorable effects of an intervention/treatment without being supported by the results, undue implications that embrace a larger or different study population/setting, and unjustified clinical practice recommendations can mislead readers, distort the scientific literature, and waste research resources.^[5,9-11,13] For example, causal language 53%,^[5] claim of equivalence/non-inferiority versus control for a negative endpoint 15%,^[12] and selective reporting of the efficacy of outcomes 6.8%^[13] were the most prevalent types of spin identified in conclusions/results of included interventional studies in medicine and orthopedic surgery. Contrastingly, interpretation bias of research readers/consumers can misinterpret conclusions made in the article that is being cited and produce the same detrimental consequences on the scientific literature in general^[16] and orthopedic literature in specific.^[17] Moreover, authors of citing articles show a specific predilection for referencing articles with positive statistical significance.^[18] This represents an additional source of bias in the published biomedical literature.^[18] The previous drawbacks prompted various academic interventions aimed at controlling bias in the published medical literature.^[19,20] However, these interventions, namely, prospective trial registration and editorial control, enjoyed little success.^[19,20]

Interpretation bias as a factor of research reporting quality has not been thoroughly investigated in the conclusions of pediatric orthopedic publications. The primary objective of this study was to report the prevalence and subtypes of interpretation bias/spin of the conclusions in abstracts and full-texts of published studies investigating the effects of treatment/intervention in pediatric orthopedic literature. The secondary objective was to report the severity of bias/spin of conclusions in published abstracts.

MATERIALS AND METHODS

We searched ten established and high-impact general orthopedic journals, three of which were Pediatric Orthopedic Society journals. According to Scopus’s updated CiteScore methodology, all except two included journals were Quartile 1 (Q1). The two exceptions were Q2 journals, which were the official publications of leading, highly sought-after continental and international pediatric orthopedic societies. In addition, all included journals were listed in the release of Journal Citation Reports. Demographics of included journals are shown in Table 1. Inclusion criteria were primary interventional – randomized or non-randomized – pediatric orthopedic studies investigating the effects of a treatment or an intervention (surgical or non-surgical). That is, matched comparative studies, classic interventional, and controlled clinical trials. We excluded classic – placebo/no treatment – RCTs. However, interventional studies that did not fit the traditional RCTs’ strict definition or principles were still considered based on our inclusion criteria (see working definitions).

In addition, we excluded case reports, editorials, narrative reviews, diagnostic, prognostic, outcome instrument validation studies, SRs/meta-analyses, those whose study samples consisted exclusively of adults, and studies that were exclusively focused on cost analysis without reporting outcomes and irretrievable studies. We searched the MEDLINE/PubMed using the PubMed Advanced Search Builder. We conducted the initial search on February 19, 2023, and set the publication date to 3 years (2020–2022). We included a sample of 93 pediatric orthopedic studies in the final analysis. The flow diagram and detailed results of the literature search and identification process are shown in Figure 1 and Additional File 1, respectively.

Close

Figure 1:

Flow diagram of the literature search and identification process. *Details can be found in Additional file 1.

Export to PPT

RESULTS

The final list of the 93 included studies in this critical analysis study is shown in Additional File 1. Demographics of included articles are shown in Table 2. The demographics of the included pediatric orthopedic subspecialties were as follows: Congenital and developmental disorders (n = 25) (27%), trauma (n = 23) (25%), spine (n = 18) (19%), hip, for example, slipped capital femoral epiphysis (SCFE), Perthes (n = 9) (10%), arthroscopy and sports injuries (n = 8) (9%), neuromuscular (n = 5) (5%), bone tumors (n = 3) (3%), and other (n =2) (2%).

Our classification identified the presence of at least one count of bias in 17 (18%) studies, 14 (15%) of which had only one count of bias, while 3 (3%) studies had two counts of bias. Nine (10%) studies had bias in both full-text and the corresponding abstract conclusions. In 4 (4%) studies, bias was restricted to conclusions of abstracts only, and in 4 (4%) studies was restricted to a bias criterion assigned to the classification criteria of conclusions of full-text only and not the abstract raw data file. Twenty-seven spin/bias items or outcomes were reported per full-text and corresponding abstract for each of the 93 included studies. A total of 2511 spin/bias items were analyzed across 93 included studies. We reported 30 (1%) counts of bias in conclusions of full-texts and/or abstracts. The severity of bias/spin in conclusions was mild level (n = 21; 70%), moderate level (n = 5; 17%), and high level (n = 4; 13%). Characteristics of reporting and interpretation bias in conclusions of included full-text articles and their abstracts are shown in Table 3. Selective citation of other studies under the category of misleading reporting in Table 3 was not reported, as it was beyond the scope of the present study. The study’s raw data with explanatory comments are shown in raw data file. A graphical abstract of discrepancies between results and conclusions in interventional pediatric orthopedic research is shown in Figure 2.

Close

Figure 2:

Graphical abstract. Discrepancies between results and conclusions in interventional pediatric orthopedic research.

Export to PPT

DISCUSSION

Our study investigated the prevalence, subtypes, and severity of misleading reporting, inadequate interpretation and extrapolation of the conclusions in full-texts and corresponding abstracts of 93 pediatric orthopedic interventional studies published in high-impact factor orthopedic journals. Most studies (82%) were free of any form of bias/spin. Furthermore, most of the reported counts of bias (87%) were of low-to-moderate severity. These overall satisfactory outcomes may be ascribed to the high-impact society journals constituting our study sample. Contrastingly, higher degrees of research interpretation bias was recorded in SRs and RCTs, reporting on the treatment of orthopedic trauma,^[12,25] arthroscopy and sports medicine,^[13] and Achilles tendon injuries.^[14] Although one of those studies targeted high-impact journals, interpretation bias was still unacceptably high.^[12] Similarly, considerable research interpretation and extrapolation bias have been reported in high-impact journals of both medical^[26] and surgical disciplines.^[27]

This underscores the fact that research interpretation bias is prevalent across the medical literature, including high-impact journals.^[26,27] Hypothetically, bias in research reporting should be less prevalent in high-impact journals than in low-impact journals. Conclusions and recommendations from studies published in high-impact journals are usually the main guide for clinical practice decision-making, public health planning, and national and global health policies. However, the high journal impact factor is not necessarily associated with high reporting quality in published articles.^{[12,14,17,27-29]}

The most common types of spin or interpretation bias reported in our study were; no consideration of the limitations, inadequate implication for clinical practice, and focus on statistical significance instead of clinical relevance. Misleading reporting, such as selective reporting and use of linguistic spin, and inadequate interpretation of statistically non-significant results, has been widely cited as an important cause of bias in conclusions of diverse medical and surgical studies as; orthopedic surgery,^[12,25] bariatric surgery,^[30] urology,^[31] cardiology,^[32] and oncology,^[33] among others.^[27,34] This contrasts with our study, where the degree of spin was generally low, with no reported spin in relation to the “claim of an effect or equivalence for non-statistically significant results.” Statistical significance was more likely to be reported in abstracts of surgical randomized trials than their corresponding full-texts.^[11] Besides, noticeable inconsistencies between the reporting quality of abstracts and corresponding full-text articles were found in pediatric orthopedic and physical rehabilitation publications.^[35,36] Our study found negligible differences between spin counts reported in abstracts and full-texts.

Using causal language, which unjustifiably and clearly asserts or strongly implies a cause-effect relationship between an intervention and an outcome, can mislead readers and misguide clinical practice recommendations and health policies. However, occasionally, there is no clear line of demarcation between causal and non-causal language, namely, the presence of mere association or correlation between an intervention and an outcome.^[23,37] In addition, it may be challenging to judge if the authors’ intent in the conclusion was to convey causation or association.^[23,24] The fact that a considerable percentage of biomedical authors are non-native English speakers adds to the challenge. In our study, the use rate of unjustifiable causal language was extremely low. Nonetheless, the unjustifiable use of causal language remains problematic in observational research across the biomedical literature,^{[23,24,37,38]} including orthopedics.^[39,40]

Pre-publication peer review is considered a means of achieving scientific quality control. It can potentially protect against misinterpreted, over-interpreted, and biased conclusions.^[41] However, the peer review system itself is not without deficiencies. Difficulties in standardizing quality assessment measures for peer review reports,^[42] insufficient peer review training programs,^[43] shortage of qualified peer reviewers,^[44] and peer review bias^[45,46] have all been cited as potential causes for deficiencies of the peer review system.

Suggested strategies to improve research reporting bias and misinterpretation/extrapolation of conclusions include (a) integrating peer review – critical appraisal skills – and research writing skills into under- and postgraduate medical curricula, (b) incentivizing and officially recognizing extra-curricular learning activities aimed at improving peer review and research writing skills of early career researchers, for example, courses, workshops, co-reviewing, and (c) raising awareness of research reporting bias among editors and peer reviewers, for example, journal instructions to authors and reviewers.

CONCLUSION

There was a low prevalence of reporting bias, namely, misleading reporting, misinterpretation, and inadequate extrapolation of conclusions, in 93 full-texts and their abstracts of interventional pediatric orthopedic studies published in high-ranking journals. A comparative study with lower-ranking journals, as a control group may reveal if our favorable results are an attribute of journal rank/quality. We recommend that authors, reviewers, and editors check the results section to ensure that – numerical – data are neither misreported nor misinterpreted and are adequately extrapolated in the conclusions, preferably in accordance with relevant reporting guidelines.