Agreements and Discrepancies between FDA Reports and Journal Papers on Biologic Agents Approved for Rheumatoid Arthritis: A Meta-Research Project

Background: Sponsors that seek to commercialize new drugs apply to the Food and Drug Administration (FDA) which independently analyzes the raw data and reports the results on its website. Objectives: This study sought to determine if there are differences between the FDA assessments and journal reports on biologic agents developed for the treatment of rheumatoid arthritis. Methods: Available data on FDA-approved drugs were extracted from the website, and a systematic literature search was conducted to identify matching studies in peer-reviewed medical journals. Outcome measures were the American College of Rheumatology response criteria ACR20 (efficacy) and withdrawal due to adverse events (safety). As effect size odds ratios were estimated for each active trial arm vs. control arm (i.e. for both sources: FDA and journal report), followed by calculation of the ratios of the FDA and journal report odds ratios. A ratio of odds ratios not equal to 1 was categorized as a discrepancy. Results: FDA reports were available for 8 of 9 FDA-approved biologic agents for rheumatoid arthritis; all identified trials (34) except one were published in peer-reviewed journals. Overall, discrepancies were noted for 20 of the 33 evaluated trials. Differences in the apparent benefit reporting were found in 39% (24/61) pairwise comparisons and in 11 cases these were statistically significant; the FDA report showed greater benefit than the journal publication in 15 comparisons and lesser benefit in 9. Differences in the reported harms were found in 51% (28/55) pairwise comparisons and were statistically significant in 5. The “signal” in FDA reports showed a less harmful effect than the journal publication in 17 comparisons whereas a more harmful effect in 11. The differences were attributed to differences in analytic approach, patient inclusion, rounding effect, and counting discrepancies. However, no differences were categorized as critical. Conclusion: There was no empirical evidence to suggest biased estimates between the two sources. Increased and detailed transparency in publications would improve the understanding and credibility of published results. Further, the FDA report was found to be a useful source when data are missing in the published report (i.e. reporting bias).