Introduction
In the United States (US), a sponsor submits a New Drug Application (NDA) to the Food and Drug Administration (FDA) to obtain permission to market a new drug. The NDA includes details of clinical studies that support the safety and efficacy of the candidate drug. These NDAs are reviewed by the FDA for their clinical, statistical, clinical pharmacology, and other information. On approving the NDA, the FDA releases documents such as the Medical Review, Multidiscipline Review, or Clinical Review (hereafter referred to as the Review) into the public domain. The Review includes information about the clinical trials listed in the NDA. The public availability of the Review is an important step in transparency around the approval of the drug, since the documents are used as a source of information by various stakeholders such as patients, medical staff, and researchers.
The public availability of the Reviews is particularly important for researchers, given that there are often powerful commercial interests involved in a drug’s approval, and also that there have been past problems with research integrity.1 Although data submitted by the sponsor is included in the NDA document, the review of that data is conducted by the FDA. As such, the information in the Review is ratified by the FDA. The transparency of the FDA’s review is important to build trust in the data underlying the drug’s approval.
Separately, in the US, sponsors of ‘applicable clinical trials’ – a category that includes many drug trials2 – are required to register the trial in the registry ClinicalTrials.gov.3 This is the leading trial registry in the world, with over 430,000 records.4
Clinical trial registry records serve overlapping but also separate roles from the Review documents. First, such registries hold information on many more trials than are part of regulatory documents. Among other things, this is helpful to patients who are looking for trials to sign up for. Second, each trial record in the registry follows a template and is therefore much easier to understand than the lengthy Review documents. Although presented in a succinct fashion, there can be tens of fields of information, providing adequate detail for many purposes. Third, due to the large number of records and the structured nature of the information, it is possible to conduct meta-analyses across trial records whose findings can be incorporated into systematic reviews.5 Such reviews can be the basis of changes in clinical practice. Overall, registry data has been used in dozens of ways.6
Although information about the same trial may be available in both the Review and in ClinicalTrials.gov, there have been reports of discrepancies in the data available from these two sources.7 In fact, discrepancies have been found in comparisons of trial data in (i) a registry and in the subsequent publication7; (ii) the FDA documents, a registry and the publications8; (iii) the protocol and publication9; and (iv) data of a given study that is registered in more than one registry.10
This is aside from the fact that there is missing, uninformative, erroneous, or inconsistent information in individual trial records in various registries.11,12 Many individuals and organizations have called for improved registry records. Nevertheless, such problems persist. Consequently, there have been repeated calls for audits, including by the House of Commons of the United Kingdom.1,13 These efforts have primarily focused on trial registries, and ClinicalTrials.gov in particular.
Ideally, the audit of the publicly available data related to a trial should cover (i) information in the one or more registries in which the trial is registered; (ii) one or more publications linked to the trial; and (iii) the documents of the FDA and any other regulator. Of these, the regulator is the most important for the sponsor. It is therefore reasonable to assume that trial information submitted to the regulator is the most accurate. In this study, we assessed how easy it would be to use summary information available in the Review to help conduct an audit of trial data available elsewhere.
Methods
In other work,14 we performed a limited audit of the publicly available details of trials underlying certain FDA-approved Orphan Drugs. We identified 63 Orphan Drugs, approved between 1 January 2009–17 April 2020, each of which (i) was approved under a single NDA, for a single (orphan) indication, and (ii) had a single entry each in the FDA’s Orphan Drug Designations and Approvals database and the Orange Book. Forty-seven of them had a publicly available Review, a document that was often several hundred pages long. However, only 45 Reviews had a summary ‘Table of Clinical Studies’ (henceforth, the Table). We took forward these 45 Reviews.
In trying to assess how easy it would be to use the Table to conduct an audit, we captured (i) the series of headings under which the Table was listed; (ii) the number of columns in the Table; and (iii) the column headings. The column heading, for a given field of information, may have had several variants. For example, the study ID was listed under the following headings: Protocol No., Protocol Number, Study, STUDY #, Study ID, Study identifier, Study Identity, Study No., Trial, Trial ID, Trial Identity, and Trial(s). For ‘(iii) the column headings’ analysis, we captured these variants for a given field. In addition, more than one field may have been listed in the heading of one column. We parsed these fields and counted them individually in our statistics of the number of occurrences of a particular field. For example, in the Table of the Review for Ampyra, the first column had ‘Study ID’, ‘No. of Centers’ and ‘Population’. We disaggregated these three terms and sorted them individually. However if a single term referred to multiple fields of data, we did not parse those meanings, and merely took the ‘implied meaning’ of that term. For example, in the Table of Firdapse, ‘Type of Study’ included Trial identity, phase and information such as ‘safety’ or ‘efficacy’. We classified it as ‘Type of Study’.
Results
In assessing how easy it would be to use the Table to extract information that would help conduct an audit, we undertook three analyses. First, we surveyed the headings under which the Table was listed. Each Table was listed under two to four headings. An example was Xpovio’s (i) 7 Statistical and Clinical Evaluation, (ii) 7.1 Sources of Clinical Data and Review Strategy, (iii) 7.1.1 Table of Clinical Studies, and (iv) Table 13: Listing of Clinical Trials Relevant to NDA 212306. Overall, the 45 Tables from 45 Reviews were listed under a total of 138 headings, of which 49 were unique. Details, including the URLs for the availability of the Reviews are presented in Supplementary Table 1.
Second, we determined that there were 2–13 columns in a given Table. However, more than one field may have been listed in the heading of one column. An example is the Table of Adcirca, which in different columns had (i) Study Id; Status; Report Type; (ii) Enrollment; Start and End; #Planned #Entered #Completed; (iii) Design; Control Type; (iv) Test and Control Drug(s); Dose, Route, Regimen; and (v) Mean Age; Years (Range). Details are available in Supplementary Table 1.
Third, we analyzed the names of the columns. Particular column headings occurred with varying frequencies. There was also considerable variation in the wording of these headings.
Headings present six or more times across the 45 Reviews are listed in Table 1, in which we analyzed (a) the total number of occurrences of a column heading; (b) the total number of highest occurrence heading name variant; and (c) the fraction of highest occurrence name variant. The findings were as follows, with the most commonly used variant listed in Table 1: (i) Type of study – with this variant of the heading present in six (of six, or 100%), of the cases; (ii) Study status – five (seven, 71%); (iii) No. of Centers and Countries – 12 (18, 67%); (iv) Phase – six (nine, 67%); (v) Study Endpoints – 12 (21, 57%); (vi) Study population – 18 (36, 50%); (vii) Test product(s) – three (seven, 43%); (viii) Treatment Duration/Follow Up – 10 (28, 36%); (ix) Trial design – 14 (43, 33%); (x) No. of patients enrolled – 13 (40, 33%); (xi) Regimen/schedule/route – 14 (43, 33%); (xii) Trial identity – 11 (42, 26%); and (xiii) Objective(s) of the Study – three (12, 25%). Further details are available in Supplementary Table 1.
The most commonly present fields were: Trial Design (43); Regimen/schedule/route (43); Trial identity (42 cases); No. of patients enrolled (40); and Study population (36). The least often present were Phase (9); Test Product(s) (7); Study status (7); and Type of study (6).
Finally, there was occasional ambiguity in the meaning of the same, or similar, column headings. For example, ‘No. of Centers and Countries’ may or may not have included the names of the countries, and ‘Study Design’ may or may not have included Phase. The terms ‘Population’ or ‘Study Population’ usually referred to the nature of the condition or disease. However, they occasionally included the age bracket of the patients, their gender, or other criteria for inclusion in the trial. And whereas ‘Study location’ referred to the countries that hosted the trial, ‘Location’ referred to the part of the Review where further details about the trial were available.
Discussion and Conclusion
Here, we have explored the presentation of information in and around the Table, which could be centrally important in the auditing of trial-related data in registries and elsewhere. The data must be in a standardized format, be it for manual data extraction or automatic data retrieval.15 Below, we discuss three aspects of such standardization.
First, the series of headings above the Table must be standardized. The combination of a standardized series of headings together with a standard Table name will help disambiguate the location of the data in case there are similar table or heading names elsewhere in the document as well.
Second, we come to the issue of the frequency of specific headings. Although we captured only 42 occurrences of ‘Trial identity’, these were actually present in all 45 cases. Our methodology failed to capture the remaining three cases because the heading was not specific to ‘Trial identity’, and the column held other information as well. It is clear that a trial needs to be identified before it is described, and therefore the presence of the ‘Trial identity’ in each Table is not surprising. It is not as obvious as to why certain other fields are largely present or absent. For instance, one would assume that ‘Phase’ is an important descriptor of a study, and yet it is only present in nine Reviews as a distinct heading, although the information was included under other headings in some other cases.
We propose the same set of headings, as listed in Table 1, that would be part of a template Table. We suggest these headings because they are the most common ones chosen for the Table by the FDA itself, across the Review documents. However, we would add ‘NCT ID’ to the ‘Trial identity’ column. The ‘NCT ID’ is crucial to cross-reference information in other sources, and hence we have included it.
Although 13 headings may appear to be too many for a table, the Review document of Cuvposa had 13 headings. Here, the table was horizontal, rather than vertical. With such an arrangement, even more headings are possible. The Cuvposa table only had three trials, although more could have been accommodated, especially if facing pages were used. Conceivably, headings could be merged in a structured manner, and this would work fine provided the information in the column was also provided in a structured way.
Third, there is occasional ambiguity in the meaning of similar column headings. We have not compared every occurrence of a heading to quantify such ambiguities. However, even a single occurrence makes the larger point that there may be ambiguity in the use of a particular term. This would undermine data extraction efforts from a large number of documents, for instance. However, if each term had a clear definition, and if users abided by that definition, then that would solve the problem. Therefore, the proposed standardization of column headings must be accompanied by a standardization of the meaning of each heading.
Above, we have recommended ways of standardizing information related to the Table. Such standardization would make it easier for patients, providers, researchers, and others to more readily understand each table and compare tables across documents if necessary, and for the manual extraction of information. The consistent use of a template will help track some of the discrepancies in trial-related data across sources, and ultimately reduce them.
Further, such standardization ought to facilitate the use of programmatic methods for such comparisons. Standardization with an extensible document schema will enable development of software solutions for automatic retrieval of summary information across multiple heterogeneous sources, such as registries or publications. The document schema should define a comprehensive list of tags to define various elements, such as headings, subheadings, table, structure within the table, etc. Machine learning based language understanding models, such as BERT (Bidirectional Encoder Representations from Transformers), could be employed to retrieve similar texts, if needed.
In summary, we have suggested that (i) the series of headings above the Table must be standardized; (ii) the Table should have a fixed number of columns, with a fixed set of headings; and (iii) each of the terms used in the headings should be well defined. We need to build confidence in trial data by ensuring that the data from all sources are consistent. The suggestions above would help to use data in the FDA documents to conduct an audit for consistency across data sources.