A Magic Mixing Cauldron for the 21st Century: FDA’s new guidances on using real-world data in regulatory decision-making

In December 2016, fifteen Food and Drug Administration (FDA) officials, including then-Commissioner Robert Califf and current Acting Commissioner Janet Woodcock, published an article in the New England Journal of Medicine (NEJM) about the potential regulatory impact of real-world evidence and the difficulties of fitting such evidence through existing regulatory drug approval pathways.¹Sherman, et al., Real-World Evidence – What Is It and What Can It Tell Us?, 375 N. Engl. J. Med. 23, 2293-97 (Dec. 8, 2016).  A solid regulatory definition of real-world evidence (RWE) was “elusive,” they cautioned, in part because of the heterogeneity of information that could potentially be considered RWE, particularly when juxtaposed against FDA’s “gold standard” of data from randomized, controlled clinical trials.²Id. at 2293.

Enter the 21^st Century Cures Act.³Pub. L. No. 114-255 (Dec. 13, 2016), amending 42 U.S.C. § 201, et seq.  Enacted days after the NEJM article published, the Cures Act required FDA to get in the game, and to do something to advance the use of RWE for drug regulation.  At the time, it meant that FDA was directed to create “a framework for a program to evaluate the potential use of RWE” to support either the approval of a new indication for an already-approved drug or to help satisfy post-approval study requirements.⁴FDA, Draft Guidance for Industry: Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory Decision-Making for Drug and Biological Products (Sept. 2021) (“EHR/MCD Guidance”), https://www.fda.gov/media/152503/download; FDA, Draft Guidance for Industry: Data Standards for Drug and Biological Product Submissions Containing Real-World Data (Oct. 2021) (“Data Standards Guidance”), https://www.fda.gov/media/153341/download.  FDA published such a framework for drugs in December 2018⁵FDA, Framework for FDA’s Real-World Evidence Program (Dec. 2018), https://www.fda.gov/media/120060/download. (the Cures Act framework is not applicable to medical devices, for which there is separate guidance⁶FDA, Guidance for Industry and Food and Drug Administration Staff: Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices (August 2017). https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devices).

The Cures Act also required FDA to use its framework to inform guidance on the use of RWE in regulatory decision-making, and specifically to address: (1) the circumstances under which drug sponsors can rely on real-world evidence for the two purposes described above; and (2) “appropriate standards and methodologies for collection and analysis of real-world evidence submitted for such purposes.”⁷21 U.S.C. § 355g(e)(1)(B).  FDA has now published two new draft guidances addressing the collection of real-world data (RWD) with the goal of using it as RWE, i.e., to support regulatory decision-making for drug products.  The first of the two draft guidances, the EHR/MCD Guidance, is a lengthy guidance that describes FDA’s thinking on considerations for the collection and analysis of electronic health records and medical claims data as RWD, and on appropriate standards and methodologies for assessing such data in light of potential pitfalls.⁸See supra at n.4.  The second draft guidance, the Data Standards Guidance, identifies considerations for using data standards in submissions containing study data derived from RWD sources.⁹See supra at n.4. 

The September 2021 Draft EHR/MCD Guidance

The draft guidance, “Real-World Data: Assessing Electronic Health Records and Medical Claims Data to Support Regulatory Decision-Making for Drug and Biological Products” or “EHR/MCD Guidance” addresses the use of electronic health records (EHRs) and medical claims data (MCD) intended to derive RWE in support of a demonstration of safety or effectiveness. To that end, the draft EHR/MCD Guidance provides recommendations on improving the utility of EHR and MCD data by ensuring that such data are accurate, complete, and relevant.  For example, sponsors should focus on sourcing accurate data, designing studies to account for misclassified or missing information, and utilizing appropriate methodologies to achieve covariate balance.

The draft EHR/MCD Guidance goes to lengths to explain the difficulties and potential problems sponsors will need to address in order to harness RWD as RWE before admonishing sponsors to evaluate appropriateness of the RWD source in light of such limitations. For example, attention is paid in the guidance to the following potential problems with RWD:

• Both EHRs and MCD may be limited due to the purpose for which they were initially created. For example, medical claims data were created to support payment for care and may not comprehensively describe patients’ health conditions.¹⁰See EHR/MCD Guidance at 4-5.
• Because EHR and MCD data are created only when there is an interaction between a patient and the healthcare system, it may not be sufficiently comprehensive as it may not contain the detail necessary to capture the study population. In addition, there may be missing data in two broad cases: (1) the information was intended to be collected (g., a data field exists) but the information is missing; and (2) the information was not intended to be collected.¹¹See id. at 10.
• RWD may be limited by factors in health care systems around the world that influence patient population and use. Differences in patients in different health care systems (g., age, socioeconomic status) and differences in systems themselves (e.g., formulary decisions, medication tiering) can affect the relevance of the RWD source. Data heterogeneity can limit the ability to pool data, as well.¹²See id at 7.
• EHR and MCD systems can contain differing levels of detail and completeness, and not all may contain necessary detail and completeness (g., study populations, exposures) that are relevant to the study question. For example, some relevant but sensitive data may not be available through EHRs and claims data, such as data on sexually transmitted infections, substance abuse, and mental health conditions.¹³See id. at 6.
• Data from EHRs and MCD can be incomplete. Patients change health plans or providers, meaning that their claims data only tells bits and pieces of their full history. In the absence of unique national patient identifiers, the inclusion of de-identified data also can result in duplicative data entry, for example, if a patient has visited multiple health care providers.¹⁴See id. at 7.

Given these limitations, getting FDA on board with study plans will be essential.  FDA recommends that sponsors should ensure they address the appropriateness of the data in light of potential limitations of the RWD source with respect to the study question and key elements, and that they prespecify all essential elements of study design, analysis, conduct, and reporting.¹⁵See id. at 5.

According to the draft EHR/MCD Guidance, relevance analysis should include consideration of the availability of key data elements and querying whether there are sufficient numbers of representative patients for the study; reliability analysis should focus on accuracy, completeness, and traceability. To that end, as the draft EHR/MCD Guidance explains, sponsors sourcing RWD from EHRs or MCD should develop protocols and statistical analysis plans that:¹⁶See id. at 5-12.

• Explain the reason for selecting the particular data sources to address the specific hypotheses, and how those sources were selected to maximize the reliability and relevance of the RWD;
• Effectively link and synthesize data networks, including by standardizing computable phenotypes that facilitate identification of similar patient populations, using distributed data networks and common data models, and developing a plan for efficiently processing unstructured data;
• Address missing data through data linkage and/or use of proxy variables. Ensure that key study elements are well defined with both conceptual and operational definitions;
• Pay specific attention to data linkage types, data impact, and approaches to linkage, and use unique patient identifiers;
• Provide background information about the health care system, including (if available) any specified method of diagnosis and preferred treatments for the disease of interest, and the degree to which such information is collected and validated in the proposed data sources; and
• Describe prescribing and use practices in the health care system (if available), including for approved indications, formulations, and doses.

Sponsors should submit protocols and statistical analysis plans to FDA before conducting the study, and should seek FDA’s confirmation regarding the acceptability of the planned study. 

In addition, although FDA reserves considerations relevant to study design and analysis when using RWD sources for subsequent discussion, the draft EHR/MCD Guidance does provide a glimpse into the way that FDA expects study design elements to be ascertained and validated.¹⁷See id. at 13-18.  For example, sponsors should: clearly define pertinent time periods; describe how population inclusion and exclusion criteria will be implemented; clearly describe how exposure to the investigational product will be determined (e.g., drug dose, formulation, strength, etc.); demonstrate that the specific products of interest can be identified and the relevant exposure dose and duration can be captured in the proposed data source; as appropriate, validate the availability, accuracy, and completeness of the data on the outcome of interest; and adequately account for key covariates (including confounders and effect modifiers).

The final section of the draft guidance raises considerations regarding quality over the data life cycle.  Sponsors should, according to FDA, evaluate the completeness, accuracy, and plausibility of the data, using scientifically-justified standards.  At each step in the data life cycle, sponsors should: (1) characterize the data with respect to completeness, conformance, and plausibility; (2) document the quality assurance and quality control (QA/QC) plan, and (3) define a set of procedures for ensuring data integrity, including specifying the data’s provenance.¹⁸See id. at 25-30.

The draft EHR/MCD Guidance gave us a lot to think about—and it is not FDA’s only new guidance on the topic. In October, FDA published another draft guidance on RWD/RWE: Data Standards for Drug and Biological Product Submissions Containing Real-World Data,” or the “Draft Data Standards Guidance.”

The October 2021 Draft Data Standards Guidance

The draft Data Standards Guidance explains that RWD submitted as study data to drug applications generally will be required to be “in an electronic format that the Agency can process, review, and archive.”¹⁹See Data Standards Guidance at 1-2 (citing 21 U.S.C. § 379k-1(a)).  To that end, the draft Data Standards Guidance offers considerations for mapping RWD to study data submission standards.  Notably, the draft Data Standards Guidance cuts a broader swath than the draft EHR/MCD Guidance; it is applicable to RWD generally, including not only data derived from EHRs and MCD, but also data from registries, patient-generated data, and data gathered from other sources, e.g., from mobile devices.

The draft Data Standards Guidance reminds us that, to maximize the likelihood of success with an electronic submission, RWD should submitted in an electronic format that FDA can process, review, and archive using the standards specified in the Data Standards Catalog.  That may be difficult, however, given “the challenges involved in standardizing study data derived from RWD sources for inclusion in applicable drug submissions.”²⁰Id. at 3.  As in the draft EHR/MCD Guidance, the draft Data Standards Guidance highlights potential problems with RWD.  RWD may use different terminology and formats, and may be aggregated, organized, and presented in significantly different manners, than current FDA-supported data standards (e.g., Clinical Data Interchange Standards Consortium’s (CDISC’s) Study Data Tabulation Model (SDTM)).  For example, RWD might document sex based on gender identity, whereas CDISC standards document sex based on biological sex at birth—these may not be equivalent for all subjects in a study.²¹See id. at 4. Similarly, patient records and traditional clinical trial data coded using CDISC terminology might document race and ethnicity differently.²²See id. at 8-10. FDA therefore recommends that sponsors provide a data dictionary in RWD/RWE submissions to document the definition of every data element used, along with its relationships to other data, origin, usage, and format.²³See id. at 4-5.

Sponsors must also consider and prepare for potential challenges in translating RWD into FDA-supported data standards. The guidance offers the following examples of such challenges for which sponsors should prepare:

• Management of semantic concepts (terms) that are present at multiple locations in a health record (such as medication information);
• Inconsistent coding or miscoding of concepts (g., drugs or diagnoses);
• Changes in data collection or coding practices (g., International Classification of Diseases-9 (ICD-9) and ICD-10 codes) that occurred during the study; and
• Missing information (either because information is not typically recorded in health care settings or due to inconsistent data entry).²⁴See id. at 5.

In addition, FDA recommends that sponsors document the challenges they confront in transforming RWD into data consistent with FDA-supported data standards, along with the rationale for the approach taken to resolve each challenge.

Moving Forward

Incorporating RWD into clinical evidence means understanding what is represented in that data, and what is not. By their very nature, RWD sources capture different information than that from traditional randomized clinical trials. We will need to account for the impacts of those differences—things like data points lost because EHRs capture data only when patients seek medical attention, or unreported sensitive data. The two recent draft guidances offer suggestions for such accounting in order to enable reliance on these data, even if they don’t provide a complete recipe.

*           *           *

King & Spalding LLP regularly counsels pharmaceutical and device manufacturers on development and submission of drug, biological product, and medical device marketing applications to FDA.  Comments on the draft EHR/MCD Guidance are due on November 29, 2021; comments on the draft Data Standards Guidance are due on December 21, 2021.

Please let us know if you have any questions regarding these draft guidances, or if you would like to consider submitting a comment.