We have previously discussed the potential roles of real-world data (RWD) and real-world evidence (RWE) for drug access. However, the precise roles of RWE in facilitating regulatory and reimbursement decisions remain poorly defined. Specifically, the types of studies, quality of data, and level of evidence to consider in the reimbursement decision process remain largely unclear. Moreover, the appropriate places for RWE throughout the product life cycle also need to be established.
In this article, I delve deeper into these issues, focusing particularly on the need for national and international collaboration, harmonization, and frameworks.
Why is RWE Needed for Regulatory Decision-making?
While randomized controlled trials (RCTs) have high internal validity and remain the gold standard for informing both pre- and post-market regulatory decision-making, they have limited external validity. This is because the results from a trial conducted in a pre-specified, homogeneous patient subpopulation under controlled circumstances often does not extrapolate well to the general patient population. In the real world, patients have different ethnicities, socioeconomic backgrounds, performance statuses, comorbidities, and/or concomitant medications. Especially, in the era of personalized medicine, RCTs are becoming more difficult and expensive to conduct. Moreover, RCTs do not take into account the manner in which the approved product is used by prescribers and their patients.
On the other hand, observational studies can address important clinical questions that RCTs cannot answer. Circumstances in which observational studies and RWE are particularly valuable include when:
- Evidence regarding the safety or efficacy of a treatment in a broader, non-target population is required
- Assessing the safety and efficacy of products that have received accelerated and conditional regulatory approval based on limited data
- Large studies are needed in order to assess infrequent events or long-term effects of a treatment
- Studying rare diseases or other conditions that are difficult to study in RCTs
- Adherence might have an impact on the treatment outcome
- A prompt result is needed
- When multiple treatment solutions are available
- Exploring population subsets such as patients with multiple comorbidities or ethnic minorities
For these reasons, RWE is starting to gain acceptance as an important part of regulatory and reimbursement decision-making. Now, the next step will be to develop clear national and international guidelines on the optimal collection and analysis of RWD.
Level of Evidence
The level of evidence is particularly important when international scientific organizations are developing guidelines. However, although interrelated, a high level of evidence does not always equate high-quality data. Even if a study technically provides the highest level of evidence (e.g. a homogeneous systematic review of randomized clinical trials), it may be poorly designed and/or implemented.
The levels of evidence required for reimbursement and regulatory decision-making vary according to multiple factors, including the specific question to address and therapeutic, clinical, and ethical factors (e.g., the intended patient population, relevant clinical trial evidence, availability of comparators or alternative therapies, and the disease burden/severity). Importantly, in all situations, the research must follow high standards of evidence generation, from first proposing the research question to reporting the results.
Quality and Validity of Data
The quality of RWD is essential when considering RWE for regulatory decision-making. Methodological advances, including more sophisticated datasets, better statistical methods, and the ongoing development of more consistent methodological approaches, are contributing to observational studies becoming not only more relevant, but also much more credible. Especially, the evolution of big data has contributed to improving the perception, uptake, and application of observational data.
RWE for regulatory and reimbursement decision-making must be generated from high‐quality data that are (1) obtained from relevant RWD sources; (2) cleaned, harmonized, and linked to fill in any gaps; and (3) include endpoints. To ensure regulatory‐grade data quality, all policies and procedures must be well documented, each dataset must be tested, and the results should be reported when appropriate. Furthermore, it is essential that the RWD are valid.
Ensuring Data Validity
Data validation – the verification of reproducibility or comparability with data from other studies – is commonly performed by comparing data recorded in one database to those in other databases, whether paper-based or electronic. This is also key to confirming the validity of the methods used in RWE studies. The methodology of the RWD validation process should be reported, clear, and systematic. Moreover, it should include descriptions of any assumptions and inferences, as well as interpretation of potential under- or overestimations of the study outcome. If the RWE differs from the results of the relevant RCTs, additional in-depth analyses will be required to examine and understand why the results diverged.
Ensuring Regulatory-grade RWE
In addition to data validity, the following checklist of key items for ensuring regulatory-grade RWE has been proposed:
- High quality data. The data must be clear, traceable, and auditable. The data quality must be systematically measured with predetermined frameworks and against predetermined benchmarks.
- Complete data. This requires predefined rules for abstraction of structured and unstructured data, data harmonization, and quality monitoring.
- Transparent study designs and analyses. This is critical for ensuring robust RWE. In particular, the specific aims and cohort selection criteria need to be precisely defined.
- Generalizability to broad patient populations. Potential biases to generalizable results must be identified and reported to allow for appropriate statistical and clinical interpretations.
- Timely and recent data. Details related to the time period that the data analysis represents must be reported, with the use of recent, up-to-date data being essential.
- Scalable data. With large numbers of patients and variables, the data analysis becomes increasingly complex. Hence, a balance between high touch and automation; a modular data model that can be used in multiple contexts and that facilitates model evolution; and unambiguous variable definitions, particularly for endpoints, are needed.
Importance of Harmonized Terminology, Methods, and Data Sharing
In addition to the above-mentioned items, an immediate barrier to precise discussion of the potential role of RWE in regulatory and reimbursement decision-making is the inconsistent terminology used. The terminology, definitions, and treatment and disease codes used should ideally be harmonized and standardized to facilitate international collaboration. Moreover, we need harmonized reporting processes and an effective audit system for verifying the accuracy and completeness of registry data.
Currently, no standardized criteria exist for validating and ensuring high-quality RWD. Thus, the above factors should all be considered when establishing guidelines for the role of RWE in regulatory and reimbursement decisions.
The Type, Level, and Quality of Evidence Needed Depend on the Purpose
In future guidelines, it is also important to assess the credibility and validity of RWE according to their purpose. The Duke-Margolis RWE Collaborative has established a useful framework for assessing RWE for regulatory decision-making. This represents a good starting point for further discussion and development of national and international guidelines related to regulatory and reimbursement decision-making. In their white paper on RWE, the Duke-Margolis RWE Collaborative suggests that RWE should be defined as the output from the combination of a variety of different RWD sources and methods for evidence development. Consequently, the resultant applicability of RWE will vary across regulatory contexts of use.
In brief, RWE should be assessed by considering the following:
- The regulatory context. That is, the specific regulatory decision under consideration.
- The clinical context, including the disease prevalence and treatment effects.
- Data collection considerations, including the quality, level, and validity of RWD, as discussed above.
- Study design and conduct considerations, including the credibility of the methodology used to collect RWD and analyze RWE.
Appropriately matching data and methods to potential regulatory uses will take time and require ongoing input from all involved stakeholders. However, progress can be made by thoughtful and transparent collaborations to address the underlying challenges related to data collection, study design, and shared infrastructure. The wealth of RWE currently being generated through routine patient care is already proving useful to many key stakeholders. Hence, determining ways to make RWE sufficiently reliable for regulatory bodies will be the next step.
International RWE Policy Frameworks
International clinical research networks and agencies are key players in determining the place of RWE in regulatory decisions. Their current programs and frameworks can help provide support for harmonizing the rules and guidance for RWE generation.
Europe
In Europe, a number of groups are, or have previously been, working on establishing the role of RWE in regulatory and reimbursement decision-making by the European Medicines Agency (EMA). Among many others, these include the Innovative Medicines Initiative (IMI) through the IMI GetReal project, European Medical Information Framework (EMIF), and European Network of Centers for Pharmacoepidemiology and Pharmacovigilance (ENCePP).
In particular, ENCePP’s Guide on Methodological Standards in Pharmacoepidemiology has helped greatly advance the understanding of RWE. This guide explores primary data collection vs. secondary data use, statistical and epidemiological methods, and study designs related to RWE generation.
More recently, the EMIF was launched to help improve access to patient-level data. This 5-year project, which should be wrapping up this year, aims to develop common technical and governance solutions and improve access and use of health data. To this aim, a common information framework is being created to link and facilitate access to diverse medical and research data sources.
Finally, InSite (not to be confused with the Impetus InSite Platform©️!) is another innovative platform that enables efficient and secure re-use of electronic health record data for research. This is a Champion Programme building on the EHR4CR, designed to prove the value of RWD for clinical research on a wide scale. InSite facilitates the collaboration between clinicians and researchers and aims to maximize the output of clinical research through new technology.
However, as promising as this all sounds, a recent review of the landscape of EU-funded initiatives linked to RWE found that the immediate utilization of their outputs to support regulatory decision-making has been limited so far. This seems to be largely due to insufficient available information and discrepancies between the outputs and original objectives.
North America
In the US, the Food and Drug Administration (FDA) published their non-binding guidelines on the use of RWE to support regulatory decision-making for medical devices. These guidelines describe the factors that the FDA considers when evaluating RWD or RWE. However, they do not provide specific criteria for determining the suitability of RWD or RWE for a particular regulatory decision.
In addition to the FDA and the above-mentioned Duke-Margolis RWE Collaborative, the Institute for Clinical and Economic Review (ICER) is another major player assessing the potential role of RWE for regulatory decision-making in the US. ICER is an independent not-for-profit research institute that produces reports analyzing the available evidence related to the effectiveness and value of drugs and other medical services. In their white paper on RWE, ICER identifies the key opportunities and associated challenges around the use of RWE for regulatory decisions. This is another great resource for agencies and working groups seeking to determine the place for RWE in these decisions.
In Canada, CADTH, an independent not-for-profit organization responsible for providing healthcare decision-makers with objective evidence for informed decisions about the optimal use of health technologies, recently published their report on RWE. In this report, CADTH concludes that the way in which RWE is used and its value to decision-making depend on three main factors. Namely, the clinical context, availability or feasibility of conducting RCTs, and the agency’s policies and practices must be considered.
Need for Multi-stakeholder National and International Collaboration
As mentioned, despite the many potential advantages of RWE, only few clear guidelines on RWD and RWE currently exist. Especially, the use of RWE for guiding clinical practice recommendations is rare. Many experts agree that, as the robustness of RWD and RWE increases, scientific organizations and regulatory agencies should consider elevating the status of RWE.
To achieve this, however, the industry must first overcome several obstacles. These include challenges related to accessing data by different stakeholders, uncertainty of the acceptability of RWE for regulatory/reimbursement decisions, and practical challenges in integrating databases across different geographical areas and disease states. Accordingly, there is a strong need for agencies to develop consensus on the current frameworks to ensure more aligned approaches to RWE. In turn, this requires collaboration between all relevant stakeholders (academia, industry, physicians, payers, patients etc.), both nationally and internationally. Streamlining terminology, data standards, technologies, and database infrastructure is essential. Furthermore, education for researchers, healthcare professionals, and patients will also help emphasize the importance of ongoing data collection.
The Impetus Solution!
Here at Impetus, we specialize in bringing key stakeholders together to share insights and co-create materials. In the setting of RWE, the Impetus InSite Platform©️ is ideal for engaging stakeholders or advisors from across the world to co-create guidelines, consensus statements, white papers, or educational materials. The online format lets the advisors provide feedback and edit documents on their own time, without the need to travel or adjust their schedule. If needed, web-based or in-person meetings can be organized at critical time points to allow for synchronous discussions on urgent matters.
We are committed to ensuring the success of our clients’ projects. The well-established and easy-to-use platform and processes in place make it easy for both our clients and advisors. The virtual nature of the advisory boards helps increase the engagement rates of the advisors. Moreover, the asynchronous nature of the assignments give the advisors time to pause, reflect, process, and review their colleagues’ comments on their own time. In turn, this allows for more thoughtful and granular insights shared through the online forums. Impetus’ dedicated team of digital and medical experts create, program, project-manage, and report all assignments. Hence, the manufacturer’s workload is minimal and so are the costs when comparing to more traditional in-person consultancy meetings.
What’s Next for RWE?
It is clear that more work is needed before RWE will routinely be used to guide regulatory and reimbursement decisions. Nevertheless, it is equally clear that RWE will be playing an increasingly important role in these processes. In general, RWE will only help confirm or supplement RCT findings. In some situations, however, RWE will be especially valuable, including when RCTs are not feasible or ethical, when there are substantial unmet needs, and to identify infrequent adverse events and long-term outcomes.
Regulatory agencies are exploring ways that RWE can help influence regulatory and reimbursement decisions at all stages. A greater understanding, dissemination, uptake, and use of observational data may help drive the ongoing evolution of research, data collection, analysis, and validation. Consequently, this will improve the quality, credibility, and application of RWD. As the roles of RWE are clarified, this may translate into the development of new processes and standards worldwide.
References
CADTH. (2018). About CADTH. Retrieved from https://www.cadth.ca/about-cadth
CADTH. (2018). Use of Real-World Evidence in Single-Drug Assessments Environmental Scan. Retrieved from https://www.cadth.ca/use-real-world-evidence-single-drug-assessments-environmental-scan
Duke Margolis Center for Health Policy. (2017). A Framework for Regulatory Use of Real-World Evidence. Retrieved from: https://healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_paper_2017.09.06.pdf
EMIF. (2018). About the EMIF Project. Retrieved from http://www.emif.eu/about
Gavini, F. (2017). Real World Data: A Pressing Need for Harmonizing the Standards of Evidence Generation. Retrieved from https://www.clinicaltrialsarena.com/news/clinical-trials-arena/real-world-data-a-pressing-need-for-harmonizing-standards-of-evidence-generation-5717363-2/
Hampson, G., Towse, A., Dreitlein, B., et al. (2018). Real World Evidence for Coverage Decisions: Opportunities and Challenges – A Report from the 2017 ICER Membership Policy Summit. Retrieved from https://icer-review.org/wp-content/uploads/2018/03/ICER-Real-World-Evidence-White-Paper-03282018.pdf
Heikinheimo, O., Bitzer, J., Rodríguez, L.G. (2017). Real-world research and the role of observational data in the field of gynaecology – a practical review. Eur J Contracept Reprod Health Care, 250-259.
Innovative Medicines Canada, BIOTECanada. (2018). Innovative Medicines Canada and BIOTECanada Comments CAPT Real World Evidence Workshop October 9, 2018.
Katkade, V.B., Sanders, K.N., Zou, K.H. (2018). Real world data: an opportunity to supplement existing evidence for the use of long-established medicines in health care decision making. J Multidiscip Healthc, 11, 295-304.
Miksad, R.A., Abernethy, A.P. (2018). Harnessing the Power of Real‐World Evidence (RWE): A Checklist to Ensure Regulatory‐Grade Data Quality. Clin Pharmacol Ther, 103, 202–205.
Plueschke, K., McGettigan, P., Pacurariu, A., et al. (2018). EU-funded initiatives for real world evidence: descriptive analysis of their characteristics and relevance for regulatory decision-making. BMJ Open, 8, e021864.
U.S. Department of Health and Human Services, Food and Drug Administration, Center for Devices and Radiological Health, Center for Biologics Evaluation and Research. (2017). Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices – Guidance for Industry and Food and Drug Administration Staff. Retrieved from https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM513027.pdf