Services
Company
Legal and Governance
Insights / Articles
The Rise of Artificial Intelligence for Evidence Synthesis and Analysis
Written by Emanuele Arcà, Nick Halfpenny & Elisabeth Fenwick on Wednesday, June 26, 2024
Current challenges
The rise of artificial intelligence (AI) heralds a transformative era in health economics and outcomes research (HEOR). AI technologies are fundamentally changing how stakeholders analyze, interpret, and capitalize on large amounts of healthcare data, empowering them to make well-informed decisions. This revolution is reshaping the entire HEOR landscape, from optimizing clinical trial designs to predicting patient outcomes and evaluating real-world treatment effectiveness. Additionally, as an indispensable tool in evidence synthesis, AI is streamlining literature review, enhancing efficiency, and contributing to the robust generation of evidence for informed decision-making in healthcare.
The increasing volume and rapid expansion of scientific literature, coupled with short turnaround times, make systematic literature reviews (SLRs) more complex. The necessity for highly trained individuals to navigate databases with complex search syntax contributes to resource and budget constraints. Moreover, the high volume poses challenges in maintaining consistency across interpretation and data extraction variables.
“AI is going to make some SLR work quicker and easier to do because the machine will do it, but the important thing is to have trained people asking the questions and working out what we want the AI to do and then trained people interpreting the results,” explains Elisabeth Fenwick, Chief Scientific Officer at OPEN Health. “It isn’t a case that you can feed data into a machine and out comes the answer and it’s done. OPEN Health has the experts who can translate the questions from our clients into something we can give the machine to do and interpret the answers that come back.”
The integration of AI, particularly large language models (LLMs), such as GPT, into HEOR introduces both excitement and challenges. While LLMs offer powerful capabilities, concerns about reproducibility and hallucination effects highlight the need for careful application. Recent publications showcasing the successful use of LLMs in data extraction for network meta-analyses signal substantial time-saving potential.1
Implementing AI in operational processes requires a focus on transparency, accuracy, and reproducibility. Despite the efficiency gains with AI, maintaining human oversight is crucial. Striking a balance between leveraging AI for certain aspects of literature reviews, such as screening and data extraction, and ensuring human involvement in decision-making will help achieve optimal efficiency without compromising quality.
“There is inherently a human bias to systematic literature reviews,” says Emanuele Arcà, Scientific Office Lead for the Strategic Market Access group at OPEN Health. “It’s time-consuming, repetitive work, and the way data is often heterogeneously reported across publications makes it difficult for human reviewers to systematically identify the relevant information. AI tools might support in the identification of relevant data that might otherwise be overlooked.”
In the context of Health Technology Assessment (HTA) submissions, AI holds the promise of efficiency and scalability, addressing varying requirements across different HTAs. However, pitfalls include concerns about accuracy, interpretability, the complexity of AI models, legal and liability issues, and privacy considerations. Despite these challenges, AI’s potential to ease time pressures on researchers and allow for more flexibility in responding to HTA requests underscores its importance in the evolving regulatory landscape.
OPEN Health HEOR team approach
The OPEN Health HEOR team working on AI adopts a discerning and critical-thinking approach to envision the future of evidence synthesis, particularly SLRs, and the integration of AI within this domain. Beyond process automation, the team prioritizes investigating scientific, technical, regulatory, and policy considerations of AI use for SLRs.1,2
This approach has led to the establishment of four distinct internal workstreams, which focus on testing different AI tools, exploring perceptions around AI, understanding the intersection of AI with EU HTA regulations, and examining AI’s role in the interaction between SLRs and indirect treatment comparisons (ITCs), respectively. Each workstream has defined scopes and objectives, with results expected imminently (see Figure 1 below for more details).
Figure 1 AI in evidence synthesis – Scientific and thought leadership strategy at OPEN Health
Testing different AI tools for SLRs
In a strategic move towards heightened efficiency and adaptability, OPEN Health is actively pursuing an initiative to enhance the automation of SLRs through the integration of AI. The primary goal is to transform SLRs into more dynamic and living deliverables, ensuring that our analyses and insights remain agile in response to the evolving business landscape (e.g., the challenges posed by the new EU HTA Regulation). By leveraging AI, we aim to expedite the SLR process significantly. Machine learning algorithms will play a pivotal role in automating searches, the selection process, and data extraction, allowing for real-time updates and a more comprehensive understanding of the literature. This will not only accelerate the review process but also enable us to adapt swiftly to emerging trends and insights.
Over the past three years, OPEN Health has been formally testing different advanced AI platforms. The initial focus was on automating the search and selection processes, with ongoing exploration into the potential use of AI for data extraction. Notably, the workstream concentrating on testing various AI tools is finalizing an SLR assessing the accuracy and efficiency of machine learning and natural language processing tools for publication screening and selection in the preparation of SLRs. This work, along with case studies showcasing testing results with AI, has been accepted for an oral presentation at the HTAi 2024 conference.
“At OPEN Health, we’re experts in evidence generation and evidence analysis and synthesis,” says Arcà. “We believe that taking a product-agnostic approach that tests all the available products will allow us to weigh the advantages and disadvantages of each.”
AI between SLRs and ITCs
The overarching goal of this workstream is to critically evaluate the application of AI in evidence synthesis, specifically SLRs, and evidence analysis, including ITCs. Through this initiative, OPEN Health envisions SLRs and ITCs as synchronized and evolving exercises, constantly monitored and updated to reflect the latest insights. Our focus also extends to the implementation of interactive dashboards. These dashboards, complemented by AI, serve as intuitive interfaces for stakeholders to explore and digest SLR and ITC outputs in real time. This dynamic visualization not only enhances accessibility but also facilitates collaborative decision-making. Teams can interact with the dashboard to delve into specific areas of interest, fostering a more engaged and informed strategic discourse. The integration of AI and interactive dashboards will not only make the SLR and ITC processes more efficient but also elevate the status of these deliverables as living tools that drive continuous improvement and strategic alignment. In doing so, we aim to navigate the complexities of evidence synthesis and analysis with enhanced speed, precision, reliability, and adaptability.
AI use and perceptions across assessors and payers
Through our discussions and examination of HTA/payer websites, we have discerned that these organizations are interested in understanding the potential applications of AI in generating SLRs. However, they are currently in the nascent stages of familiarizing themselves with these methodologies. Notably, existing HTA/payer websites do not explicitly mention the utilization or endorsement of AI concerning the execution of SLRs for HTA. One notable exception is the Institute for Clinical and Economic Review, an influential drug pricing watchdog in the United States, that uses Nested Knowledge for conducting SLRs.
“It’s not clear how HTA agencies are going to react to submissions incorporating SLRs undertaken by AI,” Fenwick says. “There’s work to do in educating the agencies themselves in terms of what these methods can do, and how they can replicate what humans do, so that they become acceptable in submissions.”
Overall, HTA agencies and national health payers seem generally to view the use of AI in SLRs positively, recognizing its potential benefits for efficiency, data analysis, and resource optimization. However, caution is warranted, given concerns about data quality, interoperability, biases, and explainability / reproducibility.2 OPEN Health believes that a collaborative approach involving HTA experts, data scientists, and regulatory bodies is essential. There is a need to establish guidelines and standards for the responsible implementation of AI in SLRs. For example, within the current framework, HTA bodies anticipate the involvement of two reviewers in the SLR process, yet they do not specify whether, or how, AI tools can be integrated into this process. There exists an initial curiosity about the capabilities of AI in the SLR domain, but as of now, formalized guidance or acknowledgment of its role in the review process is yet to be articulated by HTA / payer organizations.
The field is evolving, and stakeholders are balancing enthusiasm for AI with a commitment to quality, transparency, and ethical use, indicating a careful and collaborative path forward. OPEN Health is engaging with HTA bodies and payers with the objective of exploring perceptions and challenges and developing guidance documents.
AI & the EU HTA Regulation
Looking ahead, the impact of European Union Health Technology Assessment (EU HTA) Regulation processes emerges as a critical consideration. With the potential for multiple PICOs (Patient, Intervention, Comparator, Outcome) and stringent timelines, the application of AI in screening and data extraction becomes instrumental. The OPEN Health team sees the EU HTA Regulation as a catalyst for adapting AI in evidence synthesis and generation, providing a feasible solution to manage work volume and enhance efficiency.
“The timelines are even tighter under the new regulations, and every is going to face challenges with making sure they’re able to update their reviews in a timely manner,” says Nick Halfpenny, Senior Director and joint EU Lead, Strategic Market Access, at OPEN Health. “In the context of an EU HTA submission, replacing that one level of human review with AI can work with a high degree of accuracy and maximize efficiencies to allow us to be able to update those reviews when there are just over three months between the last update and the point of submission.”
This workstream’s strategic approach revolves around a thorough exploration of the risks and benefits associated with the integration of AI to meet the timelines and manage the workload imposed by the new European Union Health Technology Assessment Regulatory framework (EU HTAR). This initiative involves a comprehensive review of EUnetHTA guidelines and HTA Regulation documents to gain a nuanced understanding of the stipulations and recommendations regarding the use of AI in this context. By delving into these guidelines, the team aims to align its AI implementation strategy with the regulatory landscape, ensuring compliance while harnessing the advantages that AI can offer.
As a pivotal part of the strategy, the team plans to rigorously test AI capabilities in meeting the heightened PICO requirements within anticipated and reduced timelines. This involves assessing the adaptability of AI tools to enhance the efficiency of SLRs and ITCs with increased flexibility and a focus on data reuse. By testing AI in this context, the team aims to not only streamline the HTA process but also ensure the accuracy and reliability of AI-driven solutions in handling several PICOs, ultimately contributing to a successful joint clinical assessment.
Transforming health through intelligent innovation
AI is revolutionizing HEOR and market access, primarily by improving evidence synthesis and analysis. Despite AI’s potential, challenges such as reproducibility, bias, and the need for human oversight persist, necessitating a balanced approach where AI supports but does not replace human decision-making. As discussed in this article, OPEN Health is critically evaluating AI tools and integrating them with SLRs and ITCs, with the goal of increasing efficiency, transparency, and alignment with evolving regulatory frameworks like the EU HTA.
OPEN Health will continue to lead discussions on using AI in EU HTA evidence generation, focusing on predicting PICOs and digesting larger bodies of evidence in shorter timelines. Additionally, we are integrating AI-driven extraction with interactive dashboards to make it easier to visualize and interpret data. Ultimately, our future efforts will be directed to understanding and driving the acceptance and usability of AI tools among stakeholders involved in HTA.
Working in partnership with our clients, we embrace our different perspectives and strengths to deliver fresh thinking and solutions that make a difference.
Together we can unlock possibilities.
For information about OPEN Health’s services and how we could support you, please get in touch.
References
- van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, van der Palen J, Doggen CJM, Lenferink A. Artificial intelligence in systematic reviews: Promising when appropriately used. BMJ Open. 2023;13(7):e072254. doi:10.1136/bmjopen-2023-072254
- Tachkov K, Zemplenyi A, Kamusheva M, et al. Barriers to use artificial intelligence methodologies in health technology assessment in Central and East European countries. Front Public Health. 2022;10:921226. doi:10.3389/fpubh.2022.921226