Artificial Intelligence (AI) in Learning and Discovery

OpenEvidence as Learning Tool

Introduction to OpenEvidence

OpenEvidence is an AI-powered platform that aggregates, synthesizes, and visualizes peer reviewed medical literature in a format intended to be easily understood and support healthcare professionals in learning and making evidence-based decisions [1]. Daniel Nadler, PhD founded Kensho Technologies while immersed in his studies. Nadler founded OpenEvidence in November 2021 and was selected to participate in a Mayo Clinic Platform accelerator. As of October 2021, more than 10,000 clinicians have registered for accounts [2], [3].  

The Levy Library was informed that students and clinicians have reported using OpenEvidence. To better understand its benefits and key considerations, it is worth conducting an in-depth exploration of OpenEvidence. This and future posts will include a detailed investigation of innovative generative AI tools for teaching, learning, and research. The Levy Library is your partner in thoughtfully and critically reviewing new tools and integrating them into information literacy education for the Mount Sinai community. 

OpenEvidence is an experimental technology demonstrator service. It is not meant to provide medical advice, diagnosis or treatment. Users must assess the information they obtain from OpenEvidence. Providers must acknowledge their obligation to adhere to all laws and professional standards applicable to their field of practice of medicine or other relevant health profession [4]. 

OpenEvidence is not Health Insurance Portability and Accountability Act (HIPAA)-compliant and Protected Health Information (PHI) should NOT be included in queries or prompts to the system. Agreeing to the Terms of Use is required [5]. At the time this was written, OpenEvidence may collect data from the prompts asked and the interactions had with the service. They claim this data is collected to better understand customers’ behavior and use of the product. Their terms state they may disclose, distribute, transfer, and sell collected data in connection with customers’ use, provided personal information of users is not disclosed to third parties [4].

Terms may change from the time of this blog post. We strongly recommend Mount Sinai users carefully review and decide whether to accept or decline the terms of any AI platform regarding privacy and subsequent use of queries/prompts as training data. 

In investigating OpenEvidence in April-May 2024, as a librarian without a National Provider Identifier (NPI), I have been limited to two searches per day. Any non-clinician learners or clinicians who do not wish to register their account with an NPI will be limited. The limitation restricts accounts for 24-hour period since the second question of the day was submitted. 

Using OpenEvidence Effectively with PICO Example 

Taking time to methodically craft your questions ensures thoroughly generated outputs, especially if you are limited to inputting two daily prompts. To use services effectively, please note the following steps and options: 

• Account Registration: Required for access.  

• Search Filtering: Use the drop-down tab to the left of the text box to filter your search. 

  • Filtering options include: 

  • All: (default setting) 

  • Guidelines & Standard of Care 

  • Clinical Evidence 

• Follow-up Questions: Users can ask a follow-up question after an output is generated.  

When articles are retrieved, they may be tagged with Highly Relevant, Leading Journal, or New Research, which could be helpful when verifying the information you are reading to be relevant and accurate –  

• Highly Relevant: The proprietary evidence retrieval algorithm indicates the evidence contained in the source can be directly used to answer the question 

• Leading Journal: At or above the 90th percentile Impact Factors [1].  

• New Research: The evidence comes from research published within the last year. 

Below are screenshots that display an example PICO (population, intervention, comparison, outcome) question input into OpenEvidence and the generated response using OpenEvidence on May 20th, 2024. 

The question reads, "In elderly patients with chronic lower back pain, does yoga therapy compared to physical therapy reduce pain and improve functional mobility?"

The response will generate 3-4 paragraphs and include an abstract for the references cited. Simply select the Show Abstract button next to reach reference to expand the information.

OpenEvidence is one source, and as an experimental learning tool, should not be the only source of information you consult. It's crucial to check what the original references say and document the AI-generated information (such as taking screenshots) if you are going to bring it into a clinical discussion or assignment. All AI systems may make errors, and without knowing which sources it is consulting, you should continue to compare outputs with trusted sources to ensure you have obtained the highest level of evidence for your clinical questions.

Final Thoughts

As a tool open to all learners and providers, OpenEvidence can play a helpful role in addressing clinical questions by aggregating and visualizing clinical evidence. It is critical that users ensure their questioning and prompting strategies do not include PHI to safeguard the identity of their patients. Always document prompts, generated outputs, and the date. AI platforms are always changing and most of the time we don't even know it. Be mindful of terms of service and privacy policy, which could be updated frequently, and how your information is used by the platform.

The Levy Library welcomes you to ask questions about OpenEvidence or any other generative AI tool. Be sure to check out the Library's Artificial Intelligence (AI) in Learning and Discovery LibGuide to see the most recent journal publications and other resources about AI in medical education, research, and publishing.

References 

[4] Xyla Inc. Network Terms of Use [Internet]. [cited 2024 May 9] OpenEvidence. Available from: https://www.openevidence.com/policies/terms