AI in Teaching and Learning Blog

Introduction to OpenEvidence

OpenEvidence is an AI-powered platform that aggregates, synthesizes, and visualizes peer reviewed medical literature in a format intended to be easily understood and support healthcare professionals in learning and making evidence-based decisions [1]. Daniel Nadler, PhD founded Kensho Technologies while immersed in his studies. Nadler founded OpenEvidence in November 2021 and was selected to participate in a Mayo Clinic Platform accelerator. As of October 2021, more than 10,000 clinicians have registered for accounts [2], [3].  

The Levy Library was informed that students and clinicians have reported using OpenEvidence. To better understand its benefits and key considerations, it is worth conducting an in-depth exploration of OpenEvidence. This and future posts will include a detailed investigation of innovative generative AI tools for teaching, learning, and research. The Levy Library is your partner in thoughtfully and critically reviewing new tools and integrating them into information literacy education for the Mount Sinai community. 

OpenEvidence is an experimental technology demonstrator service. It is not meant to provide medical advice, diagnosis or treatment. Users must assess the information they obtain from OpenEvidence. Providers must acknowledge their obligation to adhere to all laws and professional standards applicable to their field of practice of medicine or other relevant health profession [4]. 

OpenEvidence is not Health Insurance Portability and Accountability Act (HIPAA)-compliant and Protected Health Information (PHI) should NOT be included in queries or prompts to the system. Agreeing to the Terms of Use is required [5]. At the time this was written, OpenEvidence may collect data from the prompts asked and the interactions had with the service. They claim this data is collected to better understand customers’ behavior and use of the product. Their terms state they may disclose, distribute, transfer, and sell collected data in connection with customers’ use, provided personal information of users is not disclosed to third parties [4].

Terms may change from the time of this blog post. We strongly recommend Mount Sinai users carefully review and decide whether to accept or decline the terms of any AI platform regarding privacy and subsequent use of queries/prompts as training data. 

In investigating OpenEvidence in April-May 2024, as a librarian without a National Provider Identifier (NPI), I have been limited to two searches per day. Any non-clinician learners or clinicians who do not wish to register their account with an NPI will be limited. The limitation restricts accounts for 24-hour period since the second question of the day was submitted. 

Using OpenEvidence Effectively with PICO Example 

Taking time to methodically craft your questions ensures thoroughly generated outputs, especially if you are limited to inputting two daily prompts. To use services effectively, please note the following steps and options: 

• Account Registration: Required for access.  

• Search Filtering: Use the drop-down tab to the left of the text box to filter your search. 

  • Filtering options include: 

  • All: (default setting) 

  • Guidelines & Standard of Care 

  • Clinical Evidence 

• Follow-up Questions: Users can ask a follow-up question after an output is generated.  

When articles are retrieved, they may be tagged with Highly Relevant, Leading Journal, or New Research, which could be helpful when verifying the information you are reading to be relevant and accurate –  

• Highly Relevant: The proprietary evidence retrieval algorithm indicates the evidence contained in the source can be directly used to answer the question 

• Leading Journal: At or above the 90th percentile Impact Factors [1].  

• New Research: The evidence comes from research published within the last year. 

Below are screenshots that display an example PICO (population, intervention, comparison, outcome) question input into OpenEvidence and the generated response using OpenEvidence on May 20th, 2024. 

The question reads, "In elderly patients with chronic lower back pain, does yoga therapy compared to physical therapy reduce pain and improve functional mobility?"

The response will generate 3-4 paragraphs and include an abstract for the references cited. Simply select the Show Abstract button next to reach reference to expand the information.

OpenEvidence is one source, and as an experimental learning tool, should not be the only source of information you consult. It's crucial to check what the original references say and document the AI-generated information (such as taking screenshots) if you are going to bring it into a clinical discussion or assignment. All AI systems may make errors, and without knowing which sources it is consulting, you should continue to compare outputs with trusted sources to ensure you have obtained the highest level of evidence for your clinical questions.

Final Thoughts

As a tool open to all learners and providers, OpenEvidence can play a helpful role in addressing clinical questions by aggregating and visualizing clinical evidence. It is critical that users ensure their questioning and prompting strategies do not include PHI to safeguard the identity of their patients. Always document prompts, generated outputs, and the date. AI platforms are always changing and most of the time we don't even know it. Be mindful of terms of service and privacy policy, which could be updated frequently, and how your information is used by the platform.

The Levy Library welcomes you to ask questions about OpenEvidence or any other generative AI tool. Be sure to check out the Library's Artificial Intelligence (AI) in Learning and Discovery LibGuide to see the most recent journal publications and other resources about AI in medical education, research, and publishing.

References 

[4] Xyla Inc. Network Terms of Use [Internet]. [cited 2024 May 9] OpenEvidence. Available from: https://www.openevidence.com/policies/terms 

By Carrie Levinson, MSLIS

Every month, our AI blog provides a selection of literature on artificial intelligence in teaching and learning. Here’s the roundup for April 2024:

Maiberg E. Google Books is indexing AI-generated garbage. 404 Media [Internet]. 2024 Apr 4. Available from: https://www.404media.co/google-books-is-indexing-ai-generated-garbage  

Excerpt: “Google Books is indexing low quality, AI-generated books that will turn up in search results, and could possibly impact Google Ngram viewer, an important tool used by researchers to track language use throughout history.” 

Knox L. The (AI) counselor is in. Inside Higher Ed [Internet]. 2024 Apr 8. Available from: https://www.insidehighered.com/news/admissions/traditional-age/2024/04/08/can-ai-make-college-counseling-more-equitable  

Excerpt: “AI-powered college advising tools promise to free up time-strapped counselors and “democratize” admissions expertise for less-privileged high schoolers. Will they?” 

Note: Create a free account on the Inside Higher Ed site to access articles.   

Warner J. What’s off limits from AI? Inside Higher Ed [Internet]. 2024 Apr 9. Available from: https://www.insidehighered.com/opinion/blogs/just-visiting/2024/04/09/some-uses-ai-education-should-be-nonstarters-opinion  

Excerpt: “I had hoped that with generative AI having come to (and for) higher education that there would be some applications that would collectively be judged beyond the pale, nonstarters, no-go zones.” 

Note: Create a free account on the Inside Higher Ed site to access articles.   

Coffey L. Higher education is most trusted source to handle AI. Inside Higher Ed [Internet]. 2024 Apr 9. Available from: https://www.insidehighered.com/news/quick-takes/2024/04/09/report-higher-education-most-trusted-source-handle-ai  

Excerpt: “While the public remains cautious of artificial intelligence (AI), most trust higher education institutions to use AI responsibly, according to a new report...” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Schroeder R. Preparing for our new AI workforce in higher education. Inside Higher Ed [Internet]. 2024 Apr 10. Available from: https://www.insidehighered.com/opinion/blogs/online-trending-now/2024/04/10/preparing-new-ai-workforce-higher-education  

Excerpt: “The rapid development of Generative AI has opened the possibility of far more efficient and cost-effective use of technology to assume roles currently performed by people.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Frazee J. Using Student Data to Bridge the AI Divide. Educause Review [Internet]. 2024 Apr 10. Available from: https://er.educause.edu/articles/2024/4/using-student-data-to-bridge-the-ai-divide  

Excerpt: “Knowing your students is essential for bridging the AI divide and paving the way for a more inclusive and equitable future.” 

Baule S. How open should teachers be about using AI? Inside Higher Ed [Internet]. 2024 Apr 10. Available from: https://www.insidehighered.com/opinion/career-advice/2024/04/11/should-instructors-disclose-their-own-use-ai-teaching-opinion  

Excerpt: “A lot has been written about the impact of AI on student work, but relatively little has focused on the role it should play for instructors...” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Torres JT & Nemeroff A. Are we asking the wrong questions about ChatGPT? The Chronicle of Higher Education [Internet]. 2024 Apr 15. Available from: https://www-chronicle-com.us1.proxy.openathens.net/article/are-we-asking-the-wrong-questions-about-chatgpt  

Excerpt: “Since the emergence of ChatGPT, one of the most frequent questions we hear from faculty members who request instructional support is, 'What should I say about AI on my syllabus?' Most of the time, what they’re really asking is: 'How do I police the use of AI in my classes?'” 

Note: Login when prompted with your Mount Sinai email and password to access full article.    

Quinn R. Annual provosts’ survey shows need for AI policies, worries over campus speech. Inside Higher Ed [Internet]. 2024 Apr 16. Available from: https://www.insidehighered.com/news/tech-innovation/artificial-intelligence/2024/04/16/provosts-survey-shows-need-ai-policies  

Excerpt: “Many institutions are not yet prepared to help their faculty members and students navigate artificial intelligence. That’s just one of multiple findings from Inside Higher Ed’s annual survey of chief academic officers.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Coffey L. Religious institutions embrace AI as an educational tool. Inside Higher Ed [Internet]. 2024 Apr 16. Available from: https://www.insidehighered.com/news/tech-innovation/artificial-intelligence/2024/04/16/religious-universities-embrace-ai-another 

Excerpt: “Despite widespread concerns about AI among secular and nonsecular institutions, religious colleges are treating generative AI as a tool for lessons that go beyond academics and also focus on the whole person.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Coffey L. Faculty unions seeking formal AI guidelines. Inside Higher Ed [Internet]. 2024 Apr 18. Available from: https://www.insidehighered.com/news/tech-innovation/artificial-intelligence/2024/04/18/faculty-unions-seek-ai-guidelines-contracts  

Excerpt: “Most faculty union contracts make no mention of artificial intelligence, but rising concerns about faculty autonomy and job security are starting to shape discussions about labor protections.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Lederman D. New data consortium wants colleges to ‘own’ their AI future. Inside Higher Ed [Internet]. 2024 Apr 19. Available from: https://www.insidehighered.com/news/tech-innovation/artificial-intelligence/2024/04/19/data-consortium-aims-colleges-own-their-ai  

Excerpt: “A group of higher education leaders are joining forces to try to make sure colleges and universities—especially under-resourced ones—aren’t left behind as generative artificial intelligence transforms our work and our lives.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Warner J. Making progress against ChatGPT. Inside Higher Ed [Internet]. 2024 Apr 19. Available from: https://www.insidehighered.com/opinion/blogs/just-visiting/2024/04/19/chatgpt-does-not-rule-over-how-we-teach-and-students-learn  

Excerpt: “I did not think I would witness more doubt and despair about the work of teaching than what was going on during the pandemic, but I now believe that the presence of large language models has surpassed that period in terms of triggering negative emotions among those working directly with students. My chief response is sympathy. The feeling that you are putting great effort into delivering experiences that should be—and have been—meaningful to students, but which students are bypassing almost entirely by employing ChatGPT or their ilk, is simply awful.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Plotts C & Gonzalez L. Creating a culture around AI: Thoughts and decision-making. Educause Review [Internet]. 2024 Apr 22. Available from: https://er.educause.edu/articles/2024/4/creating-a-culture-around-ai-thoughts-and-decision-making  

Excerpt: “Given the potential ramifications of artificial intelligence (AI) diffusion on matters of diversity, equity, inclusion, and accessibility, now is the time for higher education institutions to adopt culturally aware, analytical decision-making processes, policies, and practices around AI tools selection and use.” 

Bowen JA & Watson CE. Is AI finally a way to reduce higher ed costs? Inside Higher Ed [Internet]. 2024 Apr 23. Available from: https://www.insidehighered.com/opinion/views/2024/04/23/ai-finally-way-reduce-higher-ed-costs-opinion  

Excerpt: “Jobs consist of groups of tasks. AI can’t do everything better, but almost every job has some tasks that AI can already do better. The ability to outsource tedious tasks (either to others or machines, like a spellchecker) often makes us happier and more efficient, and it is no surprise the same has already been demonstrated when we use AI well.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Muscanell N. 2024 AI breakout report: Perceptions of planning and readiness. Boulder, CO: ECAR, April 2024. Available from: https://www.educause.edu/ecar/research-publications/2024/2024-ai-breakout-report-perceptions-of-planning-and-readiness/introduction-and-key-findings  

Excerpt: “In this breakout report, we provide a closer look at higher education professionals' sentiments toward their institution's strategic planning and readiness, examining similarities and differences based on respondent position and areas of responsibility.” 

Blumenstyk G. The Edge: Some colleges are getting AI-ready. Are you? The Chronicle of Higher Education [Internet]. 2024 Apr 24. Available from: https://www-chronicle-com.us1.proxy.openathens.net/newsletter/the-edge/2024-04-24

Excerpt: “This week I share some ideas for getting professors and others up to speed on the basics of artificial intelligence. Separately, I highlight an idea to help colleges build revenue and expand experiential-learning opportunities.” 

Note: Login when prompted with your Mount Sinai email and password to access full article.  

McMurtie B. Teaching: How has ChatGPT affected your teaching this semester? The Chronicle of Higher Education [Internet]. 2024 Apr 25. Available from: https://www-chronicle-com.us1.proxy.openathens.net/newsletter/teaching/2024-04-25

Excerpt: "This week, I: Ask you to tell us how generative AI has affected your teaching; Continue the discussion around teaching study skills to students."

Note: Login when prompted with your Mount Sinai email and password to access full article.    

Coffey L. Robot speaker at commencement? Some human students balk. Inside Higher Ed [Internet]. 2024 Apr 26. Available from: https://www.insidehighered.com/news/tech-innovation/artificial-intelligence/2024/04/26/students-balk-after-university-picks-robot  

Excerpt: “Each year, D’Youville University, a small, private institution in upstate New York, strives to bring in a commencement speaker to address a topic that’s both timely and relevant to its roughly 500 graduates. Past topics have run the gamut, including gender equity and the opioid epidemic. This year, the obvious choice was artificial intelligence, said Lorrie Clemo, its president.” 

Note: Create a free account on the Inside Higher Ed site to access articles. 

Zhou H & Bishtawi H. 2024 Apr 30. The impact of AI on information discovery: From information gathering to knowledge application. In: Society for Scholarly Publishing. Scholarly Kitchen [Internet]. Mount Laurel, NJ: Society for Scholarly Publishing. Available from: https://scholarlykitchen.sspnet.org/2024/04/30/the-impact-of-ai-on-information-discovery-from-information-gathering-to-knowledge-application

Excerpt: "We all know that the ability to search for and find information quickly and efficiently is vital for researchers worldwide. And as we’re also aware, AI applications are increasingly transforming the way researchers do their work and interact with published content. But how is AI – and particularly generative AI – changing the way we find information?"