In a groundbreaking study, UCSF researchers used a new clinical large language model (LLM) to identify serious adverse events (SAEs) occurring in patients treated with immunosuppressants to manage inflammatory bowel disease (IBD). The results pave the way for using artificial intelligence (AI) to help clinicians choose the safest treatments for their patients.

“This study shows how AI can learn from medical records to uncover new signals about drug safety in the context of routine clinical care,” said UCSF gastroenterologist Vivek Rudrapatna, MD, PhD, senior author on the study.

Trained on 75 million clinical notes

The LLM used in the study, UCSF BERT (Bidirectional Encoder Representations from Transformers), is designed to interpret clinical text as typically documented in electronic health record (EHR) systems. It was trained on 75 million clinical notes documented across a range of specialties over the last 10 years at UCSF.

For the study, the researchers adapted UCSF BERT to analyze notes from the UCSF Colitis and Crohn’s Disease Center and identify episodes where patients were hospitalized after receiving immunosuppressive treatment for IBD.

“Adverse events that result in patients being hospitalized are considered serious,” Rudrapatna said. “So we trained our AI model to identify all hospitalization events that occurred while a patient was actively receiving an IBD immunosuppressant and asked it to identify the reasons for those hospitalizations.”

UCSF BERT’s accuracy rate at identifying hospitalization events following IBD medication use was 88 to 92%, which is 5 to 10% better than existing models used for SAE detection. No new safety concerns were found, but this novel method has the potential to detect new safety signals and address unmet needs in pharmacovigilance.

“This proof-of-concept study had very favorable results,” Rudrapatna said. “UCSF BERT beat other models by a pretty reasonable margin in terms of accuracy. It’s one of only a handful of models that has been trained using health records data.”

AI-enhanced decision support at point of care

In addition to practicing gastroenterology at UCSF Health, Rudrapatna is a clinical data science investigator at the UCSF Bakar Computational Health Sciences Institute and co-director of the UCSF Center for Real World Evidence, which includes expert data scientists, epidemiologists, biostatisticians and faculty clinicians in all fields of medicine who work together to deliver actionable answers from rich clinical data and incisive AI.

“Models like UCSF BERT will eventually make their way into EHRs and can then help form the backbone of decision-support tools at the point of care,” he said. “We should be able to lean on artificial intelligence and computational algorithms to help make our decisions more robust and better for patients.”

To learn more

UCSF Colitis and Crohn’s Disease Center
Phone: (415) 353-7921 | Fax: (415) 502-2249
Refer a patient
Clinical trials