a, BiomedGPT focuses primarily on visual and textual inputs, but can also process tabular data through serialization. CT, computed tomography; EHR, electronic health records; EKG, electrocardiogram; MRI, magnetic resonance imaging. b, Examples of the supported downstream visual-language tasks of BiomedGPT demonstrate its versatility. Additional tasks can be incorporated to meet further clinical needs through lightweight, task-specific fine-tuning. c, Examples of clinically relevant use-cases for BiomedGPT include tasks in which the input consists of both image and text or only text; the model responds to queries (Q) by generating responses (A). Thanks to its unified framework design and comprehensive pretraining on biomedical data, BiomedGPT is highly adaptable and can be applied to a variety of downstream tasks. BP, blood pressure; CABG, coronary artery bypass graft surgery; CAD, coronary artery disease; ER, estrogen receptor; GnRH, gonadotropin-releasing hormone; HR, heart rate; NRB, non-rebreather mask; PR, progesterone receptor; RR, respiratory rate; Reg#, de-identified “Medical Record Number.” Credit: Nature Medicine (2024). DOI: 10.1038/s41591-024-03185-2
A picture may be worth a thousand words, but they both have a lot of work to do to catch up to BiomedGPT. A Lehigh University research team has now collaborated with Massachusetts General Hospital in an effort to transform medical text and images into faster disease diagnosis, enhanced medical reporting, improved drug discovery, and more.
Covered recently in the journal Nature Medicine, BiomedGPT is a new type of artificial intelligence (AI) designed to support a wide range of medical and scientific tasks. This new study, conducted in collaboration with multiple institutions, is described in the article as “the first open-source and lightweight vision–language foundation model, designed as a generalist capable of performing various biomedical tasks.”
“This work combines two types of AI into a decision support tool for medical providers,” explains Lichao Sun, an assistant professor of computer science and engineering at Lehigh University and a lead author of the study. “One side of the system is trained to understand biomedical images, and one is trained to understand and assess biomedical text.
“The combination of these allows the model to tackle a wide range of biomedical challenges, using insight gleaned from databases of biomedical imagery and from the analysis and synthesis of scientific and medical research reports.”
State-of-the-art results for medical practitioners and patients
The key innovation described in the article is that this AI model doesn’t need to be specialized for each task. Typically, AI systems are trained for specific jobs, like recognizing tumors in X-rays or summarizing medical papers. However, this new model can handle many different tasks using the same underlying technology. This versatility makes it a “generalist” model—and a powerful new tool in the hands of medical providers.
“BiomedGPT is based on foundation models, a recent development in AI,” says Sun. “Foundation models are large, pre-trained AI systems that can be adapted to various tasks with minimal additional training. The generalist model described in the article has been trained on vast amounts of biomedical data, including images and text, enabling it to perform well across different applications.”
“By evaluating 25 datasets across 9 biomedical tasks and different modalities,” says Kai Zhang, a Lehigh Ph.D. student advised by Sun, who serves as first author of the article.
Zhang says that he is proud that the open-source codebase is available for other researchers to use as a springboard to drive further development and adoption.
The team reports that the technology behind BiomedGPT may one day help doctors by interpreting complex medical images, assist researchers by analyzing scientific literature, or even aid in drug discovery by predicting how molecules behave.
“The potential impact of such technology is significant,” Zhang says, “as it could streamline many aspects of health care and research, making them faster and more accurate. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.”
A team effort for clinical validation, and more
A crucial step in the process was validation of the model’s effectiveness and applicability in real-world health care settings.
“Clinical testing involves applying the AI model to real patient data to assess its accuracy, reliability, and safety,” Sun says. “This testing ensures that the model performs well across different scenarios. The outcomes of these tests helped refine the model, demonstrating its potential to improve clinical decision-making and patient care.”
Massachusetts General Hospital (MGH) played a crucial role in the development and validation of the BiomedGPT model. The institution’s involvement primarily focused on providing clinical expertise and facilitating the evaluation of the model’s effectiveness in real-world health care settings.
For instance, the model was tested with radiologists at MGH, where it demonstrated superior performance in tasks like visual question answering and radiology report generation. This collaboration helped ensure that the model was both accurate and practical for clinical use.
Other contributors to BiomedGPT include researchers from University of Georgia, Samsung Research America, University of Pennsylvania, Stanford University, University of Central Florida, UC-Santa Cruz, University of Texas-Health, Children’s Hospital of Philadelphia, and the Mayo Clinic.
“This research is highly interdisciplinary and collaborative,” says Sun. “The research involves expertise from multiple fields, including computer science, medicine, radiology, and biomedical engineering. Each author contributes specialized knowledge necessary to develop, test, and validate the model across various biomedical tasks. Large-scale projects like this often require access to diverse datasets and computational resources, along with access to skills in algorithm development, model training, evaluation, and application to real-world scenarios, as well as clinical testing and validation.”
“This was a true team effort,” he says. “Creating something that can truly help the medical community improve patient outcomes across a wide range of issues is a very complex challenge. With such complexity, collaboration is key to creating impact through the application of science and engineering.”
More information:
Kai Zhang et al, A generalist vision–language foundation model for diverse biomedical tasks, Nature Medicine (2024). DOI: 10.1038/s41591-024-03185-2
Provided by
Lehigh University
Citation:
Open-source AI model can assess biomedical images and text to provide real-time, patient-focused insight (2024, November 4)
retrieved 4 November 2024
from https://medicalxpress.com/news/2024-11-source-ai-biomedical-images-text.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.
