Multimodal AI to simulate medical student competency

Overview

The assessment of medical graduate competency is a cornerstone of medical education and a critical safeguard for patient safety. Newly qualified physicians must demonstrate a broad range of skills and knowledge, including diagnostic reasoning, clinical decision-making, communication, procedural skills, and professionalism before independently practicing medicine. Traditional assessment methods often include standardized multiple-choice examinations, objective structured clinical examinations (OSCEs), direct observation of procedural skills (DOPS), and portfolio reviews. While these methods offer valuable insights, they have inherent limitations. Standardized tests may not fully capture practical skills, and OSCEs can be resource-intensive. They may lack the complexity of real-world scenarios, and direct observation can be subjective and influenced by the observer.

The increasing availability of diverse clinical data and advancements in artificial intelligence presents an opportunity to develop more sophisticated and comprehensive methods for evaluating medical competency. AI, particularly multimodal AI, which can process and integrate information from various sources, holds the potential to simulate complex clinical situations and provide nuanced assessments that more closely reflect real-world performance. By leveraging data such as patient notes, recorded clinical encounters, medical images, and potentially physiological data, a multimodal AI system can offer a more holistic view of a graduate's abilities across different competency domains. This project aims to explore the feasibility and potential benefits of such an approach, contributing to the ongoing efforts to enhance medical education and ensure the highest standards of patient care.

Research activities

The technical implementation of this project will involve several key stages:

• data acquisition and preprocessing
• multimodal data integration
• AI model development
• system evaluation.

Data acquisition

• Clinical case notes:
  • access to de-identified or synthetic clinical case notes, including patient history, physical examination findings, laboratory results, and treatment plans. These could be in structured or unstructured text format.
• Standardised patient interactions:
  • using recordings (video and audio) of medical graduates iteracting with standardized patients in simulated clinical scenarios. These recordings will capture verbal communication, non-verbal cues, and procedural skills.
• Medical imaging:
  • incorporating relevant medical images (e.g. X-rays, CT scans, MRIs) associated with the simulated cases. These images will require appropriate anonymization and may be paired with textual reports.
• Physiological signals (optional):
  • exploring the potential inclusion of physiological data such as heart rate, blood pressure, and respiratory rate collected during simulated encounters, if feasible and relevant to the assessed competency.

Data preprocessing

• Natural language processing (NLP):
  • applying NLP techniques to clinical case notes and transcripts of patient interactions for tasks such as entity recognition (identifying medical terms, symptoms, diagnoses), sentiment analysis, and information extraction.
• Computer vision:
  • processing video recordings of standardized patient interactions to analyse non-verbal communication and procedural skills and potentially identify relevant objects or actions.
• Image processing:
  • preprocessing medical images to ensure format, size, and quality consistency. This may involve normalization, augmentation, and feature extraction.
• Audio processing:
  • analysing audio recordings for speech patterns and tone and potentially identifying key communication elements.
• Data alignment and synchronisation:
  • developing methods to align and synchronize data from different modalities based on the simulated clinical scenario and timeline.

AI platform and frameworks

• Cloud-based AI platforms (e.g. Google Cloud AI Platform, AWS SageMaker, Azure Machine Learning) are used for scalability and access to advanced machine learning resources.
• Employing deep learning frameworks such as TensorFlow or PyTorch for model development and training.
• Leveraging relevant libraries for NLP (e.g. spaCy, NLTK, transformers), computer vision (e.g. OpenCV, torchvision), and data manipulation (e.g. pandas, NumPy).

System architecture

The system architecture may involve a modular design with components for data ingestion, preprocessing, feature extraction, modality-specific model training, multimodal fusion, competency assessment, and output generation.

A potential architecture could include:

• data ingestion layer:
  • responsible for collecting and storing data from various sources.
• preprocessing layer:
  • performing modality-specific preprocessing steps.
• feature extraction layer:
  • extracting relevant features from each modality.
• modality-specific model layer:
  • training individual AI models for each data modality to learn representations relevant to specific aspects of competency.
• multimodal fusion layer:
  • integrating the outputs or learned representations from the modality-specific models using techniques such as concatenation, attention mechanisms, or late fusion.
• competency assessment layer:
  • using the fused representation to predict or classify the medical graduate's competency level across different domains.
• output and feedback layer:
  • generating a comprehensive assessment report, potentially including visualisations and specific feedback based on the AI's analysis.

Outcomes

The anticipated outcome is a sophisticated AI framework capable of simulating real-world clinical scenarios and providing nuanced feedback on a medical graduate's readiness for independent practice, potentially leading to more effective training programs and improved patient outcomes.

Skills and experience

• Data science.
• Machine learning.
• Python programming.
• Generative AI.

Scholarships

You may be eligible to apply for a research scholarship.

Explore our research scholarships

Keywords

• medical professional competencies
• multimodal AI
• artificial intelligence

Contact

Contact the supervisor for more information.