Syllabus

• Week 1

  Introduction of Large Language Models (LLMs)

  • Welcome

    Begin your journey into multimodal AI for healthcare with an overview of the course structure, learning goals, and emerging opportunities in the field.

  • Week 1 Introduction

    In Week 1, explore LLM foundations, self-attention mechanics, in-context learning, prompt engineering, instruction tuning, RAG, open-weight models like Llama and Phi, and medical AI evaluation using MedQA and MedExQA.

  • What is a multimodal large language model and its brief history?

    This activity covers LLM and MLLM basics and history, showing how scaling and Transformers enable models like Med-Gemini to reason across medical text, 2D/3D images, and genes.

  • What is (Self-)Attention?

    This deck introduces self-attention in Transformers, framing it like a database retrieval query where Queries, Keys, and Values compute token relationships to enable rich context and semantic disambiguation.

  • How are LLMs used and adapted?

    This four-step activity explores the core lifecycle of Large Language Models—from base pre-training and in-context prompt engineering to model adaptation and post-training alignment techniques like SFT, RAG, and RLHF.

  • Proprietary vs Open-weight LLMs

    In this activity, you will compare proprietary LLMs with open-weight models discussing the importance of data privacy in healthcare. We introduce Llama, Phi and Qwen as exempalr open-weight models.

  • Benchmarking LLMs

    In this, you will analyze benchmarks for medical LLM evaluation, covering knowledge extraction (MedQA), reasoning explanations (MedExQA), holistic multi-task evaluation (MedHELM), and multi-metric platforms like OpenCompass.

  • Test Your Knowledge

    Consolidate your understanding of LLM fundamentals and prompting techniques through a short knowledge check.

  • Week 1 Conclusion

    Reflect on the key concepts introduced this week and their significance for modern healthcare applications.

• Week 2

  Connecting Knowledge: LLMs and Knowledge Graphs

  • Week 2 Introduction

    Discover how multimodal large language models are applied across radiology and pathology, then follow the pipeline — from data and model design through training to evaluation — behind a working radiology AI system.

  • What is Multimodal AI?

    Explore how multimodal AI combines language and vision, how models such as LLaVA are trained, and why careful evaluation is essential when these systems are applied in medicine.

  • Radiology AI End-to-End

    This activity details the end-to-end pipeline for radiology AI, covering data curation, model evolution (CNNs to agents), efficient training, key clinical applications, multi-metric evaluation, and trust.

  • Pathology AI End-to-End

    Explore the end-to-end pathology AI workflow, from image processing and multimodal model training to report generation, evaluation and domain adaptation.

  • Test Your Knowledge

    Please try to use what you learned to do the scenario based exercise: think, solve and share!

  • Week 2 Conclusion

    Trace the journey from raw pixels to clinical insight, across radiology and pathology. Weigh what each step, from data to deployment, costs and earns toward a system worth trusting in the clinic.

• Week 3

  Image recognition by LLMs, recent advances and potential AI pitfalls

  • Week 3 Introduction

    Discover how LLMs, knowledge graphs and multimodal data are transforming healthcare, while considering both recent advances and emerging challenges.

  • LLMs, KG, and EHRs

    Explore how large language models and knowledge graphs can be applied to electronic health records, enabling information extraction, clinical reasoning and more effective use of healthcare data.

  • Integrating Multimodal Healthcare Data with LLMs and KGs

    Examine how clinical, genomic and molecular data can be combined with LLMs and knowledge graphs to support precision medicine, personalised care and multimodal healthcare AI.

  • History of AI Use in Healthcare and Recent Developments

    Trace the evolution of AI in healthcare from early systems to modern applications. You will examine major milestones, analyse recent developments, and evaluate case studies across six parts.

  • Pitfalls in AI: Ethical Concerns, Bias, and Hallucinations

    In this section, you will identify ethical concerns, examine bias in AI systems, and evaluate the risks of AI hallucinations in healthcare. The section is divided into two parts to support your learning.

  • Test Your Knowledge

    Test your understanding of this week's topics by applying key concepts from the course. Reflect on how AI, LLMs, knowledge graphs and multimodal data can support healthcare while addressing important limitations and risks.

  • Week 3 Conclusion

    Reflect on how AI is shaping healthcare. Consider the opportunities, limitations and future potential of LLMs, knowledge graphs and multimodal AI in clinical practice and research.

• Week 4

  From Lab to Clinic: Adapting and Deploying Multimodal LLMs in Real-World Healthcare

  • Week 4 - From passive to proactive: LLM enabled agentic framework

    Explore the evolution of LLMs from static models to proactive agents. Learn ReAct loops, deep research frameworks like BioMedArena, and autonomous runtimes like OpenClaw for complex biomedical and clinical workflows.

  • Test Your Knowledge - final assessment

    Time to test and apply what you have learned

  • Course Conclusion & Summary

    Synthesizes the 4-week MOOC on healthcare MLLMs. Reviews Transformers, RAG, medical LVLMs, EHR knowledge graphs, multi-omics, and proactive agentic frameworks—tracing the path from base LLMs to autonomous clinical runtimes.

Who will you learn with?