CHONGYANG.IO

Dr. Clara: LLM-Driven Embodied Agent for Patient Education in VR Under review
Turning the pamphlet a patient takes home into a guided lesson with an embodied agent who explains, points, answers questions, and checks understanding. Funded in part by the U.S. National Institutes of Health.
08/2024 - Now, Associated with: Relational Agents Group, Northeastern University; Prof. Timothy W. Bickmore; Michael Paasche-Orlow (Tufts Medical Center); Mina Fallah; Akilan Baskar
Background: the pamphlet problem
When patients leave a clinic, they usually leave with a printed pamphlet. Guidelines ask for these materials to be written at a sixth-grade reading level, yet most of them are not, and a large share of adults read below that level. Even a well-written pamphlet cannot slow down, re-explain a confusing term, answer a follow-up question, or tell whether the reader understood what matters most. A nurse sitting next to the patient can do all of this, but clinicians rarely have the time.
Ideation: what would a good educator do?
We asked what it would take for an agent to do the part of patient education that clinicians most often have to skip: walking a patient through the material they were already given. A human educator adjusts vocabulary, points at a diagram while talking, pauses when confusion shows, and checks understanding before moving on. Virtual reality lets an agent do some of the same things. It can stand next to a 3D anatomical model, follow where the patient is looking or pointing, and resolve a question like "what is this part?" from that context.
One design rule held from the start: every piece of LLM-generated lesson content can be reviewed and approved by a clinician before any patient sees it, and the agent declines questions that need clinical judgment, logging them for the care team instead.
From pamphlet to VR lesson: LLM transformation, clinician review, delivery in VR

From pamphlet to VR lesson: LLM transformation, clinician review, delivery in VR

System
1. Pamphlet transformation. An LLM pipeline reads an existing pamphlet PDF, extracts its text and images with their layout and captions, and rewrites the content into a structured lesson: topics, short spoken utterances each paired with one visual, a bullet-point takeaway page, and three to five review flashcards per topic. The structure follows established learning theory (constructivism, dual coding, multimedia learning, cognitive load theory), for example by sequencing topics from recognition to application and capping how much each topic holds. A web-based visualizer lets a clinician read and approve the whole lesson before use.
2. Pedagogical agent in VR. Dr. Clara, an animated agent with synthesized speech and generated conversational gestures, delivers the approved lesson in an immersive room. Each topic moves through an interactive lecture (speech paired with a highlighted keynote, an illustration, or a 3D anatomical model), a takeaway reading page with replay and navigation, and flashcards with feedback on wrong answers. A final review covers all topics, and at any pause the patient can ask a question in natural language.
Dr. Clara explaining with an illustration and with a 3D anatomical model

Dr. Clara explaining with an illustration and with a 3D anatomical model

To understand what a patient is referring to, the agent fuses several input channels ranked by how explicit they are: speech and controller clicks first, then pointing, then current gaze, then recent attention history. The most attended objects are passed to the LLM together with the question, and the reply is parsed into speech, visuals, and actions in the scene.
The four learning phases: lecture, 3D model, takeaway reading, flashcards

The four learning phases: lecture, 3D model, takeaway reading, flashcards

How each turn is assembled: dialogue state, multimodal input, LLM, parsed output

How each turn is assembled: dialogue state, multimodal input, LLM, parsed output

Study
We ran a formative mixed methods usability study with 21 adult volunteers. Each completed one session of about 18 minutes on colonoscopy, appendectomy, or kidney cancer, followed by surveys and a semi-structured interview.
Satisfaction, perceived quality, and knowledge before and after the session

Satisfaction, perceived quality, and knowledge before and after the session

As a single-session study without a comparison group, it shows that the experience was usable and well accepted. It does not show that the agent teaches better than other formats, and an earlier comparison with the paper pamphlet suggested it does not yet: people preferred the agent, but did not learn more from it.
What surprised us
The session logs told a quieter story than we expected. Participants almost never asked content questions; nearly everything they said was "continue". Their attention went mostly to the text panel rather than to the images or 3D models. In practice the system worked as a patient-paced multimedia lecture, not as a conversational tutor. One participant put it plainly: patients assume everything they need is already there, so a system should not count on them to ask.
An agent that waits for questions will mostly not get them. It has to take the initiative.
That is where my current work on timely intervention starts. One direction is timing: when an embodied tutor should step in with a question, and when it should stay quiet and wait. The other is the how: a virtual human has a body, so an intervention does not have to be spoken. It can walk over, point at the step on the board, or act out an analogy with its hands. When these lighter moves actually help is the next question I want to study.
This result is why the next step was a pedagogical layer that decides when to ask instead of tell, and why the lab is also testing teach-back. The project informs a planned NIH-funded randomized controlled trial comparing reading the pamphlet alone, agent-assisted reading on a 2D screen, and agent-assisted reading in VR.
My Major Roles & Contributions
Highlights
Dr. Clara with a kidney illustration in VR
Illustration and 3D model views
Learning phases
Pipeline