Background: the pamphlet problem
When patients leave a clinic, they usually leave with a printed pamphlet. Guidelines ask for these materials to be written at a sixth-grade reading level, yet most of them are not, and a large share of adults read below that level. Even a well-written pamphlet cannot slow down, re-explain a confusing term, answer a follow-up question, or tell whether the reader understood what matters most. A nurse sitting next to the patient can do all of this, but clinicians rarely have the time.
Ideation: what would a good educator do?
We asked what it would take for an agent to do the part of patient education that clinicians most often have to skip: walking a patient through the material they were already given. A human educator adjusts vocabulary, points at a diagram while talking, pauses when confusion shows, and checks understanding before moving on. Virtual reality lets an agent do some of the same things. It can stand next to a 3D anatomical model, follow where the patient is looking or pointing, and resolve a question like "what is this part?" from that context.
One design rule held from the start: every piece of LLM-generated lesson content can be reviewed and approved by a clinician before any patient sees it, and the agent declines questions that need clinical judgment, logging them for the care team instead.
System
1. Pamphlet transformation. An LLM pipeline reads an existing pamphlet PDF, extracts its text and images with their layout and captions, and rewrites the content into a structured lesson: topics, short spoken utterances each paired with one visual, a bullet-point takeaway page, and three to five review flashcards per topic. The structure follows established learning theory (constructivism, dual coding, multimedia learning, cognitive load theory), for example by sequencing topics from recognition to application and capping how much each topic holds. A web-based visualizer lets a clinician read and approve the whole lesson before use.
2. Pedagogical agent in VR. Dr. Clara, an animated agent with synthesized speech and generated conversational gestures, delivers the approved lesson in an immersive room. Each topic moves through an interactive lecture (speech paired with a highlighted keynote, an illustration, or a 3D anatomical model), a takeaway reading page with replay and navigation, and flashcards with feedback on wrong answers. A final review covers all topics, and at any pause the patient can ask a question in natural language.
To understand what a patient is referring to, the agent fuses several input channels ranked by how explicit they are: speech and controller clicks first, then pointing, then current gaze, then recent attention history. The most attended objects are passed to the LLM together with the question, and the reply is parsed into speech, visuals, and actions in the scene.
Study
We ran a formative mixed methods usability study with 21 adult volunteers. Each completed one session of about 18 minutes on colonoscopy, appendectomy, or kidney cancer, followed by surveys and a semi-structured interview.
- Satisfaction averaged 6.35 out of 7, and perceived quality of the material averaged 6.21 out of 7.
- Knowledge scores rose from 23.5 to 34.9 out of 50 between pre-test and post-test, with the largest gains among participants who started with the least knowledge.
- Participants valued seeing anatomy in 3D space, the agent's patient and calm manner, and the lecture, summary and flashcard rhythm. They also reported headset weight, unfamiliar controls, and a synthesized voice that was slow and sometimes mispronounced medical terms.
As a single-session study without a comparison group, it shows that the experience was usable and well accepted. It does not show that the agent teaches better than other formats, and an earlier comparison with the paper pamphlet suggested it does not yet: people preferred the agent, but did not learn more from it.
What surprised us
The session logs told a quieter story than we expected. Participants almost never asked content questions; nearly everything they said was "continue". Their attention went mostly to the text panel rather than to the images or 3D models. In practice the system worked as a patient-paced multimedia lecture, not as a conversational tutor. One participant put it plainly: patients assume everything they need is already there, so a system should not count on them to ask.
An agent that waits for questions will mostly not get them. It has to take the initiative.
That is where my current work on
timely intervention starts. One direction is timing: when an embodied tutor should step in with a question, and when it should stay quiet and wait. The other is the how: a virtual human has a body, so an intervention does not have to be spoken. It can walk over, point at the step on the board, or act out an analogy with its hands. When these lighter moves actually help is the next question I want to study.
This result is why the next step was a
pedagogical layer that decides when to ask instead of tell, and why the lab is also testing
teach-back. The project informs a planned NIH-funded randomized controlled trial comparing reading the pamphlet alone, agent-assisted reading on a 2D screen, and agent-assisted reading in VR.
My Major Roles & Contributions
- Joined the project midway, when the lab needed someone to own the VR side, and took over the system.
- Co-conceived the current version and designed the study methodology with my advisor.
- Designed and built the full system as its developer: the LLM pamphlet transformation pipeline, the clinician-facing lesson visualizer, and the VR agent with its learning workflow and multimodal perception.
- Ran the study sessions and interviews, led the quantitative and qualitative analysis, and wrote the original manuscript draft.
×