Group of two project · 2025 Apr - June
Role
Collaboration
Tool
Figma, Unity, C#, Llama, Vosk, Kokoro, AI-assisted development
Outcome
Experts found the agent credible for guiding the assessment, while highlighting turn-taking and response reliability as priorities for further development.
Background
Conducted with Tactus, an addiction healthcare organization in the Netherlands, this research project explored how an embodied conversational agent could complement existing care by making alcohol-withdrawal assessments more accessible and flexible for people with MBID.
Topic
The CIWA-Ar monitors ten alcohol-withdrawal symptoms and may require reassessment every one to four hours, creating a substantial clinical workload while posing accessibility challenges for people with MBID undergoing alcohol withdrawal.
How might an embodied chatbot support accessible, repeated assessments?
Approach
Research-informed design direction
With no direct access to patients during the early phase, we reviewed research on MBID accessibility, medical conversational agents, and alcohol-withdrawal assessment to establish the initial design direction.
Understanding the users
People with MBID benefit from simple, and structured interactions
People with mild to borderline intellectual disabilities (MBID) may experience limited attention, restricted vocabulary, memory difficulties, and difficulty distinguishing relevant from irrelevant information. Conventional assessments can therefore be challenging to follow. This calls for interactions that use short questions, concrete language, repetition, and immediate feedback.
Researching Medical ECAs
Medical ECAs can support users through multiple modalities, a clear role, and appropriate emotional support
We reviewed prior research on medical embodied conversational agents (ECAs) to understand how their communication and presentation may support comprehension, engagement, and trust. Three themes informed our initial design direction.
01 Multimodal guidance
Research insight
Speech, subtitles, and visual cues can reinforce comprehension.
Design implication
Present each question through both speech and on-screen text.
02 A clear clinical role
Research insight
Prior healthcare ECA studies often use a professional, middle-aged female character to communicate familiarity and care.
Design implication
Explore a professional healthcare character.
03 Motivational and empathetic communication
Research insight
Supportive and motivational language can help users feel heard, but excessive empathy may distract from the assessment or feel inappropriate.
Design response
Use brief, supportive responses while keeping the assessment focused.
Design Goal
Design a structured, multimodal conversational assessment
Conversation Guidelines
Use simple, concrete language
Ask one question at a time
Repeat or rephrase when needed
Provide immediate, clear feedback
Maintain a predictable conversational structure
Interface Guidelines
Present speech together with on-screen text
Provide clear, limited response options
Use a professional and approachable agent
Keep the interface visually simple
Rapid Prototyping
Translating research insights into a functional prototype
To test the concept early, we initially developed the conversational AI and interface as separate components. My teammate prototyped the Llama-based AI behavior, while I designed and implemented the screen-based interface in Unity.
01 AI behavior prototype
02 Unity interface
03 Integrating the multimodal experience
I connected the interface with speech-to-text, the language model, and text-to-speech, turning the separate prototypes into a functional assessment flow for expert review.
▴Multimodal Prototype Architecture
Focus Group
Reviewing the prototype with clinical experts
We conducted an online focus group with five clinical detoxification experts, including nurses, e-health specialists, and a researcher. We used the initial prototype as a discussion probe to assess its fit with clinical practice and identify improvements to the agent’s communication and interaction design.
Feedback to AI behavior
AI responses felt overly empathetic for the clinical context
Experts felt that the agent’s tone was sometimes overly empathetic for the direct communication expected in Dutch clinical care.
Feedback on agent appearance
The agent’s visual style did not align with the clinical context
Experts valued the multimodal interaction but felt that the agent’s anime-inspired visual style did not align with the clinical context. They preferred a more realistic and professional-looking character.
Design response
We introduced clearer behavioral guidelines and limited empathy to brief, task-focused acknowledgements.
Before
→
After
Design response
I replaced the stylized avatar with a more neutral, professional healthcare character.
Before
→
After
Design Features
A predictable three-stage assessment flow
The assessment guides users through three clear stages, briefing, assessment, and completion. Each screen presents one task at a time to reduce cognitive load and make progress easier to follow
Briefing
Introduce the assessment
Ask for the user’s name
Check whether they have questions
Assessment
Present one question at a time
Clarify the question when needed
Confirm the user’s response
Completion
Confirm that the assessment is complete
Explain what happens next
Structured assessment with flexible conversational support
To preserve the required CIWA-Ar questions and sequence, the interface controls the core assessment flow. The conversational AI supports this flow by clarifying questions, answering additional queries, and asking follow-up questions when needed, without changing or skipping required items.
Fixed assessment structure
Briefing → Question 1 → Question 2 → Question 3 → … → Completion
Flexible conversational support
Clarify questions · Answer additional queries · Ask follow-ups
Clear about its AI identity and clinical role
To avoid misleading users, the agent clearly introduces itself as an AI system and explains how their responses will be used to support clinical care.
SPEECH OUTPUT
▴Example of the agent’s spoken introduction
Qualitative Evaluation
Clinical expert evaluation of the prototype
Three clinical detoxification experts completed a full walkthrough of the prototype while I observed their interactions. Follow-up interviews were thematically analyzed to identify key usability and design insights.
3 clinical experts
≈15 min/session
Observation + interview
Results
The prototype felt credible and supported task completion, but unclear turn-taking and occasional context errors limited its reliability.
Credible guidance
The agent felt credible and appropriate for guiding the assessment.
Participants found its questions, clarifications, and guidance suitable for the assessment. Its human-like appearance, calm voice, and professional tone contributed to credibility, while its responses were generally clear and easy to follow.
Turn-taking uncertainty
Multimodal input created uncertainty about when and how to respond.
All participants completed the assessment, but initially hesitated over whether to speak or use the on-screen controls—and whether the agent was ready to listen.
Context errors
Occasional context errors weakened trust in the agent.
Most responses were clear and relevant, but the LLM occasionally misinterpreted user input and generated responses that did not fit the assessment context.
Future improvements
Based on the evaluation, we identified three priorities for future development.
Clarify turn-taking and input options
Introduce clearer onboarding and explicit interaction states, including a dedicated button for asking questions and clearer indicators of when voice or on-screen input is available.
Improve AI response reliability
Evaluate alternative language models and strengthen prompt constraints, context management, and fallback responses to reduce misinterpretations and off-topic answers.
Evaluate with the target users
Test the prototype with people with MBID in a supervised setting to assess comprehension, usability, safety, and its ability to support assessment completion.
Retrospective
The project has taught me few important things.
It is important to be clear that AI-generated, it can be presented as the
