Master Thesis · 2025 Nov -2026 May
Role
Collaboration
Centre of Expertise in Learning and Teaching, University of Twente
Tool
Figma, Lovable, Claude Next.js, Gemini API, AI-assisted development
The Interaction Gap
→
apply without previewing
→
identify misalignments during grading
Why This Gap Matters?
Prompt design shapes AI grading behavior
Research on LLM-based grading suggests that prompt structure and content can substantially affect scoring accuracy, consistency, and generated feedback quality. However, effective configurations vary across assignments, rubrics, and grading contexts.
Graders need support to evaluate and refine AI behavior
In formative research, educators wanted more guidance when configuring grading rubrics for AI. They were unsure what could be modified and whether the resulting scores were trustworthy. This indicated a need to preview AI outputs and evaluate the effects of prompt changes before grading.
Design Challenge
“How might we help graders refine their prompts to better align AI behavior with their grading expectations?”
Design Approach
Researching the user, the interaction, and the AI together
Prompt calibration is shaped not only by graders’ needs, but also by how prompts influence model behavior and what the system can reliably support. I therefore investigated three perspectives in parallel, user needs, the design space, and technical feasibility, and synthesized the findings into design requirements.
01 User Needs
Graders needed an accessible way to refine prompts and verify consistent AI grading.
Method
A focus group with three experienced graders explored their grading practices and expectations for AI assistance.
Findings
Graders prioritized consistent evaluation and high-quality feedback over perfect accuracy. They acknowledged that human grading is not flawless and therefore did not expect flawless AI performance. Instead, they wanted the system to apply criteria consistently and clearly distinguish between levels of student performance. When exploring the early concept, they also preferred structured controls over editing raw prompt text.
Design Implication
Structured refinement
Let graders adjust predefined grading and feedback components instead of editing raw prompts.
Consistency testing
Let graders test the same prompt repeatedly across multiple essays.
02 Design Space
Existing grading tools supported structured setup, while prompt tools supported experimentation—but few combined both.
Method
Competitive analysis and literature review
Findings
AI-powered grading tools used predefined prompt structures, making setup easier but offering few ways to test or compare prompts.
General-purpose prompt tools supported experimentation through output comparison, version history, and automated suggestions, but were not designed for grading workflows.
Design Implication
Structured experimentation
Let graders modify predefined rubric and feedback components without editing an unstructured prompt.
Output-based evaluation
Show prompt alongside the resulting scores and feedback so graders can evaluate their effects.
Guided iteration
Support AI-assisted refinement, output comparison, and version history across multiple prompt iterations.
03 Technical Feasibililty
AI could propose prompt revisions, but graders still needed to evaluate and approve them.
Method
A literature review informed the prompt architecture and identified feasible mechanisms for AI-assisted prompt revision.
Technical decision
AI proposes revisions; the grader evaluates, iterates, and decides.
Prompt architecture
Fixed system structure
Evaluation task
Assignment and essay context
Required output format
Score and explanation structure
Grader-configurable guidance
Rubric and performance levels
Reference examples
Grader perspective
Feedback tone, detail, and focus
Design Brief
Support free experimentation
Allow graders to quickly edit prompts, test them on essays, and observe how AI grading changes.
Make grading behavior inspectable
Help graders assess consistency and compare AI performance across essays of different quality.
Guide prompt improvement
Let graders express what they want to change, generate a revised instruction, and compare its impact before applying it.
From concept to working prototype
Step 1. Ideate the experience
Explored interface concepts and translated the design requirements into an initial flow in Figma.
Tool:
Figma
Step 2. Prototype interactions
Built and iterated on interactive workflows using Lovable to test navigation, layout, and feature behavior.
Tool:
Figma
Step 3. Build the functional prototype
Moved the prototype into code and refined the implementation with Claude-assisted development.
Tool:
Figma
Usability testing with graders
I tested the functional prototype with three participants who had grading experience, observing task completion, navigation, hesitation, and think-aloud feedback. Follow-up interviews helped identify where the workflow conflicted with their mental models or made core features difficult to discover and use.
Issue 1
Prompt-related terminology caused hesitation
All three participants hesitated around prompt-related terminology. “Prompt” reflected the system model, but not the language they naturally used for grading.
Design response
I replaced prompt-related terms with language that better reflected graders’ existing practices.
Technical terms
Before
→
Grader-oriented language
After
Issue 2
Key grading controls were easy to overlook
All three participants overlooked controls for adding essays and running consistency tests. The interface also did not clearly distinguish between different grading modes.
Design response
I reorganized scattered grading controls into three clearly defined modes, separating single-essay grading, multiple-essay comparison, and consistency testing.
Scattered grading controls
Before
→
Clearly defined grading modes
After
Prompt Playground
The final design combines two connected workflows: explore AI grading behavior, then refine the grading instructions with AI assistance.
Design Feature
Inspect individual results, compare outputs across essays, and test consistency across repeated evaluations.
One Essay
Inspect one essay’s scores and feedback in detail.
Step 1.
Describe what should improve
Graders specify how they want the rubric or feedback instruction to change.
Step 2.
Compare revisions and grading results
The system generates a possible revision and shows the original and revised instructions alongside their grading outputs.
Step 3.
Iterate or apply
Graders can continue refining the instruction or import the preferred revision back into the grading workflow.
Try the Prompt Playground
I conducted a between-subjects study with 22 participants to examine how access to prompt calibration influenced AI-assisted grading.
Procedure & Measurement
The experiment included four phase: Training, Preparation, Grading, Post-survey.
The playground condition used
Playground
N = 11
Control
N = 11
Training Phase
The experiment started with a training phase to prepare the participants to grade essays, and learn the usage of AI grading Assistant
Preparation Phase
Group playground use the tool to revise prompts for 15 mins, while the other group did not have the access
Optimized prompts
Predefined prompt
Grading Phase
Both group grade three essays in AI grading Assistant. While
Post-survey Phase
Conduct a 7-scale likert survey of AI grading Assistant
What I measured
I collected two types of data:
Behavioral data: prompt revisions in Prompt Playground during preparation phase, edits to AI-generated feedback, grading time, and changes from the initial AI-generated scores to participants’ final scores during grading phase.
Perception data: post-task ratings of output quality, understanding of the system, perceived control, and reliance on the AI Grading Assistant.
Results
AI-generated feedback was rated higher when prompt calibration was available
Prompt Playground supported iterative refinement of grading instructions. Its clearest measured effect was higher perceived quality of AI-generated written feedback, while broader effects on grading performance, efficiency, understanding, and control remained inconclusive.
01 How participants used the Playground
Calibration helped participants made their grading expectations more explicit.
The 11 Playground participants iteratively revised their grading instructions before grading. Their revisions made expectations around feedback tone, structure, and grading criteria more explicit.
02 Playground vs. control condition
Participants with access to calibration rated AI-generated feedback significantly higher
Playground participants rated the quality of the AI-generated written feedback significantly higher than the control group on a seven-point scale (M = 5.00 vs. M = 2.91; U = 18.0, p = .004). This suggests that refining the grading instructions may have helped bring the generated feedback closer to participants’ expectations.
Feedback edits shifted from rewriting toward refinement
Both groups edited the AI-generated feedback at similar rates. Qualitatively, however, Playground participants tended to make smaller refinements while preserving its structure, whereas control participants more often rewrote the feedback.
Broader effects remained inconclusive
No statistically significant differences were found in final grading accuracy, grading time, understanding, perceived control, or reliance. The study therefore provides evidence for improved perceived feedback quality, but not for broader improvements in grading performance or efficiency.
Full study and analysis
The complete methodology, statistical analysis, and limitations are available in my thesis.
Retrospective
One
Balance between transparent and easy to use
While I was designing the playground. one possible reason that playground didn't increase the perceived understanding may because of the prompt hiding behind the structure
