Towards Meta-evaluation for LLM-as-a-judge in Web Development Quality

The Hong Kong University of Science and Technology
Department of Computer Science and Engineering


MPhil Thesis Defence


Title: "Towards Meta-evaluation for LLM-as-a-judge in Web
Development Quality"

By

Mr. Chunyang LI


Abstract:

Large language models (LLMs) are increasingly used as automated evaluators 
(LLM-as-a-judge) to reduce the cost of human annotation in model development 
and assessment. However, their reliability remains unclear in complex, 
open-ended domains where quality is subjective and cannot be fully observed 
from static text. This thesis studies LLM-as-a-judge in web development 
quality evaluation, a setting that requires reasoning over user intent, 
source code, visual rendering, and interactive behavior. To support this 
study, we construct WebDevJudge, a meta-evaluation benchmark for assessing 
whether automated judges can approximate expert human preference judgments 
between paired web implementations. Starting from a crowdsourced web 
development preference dataset, we filter unsafe, unclear, non-interactive, 
and deployment-invalid instances, then annotate a curated subset with expert 
judgments guided by hierarchical query-grounded rubric trees. The final 
benchmark contains 654 preference instances with balanced labels and supports 
multiple observation forms, including code, screenshots, and live interactive 
evaluation. Experiments with different evaluators show that current automated 
judges still fall short of expert human judgment. Pairwise comparison is 
consistently more reliable than single-answer grading because it provides 
relative evidence between two implementations, and rubric helps in single 
answer grading due to more concrete decision boundary. Comparison between 
different observation forms reveals that source code is the most important 
observation form, while screenshots provide complementary visual evidence in 
pairwise settings. An agentic workflow can collect interaction-based 
evidence, but does not outperform strong vanilla LLM judges because errors in 
planning, execution, and summarization accumulate. Further analysis 
identifies several failure patterns. Positional bias remains present but is 
not the dominant source of error. More importantly, current evaluators often 
struggle to recognize functional equivalence and to verify whether web 
functions are actually feasible. These findings suggest that reliable web 
development evaluation requires stronger integration of code understanding, 
visual perception, contextual requirement interpretation, and robust 
interactive verification.


Date:                   Friday, 7 August 2026

Time:                   10:00am - 12:00noon

Venue:                  Room 3494
                        Lifts 25/26

Chairman:               Dr. May FUNG

Committee Members:      Dr. Yangqiu SONG (Supervisor)
                        Dr. Binhang YUAN