A practical guide
Azloop · Engineering notes
Build an evaluation set
Collect realistic questions, ambiguous requests and cases where no answer is available. Remove personal information that is not necessary for testing.
Score more than fluency
Evaluate correctness, source support, appropriate uncertainty and whether an answer helps the user complete the task.
Track speed and cost
Measure end-to-end latency and resource consumption alongside answer quality. A faster response is useful only if the result remains dependable.
Review changes against a baseline
Compare model, prompt and retrieval changes on the same examples. Keep a human review step for regressions that automated checks miss.