Executive summary
A prompt can be the test input, but it is rarely the complete test case. Variable outputs require explicit context, configuration, evaluation criteria, and an acceptance threshold.
Method
Synthesis of established model-evaluation practices and practical quality-engineering controls.
What the evidence suggests
The prompt is only the stimulus
The same wording can behave differently with another model, system instruction, retrieved context, tool state, temperature, or safety configuration.
Exact strings are often unsuitable
More than one response may be acceptable, so tests need dimension-level rubrics rather than one expected sentence.
Risk determines rigor
A creative response and a regulated factual answer should not share the same tolerance for unsupported claims.
Practical test protocol
- 01Record prompt, context, model, version, and settings.
- 02Define correctness, relevance, completeness, grounding, safety, and style criteria.
- 03Add pass thresholds and examples of pass, fail, and borderline behavior.
- 04Repeat representative cases and compare distribution, not one lucky run.
Prompt + Context + Model Configuration + Evaluation Criteria + Pass Threshold = an auditable AI test case.
Limitations
Rubrics still require calibration and may not transfer unchanged across domains, languages, or model versions.