> ## Documentation Index
> Fetch the complete documentation index at: https://handbook.sutro.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# What's in a Judge?

> The model, context, input, and output schema that make up a typical LLM judge.

We recommend building judges comprised of the following components and properties:

| Component     | Description                                                                                                                                                                                                                           | Example                                                                                                                                                                         | Guidance                                                                                                                                                                                                                                                                                                                                                                                                            |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Model         | A strong, instruction-tuned LLM.                                                                                                                                                                                                      | GPT-5.4-mini, Gemma-4-31B.                                                                                                                                                      | Do not overthink the [choice of model](/deployment/model-selection). Most modern LLMs are strong instruction-followers, so any foundation model of sufficient size (we recommend at least 30B total parameters as of this writing) should be able to handle a well-defined judge task. Choose something within the latency and cost budget your application requires.                                               |
| Context       | Typically a strong system prompt, with no fine-tuning.                                                                                                                                                                                | "You are evaluating the outputs of another AI model. Your job is to determine if it helped the customer return their order successfully. Evaluate based on three components..." | We recommend against manual prompt engineering to build judges. Use [human annotations](/patterns/context/expert-annotation) and an automated prompt optimization tool to automatically build a strong system prompt for the judge model you have selected.                                                                                                                                                         |
| Input         | If used for evals, typically a single user conversation with the model, including inputs and outputs, or an agent trace. If used for other purposes, typically one record of the unstructured or semi-structured data being analyzed. | User: "Can you help me return order ABC12345?" Model: "I would be happy to help. Can you provide confirmation of delivery and the address it was delivered to?"                 | Make sure to provide all necessary information to a judge, and do not hide evidence that would be useful in making a decision. You can optionally supplement a judge with web search or other external grounding tools, but these can be hard to audit and highly variable in pulling in necessary information.                                                                                                     |
| Output Schema | A decision label, ideally binary or ternary, and a rationale.                                                                                                                                                                         | `{"rationale": "The model asked for all three required components to assist the user with their return.", "label": "pass"}`                                                     | Provide an output schema with rationale first, then label second. Frame the task as a single-label classification problem with as few options as possible. [Binary or ternary label sets](/primitives/classifiers/types-of-classifiers) are ideal. Avoid numerical scores when possible; if needed, use a 1-5 Likert scale. Do not ask the model for a [confidence score](/patterns/consistency/confidence-scores). |