Skip to main content

​
Task Specificity

SutroBuild consistent models with SutroSutro provides annotation and automated prompt optimization infrastructure to create task-specific prompts that induce consistent behavior in production.Learn how Sutro works
The cheapest way to induce consistency is to be extremely specific in your prompting logic. This means defining the operational procedure the model should follow to complete the task, few-shot examples of what good and bad looks like, enumerating corner and edge cases, and generally abstracting good rules-of-thumb for problem solving on that task. Think about it this way: if you were to drop a PhD-level intelligence human into your company and ask them to carry out a repeated task that you already knew how to solve, how would you teach it to them? I suspect you’d write a specific guide - perhaps a 10-page PDF explaining how to perform that task, where it can get tricky, what issues you’ve come across in the past, and other useful context. Would you sit with them for hours, days, or weeks, teaching them what good and bad looks like? Probably not. Models these days are generally very strong instruction-followers, so just getting specific upfront is typically sufficient. Automated prompt optimization can also help create super specific prompts from annotated datasets. We’ll discuss that in another section of this guide.