Skip to main content
Most analytical AI workloads can be run as batch inference jobs. Like historical data processing analogs, batch processing has a number of appealing properties - faster job completion times (via higher throughput on large sets of inputs), lower costs, and lower need for constant resource allocation.
SutroUse Sutro for Batch InferenceSutro provides a simple, usage-based batch inference service. 1-hour job SLA’s, >80% cost reductions, a web app for job observability, simple Python SDKTry Sutro batch today
So why don’t all teams rely on batch inference for non user-facing workloads? We generally find that it comes down to a few reasons:
  1. Real-time inference APIs have been the norm since model companies began offering foundation models as a service. Many teams simply architected their systems around these APIs, and switching costs haven’t seemed worthwhile.
  2. Many systems are high-volume - meaning many requests arrive over time, but not necessarily batch-oriented (all at once). It’s possible to process these requests somewhat asynchronously or queue them to run as a batch, but they’re realistically best handled as event-driven.
  3. Many teams simply aren’t at a scale where batch inference, even if saving >50% on inference costs, would help them materially.

​
When batch is better

A simple rule of thumb: use batch inference when the work looks like a data pipeline; use real-time when the model sits inside an interactive user-facing application.

​
Batch limitations

There are a handful of system tradeoffs that should be made when considering the use of batch inference.
  1. Often, batch inference is best for single-turn LLM calls, where all of the context is known upfront. If there is some unknown number of tool calls, sandboxed code executions, or other steps that interrupt inference, some of the benefits of high-throughput inference can be defeated or made more challenging to architect around.
  2. Many providers offering batch inference services have 24-hour, or worse, 72-hour SLAs and flaky success guarantees. Not only can this be problematic to design around, it makes prototyping harder, and nearly defeats the possibility of using batch inference services for experimental work.
  3. Most batch inference services are also often second-class products: little observability, rigid handling of data, and inconsistent response times.

​
Flex/Async Processing

Although not extremely popular, a middle-ground that some inference providers have started to offer is flexible or asynchronous processing modes. These are one-off calls (rather than batch) that are designed to have longer processing times and run in the background, or with some longer SLA in exchange for lower costs. If your application is event-driven but can tolerate higher latency, this may be a good solution as an alternative to batch.