Where “AI” meets classification
As we mentioned, classifiers far predate modern notions of AI. But what shifted is how they’re built, and where time is best spent optimizing their performance. The rise of foundation models largely changed the way new classification systems get built. In the last generation of machine learning, the goal was to curate large corpuses of training data to create a model from scratch. The architectures were well known, but most of the work revolved around curating the training data itself. But foundation models are already pre-trained, meaning someone else did the work of curating data for you. Some will be relevant to the task at hand, some not. The higher costs for inference reflect what effectively amounts to the utilization of the sum of that useful and not-so-useful pile of data all at once. But regardless, the task shifts from “pre-training” the model, to “post-training” the model - effectively a steering exercise. This means, depending on task complexity, that you only need a small amount of data to steer the model towards the behavior you want. And the good news: this can almost always be done in-context, meaning you don’t really need to update the weights of the foundation model itself, but rather just tell it how you want it to behave via prompting or other in-context learning methods we’ll discuss in this guide.Features vs. Reasoning
In the older days of ML, and even more recent encoder-only language models, classifications are made based on semantic features rather than instruction-following and reasoning. This distinction sounds simple, but is critical from a generalization and task-scoping standpoint. Let’s say we are trying to classify whether a piece of text is related to spaceflight. We might have a piece of text like:I saw a rocket launch today!Given a number of pieces of text like this, a classical ML model will start to recognize “rocket” as a distinct feature. But given another input:
I love Rocketman! Such a good song.This might produce a “true” result based on the presence of the feature it learned. However, a reasoning-based approach might yield a different result. The model will scan the input from left to right, taking into account everything that came before it. It may then produce another set of reasoning tokens to further expand on the input before making a final prediction, like:
The user wants me to classify whether the input text “I love Rocketman! Such a good song.” is related to spaceflight. While there is an allusion to spaceflight within the title of the song, this likely refers to a hit song of Elton John, and does not directly refer to spaceflight. Therefore, since the text is more related to music instead of spaceflight I will classify this as false.It will then be able to scan over the input and expanded reasoning. Its final prediction will be based on all the tokens that came before it, and not just on specific features from the text. But perhaps more importantly, we can also expand out a very specific decision policy within the context of the prompt that might include something like:
You should classify as true even if the text is an indirect reference, such as a popular book or song mentioning spaceflight.In this case, the model will use its reasoning capabilities to follow the instruction, and classify as true. This makes reasoning-based AI models incredibly generalizable, and increases the decision scope they can reasonably carry out. In many cases, classical ML approaches will fail to generalize where LLM-based approaches can work. So while it might seem like ML-based classification is a solved problem of the past, reasoning-based AI decisioning is quite new.
When to (and not to) use Foundation models for classification tasks
Some classification problems are inherently simple, and more or less objective. As we mentioned earlier in the guide, much of the goal of analytical AI is to compress a task into the smallest model possible that performs well on its evals and production performance tradeoffs. The core tradeoff is whether the task benefits enough from foundation model capability to justify paying for inference instead of spending time curating data for a smaller, narrower system:
Generally speaking, more and more teams are reaching for foundation models instead of traditional ML for classification. As foundation models continue to improve and inference costs decrease, the justification for time spent on curating data rather than just paying higher inference costs is not seen as worthwhile (ML engineers and expert annotators are expensive!)