Hard Gates, Not Soft Preferences: What Production Taught Me About LLM Classifiers

An AI sales agent lives or dies on classification. Before it writes a single word, it has to answer questions like: Is this email from a real prospect? What are they asking? Which response template fits? Get the classification wrong and everything downstream is wrong — a polished, confident, completely inappropriate reply.

I learned this the expensive way, running an autonomous email agent in production. We had misclassifications across the board: wrong template selection, false positives on "this was manually replied to already," intent labels that didn't match the email. The model wasn't dumb. My instructions were.

The problem: LLMs treat preferences as suggestions

The original classification prompt was full of rules that sounded reasonable:

"Prefer template A when the prospect asks about pricing."
"Generally treat replies within an existing thread as follow-ups."

Words like prefer and generally feel precise when you write them. To an LLM, they're an invitation. A soft preference is a default the model is allowed to override whenever another option seems more contextually appropriate — and LLMs find something "more contextually appropriate" constantly. That's what they're built to do.

So the agent would look at an email, notice some nuance, and creatively route around my preferences. Each individual decision was defensible. The aggregate was chaos.

The fix: sequential decision trees with hard gates

The restructure that fixed it had three parts:

1. Strict sequential gates

Instead of a flat list of preferences, classification became a decision tree the model must walk in order. Each gate is binary and blocking: "Does the sender exist in the outreach list? If NO, stop — classify as NOT_OUTREACH. Do not continue." The model never gets to weigh options it should never have seen.

2. Explicit negative examples

For every category, the prompt now defines what it is not. "What is NOT a pricing question: a prospect mentioning the price while asking about contract length…" Negative examples close the exact loopholes the model was using to be helpful in the wrong direction.

3. Most common rejection first

Our most frequent case was "this email isn't from a prospect at all" — intro emails from colleagues, newsletters, notifications. That check moved to gate #1. The highest-volume mistake gets eliminated before any subtle reasoning starts.

One more trap: pass-only-YES gates

A related lesson from the same system. We added an LLM QA check that had to return a verdict before a response was sent, and the router only passed traffic when the verdict was exactly YES. Any other output — "YES, but…", formatting drift, an unexpected token — silently fell through to the escalation path. Within days, nearly all traffic was quietly rerouting, and it looked like the agent had simply stopped working.

If you gate on an LLM verdict, gate on the failure case, not the success case. Route when the verdict is explicitly negative; let everything else pass with logging. And put a counter on every branch — silence is not success.

The principle

Treat an LLM classifier like a junior employee with infinite confidence: give it a checklist, not a philosophy. Hard gates, walked in order, with the exits labeled. Save the model's judgment for the steps that actually need judgment — writing the response — and keep it out of the routing.