Choosing Smaller Models for Routine Tasks in an Evidence Application

Reduce unnecessary work before changing the model

First check whether the result is already saved and current. Then limit context to eligible, relevant evidence and request only the output the interface needs. These changes can reduce latency and cost without introducing another model route.

A routing experiment might compare candidates on issue titles, bounded category review and cited answers separately. A model that performs well at concise labels may be unacceptable at distinguishing an acknowledgement from a substantive reply. The task contract, not a global “cheap mode,” should determine the evaluation.

Make qualification explicit

This toy selector demonstrates a quality gate before a cost comparison. Its scores are invented and are not MailTrace benchmarks. Real gates need multiple task metrics, confidence intervals where appropriate and unacceptable-error checks.

def choose_candidate(results, minimum_quality):
    qualified = [r for r in results if r["quality"] >= minimum_quality]
    return min(qualified, key=lambda r: r["cost"]) if qualified else None

sample = [
    {"name": "candidate-a", "quality": 0.60, "cost": 1},
    {"name": "candidate-b", "quality": 0.95, "cost": 3},
]
assert choose_candidate(sample, 0.90)["name"] == "candidate-b"

Account for retries and review effort

A cheaper first response can become more expensive if it repeatedly fails validation or requires extensive human correction. Measure cost per completed useful task, including retries and review, and compare end-to-end latency rather than tokens alone.

Any adopted route should be versioned in the saved artifact alongside prompt and schema versions. Escalation to another model needs a defined budget and reason, not an unlimited loop. When no candidate meets the required quality, retaining a manual step or an explicit unavailable state is more honest than quietly lowering the acceptance threshold.