About the implementation and examples
Engineering notes draw on author-supplied MailTrace development reports through 13 October 2026. They describe the documented revision, including local work, rather than independently verified live deployment. Experiment designs and architecture proposals describe work to evaluate. Examples are educational, use synthetic data and are not copied production code.
Read the test result precisely
The supplied engineering report describes 69 synthetic sources with distinct request references, producing 69 candidate groups. A mocked model response proposed 20 canonical issues. The remaining 49 sources were retained together in one active review remainder, giving 21 active rows while preserving all 69 original source memberships.
That is a workflow result. The vectors were synthetic and the LLM response was mocked. The fixture used SQLite rather than proving PostgreSQL production behaviour. It does not establish model accuracy, justify a clustering threshold or demonstrate that every real case should have 21 issues.
Make coverage an invariant
The central property is that every selected source remains accounted for after a bounded response. This simplified check represents separate assigned and remainder sets; a production model must also define its policy for multi-issue membership.
selected = {f"source-{n}" for n in range(69)}
assigned = {f"source-{n}" for n in range(20)}
remainder = selected - assigned
assert len(assigned) == 20
assert len(remainder) == 49
assert assigned | remainder == selected
assert not assigned & remainder
active_rows = len(assigned) + bool(remainder)
assert active_rows == 21Give unassigned evidence a next action
A remainder should explain why it exists: the analysis was bounded, ambiguous evidence was left for review, or a subset was not assigned. It should remain accessible from the case and permit another explicit review. Calling it resolved would convert a processing limit into a false conclusion.
The valuable regression test is not “the model produced twenty outputs.” It is “the user can still find every input, see what changed and restore the earlier organisation.” This is where AI product testing differs from checking an isolated prompt. The surrounding application must remain useful when the model returns only a partial answer.