Cross-Validation for Email Data: Preventing Thread and Case Leakage

Choose the split to match the intended deployment

If a classifier must work on unseen cases, hold out entire cases. If the task predicts later messages within an established case, a time-based evaluation may be more appropriate, with no features taken from the future. Thread grouping alone does not prevent related cross-thread evidence from leaking.

The MailTrace experiment should document its unit of independence. Exact duplicates and near-duplicates can connect otherwise separate groups. Review how exported copies, forwarded messages and reused templates are handled before interpreting a score. Group-aware splitting reduces one risk; it is not an automatic proof of independence.

Assert the boundary directly

This example shows how GroupKFold can keep synthetic case identifiers separate. It requires scikit-learn and illustrates split mechanics only. The placeholder labels are not a dataset and no classifier is trained.

from sklearn.model_selection import GroupKFold

X = [[n] for n in range(6)]
y = [0, 1, 0, 1, 0, 1]
case_ids = ["a", "a", "b", "b", "c", "c"]
for train, test in GroupKFold(n_splits=3).split(X, y, groups=case_ids):
    train_cases = {case_ids[i] for i in train}
    test_cases = {case_ids[i] for i in test}
    assert train_cases.isdisjoint(test_cases)

Keep model selection away from the final test

Use training and validation partitions for features, thresholds and hyperparameters. Keep a final case-held-out test for the selected approach. When comparing many models, report the selection process rather than presenting the best repeated test result as an unbiased estimate.

Report variation across cases and class-level errors, not only an average across messages. One very large case can dominate a message-weighted score while the model performs poorly elsewhere. For MailTrace, a useful evaluation explains which kinds of evidence fail and whether the resulting reviewer workload is acceptable.