Random Forest, XGBoost, LightGBM and CatBoost for Email Triage

Define the decision before comparing libraries

A reasonable experiment could prioritise messages that need manual classification review. The target would be a reviewer judgement made from evidence available at that time. Avoid targets derived from later outcomes or from the existing model’s own predictions, which can make an apparently strong classifier circular.

Random forests and boosted tree libraries offer different training choices. Their value here should be assessed against the same simpler baseline and feature set. A sophisticated library cannot repair inconsistent labels or a split that puts near-duplicate threads in both training and test data.

Make the comparison reproducible

The example defines candidate estimators only. Each dependency must be installed in an isolated experiment environment, and each model needs its own preprocessing and tuning plan. These initial settings are deliberately modest placeholders, not recommended production hyperparameters.

from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier
from catboost import CatBoostClassifier

models = {
    "forest": RandomForestClassifier(n_estimators=100, random_state=7),
    "xgboost": XGBClassifier(n_estimators=100, random_state=7),
    "lightgbm": LGBMClassifier(n_estimators=100, random_state=7),
    "catboost": CatBoostClassifier(iterations=100, random_seed=7, verbose=False),
}

Optimise reviewer workload rather than a headline score

Report missed review-worthy messages at a fixed review budget, false alerts, per-case variation and inference cost. Keep test data untouched while choosing features and hyperparameters. Imbalance handling belongs inside the training process; oversampling before the split can leak related examples into evaluation.

If one model earns a place in MailTrace, it should expose the same versioned proposal contract as other classifiers. It should not become a separate source of truth for case ownership or evidence status. A controlled experiment can add a capability; adding four dependencies without a clear decision owner simply adds maintenance.