Can Logistic Regression or Naive Bayes Classify MailTrace Messages Cheaply?

Choose a label that reviewers can apply consistently

A first experiment might distinguish a substantive request from an acknowledgement, while allowing ambiguous examples to remain unlabelled. Predicting whether a request is legally resolved is a much more demanding task and should not be smuggled into the same binary label.

TF-IDF features with logistic regression provide a useful text baseline. Multinomial Naive Bayes is another candidate for nonnegative text features. Compare them against simple rules using the same labelled examples and case-held-out split. Neither should inherit credibility from its low inference cost.

Fit preprocessing inside the training pipeline

The example constructs two candidate pipelines. It does not include private data, train a model or claim a score. A real experiment supplies reviewer-labelled training and held-out data, with vocabulary fitting restricted to the training partition.

from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB

candidates = {
    "logistic": make_pipeline(
        TfidfVectorizer(ngram_range=(1, 2), min_df=2),
        LogisticRegression(max_iter=1000, class_weight="balanced")
    ),
    "naive_bayes": make_pipeline(
        TfidfVectorizer(ngram_range=(1, 2), min_df=2),
        MultinomialNB(alpha=1.0)
    ),
}

Adopt only a measured, bounded role

Evaluate per-class precision and recall, rare examples and the rate at which the model abstains. If scores drive automatic routing, inspect calibration and choose thresholds on validation data. A confidence value is not enough to justify changing issue membership without review.

A successful experiment could prioritise a review queue or reduce repeated LLM classification for routine messages. Keep the source record, model version and review outcome. Before deployment, test distribution changes such as new organisations, different templates and unusually short replies. The experiment succeeds only if it saves useful work without increasing missed requests or hidden errors.