ML experiments & evaluation

ML experiments & evaluation

Engineering note · Part 17

Testing an AI Application: What Mocked Tests Prove—and What They Leave Unanswered

Separate deterministic regression tests, provider integration checks and real evidence quality evaluation.

ML experiments & evaluation

Experiment design · Part 19

Can Logistic Regression or Naive Bayes Classify MailTrace Messages Cheaply?

A proposed baseline experiment for narrow email triage, with a scikit-learn pipeline example.

ML experiments & evaluation

Experiment design · Part 20

Random Forest, XGBoost, LightGBM and CatBoost for Email Triage

Where tree ensembles might help structured email triage, and how to compare them without inventing a winner.

ML experiments & evaluation

Experiment design · Part 24

Cross-Validation for Email Data: Preventing Thread and Case Leakage

How to evaluate email models without training on near-duplicates of the evidence used to test them.

ML experiments & evaluation

Experiment design · Part 25

Learning from Reviewer Corrections in MailTrace

Turn reviewed decisions into a governed evaluation dataset before using them to train another model.

ML experiments & evaluation

Experiment design · Part 27

PCA and t-SNE for Exploring MailTrace’s Embedding Space

Use embedding visualisations to investigate patterns without treating a two-dimensional picture as a clustering verdict.

ML experiments & evaluation

Experiment design · Part 28

Isolation Forest and One-Class SVM for Evidence-Import Anomalies

Explore anomaly detection for import quality while keeping unusual evidence distinct from invalid evidence.

ML experiments & evaluation

Experiment design · Part 29

When Would Fine-Tuning Improve MailTrace?

Decide whether a measured failure belongs to retrieval, prompting, validation or model training before fine-tuning.

AI Evaluation

Structured Outputs and LLM Evaluation for Production AI Products

How schemas, validation, critique, repair and regression testing improve reliability in LLM-powered workflows.

Machine Learning

DBSCAN vs K-Means: Choosing a Clustering Model for Real Product Questions

A practical comparison of DBSCAN and K-means based on safeguarding and marketplace segmentation projects.

Responsible ML

Synthetic Data for Sensitive Machine Learning: What It Can and Cannot Prove

How synthetic data can support privacy-preserving prototyping without pretending simulated labels are real-world evidence.

ML Evaluation

Machine Learning Evaluation Beyond Accuracy

Why precision, recall, F1, calibration, failure costs and operational thresholds matter more than one headline metric.

RAG Evaluation

How to Evaluate a RAG System: Retrieval, Grounding and Product Quality

A practical RAG evaluation framework covering retrieval recall, citation correctness, unsupported claims, latency and cost.

AI Evaluation

Regression Testing Prompts and LLM Workflows

How to stop prompt edits and model upgrades from silently breaking production behaviour.

RAG & AI Evaluation

How to Fine-Tune and Optimise RAG Systems for High-Stakes Decision Support

A practical guide to improving RAG systems through embedding fine-tuning, FAISS optimisation, hybrid retrieval, generative-model tuning, evaluation and continuous monitoring.