Cleaning Email Text Without Changing the Original Evidence

Store the original and the derived representation

The documented MailTrace preparation workflow keeps original evidence and creates normalised text for analysis. Quoted history and signature handling belong to this derived layer. The raw record remains available for inspection, and preparation needs a version so that changes can invalidate dependent indexes deliberately.

Cleaning is a lossy operation. A line beginning with an angle bracket may be a quotation worth reading. A signature may contain a relevant professional role. A useful implementation therefore preserves source context and does not confuse text omitted from an embedding with evidence deleted from the case.

Make even a small transformation traceable

The example below only normalises whitespace. It does not claim to solve signature detection or quotation parsing. The important part is that the derived representation records which raw content and preparation version produced it.

import hashlib

raw = "Please confirm the appointment.\r\n\r\nThank you."
prepared = "\n".join(line.rstrip() for line in raw.splitlines()).strip()
record = {
    "source_hash": hashlib.sha256(raw.encode("utf-8")).hexdigest(),
    "preparation_version": "whitespace-demo-v1",
    "text": prepared,
}
assert raw != prepared
assert record["text"].startswith("Please confirm")

Test what the cleaner removes

Useful fixtures include a short reply above a long quotation, a forwarded attachment notice, repeated footers, a mobile signature and a message where the signature itself matters. Compare the retained substantive text against reviewer expectations. A cleaner that removes everything should produce an explicit preparation state, not a plausible empty summary.

For retrieval and grouping, repeated quoted passages should not become extra votes for a source. MailTrace’s documented candidate design avoids letting subject-context chunks and duplicated substantive chunks inflate similarity. For citations, however, the user still needs to open the supporting passage in a source representation whose relationship to the original is clear.