The situation
The carrier needed to classify incoming events across general liability and housing claims. Conventional NLP handled the common categories acceptably and fell apart on the rare ones, sitting at roughly 20% accuracy on them.
That distribution is exactly backwards from what the business needs. Rare claim events are rare because they are unusual — and unusual claims are where severity, reserving error and leakage concentrate. A model that is strong on the routine and blind on the exceptional has automated the part that was never expensive.
How it works
Look at the data first
A data-quality view of the corpus, before touching the model.
Capture the adjusters' judgement
Expert knowledge goes in as training signal, not after-the-fact review.
Trust the sparse tags
Few keywords means a specific claim, so those few carry more weight.
Run both together
Keyword weighting needs no examples; the transformer reads context.
Why this was harder than it looksThe technical detail
Class imbalance is a familiar problem with familiar remedies: resampling, class weights, synthetic augmentation. None of them was going to work here, because the problem was not really statistical.
For a rare class there were only a handful of examples, each described differently by a different agent. There was no stable linguistic pattern for a model to learn, because the pattern lived in the adjusters' judgement rather than in the text.
We weren't short of data. We were short of the knowledge that told you what the data meant.
What we did
Built the diagnostic before the model
Before touching the classifier we built a visualisation tool for data quality, so the team could see what was actually in the corpus. That alone cut several weeks out of the proof of concept — time that had been going into experiments run against data nobody fully understood.
Treated expert knowledge as a training signal, not a review step
Subject-matter expertise was structured and fed into the classification approach directly rather than collected afterwards as corrections. That lifted average classification accuracy by 30% across categories.
A hybrid: a transformer classifier, plus a weighting scheme built on how agents annotate
Field agents tag each claim with case-specific keywords describing what happened. A single claim can carry several, and any of them could serve as a classification tag — but their diagnostic value is wildly uneven, and this is the part the standard approach misses.
A claim carrying many keywords is diffuse: the terms point in several directions at once and none of them pins the claim down. A claim carrying only a few is specific, and those few terms are strongly indicative of what it really is. We built the weighting on exactly that asymmetry — the sparser a claim's tag set, the more weight its terms carry. It inverts the usual instinct: rather than treating more annotation as more signal, it treats restraint as precision.
That weighting runs alongside a transformer-based classifier, and the two cover each other's weaknesses almost exactly. The transformer reads context and absorbs the varied phrasing across agents, regions and years — but like any learned model it needs examples, which is precisely what a rare class does not have. The keyword weighting needs no examples at all: it carries the adjusters' prior about what matters, and it is strongest exactly where the transformer is weakest. Together they took rare-event accuracy from 20% to 75%.
The outcome
- Rare-event accuracy from 20% to 75% — a 3.75x improvement on the segment carrying the cost.
- +30% average classification accuracy across categories.
- Weeks removed from the proof-of-concept cycle through data-quality tooling built up front.
If you're facing something similar
When a model fails on a rare class, adding data is usually the wrong instinct — the missing input is domain expertise, and there are ways to encode it rather than merely consult it. And if a team is iterating slowly, check whether they can actually see their own data before you look at the model.