The transformer wasn't worth it

This is the sequel to a promise, and the next stage of my master's thesis benchmark. In the last note, a linear model on TF-IDF matched everything else at classifying bank-transaction text, for a fraction of the cost. I said the open question wasn't whether a fine-tuned transformer could beat it — there was almost no room to — but whether it could earn its price. So I fine-tuned DistilBERT on the exact same pipeline. Sixty-seven million parameters against a hundred and ninety thousand.

On clean text, it was a dead heat. DistilBERT scored 0.9983 macro-F1. The linear model, 0.9984. I ran McNemar's test — the right one when two classifiers see the same test set — and it came back p = 1.00. Not "close." Of the cases where the two disagreed, the transformer won 4 and the linear model won 5. Statistically, the same model wearing different clothes.

A tie should have bothered me more than a loss would have.

Two systems with completely different views of language — one counting weighted word frequencies, the other running attention over subword tokens — scored the same. Models that read the problem this differently shouldn't land in the same place. When they do, the task isn't exercising what makes them different.

A tie between models that see language differently isn't equivalence. It's a benchmark that stopped measuring.

The benchmark was leaking

The public datasets in this domain are built from templates: a fixed set of merchants pasted into a fixed set of bank formats. Split them at random and almost every merchant in the test set also sits in training. The model doesn't need to understand anything — it memorizes merchant → category. And for memorizing a lookup table, a linear model is already optimal.

The 99.9% the dataset card advertises isn't skill. It's leakage. Once I saw that, the clean scoreboard was useless: it couldn't tell the models apart, so it couldn't answer the question I came with.

So I broke it on purpose.

Breaking it on purpose

I degraded the test set with four kinds of noise pulled from what real statements actually look like: keyboard typos, truncated descriptions, vowel-dropping abbreviations, and spurious ID numbers. Applied only at test time. No retraining — the point was to measure what each model had already learned, not to let it adapt.

And I wrote my prediction down before running it: the subword models would win. DistilBERT breaks an unknown word into known fragments, so a typo shouldn't faze it. The linear model keys on whole terms, so one typo should destroy the feature it leans on. Textbook.

It lost.

At 30% noise: the linear model kept 89.9% of its score. DistilBERT kept 84.2%.

Same order across all three corpora, at every noise level above zero. And McNemar, which couldn't separate the two on clean text, flipped hard: from p = 1.00 to p = 1.3·10⁻⁷⁸, in the linear model's favor.

McNemar · linear vs DistilBERT	Won by	p-value	Verdict
Clean text	5 vs 4	1.00	Tie — indistinguishable
30% noise	886 vs 265	1.3·10⁻⁷⁸	Linear wins

The two models the clean test called identical are, under noise, different beyond any reasonable doubt. Counts are cases only one model got right.

Why the simple one held

The explanation is about how the evidence is spread, not about how each model handles an unfamiliar word.

A five-word description gives TF-IDF about nine features — five single words and four adjacent pairs. Corrupt one word and you lose three of those features; six survive. Because a linear model sums whatever is present, the survivors still push toward the right category. It degrades gently, in proportion to the damage.

Averaging embeddings does the opposite. It crushes five words into a single vector, so corrupting one shifts the average by a fifth and leaves nothing behind to compensate. The transformer tokenizes the typo cleanly enough — but it never saw that sequence in training, and attention spreads the corruption to neighboring tokens. It doesn't degrade. It slips.

The robustness came from redundancy in the representation, not sophistication in the model.

The line I won't cross

One boundary, stated plainly, because it's the honest limit of what I measured. I tested resistance to noisy text, not generalization to merchants the model has never seen. That second thing — a brand-new store the training set never contained — is exactly where a pretrained transformer's outside knowledge might still earn its keep. Nothing here tests it.

So I won't tell you the transformer is useless for this problem. I'll tell you it added nothing under the conditions a real bank statement actually throws at you.

Back to the question

Was it worth it? The transformer cost 356× the parameters, 15× the inference latency, and a GPU to serve — to tie on clean data and lose on dirty data. On this problem, in these conditions, the answer is a clean no. The expensive architecture turned out to be the least accurate one in the only condition where the models actually differed.

None of that is an argument against transformers. It's an argument against reaching for one before your benchmark can prove you need it.

When a test can't separate two very different models, do you ship the cheap one and move on — or stop and ask what your benchmark stopped measuring?

Next: the model that won this is 1.3 MB. Small enough that it doesn't need a server at all — it runs inside the browser tab, and that turned out to change the economics far more than the accuracy ever did.