When classical wins

This is from my Master's thesis: a benchmark of which NLP technique best classifies bank-transaction text — the cryptic short strings a finance app has to read, like BIZUM ENVIO 15.00 or TRF NOMINA EMPRESA X. The most useful result so far wasn't an accuracy score. It was what happened when I tried to raise one.

I took the working classifier, froze everything around it, and fed it 7.4× more data — from 68,000 rows to 500,000. The F1 score went from 0.9845 to 0.9849. Four hundredths of a point. Training time went from 67 seconds to 304. Seven times the data bought nothing I could measure, at four and a half times the cost.

In machine learning the two reflexes are more data and a bigger model. On this problem both did almost nothing — and the benchmark shows exactly where each one stopped paying.

The setup

The goal was never a leaderboard. It was the honest balance between accuracy, compute, and scale — the call you actually make shipping a model, not the one a single accuracy number answers.

So the design isolates variables. One preprocessing pipeline, one evaluation function, one fixed seed, shared across every model and every corpus. Anything that moves is the model or the data — never the plumbing. Two public datasets from HuggingFace; one synthetic, one real.

The three runs, and what's actually in them:

Corpus	Nature	Rows	Categories	Vocab (TF-IDF)	Avg words
A	synthetic	68,000	17	11,508	5.2
B	real	68,000	10	5,901	3.1
C	real	500,000	10	14,816	3.1

And how the two classical models did across all three:

Corpus	Model	F1 (macro)	Train	Infer (ms/tx)
A · synthetic · 68K	SVM	0.9983	10.5s	0.0004
	XGBoost	0.9963	171.3s	0.1268
B · real · 68K	SVM	0.9845	1.2s	0.0001
	XGBoost	0.9848	67.0s	0.0871
C · real · 500K	SVM	0.9849	22.5s	0.0004
	XGBoost	0.9847	303.9s	0.1022

Three things the numbers said

1. More data hit a ceiling, fast

Same corpus, same model, 68,000 rows then 500,000. F1 0.9845 → 0.9849. The model had already learned everything the data had to teach it by the smaller size. The extra 432,000 rows were redundant — they re-taught what was already known.

7.4× the data → +0.04 F1 points. And 4.5× the training time.

More data isn't free. When it stops moving the metric, it's still moving the bill.

2. The simplest model won

Two classical approaches on the same features: a linear SVM and gradient-boosted trees (XGBoost). The SVM matched or beat XGBoost on every corpus — and it wasn't close on cost.

SVM vs XGBoost: 20–150× cheaper to train, 200–600× faster at inference, equal or better F1.

The fancier model bought a slower system to be, at best, exactly as accurate. In production that's not a tie. That's a loss you pay for on every request.

3. The synthetic data flattered the model

The highest score in the table, 0.9983, is the synthetic corpus (A). The moment I ran the same pipeline on a real dataset of the same size, F1 dropped about 1.4 points to 0.9845. Same model, same plumbing. The synthetic set was simply easier: more regular, cleaner, longer descriptions (5.2 words against 3.1). Its near-perfect number described the data, not the model.

A score from synthetic data is a promise, not a measurement.

Before anyone says it

Yes — even 98% F1 is suspiciously high (and the 99.8% was synthetic). It is. Even the real dataset here is clean, balanced, public data, and transaction text is highly separable: the merchant token usually gives the category away. Production bank data is messier: typos, missing fields, categories that genuinely overlap.

So the number isn't the point, and I wouldn't quote it as one. The shape is the point: the problem saturates early and cheap. Once you see that shape, throwing data and model capacity at it is spending where there's nothing left to buy.

The engineering call

This is the same decision I make on production systems, just with a clean academic frame around it. The boring model that fits the problem beats the impressive one that doesn't earn its cost. Accuracy is the headline; the bill arrives on every inference, forever.

The classics here already sit at 98–99%. So the honest question for the next phase isn't can a transformer do better — there's almost no room to. It's can a transformer justify its cost.

When did more data, or a bigger model, last pay for itself on your problem — and did you measure it, or just assume it would?

Next: I fine-tune a transformer (DistilBERT) on the exact same pipeline — and judge it not on the accuracy it adds, but on the latency and compute it charges for it. The interesting answer isn't whether it wins. It's whether it's worth it.