(08)
Commissioned by Topline, Inc.
April 2026 · Preprint
An Empirical Evaluation of Parametric Agent Memory
What does and doesn't work when a small model is fine-tuned nightly on personal facts · an independent review.
Independent PhD researcher · external review
Abstract
We evaluate whether durable agent memory can be stored in the parameters of a small,
nightly-fine-tuned language model. Across three model families and 40 factual probes, the
fine-tuned models confabulated when queried directly, lost general capability with each
retraining cycle, and showed no measurable benefit from the proposed “organic
reinforcement” mechanism. A plain retrieval baseline over the same memory store
outperformed the parametric approach on factual recall. We conclude that the value lay in
the retrieval layer, not the weights.
- 1.Introduction & hypothesis—
- 2.Method & probe set—
- 3.Confabulation under direct query—
- 4.Capability regression per cycle—
- 5.Organic reinforcement: null result—
- 6.Retrieval baseline comparison—
- 7.Conclusion: retrieval over a ledger—
§ 3
It confabulated under direct query.
Asked directly for stored facts, the nightly-fine-tuned model invented plausible-but-false phone numbers, SSNs, and preferences on 31 of 40 probes — confidently. Parametric recall is lossy in exactly the way you can't afford for personal data.
§ 4
It lost capability every cycle.
Each retraining pass cost roughly 11 percentage points of general capability. An agent that retrains nightly gets a little worse at everything else, night after night.
§ 5
“Organic reinforcement” did nothing.
The mechanism our original whitepaper touted showed no benefit across three model families — the effect was statistically null or negative. The story was elegant; the data didn't support it.
§ 6
Plain retrieval beat it outright.
Search over the memory store recalled facts at 0.70 versus the fine-tuned model's 0.47. The retrieval layer was doing the real work all along — so we kept it, dropped the nightly fine-tuning, and built the product on the local ledger.