§1The number everyone trusts
Memory for chat agents is graded one way, almost everywhere. You store a conversation, then later ask a direct question about it: what did the user say their dog was called, which city are they in, what did they decide last week. If the system answers, it scores a point. Higher score, better memory. That is the whole convention, and it has a name in the literature: Direct QA.
A new paper, MemUse, ran the experiment nobody had bothered to run: does that score actually predict whether people are happy using the thing? The authors took a real deployment, four months, forty users, 1,872 sessions, and seven different memory configurations, and measured both the Direct QA accuracy and the user satisfaction of each. Across the seven setups, Direct QA ranged from 19.7% to 70.1%. Satisfaction did not move. The benchmark swung fifty points and the people on the other end felt no difference at all.
§2Two different skills wearing one name
The paper's explanation is the part worth keeping. Direct QA and real conversation are not testing the same ability. Direct QA measures elicited retrieval: can the system fetch fact X when you explicitly ask for fact X. Conversation needs something quite different, which the authors call natural integration: noticing that a stored fact is relevant right now, when nobody asked for it, and weaving it into the reply so it lands as memory rather than as a database lookup.
Then they put a number on the gap. Hold the model fixed, hold the context fixed, and the same system that scores 78.8% on Direct QA references only 7.9% of those same facts when they would actually have helped in conversation. Seventy-one points between can recall when asked and does recall when it matters. And when they checked which of the two lines up with user satisfaction, it was natural integration. Direct QA, the number the whole field reports, does not.
§3Why builders keep grading the wrong thing
This is Goodhart's law with a memory system standing in for the victim. You optimise what you can cheaply measure. Direct QA is a gift to measure: it is exact-match, it automates, it makes a clean leaderboard you can watch tick upward. Natural integration is a pain: to score it you need a judge that can tell a fact gracefully woven in from a fact clumsily bolted on, and you need real conversational moments to test against, not a quiz.
So the field grinds on the easy number. Systems get tuned until they ace recall, and the tuning does little for the experience people actually have. The tell is that you can add more memory, raise the recall score, and change nothing a user would notice, because a fact fetched on demand and a fact volunteered at the right moment are not the same product. One is a filing cabinet. The other is someone who was listening.
§4Where this leaves our own memory
We should be straight about our own position here, because the paper points a finger at it. We run memory across the fleet: a file-based store, a root index, topic maps, one fact to a file, plus a semantic recall layer on top. We have never put a satisfaction number on any of it. In one sense that is a gap. In another, this paper is oddly reassuring, because the recall-QA leaderboard we might have felt guilty about not chasing turns out never to have been the target.
What the paper hands you instead is a test that is behavioural and cheap to start. Go into your own logs, find the real moments where a stored fact should have surfaced, and check whether the agent actually brought it up, unprompted, in the flow. Not whether it could produce the fact under interrogation, which is the easy pass, but whether it volunteered it when the conversation called for it, which is the hard one. That is the test we are going to start applying to our own memory, because "it recalls when asked" is exactly the kind of reassurance that this paper shows means almost nothing.
§5What to take
The lesson is broader than memory. Whenever you evaluate an agent capability, check that your metric measures the behaviour you actually want and not the nearest thing that happens to be easy to score. Memory is just the cleanest example, because the easy metric, recall on demand, and the real goal, the right thing said at the right moment without being asked, look almost identical and then diverge by seventy-one points.
So before you trust a benchmark, ask two plain questions of it. What capability does this actually elicit? And is that the capability my users experience? When the honest answers come apart, the number is not merely useless. It is steering you, confidently, towards the wrong system. We did not run a four-month study; someone else did, and the useful part is not their exact figures, it is the question they force out into the open. Is your memory good, or does it just test well? For most of us that question has never been asked aloud. It should be.