LoCoMo Benchmark Explained: What It Tests and How It Scores
A 92% on LoCoMo can mean three different tests. The paper that defines the LoCoMo benchmark scores answers with F1 on all five question types. Other reports use an LLM judge instead of F1. Some drop a question type, and some do not name the judge.
Picture Priya, a staff engineer who must pick a memory layer by Friday. Two vendor pages each claim a LoCoMo score above 90%. She can't tell which one used the paper's metric, which one dropped the adversarial questions, or which model judged the answers. The two numbers look alike. They say different things.
You have probably seen the same thing. LoCoMo scores sit in the launch post of nearly every AI agent memory product. After this page you can read any of them and say what it measures. I'll cover what the test is, how it is built, how it is scored, why scores don't compare, what it misses, how to run it, and where our own score comes from.
Short answer first: the LoCoMo benchmark is a 2024 test of long-term conversational memory. It asks questions about multi-session chats between two people. I build Hindsight, and we run LoCoMo on every release. You can read our Hindsight benchmark results next to the method.
What the LoCoMo benchmark is
The LoCoMo benchmark comes from the paper "Evaluating Very Long-Term Conversational Memory of LLM Agents" by Maharana, Lee, Tulyakov, Bansal, Barbieri and Fang, posted to arXiv in February 2024. The authors wanted a test for conversations far longer than the five-session chats earlier work used.
The paper defines three tasks: question answering, event summarization and multimodal dialogue generation. Vendors report one of them. When a memory product shows a LoCoMo score, it is a question answering score.
There are also two sizes of the dataset. The paper's statistics table counts 50 conversations. The public repository, snap-research/locomo, says: "The benchmark consists of ten conversations." Those ten sit in data/locomo10.json. The ten-conversation file is the one most memory products run.
So the LoCoMo benchmark answers a narrow question. Given a long chat history between two people, can a system find and use the right facts when asked? That is a useful question for any agent that talks to a user over many sessions. It is not the whole job of memory, and I'll come back to that.
If you want to see what retaining a conversation looks like in practice, the Hindsight retain guide shows how raw conversations become searchable memories.
How LoCoMo conversations and questions are built
Two LLM agents, each given a persona and a temporal event graph, talk to each other across sessions. They can share images and react to them. Human annotators then check and edit the dialogue for consistency. That is the machine-human pipeline the paper describes.
The size is modest by production standards. Table 1 of the paper gives an average of 304.9 turns, 19.3 sessions and 9,209.2 tokens per conversation, with up to 35 sessions. The Mem0 paper, arXiv 2504.19413, describes the ten conversations it uses as having "approximately 600 dialogues and 26000 tokens on average." The two papers disagree by almost a factor of three.
Each question belongs to one of five types. The paper defines them in Section 4.1:
- Single-hop: the answer sits in a single session.
- Multi-hop: the answer needs information from several different sessions.
- Temporal reasoning: the answer depends on time cues in the conversation.
- Open-domain knowledge: the answer joins what a speaker said with outside knowledge, such as common sense.
- Adversarial: the question is built to trick the system. The right answer is to say it can't be answered.
The mix is not even. Table 5 of the paper counts 7,512 questions over 50 conversations. Single-hop is the largest group at 2,705 questions. Open-domain is the smallest at 285.
Images matter less than the dataset's design suggests. In the question answering task the paper replaces each image with a BLIP-2 caption, so the model reads text with captions mixed in.
How the LoCoMo benchmark is scored
Here the name stops meaning one thing. A run has the same shape every time, and the diagram shows it. Sessions go into a memory system. A question retrieves context. An answer model writes the answer. A scorer then marks it, and the scorer is where reports differ.
The paper computes F1 on normalized answers and uses a partial-match F1 for model predictions, because LLMs paraphrase. For systems that retrieve, it reports retrieval accuracy as recall of the turns that hold the evidence.
Many vendor posts use a different metric. An LLM judge reads each answer and marks it right or wrong. The score is the share marked right.
The two methods disagree on easy cases. F1 compares words, so "May 7, 2023" and "7 May 2023" overlap only in part. A judge accepts both. A judge also adds its own variable. A different judge model can shift the score, and the model that writes the answers shifts it again.
The question set changes the score too. The Mem0 paper says the adversarial category "was excluded from our evaluation because ground truth answers were unavailable." It reports four types, and about 200 questions per conversation. A run on all five types, as in the paper, is a different test. A LoCoMo benchmark score without its metric, its judge, its answer model and its question set is a number with no meaning.
The next section turns this into a checklist.
Why LoCoMo scores from different vendors do not compare
Ask four questions of any LoCoMo score:
- Which metric: F1, or a judge's accuracy?
- Which judge model, if a judge was used?
- Which answer model wrote the answers?
- Which question set: all five types, or four?
The table shows what each report states. It holds no scores on purpose. Cells say "not named" where the source gives no name.
| Report | Metric | Judge model | Answer model | Question types | Conversations |
|---|---|---|---|---|---|
| LoCoMo paper (arXiv 2402.17753) | F1 partial match, plus retrieval recall | no judge | set per run | all five | 50 in the paper, 10 released |
| Mem0 paper (arXiv 2504.19413) | F1, BLEU-1 and an LLM judge | not named | GPT-4o-mini | four, adversarial excluded | 10 |
| Hindsight Agent Memory Benchmark post | accuracy | not named | not named | not named | locomo10 |
| OpenViking README | accuracy | not named | Doubao models | not named | not named |
Here is how that plays out. Mem0 states 92.5% on LoCoMo on its research page. That page does not name the metric or the judge, and Mem0's paper drops the adversarial category. OpenViking states LoCoMo accuracy for each of three agents with and without OpenViking, on Doubao models. Those figures compare an agent with itself. They do not compare one memory system with another. I go into this in the OpenViking review.
I don't claim any vendor changed a number it should not have. I claim something plainer: you can't line these figures up in one chart. The same limit applies when you read Mem0 vs Supermemory or any other comparison that quotes benchmark scores.
Read the benchmark code yourself. View Hindsight on GitHub and run it against your own history.
What a LoCoMo score does not show
This is where I stop explaining the test and start arguing with it. Memory is not search. A LoCoMo score shows how well a system finds facts in one person's past chats. It does not show whether an agent learns from outcomes.
Our own Agent Memory Benchmark post says that on most LoCoMo and LongMemEval instances, a naive "dump everything into context" approach scores competitively. It still calls both datasets "a valid foundation" that "only cover one slice of the problem." I agree with both halves.
Say you're Marcus, and you put all 19 sessions of a LoCoMo conversation into one long prompt. The score looks strong. Then your support agent goes to production, where each customer has six months of tickets. The same approach runs out of context. The agent repeats a fix that failed last week, because nothing in the prompt tells it the fix failed.
The benchmark rewarded search over a short history. The job needed memory that learns.
A LoCoMo benchmark run has a few blind spots for that reason:
- It covers hundreds of turns, not months of chats.
- It has no stream of conflicting facts that a system must reconcile.
- It does not test whether an agent changes what it does after a mistake.
This is the gap between memory and retrieval. Hindsight learns from outcomes, not only stores and retrieves facts. I wrote about the difference in agent memory and RAG.
How to run the LoCoMo benchmark yourself
The snap-research README gives the steps. Set your API keys and output folders in one file, then run the script for your model family.
- Clone the repository and open the data folder. The ten conversations are in
./data/locomo10.json. - Set API keys and output directories in
scripts/env.sh. The README says every other script runs it first. - Run the script for your model. Each one evaluates question answering.
- Read the output files before you read the score.
git clone https://github.com/snap-research/locomo.git
cd locomo
# set API keys and output directories in scripts/env.sh
bash scripts/evaluate_gpts.sh # OpenAI models
bash scripts/evaluate_claude.sh # Anthropic models
bash scripts/evaluate_gemini.sh # Gemini models
bash scripts/evaluate_hf_llm.sh # Hugging Face models
bash scripts/evaluate_rag_gpts.sh # retrieval-augmented gpt-3.5-turbo
The README marks the event summarization and multimodal dialogue generation tasks as "Coming soon!", so these scripts cover question answering only.
Four mistakes show up again and again. People mix F1 and judge scores in one table. They drop the adversarial type without saying so. They change the judge between runs. They forget to record the answer model.
Say you're Dana, and you run evaluate_gpts.sh on a Friday afternoon. The F1 comes back lower than the vendor figures you read last week. You open the outputs. Many answers are right in meaning, and F1 marks them wrong on wording. So you report F1 and a judged score side by side, and you write down which judge you used.
That habit is what makes a LoCoMo benchmark number worth quoting.
Where Hindsight's LoCoMo score comes from
Hindsight scores 92.0% on LoCoMo, version v0.4.19, in single-query mode, as of March 2026. The Agent Memory Benchmark post defines single-query mode as "one retrieval call against the memory system, results passed directly to the LLM for answer generation." The score is for the ten-conversation file, locomo10.
The run output is public. The Hindsight benchmarks page links the output file for the LoCoMo run, and the evaluation harness is on GitHub as agent-memory-benchmark. I'd rather you check the answers than trust me.
We use LoCoMo as a regression check on each release, not as proof that memory is solved. For how we compare with other systems on the benchmarks that matter, see agent memory systems compared, and for another memory benchmark discussion, read about MemPalace benchmarks.
Frequently Asked Questions
Who created the LoCoMo benchmark?
Maharana, Lee, Tulyakov, Bansal, Barbieri and Fang created it. They describe it in the 2024 paper "Evaluating Very Long-Term Conversational Memory of LLM Agents" (arXiv 2402.17753). The public dataset and evaluation scripts sit in the snap-research/locomo repository on GitHub, in the file data/locomo10.json.
How is LoCoMo different from LongMemEval?
LongMemEval has 500 questions on chat histories, and its official code scores answers with a GPT-4o judge. LoCoMo has ten public conversations, and its paper scores answers with F1. They test different histories with different metrics, so a score on one says little about the other.
Does LoCoMo test images?
The paper includes shared images in its conversations, but the question answering task does not use them directly. It replaces each image with a BLIP-2 caption, and the model reads text with captions mixed in. Images are used directly only in the multimodal dialogue generation task.
Is LoCoMo still useful in 2026?
Yes, as a regression check. Hindsight's Agent Memory Benchmark post calls LoCoMo and LongMemEval a valid foundation. It also says they cover one slice of the problem. They don't test months of history, conflicting facts or an agent that learns from mistakes, so don't use LoCoMo alone to choose a system.
What is a good LoCoMo score?
The paper reports a human baseline of 87.9 overall F1 (Table 2). That number uses the paper's metric. A judge-based accuracy score uses a different scale, so compare a score only with scores that use the same metric, judge, answer model and question set.
Conclusion
The LoCoMo benchmark is a fair test of one skill. It is a poor basis for comparing vendors unless the reports match. Here is what to do:
- Ask which metric a LoCoMo score uses.
- Ask which judge and answer model produced it.
- Check the question count and whether adversarial questions are in.
- Read the run output before you trust the number.
- Test with your own agent's history, not only a public dataset.
Next steps, in order. Read the Hindsight benchmark results and the method behind them. Run locomo10 yourself with the commands above. Then try Hindsight on your own agent's history.
Hindsight is open source under the MIT licence and runs on Postgres, so you can test it on your own data before you commit to anything. If you'd rather skip the setup, sign up for Hindsight Cloud. A LoCoMo benchmark score tells you how a system searches. Your own history tells you whether it learns.