LongMemEval Explained: What It Tests and How to Read Scores

LongMemEval Explained: What It Tests and How to Read Scores

Picture Marcus. In March he tells his assistant that he moved from Austin to Denver. In June he asks it to book a dentist, and it books one in Austin. The memory system found both conversations. It just didn't know which fact was current.

That is a memory failure, not a search failure, and it's the reason LongMemEval exists. LongMemEval is a benchmark of 500 questions that tests five long-term memory abilities across long chat histories. It comes from an ICLR 2025 paper. Supermemory and Mem0 both print a LongMemEval score, and the numbers don't line up.

This page tells you what LongMemEval tests, how an answer gets scored, and how to compare two scores without fooling yourself. I build Hindsight, so I have a stake in this. My team published its own results in the Hindsight paper, and we run the benchmark on each release. I'll show you every number with its label. See how we run LongMemEval on each Hindsight release.

Here's my argument, in one sentence: memory is not search, and LongMemEval is the clearest proof. If you want the wider picture of AI agent memory, start there. If you want our numbers, they're on the Hindsight benchmark results page. Below I cover what the benchmark tests, how it scores, why it isn't a search test, how to read a score, what a score hides, and how to run it yourself.

What LongMemEval Tests

The LongMemEval paper describes "a comprehensive benchmark designed to evaluate five core long-term memory abilities." The authors wrote 500 curated questions. Each question sits on top of a long history of chat sessions between one user and one assistant. The system under test reads that history, stores what it wants, and then answers.

The five abilities are:

AbilityWhat the question asksExample question
Information extractionRecall a detail the user or assistant stated"What was the name of the restaurant I mentioned?"
Multi-session reasoningCombine facts from several sessions"How many projects have I started this year?"
Temporal reasoningUse time order or dates"Which did I finish first, the course or the report?"
Knowledge updatesReport the newest version of a changed fact"Where do I live now?"
AbstentionSay the system doesn't know"What is my sister's birthday?" when the user never said

I wrote these example questions myself. They are not from the dataset.

In the data, the official repository labels each question with one of six question_type values: single-session-user, single-session-assistant, single-session-preference, temporal-reasoning, knowledge-update and multi-session. Abstention questions are marked by an _abs suffix on the question_id. So you'll see seven kinds of question in the files, grouped into five abilities in the paper.

This is a different job from retrieval-augmented generation. A RAG system looks up passages and hands them to a model. LongMemEval asks the whole stack to remember, update and decline. If you want the difference in plain terms, I wrote it up in agent memory and RAG.

LongMemEval-S, M and oracle

The repository ships three files, each with 500 evaluation instances.

  • LongMemEval-S holds about 115k tokens of chat history per question, roughly 40 sessions.
  • LongMemEval-M holds roughly 500 sessions per history.
  • Oracle holds only the sessions that contain the evidence.

Vendors report S. Oracle isn't a memory test, because the system never has to find anything. It tells you how well the answer model reads evidence it's handed. M is the hard setting, and few vendors report it.

One more detail matters. The README notes that in September 2025 the authors cleaned up the history sessions "to prevent interference on answer correctness." The download commands now use _cleaned file names. A score should name the dataset revision it ran on. Most don't.

How a LongMemEval Answer Is Scored

Scoring has more moving parts than a single percentage suggests. The diagram shows the pipeline.

LongMemEval evaluation pipeline: chat history goes to a memory system, an answer model and an LLM judge for answer accuracy, with retrieval recall read separately.

The chat history goes into the memory system. The system stores it, then retrieves what it needs for each question. An answer model writes the final answer from what was retrieved. A judge model compares that answer with the gold answer and gives a binary verdict, correct or not. Answer accuracy is the share of correct verdicts.

The official script does this with a GPT-4o judge. This is the command from the repository README:

export OPENAI_API_KEY=YOUR_API_KEY
cd src/evaluation
python3 evaluate_qa.py gpt-4o your_hypothesis_file ../../data/longmemeval_oracle.json

Your hypothesis file is a jsonl file. Each line holds a question_id and a hypothesis. The README example points at the oracle file, so swap the last argument for the S file when you run a memory test. The script writes a log with an autoeval_label for each line, and a second script, print_qa_metrics.py, adds them up.

Accuracy and recall are different measurements

Retrieval recall is a separate evaluation. The dataset labels answer_session_ids and has_answer, so you can check whether the system found the right session. That tells you the system found the text. It doesn't tell you the system answered.

Think of an open-book exam. Finding the right page in the textbook is recall. Writing the right answer on the exam paper is accuracy. A student can open to the correct page and still write the wrong answer. A memory system can do the same: retrieve the March session and the June session, then answer "Austin."

That's Marcus's problem. His system had full recall for his question. It had no way to rank the two facts by time. Recall can't see that failure. Accuracy can.

Why LongMemEval Is Not a Search Test

I'll make the case plainly. Three of the five abilities can't be passed by search alone.

  • Knowledge updates. Search returns every mention of Denver and every mention of Austin. It doesn't tell you which one is current.
  • Temporal reasoning. Search returns passages. It doesn't put events in order or subtract dates.
  • Abstention. Search always returns something. A search-only system has no way to say "this was never said."

The paper backs this up. Its authors report "a 30% accuracy drop on memorizing information across sustained interactions" for commercial assistants and long-context models. Those systems had the text. Having all the text wasn't enough. The model still had to decide what mattered, what changed and what never happened.

Our own number points the same way. Hindsight scores 94.6% answer accuracy on LongMemEval-S, as of November 2026, in single-query mode on v0.4.19, from the Agent Memory Benchmark post. Single-query mode means "one retrieval call against the memory system, results passed directly to the LLM for answer generation." There is no agent loop in that run.

What does Hindsight do that search doesn't? Retain turns conversations into structured memories with extracted facts, entities and links between them. It records when events happened and separately when each fact was told to it, according to the retain docs. Background consolidation refines existing observations with new evidence. That's memory work: keep what happened, keep when it happened, and revise as the story changes.

This is also why I don't treat it as a leaderboard to win. I treat it as a test of a design choice. If your system stores text and finds text, it'll do fine on information extraction and badly on updates and abstention. To see how systems with different designs line up, read agent memory systems compared.

How to Read a LongMemEval Score

Picture Priya, who picks a memory system for her team. One vendor page says 97%. Another says 94.6%. She nearly picks the higher number. Then she reads the labels. The 97% is "Recall@20 with aggregation," a retrieval figure. The 94.6% is answer accuracy. The two numbers measure different things, and neither one tells her which system will answer her users correctly.

Priya's fix is the one I'd give anyone. Check four labels on every score before you compare it with another:

  1. The metric. Is it answer accuracy or retrieval recall?
  2. The judge model. Which model decided whether an answer was correct?
  3. The answer model. Which model wrote the answer from the retrieved memory?
  4. The dataset. Is it S, M or oracle, and the original or the cleaned revision?

Here's the table I wish every vendor published. It holds answer-accuracy figures only, all on LongMemEval-S, and every row names its judge and answer model.

SystemAnswer accuracyJudgeAnswer modelSource
Hindsight v0.4.19, single-query (as of November 2026)94.6%Gemini model, AMB harnessGemini model, AMB harnessAMB post
Hindsight (Gemini-3)91.4%GPT-OSS-120bGemini-3 (memory stack: GPT-OSS-120b)Hindsight paper, Table 3
Hindsight (OSS-120B)89.0%GPT-OSS-120bOSS-120B (memory stack: GPT-OSS-120b)Hindsight paper, Table 3
Hindsight (OSS-20B)83.6%GPT-OSS-120bOSS-20B (also runs the memory stack)Hindsight paper, Table 3
Supermemory (GPT-4o), earlier report81.6%GPT-4oGPT-4oHindsight paper, Table 3, from the Supermemory technical report
Full context, no memory system60.2%GPT-4oGPT-4oHindsight paper, Table 3, from the Supermemory technical report

The OSS-20B row changes the memory model as well as the answer model. The judges differ between rows, so compare the rows with care. The paper rows use GPT-OSS-120b at temperature 0.0. The Supermemory and full-context rows use a GPT-4o judge, as the paper notes. The Agent Memory Benchmark README says a Gemini model produces the answer and a second Gemini call scores it. The first row comes from a different harness than the rows below it, so don't read the gap between 94.6% and 91.4% as one change.

Two figures stay out of the table, because they aren't answer accuracy. Supermemory states 97% Overall Recall@20 on its LongMemEval page, which describes the configuration as "Recall@20 with aggregation," with gpt-4o judging the answers. Mem0 states 94.4 on LongMemEval in the description of its research page, and the page names no metric, judge or answer model. I print both as stated and leave them out of the comparison, because a retrieval figure and an answer-accuracy figure can't share a column.

If you want the full detail on those two products, I've written a Supermemory review and a head-to-head, Mem0 vs Supermemory, that go through how each one stores memory.

What a LongMemEval Score Does Not Show

A clean score is still one data point. Here's what it leaves out.

One user's history. The benchmark tests one user talking to one assistant. It doesn't test memory shared by a team, or by several agents that write to the same store. If your product needs shared memory, the benchmark won't tell you how it behaves.

Size. S holds about 115k tokens of history per question. M holds roughly 500 sessions per history, and every score on this page is an S score.

The judge is a model. A different judge gives a different number. The official script uses GPT-4o. Papers and vendors swap in other judges, which is why the judge is the second label on my list.

The answer model moves the score. This one surprises people the most.

Picture Dana, who runs LongMemEval-S on Hindsight and changes only the answer model. The memory stack stays the same: in the paper's OSS-120B and Gemini-3 runs, GPT-OSS-120b does the fact extraction and the retrieval. The judge stays the same too. With OSS-120B writing the answers she gets 89.0%. With Gemini-3 she gets 91.4%. The memory didn't change. 2.4 points came from the model that wrote the final sentence. Those are two rows from the paper's Table 3, so you can check them yourself.

The OSS-20B row is a different test. According to Section 7.3 of the paper, OSS-20B runs both the memory stack and the answers, so that row changes two things at once. I show it in the chart and the table, and I don't use it for this comparison.

Bar chart of Hindsight LongMemEval-S answer accuracy, judged by GPT-OSS-120b: OSS-20B 83.6% (changes the answer model and the memory model), OSS-120B 89.0% (baseline), Gemini-3 91.4% (changes only the answer model, against OSS-120B).

Chart: the OSS-20B bar also changes the memory model. The step from OSS-120B to Gemini-3 changes only the answer model.

The lesson: when two vendors report different numbers, ask what answer model each used before you ask whose memory is better. The same habit applies to every vendor benchmark. For a second example of reading a vendor's figures, see MemPalace benchmarks.

Hindsight on LongMemEval, and How to Run It Yourself

Our latest result is 94.6% answer accuracy on LongMemEval-S, as of November 2026, on Hindsight v0.4.19 in single-query mode. The same post describes an agentic mode, where the LLM drives retrieval through tool calls, issuing multiple queries and inspecting results.

Everything in the harness is public. The Agent Memory Benchmark repository, vectorize-io/agent-memory-benchmark, holds the code and the instructions for running it. I'd rather you reproduce the number than trust me.

To run the benchmark against any memory system, including ours:

  1. Get the data. Download the cleaned S file from the official repository and note the revision.
  2. Produce a hypothesis file. Load each history into your memory system, ask each question, and write one question_id and hypothesis per line.
  3. Run the judge. Run evaluate_qa.py with your chosen judge, as in the command above, then aggregate with print_qa_metrics.py.

The most common mistake is comparing a retrieval figure with an accuracy figure. The second most common is changing the answer model between runs and crediting the memory.

To start Hindsight locally, the installation docs give a Docker quick start:

export OPENAI_API_KEY=sk-xxx
docker run -it --pull always --name hindsight --restart unless-stopped --shm-size=1g --stop-timeout 75 \
  -p 8888:8888 -p 9999:9999 \
  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
  -v hindsight-data:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest

Run the benchmark against Hindsight yourself. View on GitHub.

What to do next

Here are the takeaways, each one an action:

  • Check the metric first. Reject any comparison of recall with accuracy.
  • Ask for the judge model, and note when two scores use different judges.
  • Ask for the answer model, and rerun with a fixed one if you can.
  • Name the dataset: S, M or oracle, original or cleaned.
  • Run the benchmark on your own data before you trust anyone's number, including mine.

If your agents forget, contradict themselves or confidently answer things the user never said, the fix isn't a bigger search index. It's a memory that updates, orders events in time and knows when it doesn't know. You can try that today: sign up for Hindsight Cloud and have it running in minutes. Hindsight is open source under MIT and runs on Postgres, so if you leave Cloud you can self-host the same system and keep your data.

Frequently Asked Questions

Who created LongMemEval? Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang and Dong Yu created it. They posted the paper in October 2024, and it appeared at ICLR 2025. The benchmark has 500 questions and tests five long-term memory abilities in chat histories between one user and one assistant.

How is LongMemEval different from LoCoMo? LongMemEval tests five memory abilities over long, multi-session histories of one user, including updates and abstention. LoCoMo is a separate benchmark of very long conversations, introduced in the LoCoMo paper. They use different data and different question sets, so their scores don't convert.

What does LongMemEval test? LongMemEval tests five long-term memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention. Each of its 500 questions sits on a long chat history between one user and one assistant. The system must store that history, then answer with the current, correct fact or say it doesn't know.

How do I run LongMemEval myself? Download the cleaned LongMemEval-S file from the official repository, run your memory system over each history, and write a jsonl file with a question_id and a hypothesis on each line. Then score it with evaluate_qa.py and a judge model. Keep the judge, answer model and dataset revision fixed between runs.