Evaluation proposal: reproducible scope-isolation benchmark for DeerMem long-term memory

[PeaceMaker-best]

Summary

I would like to contribute a small, reproducible evaluation for DeerMem's long-term-memory admission and identity-isolation boundaries.

The proposed first slice does not change production memory behavior. It turns two already-observed failure classes into a versioned regression benchmark:

1. task-, project-, or authorization-scoped information being promoted into user long-term memory; and
2. custom-agent information being written to or recalled from the wrong user_id x agent_name bucket.

If the scope is useful, I am happy to implement it as a standalone benchmark following the repository's existing benchmark layout.

Motivation

DeerFlow now has strong deterministic safeguards after:

• [bug] DeerMem promotes task-scoped constraints into user long-term memory #4603 / fix(memory): prevent task-scoped data from entering long-term memory #4604 added extraction-time scope, durability, and authority labels plus a fail-closed write gate; and
• [bug] There is a **memory leakage / personality contamination** issue in the multi-agent memory framework. #4802 / fix(memory): scope bootstrap facts to custom agent #4804 fixed a real bootstrap path that wrote custom-agent facts into the reserved __default__ bucket, causing personality contamination in ordinary conversations.

The remaining evaluation gap is one layer above the current unit tests:

• test_memory_scope_gate.py proves that the deterministic gate behaves correctly when supplied with already-classified model output.
• The regression in fix(memory): scope bootstrap facts to custom agent #4804 pins the bootstrap middleware argument, but not an end-to-end memory-bucket outcome.
• fix(memory): prevent task-scoped data from entering long-term memory #4604 reported a 7/7 real-model extraction check, but the protocol, runner, and row-level outcomes are not versioned in the repository. A later model or prompt change therefore cannot reproduce or compare that result.

Because the extractor is model-driven, production code can remain unchanged while classification quality regresses. For example, a model may label a one-run permission (you may force-push this branch) or a repository-local constraint (this PR must not change the API) as durable user memory. The deterministic gate only protects the system if the required labels are correct.

Proposed first slice

Add a standalone evaluation under a path such as:

backend/scripts/benchmark/deermem_scope_isolation/

It would contain two deliberately small suites.

1. Semantic admission suite

Use synthetic, versioned conversations with unique non-sensitive canary terms and run them through the production DeerMem extraction prompt, normalization, deterministic gate, and temporary storage.

Representative cases:

Case	Expected persisted outcome
Durable communication preference	Persist
Durable cross-project working preference	Persist
One-off task constraint	Reject
Repository/project-local rule	Reject
Single-run edit/push/delete authorization	Reject
File-local correction	Reject
Explicit user-level correction	Atomically replace the old durable fact
Mixed durable and transient statements	Persist only the durable statement

2. Identity-routing suite

Exercise the production queue/storage boundary with deterministic model output and verify:

• facts written for custom Agent A do not appear in __default__ or Agent B;
• facts for User A do not appear for User B; and
• the custom-agent bootstrap path writes to the selected agent bucket.

This suite is deterministic and should not require a live provider.

Metrics

The report would keep the measurements small and directly tied to the failure modes:

• durable_retention_rate: expected durable canaries persisted / expected durable canaries;
• unsafe_persistence_rate: transient, project, or transactional canaries persisted / unsafe canaries;
• cross_agent_contamination_rate: facts visible outside their expected agent bucket / scoped facts;
• cross_user_contamination_rate: facts visible outside their expected user bucket / scoped facts; and
• atomic_correction_success_rate: valid user-level replacements that both remove the old fact and persist the replacement.

Reproducibility and safety boundaries

• The committed dataset would be synthetic and contain no user data.
• The runner would import production extraction/gating/storage code rather than duplicate policy logic.
• Default tests would be fully offline using deterministic fake model outputs.
• Live extraction would be explicit and environment-configured; normal CI would require no credentials or network access.
• Each live row would be resumable and bound to the suite, prompt, model settings, and source revision.
• Published artifacts would contain only synthetic canaries, verdicts, and non-secret metadata.
• Reports would keep deterministic routing and live model-quality results separate.

Non-goals

This first slice would not:

• change the production extraction prompt or write gate;
• introduce another memory backend, retrieval/ranking strategy, or vector store;
• define shared memory between agents;
• build a hosted evaluation service or UI;
• add live-provider calls to normal CI; or
• claim that a small synthetic suite is a general long-term-memory benchmark.

If the evaluation exposes a repeatable production failure, any prompt/gate behavior change would be proposed separately and would use this benchmark as before/after evidence.

Relationship to existing work

• fix(memory): prevent task-scoped data from entering long-term memory #4604 is the production semantic-admission boundary this benchmark would exercise.
• fix(memory): scope bootstrap facts to custom agent #4804 is the concrete cross-agent contamination regression this benchmark would generalize.
• eval(memory): add a reproducible hybrid eviction evaluation #4810 demonstrates the repository pattern for a standalone, reproducible memory evaluation that imports production policy code and keeps live calls optional.
• [RFC] DeerFlow Agent Evaluation Platform: Reproducible Live Runs, Evidence, and Deterministic Oracles #4083 is broader agent-evaluation platform work. This proposal intentionally stays a local, memory-specific benchmark and does not require that platform.

Questions for maintainers

1. Would a standalone benchmark in backend/scripts/benchmark/ be the preferred shape for this evaluation?
2. Should the initial contribution contain both the semantic-admission and deterministic identity-routing suites, or would you prefer the admission suite first?
3. Is a small published live baseline useful, provided the protocol is pinned and normal CI remains offline?

If this direction is acceptable, I can keep the implementation to one evaluation-only PR with no production behavior changes.

[sohamsinha2018]

One guardrail I’d add is an end-to-end “absence” assertion, not just a rejected write. For each transient canary, run a fresh session through the normal retrieval path and assert that it is not surfaced for the same user, another user, the default agent, or a different custom agent.

That distinguishes “the extractor happened not to propose a memory” from “a scoped fact can never become visible later.” I’d report those separately: admission false positives, storage-routing leaks, and retrieval leaks. A small matrix of the same canary across user × agent × fresh-session boundaries would make a regression much easier to localize.