Preserve bounded RAG evidence in durable batch exports (#6405)

[Guo-Yixin]

Scope proposal for #6405, opened here because the feature-request form asks for an Ideas discussion before implementation. This is the same proposal, not a second workstream.

Before you start

• I searched existing issues and pull requests for batch sources/artifacts and RAG citations.

Problem / motivation

At official main 042f15249da9fad1894905d2a751cb8a00fba6f2, a durable research item can return #knowledge-<id> citations, but its JSONL result contains only the report text. The evidence excerpts captured by the child are lost after the worker/registry is gone. Unlike ordinary task results, the exported batch result cannot be audited against the exact retrieved text the model used.

The current RAGFlow guide explicitly excludes durable batches from source-artifact forwarding. This is a bounded follow-up to #5551's verifiable evidence protocol, not a replacement knowledge provider or batch scheduler.

Reproduced offline through the production RAGFlow formatter, real ToolMessage/step capture, production batch worker, SQLite engine teardown/reopen, and the existing owner-scoped JSONL export. The report survives; its evidence artifact is absent. Both direct knowledge-search and already-delegated task evidence exhibit the gap. External model/provider execution was replaced by a deterministic result at the executor boundary; no live RAGFlow or model call was made.

Proposed solution

• Add a nullable JSON result_artifact to durable batch items, with an additive Alembic migration. Existing rows remain readable with no evidence snapshot; no historical backfill or fresh provider fetch.
• Forward only captured, schema-validated RAGFlow source records cited in the stored report, including delegated task evidence. Preserve source IDs and entire excerpts; omit records that cannot fit instead of shortening text under an existing ID.
• Bound the serialized evidence snapshot separately by the existing max_result_chars setting and cap records at the existing 100-source forwarding limit. Explicitly count budget omissions.
• Commit evidence in the same lease-fenced transaction as a successful result. Cancelled, retried or superseded attempts must not publish evidence.
• Include the snapshot only when full results are requested: existing owner/thread-authorized results.jsonl. Keep compact item previews and progress responses unchanged; no new document proxy or provider credentials.

Acceptance / scope

A caller submits independent knowledge research items, retrieves the existing JSONL export after restart, and resolves each retained citation ID to its original bounded excerpt and provider locator. Reopening the database must not require the provider, worker or in-memory registry. Cover owner/thread rejection, compact projections, plain/legacy results, budget/truncated reports, cancellation, stale leases, retries and migration preservation.

This stage is an export/API capability. No batch-panel citation dialog, new model-facing results tool, arbitrary artifact persistence, execution/retry/acceptance policy change, or model-quality claim. Core is appropriate because this changes the native batch worker and durable result contract; an extension cannot update their lease-fenced persistence atomically.

Alternatives considered

• Refetch provider chunks when exporting: changes historical evidence and requires credentials/provider availability.
• Persist all child messages/artifacts: stores unrelated context and makes batch outputs unbounded.
• Strip citation links from reports: avoids dangling links but removes evidence instead of making it auditable.

Affected area(s)

Backend API (additive JSONL field), Agents / LangGraph, SQL persistence, docs/tests.

[willem-bd]

I support the bounded core change proposed here. The evidence-loss problem is reproducible, and whole-excerpt retention plus atomic persistence are appropriate for this scope.

I independently reproduced both direct retrieval and delegated task evidence through the production RAGFlow formatter, real ToolMessage/step capture, production batch worker, SQLite engine close/reopen, and existing JSONL export. The report and citation IDs survive, but the source artifacts are absent. Model execution was replaced with a deterministic result, tool assembly was stubbed, and no live provider/model call was made. The focused reproduction preserved owner/batch-thread rejection; it did not exercise the route permission decorator. The existing batch repository, service, and router tests also passed: 31 passed.

A few implementation-review criteria:

• Validate the durable schema explicitly. The existing cited_source_artifact() helper copies source dictionaries with limited validation. Persistence should allow only the defined RAGFlow fields and reject malformed records or unsupported versions, rather than treating that forwarding helper as sufficient durable-schema validation.
• Measure the complete serialized snapshot. Include locators, names, pages, JSON escaping, and omission metadata in the budget, not just excerpt text. Define the serialization/unit used for the bound. Since report and evidence have separate allowances, document that their combined payload can approach twice max_result_chars, plus the surrounding export metadata.
• Select citations after report truncation. Preserve entire captured excerpts and their IDs. Explicitly report incomplete evidence when a cited record cannot fit; the stored report can still contain references whose evidence was omitted. Test truncated citation links, duplicate IDs, and zero retained records.
• Keep report and evidence transactionally consistent. Successful finalization must write both together under the existing lease/cancellation checks. Cancellation, stale workers, failed attempts, and retries must never expose evidence from another attempt. Include database reopening, compact-projection exclusion, owner/thread authorization, legacy rows, and migration preservation in the regressions.

Core is justified because the change belongs inside the native batch finalization transaction. No architectural blocker found for this scope.

I saw the update in #6405 that the implementation and regression tests are already completed locally. The upcoming PR should be reviewed against these criteria; this assessment verifies the existing gap and proposed design, not the unpublished implementation or its reported validation totals.