Retrieve evidence for the decision a reader must make
The right result includes enough context to use and challenge the answer.
A search result identifies the correct document, but the reader still cannot answer the question. The relevant rule appears several pages later, an exception sits in an appendix, and the summary does not say which version applies. Finding the file helps, but the reader still needs the applicable passage and its exception before using the answer.
An agentic retrieval system can search again, open the document, inspect surrounding material, and revise its query when the evidence is incomplete. That additional work is useful only when it responds to a specific need. A simple lookup and a comparison across conflicting sources should not automatically receive the same expensive research procedure.
In my public knowledge-engineering essays, I distinguish machine-facing representations from human-facing views and examine retrieval that continues beyond an initial search. The shared concern is whether a person can inspect the evidence relevant to their decision. Those essays discuss methods and published research, with evidence limits stated alongside their findings.
Write the task before the query
Begin by describing the reader and the action they need to take. “Find information about batteries” leaves too much unspecified. “Help a volunteer identify the approved replacement battery for this device revision” defines an object, a decision, and a practical consequence of retrieving the wrong version.
Then identify what a sufficient answer requires. The reader may need the device revision, an applicable requirement, supporting evidence, and any unresolved exception. A maintainer investigating a change may instead need a timeline and dependency history, which means both readers can use the same collection while needing different selections and arrangements of evidence to complete their work.
The interface should retain distinctions that matter to the decision. If a qualification changes whether the answer can be used, placing it behind a collapsed panel can make the visible answer misleading. A citation should help the reader inspect support, not merely decorate a sentence with an appearance of authority.
An illustrative search sequence
Consider a hypothetical volunteer asking whether a donated projector can use a replacement power supply. The collection contains an old manual, a newer compatibility notice, and an unrelated model with a similar name. The agent’s first search returns the old manual because its title matches the question closely.
The next action should follow the missing evidence. Open the manual and identify the exact model and revision. Search the collection for that identifier, inspect the compatibility notice, and compare its scope with the donated device. If the device label is unavailable, return the missing identification requirement rather than guessing from the nearest product name.
The resulting answer should make the unresolved state useful. It can explain which label needs checking, show where the compatibility condition appears, and preserve the distinction between a confirmed match and a possible one. This is an illustrative information-retrieval exercise, not advice to connect electrical equipment or a report of a successful test.
Now change the task. A cataloguer wants every document mentioning that projector family, including obsolete records. The earlier procedure may be inadequate because it was designed to find decisive evidence for one compatibility question. Evaluation should therefore include coverage-oriented tasks as well as questions that one authoritative passage can settle.
Measure the work the reader can finish
Build a test set from actual, authorized reader tasks. For each task, record the evidence required for a supported answer and the circumstances that require abstention or clarification. Include ambiguous names, missing versions, contradictory statements, and sources the test identity is not allowed to access, then inspect whether the resulting answer preserves those distinctions instead of simplifying the evidence into an unsupported conclusion.
Measure whether the answer is supported and whether the required evidence was found. Also record time, tool calls, and the reader’s ability to complete the intended task. A longer answer or a larger retrieved bundle does not establish improvement. Additional research can increase cost while leaving the crucial exception unread.
Keep a fast route for questions that an exact, current source can answer. Escalate to repeated search and document inspection when the task genuinely requires more evidence. A budget limit should produce a clear partial result identifying what remains unresolved, rather than a confident answer assembled from whatever happened to be retrieved first.
For a first evaluation, choose one reader and a small group of recurring questions. Have that reader inspect the answer and its source route, then record where they still need to search manually. Use those failures to revise retrieval and presentation together before expanding the collection or adding another agent role.
Source notes
Public sources: One Body of Knowledge, Two Different Views and Enterprise Search Needs to Know When to Look Again. The projector example is hypothetical.
