Field Notes: AI Agents Are Moving From Retrieval to Context Assembly

Enterprise AI has spent years treating context as a search problem: receive a request, retrieve relevant documents, and place the best passages in front of a model. That pattern remains useful, but it is no longer sufficient for agents expected to complete operational work.

An agent preparing a renewal brief, approving an expense exception, or drafting a customer response does not merely need relevant text. It needs a coherent operating picture: the current record, applicable policy, recent events, unresolved exceptions, the user’s role, and the state of the workflow. Those inputs live in different systems and carry different levels of authority.

The emerging design unit is therefore not the retrieved document. It is the context package: a task-specific, time-bounded, provenance-aware representation of what the agent should know for one decision.

What changed: context became an engineered product

Earlier retrieval systems typically optimized for semantic similarity. If a user asked about a contract clause, the system found passages resembling the question. Agentic systems have a harder requirement. They must assemble facts that may not resemble the prompt but materially govern the action.

Consider the request, “Prepare the Acme renewal.” Semantic search might surface the contract and prior renewal notes. A competent context assembly process may also need to collect current product usage, open support incidents, unpaid invoices, approved discount limits, the account owner’s forecast, recent executive correspondence, and any pending security review.

Several technical changes make this possible. Tool-calling lets agents query structured systems rather than rely solely on indexed documents. Longer model inputs allow more evidence to be considered, although they do not remove the need for selection. Workflow engines expose state transitions. Enterprise identity layers make role-aware access practical. Structured outputs let an assembly layer normalize evidence before the reasoning step.

The practical change is architectural: retrieval becomes one component in a pipeline that also resolves entities, queries live systems, applies permissions, transforms records, detects conflicts, and records provenance.

The new pipeline: assemble before reasoning

A reliable context assembler performs distinct operations. Combining them into one opaque prompt makes failures difficult to diagnose.

1. Interpret the task. Identify the decision, target entity, time horizon, and required output.
2. Resolve entities. Map “Acme” to the correct account, contracts, opportunities, and legal entities.
3. Generate evidence requirements. Determine which facts would change the answer, including policy constraints and exceptions.
4. Acquire evidence. Retrieve documents, query systems of record, inspect workflow state, and collect recent events.
5. Normalize evidence. Convert heterogeneous records into consistent fields, dates, units, and identifiers.
6. Rank by authority and recency. Distinguish an executed agreement from a sales note and a current balance from an old export.
7. Compress with provenance. Summarize where necessary while preserving links to source records and timestamps.
8. Validate completeness. Check whether mandatory evidence is present before allowing reasoning or action.

This separation creates operational leverage. If the output is wrong, a team can ask whether the task was misclassified, the entity was mismatched, a source was unavailable, a stale record outranked a current one, or the model reasoned incorrectly from valid evidence.

What a context package should contain

A context package is not simply a larger prompt. It is a bounded object with an explicit purpose. Its contents should vary by task, but its structure should be predictable.

The package should also distinguish observed facts from derived claims. “Invoice 418 is overdue” may come directly from the billing system. “Payment risk is elevated” is an interpretation. Both may be useful, but the agent and reviewer must be able to tell them apart.

Worked example: assembling a renewal recommendation

Suppose a sales leader asks, “Can we offer Acme a larger discount to close the renewal this week?” A retrieval-only system may find the discount policy and account notes. It could produce a plausible answer while missing decisive operational facts.

A context assembler first resolves the account and identifies the request as a commercial exception decision. It then creates an evidence checklist: current contract value and terms, proposed package, approved discount band, prior exceptions, margin constraints, invoice status, customer health, open escalations, renewal date, and approval authority.

The billing system shows an overdue invoice. The CRM describes Acme as healthy, but that field was last updated before a critical support incident. The support platform shows the incident remains open. The executed contract contains a renewal provision that differs from a summary in the CRM. The pricing policy permits the requested discount only with finance approval.

The resulting recommendation should not flatten these facts into a generic account summary. It should state that the discount is outside the sales leader’s independent authority, identify the required approver, flag the unresolved incident and overdue invoice, and use the executed contract rather than the CRM summary as the governing source. It might also propose a conditional offer: approval subject to payment resolution and an agreed support remediation plan.

The value comes from composition. No single retrieved passage answers the question. The answer emerges from joining state, policy, authority, and exceptions.

What it means in practice: retrieval metrics are no longer enough

Teams commonly test whether a retrieval system returned the expected passage. Context assembly requires broader evaluation. The unit under test is the evidence package and its effect on a decision.

• Coverage: Did the package include every mandatory evidence class for the task?
• Authority: Did governing sources outrank informal or derived records?
• Freshness: Were volatile facts queried live or checked against an explicit age limit?
• Entity integrity: Did every record belong to the correct customer, case, employee, or transaction?
• Conflict visibility: Were contradictory facts exposed rather than silently merged?
• Permission integrity: Was each piece of evidence available to the requesting user and permitted for the intended purpose?
• Decision sensitivity: If a decisive fact changes, does the recommendation change appropriately?

The last test is particularly important. Teams can construct paired cases in which only one governing fact changes: the invoice becomes current, an approval expires, or the contract version changes. If the agent produces the same recommendation, the package may be noisy or the reasoning step may be ignoring critical evidence.

The main trade-offs are operational, not cosmetic

Completeness versus latency

Querying every possible system produces slow, expensive, and cluttered interactions. The assembler should use staged acquisition. Fetch inexpensive mandatory evidence first, then request additional sources when a trigger appears. An overdue balance, for example, may trigger a deeper finance query; a standard renewal within policy may not require it.

Compression versus fidelity

Summaries reduce context size but can erase qualifications. Preserve original values for decision-critical fields, quote controlling clauses when wording matters, and attach source references to derived summaries. Compression should remove repetition, not uncertainty.

Personalization versus governance

An agent may know that an executive prefers terse briefs, but presentation preferences must not alter evidence thresholds. Personalization can control format, ordering, and explanatory depth. It should not suppress a mandatory risk, override policy, or bypass an approver.

Reuse versus contamination

Caching assembled context improves speed, but reused packages can carry stale facts or data from the wrong purpose. Cache components with explicit freshness rules rather than treating the entire package as timeless. Recheck identity, permissions, volatile state, and governing versions at execution time.

What remains unresolved

The hardest open question is how to prove that a package is sufficiently complete. A known checklist can confirm required sources, but agents will often encounter tasks for which the decisive evidence was not anticipated. Dynamic evidence planning helps, yet it also makes evaluation less deterministic.

Conflict resolution remains difficult when systems disagree and no formal source hierarchy exists. Many companies have never documented whether the contract repository, CRM, billing platform, or operational database governs a particular field. AI exposes this organizational ambiguity; it does not remove it.

Purpose limitation is another unresolved boundary. A user may have permission to view two datasets independently while combining them for a particular decision is inappropriate. Context assembly therefore needs controls based not only on identity and source access, but also on intended use.

Provenance can also become misleading after transformation. A citation to an original record does not fully explain how filters, joins, currency conversions, or summaries produced the presented claim. Mature systems will need transformation-level lineage, especially when assembled context supports consequential actions.

The operating move to make now

Select one recurring decision that currently requires employees to open several systems. Define its minimum evidence set, governing source hierarchy, freshness rules, conflict behavior, and escalation conditions. Build the context package as a reviewable artifact before automating the final action.

Keep reasoning downstream from assembly. Log which evidence was requested, obtained, excluded, transformed, and presented. Then test the package with cases where one material fact changes.

This approach reveals whether the real limitation is model performance, fragmented systems, ambiguous ownership, or missing policy. More often than expected, the agent does not need a more eloquent prompt. It needs a better-constructed view of reality.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.