The Evidence Ladder: A Beginner’s Guide to What AI Should Trust

An AI assistant often receives several plausible versions of reality at once. A manager asks it to expedite an order. The customer record shows a standard service tier. A sales note promises priority handling. An old email suggests a special exception. Which source should govern the decision?

This is not merely a retrieval problem. Finding more information can make the conflict worse. The operational problem is deciding what evidence deserves authority, what evidence provides context, and what evidence should trigger a question rather than an action.

An evidence ladder is a ranked model for making that choice. It tells an AI which sources to trust first, how to handle disagreement, and when no available source is strong enough to support a decision.

The Core Vocabulary

Beginners should separate five concepts that are often collapsed into the word context.

• Evidence: information relevant to a decision, such as a signed contract, account record, message, or observed system state.
• Authority: the right of a source to define what is permitted or required. A policy may be authoritative even when another source is more recent.
• Reliability: the likelihood that a source accurately represents reality. A live inventory count may be more reliable than a manually maintained spreadsheet.
• Recency: how current the information is. Recency matters, but it does not automatically confer authority.
• Corroboration: independent support for a claim. Two systems repeating data from the same original record are not independent corroboration.

These distinctions prevent a common design error: allowing the newest, most detailed, or most confidently worded statement to win by default.

A Practical Mental Model: Rank Sources Before Claims

The evidence ladder ranks source classes before evaluating individual claims. A useful starting structure is shown below. The exact order should reflect the business process.

The ladder is not a universal truth. In a fraud investigation, a locked audit log may outrank a customer-facing account record. In incident response, live telemetry may temporarily outweigh a stale runbook. The principle is stable: make precedence explicit before the AI encounters a conflict.

The ladder should also be scoped to a decision. A CRM may be authoritative for a customer’s assigned account owner but not for contractual payment terms. Trust belongs to a source-field-decision combination, not to an entire application.

How the AI Should Evaluate a Claim

Once the source hierarchy is defined, the AI can process each important claim through a short sequence.

1. Identify the decision. “Should this order ship?” is clearer than “Handle the account.”
2. Extract governing claims. Examples include payment received, stock available, account not blocked, and requester authorized.
3. Attach provenance. Record where each claim came from, when it was updated, and whether it is directly observed or inferred.
4. Apply precedence. Compare conflicting claims using the relevant evidence ladder.
5. Test sufficiency. Determine whether the winning evidence actually supports the proposed action.
6. Act, ask, or escalate. Proceed only when the required claims are adequately supported.

This mechanism matters because an AI can reach a correct-looking answer through an unsafe path. If it approves a refund based on an informal note when the policy record was unavailable, the favorable outcome does not validate the method.

A Worked Example: Expediting a Customer Order

Suppose an account manager writes: “Please send Acme’s order overnight. We promised priority treatment.” The AI finds four relevant items:

• The signed agreement provides standard delivery.
• The CRM labels Acme as a strategic account.
• A sales note says, “Offer priority shipping when needed.”
• The fulfillment system shows that overnight delivery will require an exception approval.

A weak system blends these facts into a narrative and concludes that expedited shipping is appropriate. A disciplined system separates objective, authority, and operational state.

The account manager’s instruction establishes the desired outcome. The strategic-account label explains why the request may be reasonable. The sales note supplies historical context, but it does not necessarily authorize cost. The fulfillment system identifies a required control. The contract does not prohibit expedited delivery, yet it does not grant it either.

The correct next step is therefore not automatic shipment or outright refusal. The AI should route the exception to the role permitted to approve the extra service level, presenting the order, expected operational consequence, and supporting account context. The ladder has converted an ambiguous request into a bounded decision.

Notice the trade-off: escalation adds delay, but acting on weak authority creates an uncontrolled commercial commitment. If the business handles this exception frequently, the longer-term fix is not better prompting. It is a documented approval rule.

Design the Ladder Around Conflict

Teams often document where information lives but omit what happens when sources disagree. The disagreement rules are the operational value of the ladder.

Authority conflict

An authorized user requests something prohibited by policy. The AI should explain the boundary and offer permitted alternatives. User seniority should not silently override a control unless the control explicitly recognizes that authority.

State conflict

Two systems report different inventory quantities. The AI should prefer the designated system of record, check timestamps and synchronization status, then abstain if the discrepancy could materially affect fulfillment.

Temporal conflict

An older contract and a newer email specify different terms. The newer email is not automatically controlling. The AI must determine whether the sender had authority to amend the agreement and whether the required amendment process occurred.

Inference conflict

The model’s interpretation contradicts a structured field. For example, a message sounds urgent, but the case priority is marked normal. The AI may flag the mismatch; it should not quietly rewrite the priority unless the workflow permits that change.

Your First Implementation

Start with one consequential decision, not an enterprise-wide knowledge map. Good candidates include issuing a refund, changing a delivery date, approving a discount, or closing a support case.

1. Name the decision and possible actions. Include the option to ask or escalate.
2. List the claims required for each action. A refund might require purchase verification, eligibility, amount, payment method, and authorization.
3. Map each claim to its authoritative source. Specify the field or record type, not merely the application.
4. Define fallback sources. State whether they can support action or only inform review.
5. Write conflict rules. Cover stale data, missing records, contradictory instructions, and unauthorized requests.
6. Set evidence thresholds by consequence. Drafting a response may tolerate uncertain context; transferring funds should not.
7. Log the decision path. Preserve the claims used, their provenance, unresolved conflicts, and the resulting action.

Test the design with adversarial cases. Give the AI a persuasive email that conflicts with policy, a stale record that appears authoritative, and two systems that inherited the same incorrect value. The goal is not to see whether it finds an answer. The goal is to see whether it recognizes when the evidence cannot support one.

What to Ignore for Now

Do not begin by assigning numerical trust scores to every document. Apparent precision can conceal an undefined policy. First establish source roles, precedence, and abstention behavior in plain language.

Do not attempt to solve every contradiction through model reasoning. Some conflicts require transaction controls, identity checks, fresh system queries, or human approval. A fluent explanation is not a substitute for verification.

Do not ingest all available company content. More retrieval increases the number of irrelevant, duplicated, stale, and unauthorized claims the AI must reconcile. Retrieve according to the decision and its required evidence.

Finally, do not treat the ladder as permanent. Systems of record change, approval rights move, and workflows acquire new exceptions. Review the ladder when a process changes or when decision logs reveal recurring conflicts.

The Operating Standard

A well-designed AI does not merely cite sources. It understands the role each source is allowed to play. It can distinguish a binding rule from a useful precedent, a live state from a remembered fact, and an authorized instruction from a persuasive suggestion.

The practical standard is simple: for every consequential action, the AI should be able to show which claims mattered, where they came from, why those sources were entitled to govern, and what uncertainty remained. That is the foundation for an assistant that understands the business before it commits the business.

This post was drafted with AI assistance and reviewed against our editorial policy before publication. Corrections are made at the source, on the page, with the date shown.