You attach a document and ask an AI for a news briefing. It says, “There were no incidents,” but the document says nothing about incidents. A fluent sentence is not evidence. A claim the document does not address differs from a claim the document disproves.
How do you identify a claim unsupported by the attachment?
Consider this fictional article note:
The new version will roll out gradually starting Tuesday.
No date has been announced for completion of the rollout.You can answer “When does the rollout start?” with “Tuesday.” For “Has it reached everyone yet?”, the note gives no completion date, so the answer is “This note does not establish that.” The same applies to “Were there no incidents?” Silence about incidents is not evidence that none occurred.
| Question | Evidence in the note | Answer |
|---|---|---|
| When does rollout begin? | Starting Tuesday | Tuesday |
| Is rollout complete? | Completion date unannounced | Cannot determine |
| Were there no incidents? | No mention | Cannot determine |
The diagram's supported answer and deferred answers all come from this one note. Deferring a claim is a useful way to stay within the evidence.
The note supports Tuesday as the start. It does not establish that rollout is complete or that no incident occurred. Missing mention is not a negative finding.
What boundary should the prompt state?
“Answer accurately” is vague. State the allowed source and the condition for withholding an answer. For example:
Answer the question using only the note below.
For each answer, identify the sentence that supports it.
If the note provides no support, do not guess. Say: “This note does not establish that.”
Question: Has the rollout finished?
Note: The new version will roll out gradually starting Tuesday. No date has been announced for completion of the rollout.An LLM-provided supporting sentence is not proof by itself. Check that the sentence actually appears in the input and is strong enough for the claim. “Rollout started” does not mean “rollout finished.”
Which failures should a test catch first?
Use the three questions above as a small test set. If the model answers the last two with certainty, the uncertainty rule failed. Remove the completion detail from the note, then add an explicit completion sentence, and compare the outputs. If the answer never changes, the model may be favoring a familiar story over the document.
In a real service, retain the link between each answer and its evidence, but do not casually log the full attachment: it may contain personal data. If external search is available, label “attachment only” and “supplemented by search” as separate modes so sources do not blur together. Important conclusions still need a person to check the original and approve them.
Key takeaways
Information missing from a document is unverified, not necessarily false. Test supported facts, unconfirmed completion, and absent mention side by side, then check that each cited sentence supports the answer's strength. Keep external search and human approval distinct from attachment-grounded answers.
