Fact Check: Verify Before You Ship#

Verify that every claim, reference, and technical detail is backed by current evidence. AI models have training data cutoffs and frequently hallucinate version numbers, API signatures, field names, CLI flags, and dates. This skill provides a systematic process to catch these errors before they reach users.

Why This Exists#

AI models commonly produce these types of false information:

CategoryExample of HallucinationModel names/versionsReferencing "GPT-5" or "Claude 4" when they don't exist yetAPI field namesWriting likeCount when the real API returns like_countCLI flagsUsing --recursive when the tool only supports -rLibrary methodsCalling .transformAll() on a library that has no such methodDatesGetting today's date wrong, or citing a "2024 release" that happened in 2025SDK versionsReferencing v2.0 features when the latest is v1.8Repo structureClaiming a file exists at a path where it doesn'tDefault valuesStating "default is 100" when the actual default is 50

The Verification Process#

For every piece of output that contains technical claims, run this process:

Step 1: Identify Claims#

Scan the output and tag every verifiable claim:

• Version numbers (library, SDK, model, API)
• Field names (API response, config keys, database columns)
• CLI commands and flags
• URLs and endpoints
• Dates and timelines
• Default values and limits
• File paths and function names
• "It supports X" or "X is available" statements

Step 2: Classify Each Claim#

ClassificationMeaningActionVerifiedConfirmed by tool output, API call, or file readMark with evidence sourceUnverifiedNot yet checkedMust verify before outputUnverifiableCannot be checked with available toolsLabel clearly as unverifiedStaleBased on information older than 6 monthsRe-verify from current source

Step 3: Verify Using Tools#

For each unverified claim, use the appropriate verification method:

Claim TypeVerification MethodFile exists at pathls /path/to/file or Read toolFunction/method existsgrep -r "function_name" /path/to/repoAPI field nameMake a real API call and inspect the responseCLI flag existscommand --help or man commandnpm package versionnpm info package-name versionPython package versionpip show package-nameGitHub repo infogh repo view owner/repoPR/Issue statusgh pr view NUMBER --repo owner/repoCurrent datedate commandURL is reachablecurl -s -o /dev/null -w "%{http_code}" URLGit branch/tag existsgit ls-remote --tags origin

Step 4: Label the Output#

After verification, every claim in the output should be one of:

• Verified (with evidence: "confirmed by npm info output")
• Unverified (explicitly marked: "not yet confirmed")
• Corrected (original was wrong, replaced with verified data)

Do / Don't Checklist#

Do#

• Verify every version number mentioned in output with a tool or API call
• Verify URLs you generate with curl -s -o /dev/null -w "%{http_code}" URL before recommending
• Never guess URLs — if you can't see it, ask the user to provide it
• For user-provided URLs: only check HTTP status code ( -o /dev/null), don't fetch or render content from untrusted sources (phishing/malware risk)
• Check today's date with date if you reference it
• Run actual API calls to confirm field names (don't guess from memory)
• Use --help or docs to confirm CLI flags exist
• Check if URLs return 200 before recommending them
• Mark claims as "unverified" if you cannot check them
• Date your verification ("verified on 2026-04-11")
• Re-verify if the source information is older than 6 months
• Verify PR/issue status with gh commands, not from memory

Don't#

• Don't trust your memory for version numbers, API shapes, or dates
• Don't assume an API field name — always verify from real output
• Don't report "I fixed it" without verifying CI passes ( gh pr checks)
• Don't report "PR has reviews" without checking ( gh pr view)
• Don't cite documentation without confirming the URL exists
• Don't use old StackOverflow answers as current truth — platforms change
• Don't claim "push succeeded" means "task complete"
• Don't state defaults or limits without checking the actual source code or schema

Common Hallucination Patterns to Watch For#

Pattern 1: Outdated Model/Version References#

AI models are frozen at their training cutoff. Always verify:

bash

# Check latest version of a package
npm info @anthropic-ai/sdk version
pip show openai | grep Version
gh release list --repo owner/repo --limit 3

Pattern 2: Invented API Fields#

Never guess API response field names. Always run the API and read the actual response:

bash

# Real API call to verify field names
curl -s "https://api.example.com/endpoint" | jq '.[0] | keys'

Pattern 3: False Success Reporting#

The most dangerous hallucination. "I did it" when the work is incomplete:

What was saidWhat it actually meansHow to verify"Pushed successfully"Git push workedgh pr checks — is CI green?"Responded to review"Committed a changeDoes the new code actually fix the issue?"Found 10 problems"Listed 10 itemsRead the code for each — do they exist?"PR has been reviewed"Someone looked at itgh pr view --json reviews — what's the count?

Pattern 4: Phantom References#

Referencing things that don't exist:

bash

# Verify a file exists before referencing it
ls /path/to/claimed/file

# Verify a function exists in a codebase
grep -r "functionName" src/

# Verify an npm package exists
npm info claimed-package-name

Pattern 5: Date Confusion#

AI models often get dates wrong, including today's date:

bash

# Always verify current date
date "+%Y-%m-%d"

# Check when a release was published
gh release view v1.0.0 --repo owner/repo --json publishedAt

Pattern 6: URL Guessing#

AI models will fabricate plausible-looking URLs when they don't know the real one:

Fabricated URLWhy it looks rightActual URLhttps://auth.openai.com/activateFollows common OAuth patternshttps://auth.openai.com/codex/device``https://docs.clawhub.aiStandard docs subdomainDoes not exist (returns 404)https://api.example.com/v2/schemaCommon API conventionEndpoint may not exist

Prevention:

• Never guess URLs. If you can't see the full output, ask the user to expand it.
• If recommending a URL, verify it with curl -s -o /dev/null -w "%{http_code}" URL first.
• Say "I don't have the URL — can you check the terminal output?" instead of guessing.
• URLs are not inferrable from patterns. auth.openai.com/activate vs auth.openai.com/codex/device — one character path difference, completely different endpoints.

Verification Report Format#

When fact-checking, produce a report like this:

markdown

## Fact Check Report

| # | Claim | Status | Evidence |
|---|-------|--------|----------|
| 1 | "API returns `like_count`" | Verified | Real API run, dataset abc123 |
| 2 | "Latest version is v2.3" | Corrected → v2.5 | `npm info package version` |
| 3 | "Supports `--recursive` flag" | Unverified | Cannot test without install |
| 4 | "Default timeout is 30s" | Verified | Source code line 42 |

Tips#

• The more confident you feel about a claim, the more important it is to verify. Confidence ≠ correctness.
• "I'm pretty sure" is not evidence. Run the command.
• Verification has a cost (time, API calls), but false information has a higher cost (broken code, lost trust, wasted user time).
• When in doubt, say "I'm not sure — let me verify" instead of guessing.
• If you can't verify something, say so explicitly. An honest "unverified" is better than a confident hallucination.