Flow Test#

Use this skill to design tests for tasks that cannot be validated reliably with traditional unit-test assertions alone.

This skill is for flow testing: the agent performs a realistic task, records key evidence from the process, and then judges success with an explicit semantic rubric.

Invoke this skill when:

• the task depends on live or changing web content
• the output can vary but still be correct
• the workflow spans multiple model or tool steps
• intermediate evidence matters more than one exact final string
• you need to verify user intent was satisfied, not exact wording

Do not use this skill when:

• the result is deterministic and easy to assert directly
• a schema check, exact match, snapshot, or pure function test is enough
• the requirement can be covered fully by normal unit or integration tests

Objective#

Turn a fuzzy requirement into a test design that combines:

• deterministic checks for stable invariants
• evidence collection for dynamic execution
• semantic evaluation for variable outcomes
• a bounded verdict of pass, fail, or needs_review

Design Principles#

1. Keep asserts where they still work#

Do not replace traditional tests blindly. Preserve exact checks for stable facts such as:

• tool call success
• required fields
• minimum counts
• status codes
• domain restrictions
• date or freshness constraints when machine-checkable

2. Judge task completion, not exact phrasing#

Prefer questions like:

• did the agent reach the right source
• did it gather relevant information
• does the final answer satisfy the user request

Avoid requiring one exact string unless the wording itself is the requirement.

3. Require inspectable evidence#

Ask the execution flow to print or capture concise evidence such as:

• visited URL
• page title
• visible headings
• extracted entities
• timestamps or date clues
• key tool outputs
• final answer

The evaluator should be able to inspect why a verdict was reached.

4. Use explicit semantic rubrics#

Never rely on vague instructions such as "judge whether it looks good."

Always define:

• what evidence is required
• what counts as a pass
• what clearly fails
• when uncertainty should become needs_review

5. Prefer bounded confidence#

If evidence is incomplete, contradictory, or too weak, do not force a pass.

Return needs_review.

Workflow#

When invoked, design the test in the following order.

1. Identify why exact assertions are brittle#

Classify the task:

• dynamic web browsing
• search or retrieval
• LLM generation
• multi-tool orchestration
• end-to-end user flow

Then explain why literal equality or fixed snapshots are not sufficient.

2. Split deterministic checks from semantic checks#

Write two groups:

Deterministic Checks#

Use exact validation for stable parts, such as:

• tool returned successfully
• required fields are present
• minimum number of results exists
• source domain matches expectation
• response includes a valid date range

Semantic Checks#

Use agent evaluation for variable parts, such as:

• relevance to the requested topic
• freshness of the retrieved content
• whether the answer reflects the gathered evidence
• whether the workflow actually satisfies the intended task

3. Define the evidence schema#

Specify exactly what the run should log or output.

Recommended evidence fields:

• task
• source_url
• source_title
• extracted_items
• freshness_signals
• intermediate_results
• final_answer
• evaluator_notes

Keep evidence minimal but sufficient for review.

4. Define the verdict rubric#

Use this baseline:

Pass#

• the agent reached a relevant source or completed the intended flow
• collected evidence supports the conclusion
• the final output is relevant and sufficiently current for the task
• there is no major contradiction between evidence and answer

Fail#

• the agent failed to reach a relevant source or complete the flow
• the result is clearly irrelevant, stale, or fabricated
• the output contradicts the evidence
• the workflow misses a required user objective

Needs Review#

• evidence is partial or ambiguous
• freshness cannot be determined confidently
• multiple interpretations remain plausible

5. Produce a structured test spec#

Return the design in this format:

markdown

## Test Intent

## Why Exact Assert Fails

## Deterministic Checks

## Evidence To Collect

## Semantic Rubric

## Execution Notes

## Final Verdict Format

Output Template#

markdown

## Test Intent
- Validate that:

## Why Exact Assert Fails
- Dynamic factors:
- Why literal equality is brittle:

## Deterministic Checks
- Check 1:
- Check 2:

## Evidence To Collect
- Evidence 1:
- Evidence 2:

## Semantic Rubric
- Pass when:
- Fail when:
- Needs review when:

## Execution Notes
- Constraints:
- Allowed variance:
- Safety concerns:

## Final Verdict Format
- verdict: pass | fail | needs_review
- reason:
- evidence:

Example#

Task: verify that visiting a news site returns today's news rather than stale content.

Good test design:

• deterministic checks confirm the page loads and at least one article item is collected
• evidence includes the visited site, page title, visible headlines, date clues, and final summary
• semantic rubric passes when the result clearly reflects same-day or current reporting from the visited source
• semantic rubric fails when headlines are outdated, unrelated, or invented
• semantic rubric returns needs_review when freshness cannot be established from the evidence

Bad test design:

• assert returned_text == "Today's news is ..."

Guidance#

When using this skill:

• keep traditional asserts for stable invariants
• use semantic evaluation only where exact matching becomes brittle
• prefer narrow rubrics over subjective judgment
• require visible evidence before passing the test
• state uncertainty explicitly instead of masking it

Deliverables#

When asked to design a flow test, provide:

• a structured test spec
• deterministic checks
• an evidence schema
• a semantic rubric
• a final verdict format