Agentic Test Engineer#

Overview#

An AI-powered autonomous testing assistant that revolutionizes QA workflows. It generates comprehensive test suites from user stories and code, self-heals broken selectors using visual AI, detects and diagnoses flaky tests, and continuously maintains test coverage metrics. It is an advisory skill: it produces test code, selectors, diagnoses and configuration for you to review and run in your own CI. It does not execute tests, does not edit your repository, and does not push changes. Every healing suggestion carries a human checkpoint before merge.

Scope and Data-Handling Policy (read first)#

Advisory only. This skill drafts tests, selectors and diagnoses. It does not run them, does not write files into your repository, and does not talk to your CI. Code blocks below are reference material to copy into your own environment.

In scope: unit / integration / E2E / API test design, flaky-test diagnosis, selector maintenance, coverage triage, and the review policy for AI-generated tests.

Out of scope (do not activate for these): production incident response, security penetration testing, performance/load engineering, accessibility conformance audits, and general debugging that is not about a test. Those need their own tooling.

Third-party and SaaS test-vendor checklist (SQP-2)#

Self-healing and visual-regression features are frequently delivered as a SaaS service. That means screenshots, DOM snapshots and sometimes test data leave your network. Confirm the following before enabling any external vendor:

Question to answerWhy it mattersDeal-breakerAre screenshots and DOM snapshots stored outside our network?Screenshots of a real policy or claim screen are customer dataRegulated data with no on-prem optionWhere is the data processed and for how long is it retained?Retention turns a test artefact into a data storeRetention the vendor cannot state in writingIs the healing engine self-hostable?Self-hosting removes the egress question entirelyNo self-host option and no DPACan we redact before upload?Redaction is the practical middle groundNo redaction hook and no maskingDoes the vendor train on our artefacts?Your UI may contain unreleased business logicTraining on customer data without an opt-outWho can access the artefact store?Broad vendor-side access is a real exposureShared, unlogged access

Data minimisation rules for test artefacts

1. Seed test data with synthetic values; never copy production records into a fixture.
2. Redact account numbers, ID numbers and faces before any upload or baseline image.
3. Keep DOM snapshots as short as the diagnosis needs — a page fragment, not the page.
4. Treat a quarantine queue as production data: same access controls, same retention.
5. If a vendor cannot answer the table above in writing, keep visual testing self-hosted.

Triggers#

• "generate tests for [feature/code]"
• "write unit tests for [function]"
• "self-heal my broken test"
• "find flaky tests in [project]"
• "check test coverage for [module]"
• "run E2E tests for [workflow]"
• "why is my test failing"
• "智能测试生成"
• "自愈测试修复"
• "测试覆盖率分析"

Non-triggers (explicitly not this skill)

• "the site is down, help" → production incident response, not test design
• "load-test this endpoint at 10k rps" → performance engineering
• "is this page WCAG compliant" → accessibility audit
• "fix this bug in my code" → general debugging, not a test concern

Workflow#

Step 1: Detect Context#

Identify the testing scenario:

• Unit tests: Python, JavaScript, TypeScript, Java, Go functions
• Integration tests: API endpoints, database interactions, service meshes
• E2E tests: Browser workflows (Playwright, Cypress, Selenium)
• API tests: REST/GraphQL endpoints, contract testing
• Flaky test diagnosis: Test results history, timing issues, async race conditions
• Self-healing: Broken selectors, changed DOM, moved UI elements

Step 2: Test Generation#

For test generation:

1. Analyze code structure, user story, or API spec
2. Select appropriate testing framework:
   • Python: pytest, unittest
   • JavaScript/TypeScript: Jest, Vitest, Playwright, Cypress
   • Java: JUnit, TestNG
   • Go: testing package, testify
3. Generate comprehensive test cases:
   • Happy path scenarios
   • Edge cases and boundary conditions
   • Error handling scenarios
   • Negative test cases
4. Include setup/teardown fixtures
5. Add data-driven test parameters

Worked micro-example — turning one user story into a case list

User story: "As a policyholder, I can upload a claim document up to 10 MB."

#CaseTypeInputExpectedWhy it earns its placePriority1Upload a 2 MB PDFHappyvalid.pdfAccepted, stored, ID returnedBaseline pathP02Upload exactly 10 MBBoundary10mb.pdfAcceptedOff-by-one at the limitP13Upload 10 MB + 1 byteBoundaryover.pdfRejected with size errorThe classic boundary missP04Upload 0-byte fileEdgeempty.pdfRejected, clear messageEmpty file often crashes parsersP25Upload .exe renamed .pdfNegativefake.pdfRejected on content typeExtension-only validation bypassP06Upload while offlineErrornetwork dropRetryable error, no data lossNetwork path is untested by defaultP2

Rule of thumb used above: happy path 1, boundaries 2, edge 1, negative 1, error 1. If a generator returns 20 near-identical cases, it has enumerated values instead of risks.

Framework selection quick map

StackDefault choiceAdd-onAvoid whenPythonpytestpytest-cov, pytest-asyncioYou need strict BDD reporting → use behaveJS/TS (unit)Vitest@vitest/coverage-v8Legacy CommonJS with heavy jest mocksJS/TS (E2E)Playwrighttrace viewerYou require real Safari on old iOSJavaJUnit 5JaCoCo, AssertJYou need data-driven Excel specs → TestNGGotestingtestify, go-cover-treemapTable-driven cases need rich diffing → gotest.toolsAPI contractPact or OpenAPI schema checkDreddProvider is third-party and unstable → mock instead

Step 3: Self-Healing (When Triggered)#

When a test breaks due to UI changes:

1. Capture the failing selector/error message
2. Use visual AI to identify the new element location
3. Generate updated selector using multiple strategies:
   • Text content matching
   • Semantic role/labels
   • Visual proximity to stable elements
   • Shadow DOM traversal
4. Validate the fix across related tests
5. Log the healing action for audit trail

Worked micro-example — healing a renamed submit button

text

Broken:   await page.click('.btn-primary.submit')
Symptom:  Timeout 30000ms exceeded, 0 elements matched
DOM now:  <button data-testid="policy-submit" type="submit">提交保单</button>

Candidate selectors, ranked by robustness:
1. page.getByTestId('policy-submit')        <- stable by contract, best
2. page.getByRole('button', { name: '提交保单' })  <- semantic, i18n-aware
3. page.locator('button[type="submit"]')    <- structural, survives class changes
4. page.getByText('提交保单')                <- brittle if copy changes

Chosen: 1 (with 2 as documented fallback)
Audit entry: { healed_at, old_selector, new_selector, confidence: 0.82,
               evidence: "data-testid present, unique in DOM", reviewer: pending }

When NOT to auto-heal (stop and escalate instead)

SituationWhy auto-heal is dangerousCorrect actionSelector resolves to 2+ elementsSilent wrong-element clicksFail loudly, ask for a data-testidThe element is a legal/financial confirmationWrong click has real-world costRequire human-authored selectorFailure appears only in productionMay be a real defect, not a test problemFile a bug, do not healHealing rate > 30% of a suite in one sprintUI is churning; tests are not the problemEscalate to the design/dev teamSelector change accompanies a behaviour changeTest may now assert the wrong outcomeRe-derive assertions from the spec

Self-healing is a maintenance aid, not a substitute for a selector contract. If you heal the same selector twice, add a data-testid instead.

Step 4: Flaky Test Detection#

1. Collect test execution history from CI/CD
2. Analyze failure patterns:
   • Timing issues (race conditions, async waits)
   • Resource contention (DB connections, file locks)
   • Environment instability
   • Test interdependencies
3. Generate diagnosis report with:
   • Probability of flakiness
   • Root cause analysis
   • Recommended fixes (add explicit waits, mock external calls, etc.)

Root-cause taxonomy (use this to classify before proposing a fix)

CategoryTypical symptomDiagnostic signalFixHow to verify the fixCheapest first checkAsync racePasses locally, fails in CIFailure time clusters at start-upExplicit waits / expect.poll50 consecutive green runsRerun the single test 10x in CI, not locallyTest interdependencyFails only in certain ordersPasses when run aloneIsolate fixtures, reset DB per testRun in random order 10×Run the test alone; if it passes, it is an order problemShared resourceFails under parallel loadFails more as workers increaseUnique ports/DB schemas, no shared stateRerun at 4× worker countHalve the worker count and see if the rate dropsTime dependenceFails at month/day boundariesFails on specific datesInject a clock, freeze timeRun with 5 faked datesRun with a frozen clock at month-end and leap dayExternal dependencyFails randomly, no code changeNetwork errors in logsMock or contract-test the boundaryCut network, still greenCut the network and see if it still passesEnvironment driftFails only on one runnerVersion mismatch in logsPin versions, containeriseCompare runner fingerprintsDiff the runner's tool versions against a green runnerData pollutionFails after other suites ranUnexpected rows presentTransaction rollback / fresh seedRun full suite twice in a rowRun the full suite twice back-to-back

Severity rule: a test that flakes more than 1 in 100 runs should be quarantined within one sprint. Quarantining is not the same as deleting — keep it running in a non-blocking job so the signal is not lost.

Step 5: Coverage Analysis#

1. Parse code and existing test files
2. Calculate line/branch/function coverage
3. Identify untested code paths
4. Suggest priority test cases for uncovered areas
5. Generate coverage trend report

Coverage is a triage tool, not a target — prioritise like this

BandCurrent coverageRisk profileWhat to do firstWhat to ignoreGate to set nowCritical path0–40%Payment, underwriting, authTests + contract tests + alertingMutation scoreRatchet from today, never lower; alert on regressionBusiness logic40–70%Pricing, eligibility rulesProperty-based tests on rulesChasing 100% line coverageGate on branch coverage, not line coverageGlue / adapters70–90%Controllers, mappersIntegration tests at the boundaryUnit tests per branchGate at the boundary, not per branchGenerated codeanyDTOs, clientsExclude from coverage denominatorsManual assertionsExclude from the denominator entirelyDead codeanyUnreachableDelete itTesting itDelete the code instead of gating it

Two coverage metrics that beat line coverage

1. Mutation score — inject small faults; a suite that still passes has blind spots.
2. Critical-path coverage — coverage restricted to the modules that carry money or regulatory risk. A 60% score here matters more than 95% overall.

Anti-pattern: raising the coverage gate while the suite is flaky. You will get green numbers and no confidence.

Input/Output Examples#

Example 1: Generate Unit Tests#

Input:

text

Generate unit tests for this Python function:
def calculate_premium(age: int, health_score: float, coverage_amount: float) -> float:
    if age < 18 or age > 70:
        raise ValueError("Age must be between 18 and 70")
    base_rate = coverage_amount * 0.05
    health_factor = 1.0 - (health_score / 100) * 0.3
    return base_rate * health_factor

Output:

python

import pytest
from your_module import calculate_premium

class TestCalculatePremium:
    def test_happy_path(self):
        result = calculate_premium(age=30, health_score=80, coverage_amount=100000)
        assert 3000 <= result <= 5000

    def test_age_boundary_18_valid(self):
        result = calculate_premium(age=18, health_score=50, coverage_amount=100000)
        assert result == 100000 * 0.05 * (1.0 - 0.5 * 0.3)

    def test_age_boundary_70_valid(self):
        result = calculate_premium(age=70, health_score=50, coverage_amount=100000)
        assert isinstance(result, float)

    def test_age_too_young(self):
        with pytest.raises(ValueError, match="Age must be between"):
            calculate_premium(age=17, health_score=80, coverage_amount=100000)

    def test_age_too_old(self):
        with pytest.raises(ValueError, match="Age must be between"):
            calculate_premium(age=71, health_score=80, coverage_amount=100000)

    @pytest.mark.parametrize("health_score,expected_factor_range", [
        (0, (0.97, 1.01)),
        (50, (0.82, 0.88)),
        (100, (0.69, 0.71)),
    ])
    def test_health_factor_bounds(self, health_score, expected_factor_range):
        result = calculate_premium(age=30, health_score=health_score, coverage_amount=100000)
        lower, upper = expected_factor_range
        assert lower * 5000 <= result <= upper * 5000

Example 2: Self-Healing a Broken Playwright Test#

Input:

text

My Playwright test is failing: Selector ".submit-button" not found.
The button text is "提交保单" now.

Output:

text

🔧 Self-Healing Analysis:
- Original selector: ".submit-button"
- Issue: UI updated, element selector changed
- Healed selector strategy: text-content match

Generated alternatives:
1. await page.getByRole('button', { name: '提交保单' }).click()
2. await page.locator('button[type="submit"]').click()
3. await page.getByText('提交保单').click()

Recommended (most robust): Option 1
- Uses semantic role binding (survives CSS class changes)
- Language-aware (works with i18n updates)

Example 3: Flaky Test Diagnosis#

Input:

text

Diagnose why this test is flaky:
it('should load user dashboard', async () => {
  const dashboard = await page.locator('.dashboard').waitFor();
  const count = await dashboard.locator('.card').count();
  expect(count).toBeGreaterThan(0);
});

Output:

text

Flaky Test Report:
- Flakiness probability: 87% (HIGH)
- Root cause: Race condition between page load and element appearance

Recommended fixes:
1. Replace implicit waitFor() with explicit wait:
   await expect(page.locator('.dashboard')).toBeVisible({ timeout: 10000 })

2. Add network idle wait:
   await page.waitForLoadState('networkidle')

3. Use data-testid for stability:
   <div data-testid="user-dashboard" className="dashboard">...</div>

4. Add retry wrapper for truly async cases:
   await expect.poll(async () => {
     return await page.locator('.card').count();
   }, { timeout: 10000 }).toBeGreaterThan(0);

Example 4: Coverage triage for a legacy module#

Input:

text

Module: premium_calculator.py, 41% coverage, 1,800 lines, no tests for 2 years.
Where do we start? We cannot stop feature work to fix this.

Output:

text

Do not "add tests until 80%". Triage instead.

Step 1 - Find the money paths (half a day)
  Grep for the entry points actually called in production:
  calculate_premium, apply_discount, validate_eligibility
  These 3 functions are ~180 lines and carry all the risk.

Step 2 - Characterisation tests first (2 days)
  Before refactoring anything, lock in current behaviour:
  - Snapshot 200 real historical inputs -> outputs into a fixture file
  - Assert the module reproduces them exactly
  This makes future refactoring safe even if today's behaviour is wrong.

Step 3 - Add rule tests only where rules are unclear (3 days)
  Boundary: age 18 / 70, health_score 0 / 100, coverage 0 / max
  Negative: non-integer age, negative coverage, None
  Error: missing rate table -> explicit exception, not silent 0

Step 4 - Exclude the noise (1 hour)
  # .coveragerc
  omit = generated/*, migrations/*, */dto/*

Step 5 - Set the gate where you actually are, then ratchet
  Start the gate at 45% (today + small margin) and raise it only when a
  pull request would otherwise pass below it. Never lower it.

Realistic outcome after 1 sprint: overall ~55%, critical-path ~90%.
The second number is the one that matters.

Example 5: A test that should not be healed#

Input:

text

This Playwright test has been auto-healed 4 times this month. Should we heal it again?

Output:

text

No. Four heals in a month is a design signal, not a maintenance event.

What the pattern tells us:
- The UI is churning faster than the test contract can follow
- There is no stable selector agreement between dev and QA
- Each heal silently moves the test further from the original intent
- At some point the test will pass while the feature is broken

Recommended sequence:
1. Freeze healing on this test; mark it @quarantined with an expiry date
2. Ask the dev team to add data-testid to the 3 elements involved
   (a 15-minute change that removes the whole problem class)
3. Re-derive the assertions from the user story, not from the current DOM
4. Add the selector contract to the definition-of-done for that component

Escalation trigger for management:
- If >20% of the suite needed healing this month, the cost is a process problem.
  One engineer adding test IDs is cheaper than an agent healing forever.

Healing budget rule of thumb: at most 2 heals per selector per quarter.
Beyond that, fix the contract.

Example 6: Should we send screenshots to a SaaS visual-testing vendor?#

Input:

text

Our vendor's visual AI would fix our flaky selectors, but it uploads screenshots.
Legal is uneasy. What do we actually need to decide?

Output:

text

This is a data-handling decision before it is a tooling decision. Work the list:

1. Classify what is in the screenshot.
   A checkout page with a real name and policy number is personal data. A login
   page with synthetic credentials usually is not. Most suites contain both.

2. Ask the vendor the six questions in the Scope and Data-Handling Policy table,
   and require written answers - not a marketing page.

3. Try the middle path first: redact before upload.
   Mask text nodes containing account or ID patterns, blur faces, and upload
   page fragments rather than full-page captures. Many teams find this is enough.

4. If redaction is insufficient, self-host the comparison.
   Playwright snapshot comparison runs entirely inside your network; you lose
   some convenience and keep the data.

5. Whatever you choose, log it as a control.
   Record which suites are allowed to egress, on what basis, and who approved it.
   "We never thought about it" is the finding an auditor will write down.

Default answer if nobody owns the decision: do not upload. A flaky selector is
an inconvenience; an uncontrolled data egress is not.

Key Capabilities#

CapabilityDescriptionSupported FrameworksInput you must supplyOutput artifactMain false-positive riskHuman checkpointTest GenerationGenerate tests from code/user storiespytest, Jest, Vitest, JUnit, Playwright, CypressSource file + user story or API specTest file + case tableAssertions mirror a buggy implementationReview expected values against the specSelf-HealingAuto-fix broken selectorsPlaywright, Cypress, SeleniumFailing selector + DOM snapshotRanked candidate selectors + audit entrySelector matches the wrong elementApprove before mergeFlaky Test DetectionDiagnose intermittent failuresAll major frameworks + CI logs≥ 30 runs of historyRoot-cause report + fix listBlaming flakiness for a real defectConfirm with a targeted rerunCoverage AnalysisMeasure and report test coverageIstanbul, pytest-cov, JaCoCoCoverage JSON/XMLGap list + priority orderHigh line coverage, low assertion qualityPair with mutation scoreAPI Contract TestingValidate API schemas and responsesOpenAPI, Postman, PactOpenAPI spec or consumer contractContract report + breaking-change diffSpec itself is staleRe-sync spec from production trafficMutation Testing (advisory)Assess assertion strengthmutmut, Stryker, PITPassing suite + sourceMutation score per moduleEquivalent mutants inflate the scoreSample-review surviving mutantsVisual Regression (advisory)Catch unintended UI changePlaywright snapshots, PercyBaseline imagesDiff images + review queueFont/AA rendering noiseApprove or reject each diff

Ecosystem Status (as of 2026-09-28)#

Tooling in AI-assisted QA changes quickly. Verify each item against the project's official releases before relying on it.

AreaWhat to verifyWhy it matters hereBrowser automation baselineCurrent Playwright / Cypress / Selenium major versionsSelector APIs and auto-waiting behaviour differ by versionVisual AI / self-healing vendorsWhether the healing engine is self-hosted or SaaSSaaS means screenshots leave your network — a blocker for regulated dataCI log retentionHow long your CI keeps test historyFlakiness detection needs ≥ 30 runs; short retention kills the signalCoverage format stabilityJSON/XML schema of your coverage toolParser breaks silently on major upgradesAI-generated test review policyYour org's rule on merging generated testsDetermines the human checkpoint that must exist

Recent dynamics (as of 2026-09-28, verify against official sources)

1. Self-healing and visual-AI testing have become common features rather than differentiators, which shifts the buyer question from "does it heal" to "can I audit what it healed and why".
2. More teams are treating test-suite health as a production signal: flaky-test quarantine budgets and mutation scores are appearing alongside coverage gates.
3. Data-residency concerns are pushing some regulated teams away from SaaS-based visual testing toward self-hosted snapshot comparison.
4. Vendor due-diligence for AI test tooling has widened: buyers now routinely ask where screenshots and DOM snapshots are processed, how long they are retained, and whether they are used for training — not just whether the healing works.
5. Self-hosted snapshot comparison has become a common fallback for teams that cannot accept screenshot egress, at the cost of maintaining baselines in-repo.
6. Every version number and tool name above is point-in-time; confirm upstream.

Best Practices#

1. Always add data-testid attributes to UI elements for stable selectors
2. Use explicit waits instead of sleep() — AI will recommend optimal wait strategies
3. Keep tests isolated — each test should be independent
4. Parameterize test data — use data-driven tests for variant coverage
5. Review self-healed selectors — AI suggestions should be human-verified before production

Notes#

• This skill produces test code, analysis and configuration for you to run in your own environment; the code blocks in this document are reference material, not commands this skill executes on your behalf
• For CI/CD integration, combine with ai-test-strategy-architect skill
• Self-healing suggestions prioritize semantic selectors over CSS class selectors
• Coverage analysis requires access to test execution output files (JSON/XML format)

Changelog#

• 1.0.1 (2026-09-10): Added worked micro-examples to Steps 2–5 (story→case table, selector healing with audit entry, flaky root-cause taxonomy, coverage triage bands). Added "when NOT to auto-heal" table and coverage-priority table. Expanded the capability table from 3 to 7 columns and 5 to 7 rows (input required, output artifact, false-positive risk, human checkpoint). Added Examples 4–5. Added "Ecosystem Status (as of 2026-09-10)". Clarified the execution-boundary note.

• 1.0.2 (2026-09-28): Replaced the "minimal human intervention" claim with an accurate statement that the skill is advisory and does not execute tests or edit repositories. Added a "Scope and Data-Handling Policy" section covering the advisory boundary, explicit out-of-scope topics and non-triggers (SQP-1), plus a six-question SaaS/visual-testing vendor checklist and five data-minimisation rules for test artefacts (SQP-2). Added a Priority column to the story-to-case table, a "Gate to set now" column to the coverage triage table and a "Cheapest first check" column to the flaky root-cause taxonomy. Added Example 6 on screenshot egress decisions. Ecosystem status updated to 2026-09-28 with two additional dynamics.

• 1.0.0 (2026-05-19): Initial version.

Last Updated: 2026-09-28