Mutation Test Runner#

Line coverage lies. 80% coverage with weak assertions catches nothing. Mutation testing injects real bugs into your code and checks whether your tests catch them — giving you a genuine measure of test suite quality.

Use when: "how good are our tests really", "mutation testing", "test suite quality", "are these tests actually catching bugs", "mutation score", or when line coverage is high but bugs still slip through.

Commands#

1. run — Execute Mutation Testing#

Step 1: Detect Language and Test Framework

bash

# Detect project language
ls package.json 2>/dev/null && echo "NODE"
ls pom.xml build.gradle 2>/dev/null && echo "JAVA"
ls setup.py pyproject.toml 2>/dev/null && echo "PYTHON"
ls go.mod 2>/dev/null && echo "GO"
ls Cargo.toml 2>/dev/null && echo "RUST"

Step 2: Install and Run Mutation Tool

JavaScript/TypeScript (Stryker):

bash

npx stryker init 2>/dev/null || npm install --save-dev @stryker-mutator/core
npx stryker run --reporters clear-text,html 2>&1

If Stryker unavailable, manual mutation approach:

bash

# Find test files
rg -l "describe\(|test\(|it\(" --type ts --type js -g '!node_modules' 2>/dev/null

# Find source files with tests
rg -l "export (function|class|const)" --type ts --type js \
  -g '!node_modules' -g '!*.test.*' -g '!*.spec.*' 2>/dev/null

Python (mutmut):

bash

pip install mutmut 2>/dev/null
mutmut run --paths-to-mutate=src/ 2>&1
mutmut results 2>&1

Java (PIT):

bash

# Maven
mvn org.pitest:pitest-maven:mutationCoverage 2>&1
# Gradle
./gradlew pitest 2>&1

Go (gremlins):

bash

go install github.com/go-gremlins/gremlins/cmd/gremlins@latest
gremlins unleash 2>&1

Step 3: If No Tool Available — Manual Mutation

For any language, apply these mutation operators manually to critical files:

OperatorOriginalMutantWhat it testsNegate conditionalif (x > 0)``if (x <= 0)Boundary checksRemove callvalidate(input)``// removedSide effect coverageChange returnreturn true``return falseReturn value assertionsFlip booleanenabled = true``enabled = falseFlag-dependent logicBoundaryi < arr.length``i <= arr.lengthOff-by-one coverageRemove throwthrow new Error()``// removedError handling testsNull returnreturn result``return nullNull safety tests

For each mutation:

1. Apply the mutation to source code
2. Run the test suite
3. If tests PASS → mutation survived (test gap found!)
4. If tests FAIL → mutation killed (tests are effective)
5. Revert the mutation

bash

# Example: mutate a function and run tests
cp src/validator.js src/validator.js.bak
# Apply mutation (e.g., flip a condition)
sed -i 's/if (age >= 18)/if (age < 18)/' src/validator.js
npm test 2>&1
RESULT=$?
# Restore original
mv src/validator.js.bak src/validator.js
if [ $RESULT -eq 0 ]; then
  echo "⚠️  SURVIVED: flipping age check — no test catches this!"
else
  echo "✅ KILLED: tests caught the flipped condition"
fi

Step 4: Analyze Results

markdown

# Mutation Testing Report

## Summary
- **Mutation score:** 72% (target: 80%+)
- **Total mutants:** 145
- **Killed:** 104 (tests caught the bug)
- **Survived:** 38 (test gaps!)
- **Timed out:** 3 (infinite loops — likely caught)

## Survived Mutants (test gaps)
### Critical (business logic)
1. `src/billing/calculator.js:47` — Changing `>=` to `>` not caught
   → Missing boundary test for exact threshold value

2. `src/auth/validator.js:23` — Removing `validateToken()` call not caught
   → No test verifies token validation actually runs

3. `src/api/handler.js:91` — Changing `return 403` to `return 200` not caught
   → Authorization test doesn't check response status code

### Recommendations
1. Add test: `expect(calculate(100)).toBe(...)` for exact boundary
2. Add test: verify `validateToken` is called (spy/mock)
3. Fix auth test: assert response.status === 403

## File-Level Mutation Scores
| File | Mutants | Killed | Score | Priority |
|------|---------|--------|-------|----------|
| src/billing/calculator.js | 23 | 12 | 52% | 🔴 Critical |
| src/auth/validator.js | 18 | 14 | 78% | 🟡 Medium |
| src/api/handler.js | 31 | 28 | 90% | 🟢 Good |

2. suggest — Generate Tests for Surviving Mutants#

For each surviving mutant, generate the specific test that would kill it:

javascript

// Surviving mutant: src/billing/calculator.js:47
// Mutation: changed >= to >
// Generated test:
test('calculates discount at exact threshold', () => {
  // The boundary value that the mutation changes behavior for
  expect(calculateDiscount(100)).toBe(10); // exactly at threshold
  expect(calculateDiscount(99)).toBe(0);   // just below
  expect(calculateDiscount(101)).toBe(10); // just above
});

3. baseline — Establish Mutation Score Baseline#

Run mutation testing and save results as a baseline for tracking improvement over time. Set a minimum mutation score gate for CI.