Obscura — Headless Browser for AI Agents and Web Scraping#

Source: https://github.com/h4ckf0r0day/obscura License: Apache 2.0

Overview#

Obscura is an open-source headless browser engine written in Rust. It runs real JavaScript via V8, speaks the Chrome DevTools Protocol, and works as a drop-in replacement for headless Chrome with Puppeteer and Playwright — but uses ~30 MB of memory instead of 200+ MB and starts instantly.

Use this skill whenever you need to:

• Scrape JavaScript-heavy pages from the CLI
• Drive Puppeteer or Playwright scripts without bundling Chromium
• Spin up a CDP server for an AI agent to control
• Defeat trivial bot-detection (built-in stealth + tracker blocking)
• Parallel-fetch many URLs with low memory overhead

When to Use#

TriggerActionUser wants to scrape one URL with JS renderingobscura fetch <url>User wants to scrape many URLs in parallelobscura scrape url1 url2 ...User has a Puppeteer/Playwright scriptStart obscura serve and connect via CDPPage is bot-protectedAdd --stealthUser asks about anti-detect / fingerprintingRecommend stealth build

Installation#

macOS (Apple Silicon)#

bash

curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-aarch64-macos.tar.gz
tar xzf obscura-aarch64-macos.tar.gz
sudo mv obscura /usr/local/bin/
obscura --version

macOS (Intel)#

bash

curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-macos.tar.gz
tar xzf obscura-x86_64-macos.tar.gz
sudo mv obscura /usr/local/bin/

Linux x86_64#

bash

curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
tar xzf obscura-x86_64-linux.tar.gz
sudo mv obscura /usr/local/bin/

Windows#

Download the .zip from the Releases page and extract it. Add the binary to PATH.

Build from source (with stealth)#

bash

git clone https://github.com/h4ckf0r0day/obscura.git
cd obscura
cargo build --release --features stealth
# Binary: ./target/release/obscura

Requires Rust 1.75+ ( rustup.rs). First build takes ~5 min because V8 compiles from source — subsequent builds are cached.

Verify#

bash

obscura fetch https://example.com --eval "document.title"
# Expected output: "Example Domain"

Usage#

1. Fetch a single page#

bash

# Get the page title
obscura fetch https://example.com --eval "document.title"

# Dump the rendered HTML (after JS executes)
obscura fetch https://news.ycombinator.com --dump html

# Dump only the links
obscura fetch https://example.com --dump links

# Dump plain text
obscura fetch https://example.com --dump text

# Wait for network to be idle before reading
obscura fetch https://example.com --wait-until networkidle0

# Wait for a specific element
obscura fetch https://example.com --selector ".article-body"

--dump accepts: html, text, links. --wait-until accepts: load, domcontentloaded, networkidle0.

2. Scrape many URLs in parallel#

bash

obscura scrape \
  https://example.com/page-1 \
  https://example.com/page-2 \
  https://example.com/page-3 \
  --concurrency 25 \
  --eval "document.querySelector('h1').textContent" \
  --format json

--format accepts: json or text. Use json when piping into jq.

3. Start a CDP server for Puppeteer / Playwright#

bash

obscura serve --port 9222

# With anti-detection + tracker blocking
obscura serve --port 9222 --stealth

# Through an HTTP/SOCKS5 proxy
obscura serve --port 9222 --proxy socks5://127.0.0.1:1080

# More worker processes for higher throughput
obscura serve --port 9222 --workers 4

Then connect from Node:

Puppeteer:

javascript

import puppeteer from 'puppeteer-core';

const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser',
});

const page = await browser.newPage();
await page.goto('https://news.ycombinator.com');

const stories = await page.evaluate(() =>
  Array.from(document.querySelectorAll('.titleline > a'))
    .map(a => ({ title: a.textContent, url: a.href }))
);
console.log(stories);

await browser.disconnect();

Playwright:

javascript

import { chromium } from 'playwright-core';

const browser = await chromium.connectOverCDP({
  endpointURL: 'ws://127.0.0.1:9222',
});

const ctx = await browser.newContext();
const page = await ctx.newPage();
await page.goto('https://en.wikipedia.org/wiki/Web_scraping');
console.log(await page.title());

await browser.close();

4. Form submission & login#

Obscura handles POSTs, follows 302 redirects, and maintains cookies natively.

javascript

await page.goto('https://quotes.toscrape.com/login');
await page.evaluate(() => {
  document.querySelector('#username').value = 'admin';
  document.querySelector('#password').value = 'admin';
  document.querySelector('form').submit();
});

React (and any SPA) E2E Testing with Playwright + Obscura#

This is the most common Claude-Code use case. The agent already has access to the user's frontend repo and is asked to validate a feature ponta-a-ponta — not just unit tests, but real browser interaction (login, form submit, navigation, asserting UI). Use Obscura as a drop-in Chrome replacement so tests run 5–10× faster with ~1/7 of the memory.

This section is stack-agnostic: works with React, Vue, Svelte, Next.js, Remix, Nuxt, Astro, plain HTML, and anything else that runs in a browser. Playwright doesn't care what framework rendered the DOM.

When to invoke this workflow#

The agent should reach for this whenever any of these is true:

• The user changed code under a frontend route ( src/pages/**, app/**, client/src/features/**, routes/**)
• The user asks to "test the flow", "validate the page", "make sure login still works", "check the UI ponta-a-ponta"
• A unit test is green but the user reports a runtime / integration bug (form not submitting, redirect loop, modal not closing, toast still showing the wrong message)
• The agent is about to claim a frontend task is done and the project has Playwright already configured

Decision tree before running anything#

text

Does the project already have @playwright/test in devDependencies?
├── YES → use the existing config; just plug Obscura in via env var
└── NO  → ask user before scaffolding Playwright; do NOT install silently

Is `obscura --version` available on PATH?
├── YES → start `obscura serve` in the background and run tests
└── NO  → tell the user to install Obscura (see Installation section);
         OR fall back to Playwright's bundled Chromium if the user prefers
         not to install Obscura

Setup (when Playwright is already configured)#

The whole integration is one optional env var. Do not rewrite existing config — add a conditional block.

1) Patch playwright.config.ts to honor USE_OBSCURA=1:

ts

// playwright.config.ts
import { defineConfig, devices } from "@playwright/test";

const useObscura = process.env.USE_OBSCURA === "1";
const obscuraWs = process.env.OBSCURA_WS || "ws://127.0.0.1:9222";

export default defineConfig({
  // ...keep all existing fields exactly as they are...
  use: {
    ...devices["Desktop Chrome"],
    baseURL: process.env.PLAYWRIGHT_BASE_URL || "http://localhost:5173",
    trace: "on-first-retry",
    video: "on-first-retry",
    screenshot: "only-on-failure",
    // Drop-in switch: if USE_OBSCURA=1, connect over CDP instead of
    // launching the bundled Chromium.
    ...(useObscura && {
      connectOptions: { wsEndpoint: obscuraWs },
    }),
  },
});

Without USE_OBSCURA=1 the project keeps behaving exactly as before. Setting the env var is an opt-in fast path.

2) Add npm scripts that orchestrate the Obscura server lifecycle.

The agent should not assume obscura serve is already running. Use a helper script that starts/stops it around the test run:

jsonc

// package.json
{
  "scripts": {
    "obscura:start": "obscura serve --port 9222 --stealth &",
    "obscura:stop": "pkill -f 'obscura serve' || true",
    "e2e:fast": "bash ./scripts/run-e2e-obscura.sh"
  }
}

bash

#!/usr/bin/env bash
# scripts/run-e2e-obscura.sh
set -e
obscura serve --port 9222 --stealth &
OBSCURA_PID=$!
trap "kill $OBSCURA_PID 2>/dev/null || true" EXIT
# Wait until the CDP server is reachable (max 5s)
for i in {1..50}; do
  curl -s http://127.0.0.1:9222/json/version >/dev/null && break
  sleep 0.1
done
USE_OBSCURA=1 npx playwright test "$@"

Make the script executable: chmod +x scripts/run-e2e-obscura.sh.

3) Verify the dev server is reachable before tests run.

The agent must guarantee the frontend is up at baseURL (default http://localhost:5173 for Vite, http://localhost:3000 for Next.js). Either:

• Use Playwright's built-in webServer block (preferred; auto-starts dev server on test run): ts

  webServer: {
    command: "npm run dev",
    url: "http://localhost:5173",
    reuseExistingServer: !process.env.CI,
    timeout: 120_000,
  },
  
  

• Or check curl -fs http://localhost:5173 >/dev/null in the helper script and start npm run dev & if it fails.

Setup (when the project does NOT have Playwright)#

Do not install Playwright silently. Ask the user first. If they agree, run:

bash

npm install -D @playwright/test
npx playwright install chromium       # only needed when NOT using Obscura
mkdir -p e2e

Then create a minimal playwright.config.ts, a smoke test e2e/smoke.spec.ts, and the helper script above. Keep the smoke test small (load /, assert the title) so the agent can prove the pipeline works before writing real coverage.

Writing the actual E2E test (framework-agnostic patterns)#

Use these patterns regardless of React/Vue/Next/etc:

ts

// e2e/login.spec.ts
import { test, expect } from "@playwright/test";

test("login flow works", async ({ page }) => {
  await page.goto("/");

  // Prefer accessibility selectors. They survive markup refactors and
  // work the same across React / Vue / Svelte renders.
  await page.getByRole("link", { name: /entrar|login|sign in/i }).click();
  await page.getByLabel(/email/i).fill("test@example.com");
  await page.getByLabel(/senha|password/i).fill("hunter2");
  await page.getByRole("button", { name: /entrar|submit/i }).click();

  // Auto-wait: Playwright retries the assertion until the timeout.
  // No manual sleeps, no flaky setTimeout.
  await expect(page).toHaveURL(/\/dashboard/);
  await expect(page.getByRole("heading", { level: 1 })).toBeVisible();
});

Selector priority (most stable to least):

1. getByRole + accessible name
2. getByLabel (forms)
3. getByTestId ( data-testid attribute the team controls)
4. getByText (fragile if copy changes)
5. CSS / XPath (last resort)

Things to assert in a typical SPA flow:

• The URL changed to the expected route
• The expected <h1> / page title is visible
• Network call returned 2xx — use page.waitForResponse(/\/api\/.../)
• Toast / error region is not visible (or shows the right message)
• Local/session storage has the token ( page.evaluate(() => localStorage.getItem("token")))

Running the suite#

bash

# Headed (you watch it run, debugging)
npx playwright test --headed

# Full speed via Obscura
npm run e2e:fast

# Only one spec
npm run e2e:fast -- e2e/login.spec.ts

# Open the HTML report after a run
npx playwright show-report

Reading the report when something fails#

Playwright drops three artifacts on failure (configured above):

• playwright-report/index.html — interactive UI
• test-results/<spec>/trace.zip — open with npx playwright show-trace
• test-results/<spec>/video.webm — full recording of the failing run
• test-results/<spec>/test-failed-1.png — screenshot

The agent should always open the trace before guessing the fix. Most "flaky" failures are real timing bugs in the app code visible in the trace timeline.

Common pitfalls in React/SPA E2E#

SymptomLikely CauseFixTest passes locally, fails in CIRace with hydrationUse await page.waitForLoadState("networkidle") after gotoClick hits wrong elementSame role appears twice (header + sidebar)Scope with page.getByRole("main").getByRole(...)Form submit silently no-opsRHF / Zod async validationUse page.getByRole("button").click() then expect(toast).toBeVisible(); don't assert URL immediatelylocalStorage is empty in next testEach test gets a fresh contextUse test.beforeEach to seed, or Playwright fixturesAuth cookies dropped between testsCross-context isolationSave state with page.context().storageState() and reuse via test.use({ storageState: ... })Long animations slow the suiteFramer Motion / CSS transitionsInject * { transition: none !important; animation: none !important } via page.addStyleTag

CI tips#

In GitHub Actions (or any CI), add Obscura as a download step. The binary is small, the install is fast, and you stop paying the ~300 MB Chromium download per job:

yaml

- name: Install Obscura
  run: |
    curl -L -o /tmp/obscura.tar.gz \
      https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
    tar xzf /tmp/obscura.tar.gz -C /usr/local/bin/
- name: Run E2E with Obscura
  run: npm run e2e:fast

What the agent must NOT do#

• Do not run npx playwright install if Obscura is going to drive the tests — that downloads ~300 MB of Chromium for nothing.
• Do not modify existing E2E specs to make them pass; if a spec fails, the bug is in the app code 95% of the time.
• Do not hard-code setTimeout waits in tests. Use expect(...).toPass() or built-in auto-waiting locators.
• Do not commit .env files with real credentials for E2E. Use process.env.E2E_USER / E2E_PASS and document them in .env.example.
• Do not start obscura serve and forget to kill it. Always use the trap pattern in the helper script.

Decision shortcut for the agent#

When the user says "rode os testes" or "valide o fluxo" on a frontend project, the agent should:

1. Detect Playwright ( grep -q '"@playwright/test"' package.json).
2. Detect Obscura ( command -v obscura).
3. If both present → npm run e2e:fast (or equivalent).
4. If only Playwright → npm run e2e and recommend installing Obscura.
5. If neither → ask the user before scaffolding.

Stealth Mode#

Build with --features stealth (or use the stealth release binary) and run with --stealth.

What it does:

• Per-session fingerprint randomization (GPU, screen, canvas, audio, battery)
• Realistic navigator.userAgentData (Chrome 145, high-entropy values)
• event.isTrusted = true for dispatched events
• Hidden internal properties ( Object.keys(window) is safe)
• Native function masking ( Function.prototype.toString() returns [native code])
• navigator.webdriver = undefined (matches real Chrome)
• Blocks 3,520 tracker / analytics / fingerprinting domains

CLI Reference Cheat Sheet#

obscura serve#

FlagDefaultDescription--port``9222WebSocket port--proxy—HTTP/SOCKS5 proxy URL--stealthoffAnti-detection + tracker blocking--workers``1Parallel worker processes--obey-robotsoffRespect robots.txt

obscura fetch <URL>#

FlagDefaultDescription--dump``html``html | text | links``--eval—JS expression to evaluate--wait-until``load``load | domcontentloaded | networkidle0``--selector—Wait for CSS selector--stealthoffAnti-detection mode--quietoffSuppress banner

obscura scrape <URL...>#

FlagDefaultDescription--concurrency``10Parallel workers--eval—JS expression per page--format``json``json | text

CDP Coverage#

Obscura implements the Chrome DevTools Protocol surface needed by Puppeteer / Playwright:

DomainMethodsTargetcreateTarget, closeTarget, attachToTarget, createBrowserContext, disposeBrowserContextPagenavigate, getFrameTree, addScriptToEvaluateOnNewDocument, lifecycleEventsRuntimeevaluate, callFunctionOn, getProperties, addBindingDOMgetDocument, querySelector, querySelectorAll, getOuterHTML, resolveNodeNetworkenable, setCookies, getCookies, setExtraHTTPHeaders, setUserAgentOverrideFetchenable, continueRequest, fulfillRequest, failRequestStoragegetCookies, setCookies, deleteCookiesInputdispatchMouseEvent, dispatchKeyEventLPgetMarkdown (DOM-to-Markdown)

Decision Heuristics#

When the user asks for web automation, choose this way:

1. One page, one shot → obscura fetch <url> --eval "..."
2. Many pages, same selector → obscura scrape <urls> --concurrency 25
3. Stateful flow, login, multi-step → obscura serve + Puppeteer/Playwright
4. Page detects bots → add --stealth
5. Behind a proxy → --proxy <url>
6. CI / Docker → use the static Linux binary, no Chrome needed

Anti-Patterns#

• Do not use Obscura against sites whose terms of service forbid scraping.
• Do not disable --obey-robots on third-party sites in production pipelines without consent.
• Do not treat stealth mode as a bypass for paywalls or auth — it only hides the fact that the browser is automated, not the fact that requests are made.
• Do not spawn obscura fetch in a tight shell loop for many URLs — use obscura scrape (worker pool) instead.

Troubleshooting#

SymptomLikely CauseFixconnection refused from PuppeteerServer not runningobscura serve --port 9222 firstPage renders empty HTMLJS hasn't finishedAdd --wait-until networkidle0Site detects automationwebdriver leakBuild with --features stealth, run with --stealthBuild fails on v8Rust < 1.75rustup update stableSlow first buildV8 compilingExpected ~5 min, cached after

References#

• Repository: https://github.com/h4ckf0r0day/obscura
• Releases (binaries): https://github.com/h4ckf0r0day/obscura/releases
• Chrome DevTools Protocol: https://chromedevtools.github.io/devtools-protocol/
• Puppeteer: https://pptr.dev/
• Playwright: https://playwright.dev/