playwright-ai-qa-agents

A Claude Code plugin, playwright-qa-agents, that turns "test this feature" into reviewed, runnable Playwright tests in TypeScript. Agents plan the scenarios, stop for a human to review the plan, then write the API clients, page objects and specs, run them, and repair broken UI locators within a fixed budget.

Docs site (this README, the worked example and the changelog)

By Ben Weese · more projects at code.benweese.com

How it fits

It is one of three repos that work together:

The workflow

flowchart LR
    B["description of the<br/>feature or change"] --> M["page-mapper<br/>(UI only)"]
    M --> P[planner]
    B --> P
    P --> R{{"human review<br/>of plans/&lt;feature&gt;.md"}}
    R --> G[generator]
    G --> X[run]
    X -->|"locator / timing failure (UI)"| H["healer<br/>max 3 attempts"]
    X -->|"site disagrees with plan"| BUG[bug report]

The plan opens with numbered acceptance criteria (AC-1, AC-2, ...), and each scenario lists the criteria it covers. Each scenario also gets an id, a number unique across the repo (#0012), that goes into its test title. Ids don't encode the kind of test; tags such as @smoke, @api and @ui do. A failing test points straight back to its written steps, its expected result and the requirement behind it.

docs/worked-example.md follows one real feature, the cart in playwright-e2e-demo, through every stage, from page map to the healer's final diff.

Requirements

• Claude Code
• A test repo laid out like playwright-e2e-demo (see What the test repo needs)

Install

From a terminal, in the test repo:

claude plugin marketplace add benweese/playwright-ai-qa-agents
claude plugin install playwright-qa-agents@playwright-ai-qa-agents

The terminal works everywhere. The /plugin command isn't available in every client (the VS Code extension doesn't have it); where it is, these do the same:

/plugin marketplace add benweese/playwright-ai-qa-agents
/plugin install playwright-qa-agents@playwright-ai-qa-agents

Plugins load when a session starts. After installing, start a new Claude Code session before using the agents. In VS Code, run Developer: Reload Window from the command palette, or open a new Claude Code tab.

To suggest the plugin to everyone who opens the test repo, also name it in the repo's .claude/settings.json:

{
  "extraKnownMarketplaces": {
    "playwright-ai-qa-agents": {
      "source": {
        "source": "github",
        "repo": "benweese/playwright-ai-qa-agents"
      }
    }
  },
  "enabledPlugins": {
    "playwright-qa-agents@playwright-ai-qa-agents": true
  }
}

This records the marketplace and the plugin, but it doesn't install it on its own: in the VS Code extension the plugin didn't load from this file alone, only after the terminal install above. Keep the terminal steps in the test repo's README.

To pick up a new version, run claude plugin update playwright-qa-agents@playwright-ai-qa-agents (or /plugin update), then start a new session.

Check it loaded

Claude Code no longer has an /agents command. Instead, ask Claude in the new session:

List the subagents and skills from the playwright-qa-agents plugin.

It should name six agents (page-mapper, api-test-planner, ui-test-planner, api-test-generator, ui-test-generator, ui-test-healer) and the skills in the Skills table, each prefixed playwright-qa-agents:. If it finds none, the plugin isn't installed or the session started before the install: run claude plugin list in a terminal, then start a new session.

What the test repo needs

• The layout used by playwright-e2e-demo: api/ clients, tests/api/ and tests/ui/ specs, schemas/, plans/ and docs/pages/
• TypeScript, with playwright.config.ts at the repo root
• playwright-test-commons v0.7.0 or later (typed API responses, numeric test ids, and the BasePage checks, accessibility scan and uniqueEmail the generators use)
• @playwright/cli as a dev dependency, for the agents that drive a browser
• npm run lint that includes the type-check (and ideally JSDoc rules)
• A CLAUDE.md describing the repo's rules, helpers and site quirks, ideally split into a short root file and a guide per folder (api/, tests/api/, tests/ui/, plans/, ...) so each session loads only what it needs. Agents follow them over their own defaults

Usage

Start the whole flow with a description of the feature or change, a page URL, or a path to a requirements file:

/playwright-qa-agents:author-tests the product search page and POST /api/searchProduct

It maps the page, drafts plans/<feature>.md and stops for your review. Once you approve the plan, it generates the tests, runs them, and heals UI locator failures.

Agents

Skills

Guardrails

These rules exist because AI test authoring fails in predictable ways:

• Plan, review, then generate. Duplicate scenarios, untestable criteria and wrong assumptions are cheapest to catch in a plan, before code exists.
• Assert the plan, not what you see. An agent driving a live site will happily encode a bug as expected behaviour. Generators keep the plan's assertion, leave the test failing, and report the difference. The healer never changes an assertion to match the page: it marks the test test.fixme() with the expected and actual result, so the suite stays green while the bug is tracked in the test itself.
• A healing budget. Three attempts per test, then a // Fixme: note saying what was tried. Without a limit, a healer flips between locators forever.
• Two-tier locators. Elements are acted on through getBy*, plus whatever the test repo's tests/ui/CLAUDE.md also allows (such as stable data-* attributes). When no good locator is unique on its own, the agent scopes to a good container first (the nearest meaningful parent, such as a form, dialog, row or card) and finds the element inside it. No XPath, generated class names or .nth().
• Parallel, not serial. Following Playwright's best practice, every test is isolated and can run in parallel. One failure in a serial block skips the rest and hides what they'd find, so a dependent journey is one test with a test.step per stage, and tests that share data create it per worker.
• Clean up and go easy on shared sites. Each agent uses its own named browser session, records and deletes any account or record it creates, and never raises the worker count against a site other people use.
• Shared helpers go to commons. A helper any site could use belongs in playwright-test-commons, not the test repo; generators report it as a commons candidate instead of adding a local helper file.
• The test repo's CLAUDE.md wins. Agents read the repo's own rules, helpers and site quirks before applying their defaults, so the plugin works across different suites.

What review caught

These cases come from building playwright-e2e-demo (54 tests plus setup and teardown). Each one is a place where the human checkpoint or a guardrail changed what shipped. docs/worked-example.md follows one feature through every stage.

Limitations

What the agents get wrong

From the playwright-e2e-demo build. The guardrails above reduce these, but don't remove them:

• They copy what they see into what they expect. A planner with a page map open will write today's text as the expected result. The "requirement wins over the page" rule catches most cases. The reviewer still has to check the expected results against the source.
• They write assertions that pass with nothing working: a search test that accepts empty results, an update test that trusts the success message.
• They reach for serial mode and shared state when tests depend on each other.
• They suggest locators the rules forbid, or techniques nobody has tried. The page map recommended an id plus a class name, and the planner suggested a visibility filter that didn't work.
• They miss timing that depends on the site's own scripts. The page-mapper noted a click that "landed before page scripts were ready", and nothing downstream acted on it until #0037 failed. The healer fixes such failures afterwards, but it can't always reproduce them, and its prompt has no specific guidance yet for controls whose handlers bind late.
• They treat a few passing runs as proof. --repeat-each=3 doesn't prove a rare race is gone.
• They leave data behind when a script fails partway, before the record's details were saved.
• They can't tell an overloaded site from a broken test without being told to look for 503s.

Convention or enforced

Every guardrail in the agent prompts is a convention: the model is asked to follow it, and nothing stops it if it doesn't. What enforcement exists lives in the test repo's tooling, not in this plugin. playwright-e2e-demo enforces these:

A test repo without these lint rules gets only the conventions. This repo's own CI validates the plugin manifests and checks that their versions match. It doesn't test agent behaviour.

Cost

Token use from the playwright-e2e-demo build, read from its session logs: 9 pages mapped, 14 API and 38 UI tests planned and generated, 1 test healed. All agents ran on Sonnet. Most input is cache reads, which cost far less than fresh input.

The session that ran the agents and the reviews used more than all of them together: 654k fresh input, 31.0M cache reads and 166k output, on Opus. That session also rewrote the e2e README, set up CI and moved helpers into commons, so not all of it is pipeline cost. Expect orchestration to cost about as much as the agents.

Installed, the plugin adds about 900 tokens to every session (skill and agent descriptions). Each agent's prompt adds 1–2k tokens when it runs, and the bundled playwright-cli skill about 6k (from claude plugin details).

When not to use it

• There's no written requirement. The planners need docs, published test cases or a ticket to assert against. With only the live page, you get tests that encode today's behaviour, bugs included.
• The site has real users, payments or data you can't delete, including production. The agents create accounts and submit forms on the live site.
• Nobody will review the plan. Most of the cases above were caught at review.
• You need one or two quick tests. Mapping and planning cost more than writing them by hand.
• The test repo isn't set up for it: not laid out like playwright-e2e-demo, not using playwright-test-commons, not TypeScript, or no lint. The generators assume the layout and the BasePage API, and without lint nothing enforces the conventions.
• The environment is flaky or overloaded. The healer will chase infrastructure failures as if they were locator problems.

Bundled: Playwright's playwright-cli skill

skills/playwright-cli/ is Playwright's own skill for @playwright/cli: the full command reference, plus guides on element attributes, request mocking, storage state, tracing and more. It is copied unchanged from the npm package and keeps its Apache-2.0 LICENSE. Don't edit it here. To update it to the latest @playwright/cli:

tmp=$(mktemp -d)
npm pack @playwright/cli@latest --pack-destination "$tmp"
tar -xzf "$tmp"/playwright-cli-*.tgz -C "$tmp"
rm -rf skills/playwright-cli
cp -r "$tmp/package/skills/playwright-cli" skills/
cp "$tmp/package/LICENSE" skills/playwright-cli/

Layout

.claude-plugin/plugin.json        the plugin's name, version and description
.claude-plugin/marketplace.json   makes this repo a plugin marketplace (source: ./)
agents/                           page-mapper, planners, generators, healer
skills/author-tests/              the full authoring flow
skills/run-spec/  skills/check/  skills/bug-report/  skills/update-docs/
skills/playwright-cli/            Playwright's skill, vendored (see above)
docs/worked-example.md            one feature followed through every stage
CHANGELOG.md                      what changed in each version
.github/workflows/validate.yml    validates the plugin, and that both manifests have the same version

Development

claude plugin validate .

After editing, start a new Claude Code session (or run /reload-plugins where your client has it) to load the changes.

Releasing: bump version in both .claude-plugin/plugin.json and .claude-plugin/marketplace.json (CI fails if they differ), add an entry to CHANGELOG.md, push to main, then tag the commit v<version> and push the tag. Users get it with claude plugin update playwright-qa-agents@playwright-ai-qa-agents (or /plugin update where their client has it), then a new session.

Dependabot keeps the GitHub Actions in the workflow current.

Background

Adding Playwright Agents: Lessons from the Deep Trenches, on benweese.com: what came out of putting Playwright's planner, generator and healer agents to work, including token costs and weak generated checks.

License

MIT. The bundled skills/playwright-cli/ is Apache-2.0, from Microsoft's @playwright/cli.