How Teststream runs tests

This page explains the model behind the CLI, CI and coding-agent workflows: where execution happens, what a passing result means, and how test changes reach main.

The CLI is a remote client

Terminal / CI job ---- Teststream API ---- browser runner ---- application
                           |
                           +---- GitHub/GitLab status and comment

Coding agent ---- stdio MCP process ---- Teststream API

The CLI submits work and reads results. It does not build your application, deploy a git branch or launch the test browser in the CI job. This separates the job's network access to Teststream from the browser runner's access to your application.

The web app is where people configure projects, environments, credentials and provider connections, and review proposals. The CLI covers terminal/pipeline operations; MCP exposes agent-oriented tools such as step-by-step browser sessions and evidence retrieval.

Four choices that should not be confused

Choice	Controls
Project	Test inventory and project settings.
Environment	Application URL, credentials, variables and browser settings.
Test branch/change set	Which proposed test versions and branch recordings a run uses.
Pull/merge request	Provider reporting, source branch metadata and optional affected-test selection.

For example, --branch feature/SHOP-42-cart selects proposed tests, while --base-url https://preview.example.com targets a deployed application. Neither option deploys the other. A plain CLI run does not infer the branch from your checkout; --mr auto can infer it from CI metadata.

Tests are plain language

A step reads like a sentence: when: I click "Add to cart". The test author describes behavior rather than supplying selectors or browser code. A UI change can still break the recording or reveal a real regression; plain-language tests do not guarantee automatic resilience.

The price is that something has to turn the sentence into browser actions. That is the AI.

First run: the AI drives; later runs: replay

When a test needs AI execution, an agent drives a real browser, observes the page and attempts the written steps. Teststream records actions so later runs can replay them. Which locator a recorded action uses, and why its text comes last, is explained in How Teststream finds elements.

With a usable recording and replay enabled, browser actions can run without a model. Missing or changed recordings, or failed replay that falls back to AI, can require model work again. A second run is not guaranteed to be entirely replayed.

Replay avoids model usage cost for recorded actions. It still uses browser/server infrastructure, and the external coding agent has its own cost model. AI execution, drafting and some diagnoses are additional work; their cost depends on the model and work performed, not a fixed price per CLI command.

Read the run's execution modes and estimated cost rather than assuming every run is free after the first.

What "heal" means

An old recording can stop fitting: a button moved or a label changed. If replay fails and AI recovery then completes the test, the result can be passed with heal. That differs from using AI simply because no recording exists.

A healed pass deserves review: the browser reached a passing result, but not solely through the prior recording. Inspect the actions and assertions to distinguish legitimate UI changes from a test or execution problem.

--fail-on-heal lets a CI job reject that result. It changes the CLI's exit policy, not the stored verdict or the server's provider status, and it does not switch healing off. Server-side healing limits can also stop repeated recovery.

Three different success signals

Signal	Answers
Stored test verdict	Did execution pass the test's checks? A healed pass is still a pass.
CLI exit code / CI job result	Did this invocation satisfy the CLI policy, including --fail-on-heal and its wait timeout?
Provider teststream status	Did execution pass, and does the repository's test-review policy allow success?

A green CLI job can coexist with a pending provider status when test changes await review. A CLI job can be red for a healed pass while the provider reports success. A reporting connection can fail even when execution passes; inspect mergeRequestError.

Choose required merge checks accordingly. Requiring only the provider status does not enforce --fail-on-heal; requiring only the job does not enforce pending proposal review.

Similarly, a timed-out CLI is not a cancelled run. The server can continue execution until it finishes or receives an explicit cancellation.

Branches and review

The authoritative tests live in Teststream. Files you edit locally are a way to submit source; pushing the application's git branch does not by itself upload those test changes.

A change set is Teststream's collection of proposals for a branch or ticket. Feature-branch tests push, MCP proposals and propose-fixes put changes there. Runs can use proposed or accepted versions before main changes.

Propose -> verify in a browser -> human accept/reject -> merge in Teststream

These are separate states:

• Verified in browser means a passing verification session was kept. It is evidence, not human acceptance.
• Accepted means a reviewer approved the proposal. Main is not yet updated.
• Merged means accepted changes were applied to main. Unaccepted changes are dropped; conflicting newer main revisions require resolution.

Use Branches > Merge in Teststream for that final operation. A GitHub/GitLab merge is not a substitute for applying the Teststream change set. Closing a change set without merging does not apply its proposals.

An uppercase ticket key in a branch name can become the change-set key: feature/SHOP-42-cart and fix/SHOP-42-api share SHOP-42. This lets frontend/backend branches share test proposals, but also means they are not isolated just because their git branch names differ.

tests push --main deliberately bypasses proposal review. The MCP proposal tool does not offer that route. Prefer branch proposals when review matters.

What affected-test selection does

--affected is a fast heuristic based on the PR/MR's changed file names, observed app-map screens/API paths and test names. It adds critical-tagged tests within the selected candidate set.

It is not static code analysis or a proof that all impacted behavior is covered. When there is insufficient information, it runs the available candidates rather than assuming no tests are needed. Use a full suite when that heuristic is not sufficient for your release risk.

Diagnosis is not repair

diagnose explains a failure; propose-fixes drafts a possible test change. Neither edits application code, reruns tests, grants approval or changes the old verdict.

Some setup/environment problems have rule-based explanations. AI diagnosis combines run evidence with available PR intent and project knowledge. It can still be wrong. The important decision is whether to fix the app, the setup, or genuinely incorrect test wording, without weakening a valid assertion.

Roles keep tests short

Tests can reuse the logged-in state produced by a project's role/login-test setup rather than repeating login steps everywhere. Environment credentials still need to be correct for the application under test.

A test's role: is an application test role, not an organization's Admin/Client authorization role in Teststream.

Shared steps

Repeated flows such as filling an address can be shared through use:. Changing them affects their callers and can invalidate reusable recordings. The current CLI/MCP interfaces expose shared steps for discovery/reuse, not creation or editing; people manage them in the web app.

What coding-agent automation can and cannot guarantee

Skills describe the workflow; MCP provides the operations. Skill instructions such as "ask before running" depend on the agent following them. Server-side rules, such as session confirmation for identified consequential steps, are a separate mechanism.

Use safe test accounts and environments even with an agent. A proposal is not verified merely because its YAML is valid or its author says it looks correct. The handoff should contain browser evidence, verification state, remaining risks and the human decision still required.

For the operational steps, use CI execution, request reporting, failure investigation and coding agents.