Run tests in CI

Use this guide to make a pipeline wait for Teststream results, preserve the result as an artifact, and fail when the selected tests do not pass. For provider-specific jobs, see GitHub and GitLab examples.

Prerequisites

• Complete CLI setup. The runner needs Node.js 22 or newer; the job runs the CLI with npx --yes teststream@1, which fetches it from npm and pins the major version.
• Store TESTSTREAM_API_KEY as a CI secret. Set TESTSTREAM_URL to your Teststream API base URL only if you use a self-hosted server; without it, the CLI uses the hosted Teststream.
• Create an environment such as Staging, with its application URL, credentials, allowed sites and ready-to-run tests.
• Deploy the application before this job. Teststream does not build a preview from a git branch.

There are two network connections to check: the CI job must reach the Teststream API, and Teststream's browser runner must reach the application. A localhost URL refers to the machine or container making that connection, not automatically your laptop or CI runner.

The CLI itself does not run Playwright or launch a local browser. Execution is handled by the Teststream server and its runner.

Run and wait for the result

npx --yes teststream@1 run \
  --project SHOP --env Staging --wait --timeout 30m

Without --wait, exit code 0 means only that the run was started. With --wait, the CLI polls every five seconds and returns the result:

Code	Pipeline meaning
0	The selected tests passed.
1	A test failed, was blocked, errored or was skipped; also a healed pass when using --fail-on-heal.
2	A command/API error, a cancelled or failed run, or no tests to run.
3	The wait ended while the run was still unfinished.

While waiting, a status check that fails with a network error or a server error (5xx) is retried for about a minute, so a short outage or a server restart does not fail the job. If Teststream is still unreachable after that, the CLI exits with 2.

Do not append || true or use allow_failure: true on a required test job. That would discard the failure signal.

Timeout is not cancellation

--timeout 30m limits CLI polling, not server execution. It takes 90s, 20m or 1h, and a bare number means minutes; any other value is an error (exit 2) and no run is started. The run can continue after the CLI exits with 3, or after CI terminates the process. Use the run ID to inspect or explicitly cancel it:

npx --yes teststream@1 status "$RUN_ID"
npx --yes teststream@1 cancel "$RUN_ID"

Set the CI job's own timeout longer than the CLI wait so it has time to save artifacts. API requests have their own timeouts, so the wait option is not a hard wall-clock deadline.

Choose which tests run

Use one selector for a straightforward job; replace the example keys and folder with your project's values:

npx --yes teststream@1 run --project SHOP --env Staging --tag smoke --wait
npx --yes teststream@1 run --project SHOP --env Staging --folder Checkout --wait
npx --yes teststream@1 run --project SHOP --env Staging --tests SHOP-1,SHOP-2 --wait

Omit selectors to run the available tests for the chosen branch or main. Archived tests are excluded; drafts or other tests that are not ready can produce skipped results and fail the job. Tests that exist only on a branch are not included in main runs.

Point the run at a deployment

: "${REVIEW_APP_URL:?Set this to the deployed application's URL}"
npx --yes teststream@1 run \
  --project SHOP --env Review --base-url "$REVIEW_APP_URL" --wait

Have your deployment job supply REVIEW_APP_URL, and wait for the application to become ready. Without --base-url, Teststream uses the environment's configured URL. Other sites the application needs, such as an authentication provider, belong in Environments > Test browser > Allowed sites.

--branch selects test proposals, not application code. To test both the new application and its proposed tests, use its deployed URL and the appropriate branch, or let --mr auto supply the branch.

Treat healed tests as failures

If replay fails and AI recovery succeeds, the run can contain a passed test marked as healed. To require review of these results, make the CLI return 1:

npx --yes teststream@1 run --project SHOP --env Staging --wait --fail-on-heal

This flag changes the CLI exit code only. It does not disable healing, change the stored verdict, or turn the server's GitHub/GitLab teststream status red. Require the CI job itself if this is your merge policy.

Save machine-readable output without losing the exit code

exit_code=0
npx --yes teststream@1 run \
  --project SHOP --env Staging --wait --timeout 30m --json \
  > teststream-run.json || exit_code=$?

# Configure CI artifact collection to run even when this step fails.
exit "$exit_code"

The JSON contains the run ID, URL, counts, individual items and any mergeRequestError. A command that fails before receiving a run may leave an empty file; check before parsing it.

--json writes a run snapshot, not a stream of progress events. Do not combine it with --diagnose or --propose-fixes when you need one valid JSON document: those options append human-readable output.

Diagnose separately from the required job

Keep the required job focused on the verdict. After reviewing the evidence, run diagnosis or draft proposals as a separate, explicitly authorized action:

npx --yes teststream@1 diagnose "$RUN_ID" --json > diagnosis.json
npx --yes teststream@1 propose-fixes "$RUN_ID" --json > proposals.json

These operations can incur AI costs. Fix proposals require an open branch associated with the run and do not repair the application, rerun tests, or change the original failed verdict. Read Investigate failures before automating them.

For complete jobs with always-collected artifacts, continue with Test a pull or merge request.