Investigate failures
Use this guide when a run is red and you need to know whether the app is broken, the test is wrong, or the setup is at fault.
Complete CLI setup first. Set RUN_ID to the id from a run --json result or the run ID in the web URL, not the full URL:
export RUN_ID="replace-with-run-id"
Reading status and reports does not rerun tests. Diagnosis and drafting can use paid AI; review evidence before requesting either.
1. See what failed
teststream status "$RUN_ID"
teststream status "$RUN_ID" --json
The summary lists failed/non-passing items with available notes; healed passes are marked HEAL. The final line links to the run. JSON additionally exposes failureKind, failureStep and stored diagnosis fields for individual items.
status returns a test-result exit code, not always 0: 1 for failed tests and 3 while the run is unfinished. In scripts using fail-fast behavior, handle that status deliberately so evidence collection can continue.
For a full write-up in Markdown (for a ticket or a chat message):
teststream report "$RUN_ID" > teststream-report.md
Open the run in the web app and compare the first failing step's expected behavior, observation, screenshot and recorded actions with the preceding step. A final error can be a consequence of an earlier wrong action. Also check the environment, deployed revision, test branch and mergeRequestError.
If a CLI wait timed out, the run may still be executing. Cancel it explicitly only when you want it stopped:
teststream cancel "$RUN_ID"
2. Ask for a diagnosis
teststream diagnose "$RUN_ID"
For eligible failed tests, diagnosis returns a cause, a next step, confidence and evidence. Known setup and environment failures can be classified by rules without a model call; other cases use AI, the run evidence, available PR/MR context and project knowledge.
Treat the result as a second opinion, not proof. AI cost depends on the configured model and input. A budget or service error can prevent an individual diagnosis even when the command itself completes.
To save structured results:
teststream diagnose "$RUN_ID" --json > diagnosis.json
Inspect each entry in diagnosed for an error, as well as cause and next. Repeating a diagnosis can incur another model call.
| Cause | Meaning |
|---|---|
intended-change | The app changed on purpose and the new behavior is consistent. |
regression | Something that should work does not. |
test-wording | The test's own text is ambiguous or mixes actions. |
setup | Account, data, variable, file or allowed site is missing. |
environment | The app or a service was down or slow. |
teststream | Teststream got it wrong, for example a stale recording. |
unclear | The evidence does not decide. |
| Next step | What to do |
|---|---|
update-test | Change the test to match the new behavior. |
fix-app | Fix the application. The test stays red until then. |
fix-setup | Fix the account, variable, data or allowed site. |
rerun | Run again once. Report it if it fails again. |
report-teststream | Report the problem to the Teststream team. |
ask-human | A person must decide. |
3. Act on the result
fix-app: fix the bug. Do not edit the test to match it.update-test: change the test text, then validate and push it to the run's branch.fix-setup/rerun: fix the setup or run again. Blocked sites are added under Environments → Test browser → Allowed sites.
4. Let Teststream draft the fix
Use this only after deciding that test changes are appropriate. The original run must belong to an open Teststream branch/change set, or carry PR/MR branch information:
teststream propose-fixes "$RUN_ID" --json > proposals.json
Teststream can draft one update per eligible failed test, rather than one per examples row. Setup failures are skipped. If the model keeps the test unchanged because the application appears broken, no proposal is created. Invalid drafts and drafting failures are reported as skipped.
Inspect proposed and skipped with their reasons. Exit 0 does not mean every failure was fixed or that any proposals were created.
This command does not edit application code, merge tests, rerun anything, or turn the original run green. It creates test proposals that still need review and browser verification.
5. Verify and review the change
For a manual update, read the existing source and preserve its key so the push edits it rather than creating a new test:
teststream tests get SHOP-1 --project SHOP > shop-1.yaml
# Edit shop-1.yaml, keeping the existing key and the intended assertions.
teststream tests validate shop-1.yaml --project SHOP
teststream tests push shop-1.yaml --project SHOP --branch feature/SHOP-42-cart
teststream run --project SHOP --env Preview \
--branch feature/SHOP-42-cart --tests SHOP-1 --wait
Replace the environment and branch with the ones for your change. This last command executes the test and can incur costs. Do not rerun a consequential action blindly.
For the verified in browser proposal badge, use the coding-agent session workflow, or the corresponding step-by-step workflow in the web app, and keep the passing session. A normal completed run is not the same as keeping a verification session.
A reviewer then accepts or rejects the proposal under Branches. Accepted test changes reach main through Teststream's branch merge operation; do not assume merging application code in GitHub/GitLab automatically applies the proposals.
Automate only after the workflow is understood
For a human-readable run followed by diagnosis, run --wait --diagnose is available. run --wait --propose-fixes requests drafts when failed/blocked tests remain, provided the run has an appropriate branch.
These options are not an automatic repair policy. They can add cost, append non-JSON text even with --json, and still require review. A failed follow-up request can return command error 2 instead of the original test-result code. Keep them separate from a required CI job when preserving that original code matters.
Intended change or regression?
Ask whether the new behavior is consistent and useful, such as a review dialog that then completes the payment, or broken, such as validation that lets bad input through. Check the pull request description and the team's earlier decisions. If you still cannot tell, ask a person. Never call a regression an intended change just so the test can pass.