CI/CD Best Practices for Testronaut Runs and Reports

Moving an end-to-end test from a laptop into CI changes more than the machine that runs it. The browser starts clean, the network behaves differently, credentials arrive through secrets, and nobody is watching when the application takes an unexpected turn.

Agentic tests can adapt to interface variation, but a reliable pipeline still depends on deterministic foundations. The environment should be reproducible, the target should be explicit, and every run should leave enough evidence to diagnose a failure after the runner disappears.

Pin the runtime and install the browser deliberately

Use a supported Node.js version and a lockfile-based install such as npm ci. Install Playwright browsers and their operating-system dependencies as an explicit workflow step. This keeps a missing browser package from being mistaken for an application failure.

Caching the Playwright browser directory can make repeated runs faster. Include the installed Playwright version in the cache key so an upgraded dependency does not restore an incompatible browser binary.

- uses: actions/setup-node@v4
  with:
    node-version: 22
    cache: npm
- run: npm ci
- run: npx playwright install --with-deps
- run: npm install -g testronaut

Treat the target environment as test input

Do not let a mission guess which deployment it should test. Supply the base URL through the workflow environment and keep staging, preview, and production jobs visibly separate. If the pipeline creates a preview deployment, wait for it to become healthy before launching a browser.

Use dedicated test accounts and isolated fixtures. A clean runner does not guarantee clean application data. Parallel jobs that share one account can invalidate each other by changing a password, consuming a one-time link, or editing the same record.

Keep secrets out of missions and logs

API keys, passwords, and service tokens belong in the CI provider’s secret store. Pass them to the test process as environment variables. Avoid placing secret values in mission text, repository configuration, command-line output, screenshots, or uploaded artifacts.

Give CI credentials the smallest useful permissions and rotate them independently from developer accounts. Remember that screenshots and browser reports may contain customer-like data even when the workflow log does not.

Choose a useful run scope

Running every browser journey on every commit can create slow feedback and unnecessary model cost. A small set of critical smoke missions is a better pull-request gate. Broader suites can run after a deployment, on the main branch, or on a schedule.

• Pull requests: fast missions covering login and the highest-value paths.
• Main branch or preview deployment: integration journeys across changed areas.
• Scheduled runs: broader regression coverage and longer edge cases.
• Manual dispatch: targeted diagnosis or release verification.

Set realistic turn and time limits. A bounded failure is more useful than an agent consuming an unlimited budget while trying increasingly unlikely routes.

Preserve reports even when the test fails

The most important artifact is often produced by a failed run. Configure the artifact step to execute regardless of the test result, and retain the JSON or HTML report together with relevant screenshots.

- name: Run Testronaut missions
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    URL: ${{ vars.TEST_URL }}
    USERNAME: ${{ secrets.TEST_USERNAME }}
    PASSWORD: ${{ secrets.TEST_PASSWORD }}
  run: testronaut

- name: Preserve Testronaut reports
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: testronaut-reports
    path: missions/mission_reports/**

Uploading a run to Mission Control can provide longer-term history and comparison. CI artifacts are still worth keeping because they remain attached to the exact commit, workflow, and deployment that produced them.

Avoid blind retries

A pipeline-level retry can distinguish a transient infrastructure problem from a repeatable failure, but it can also turn flaky behavior into a green check. If retries are allowed, preserve every attempt and make the retry visible in the result.

The same is true inside an agentic run. Adaptation is useful, but an unusual refresh, repeated interaction, or unexpected re-authentication may be evidence of a product defect. Passing after recovery should not look identical to a clean pass.

Fail on product outcomes, not incidental variation

A useful CI gate answers whether an important user journey still works. It should not fail merely because a harmless selector or layout detail changed, and it should not pass merely because the final page eventually appeared.

That balance comes from combining well-authored missions with disciplined pipeline setup: stable infrastructure around a flexible agent, explicit validations around adaptive navigation, and durable evidence around every outcome.

Pipeline checklist

• Pin Node.js and install dependencies from the lockfile.
• Install compatible Playwright browsers and operating-system packages.
• Pass the target URL, credentials, and provider key through CI configuration.
• Use isolated accounts and uniquely identifiable test data.
• Run a scope appropriate to the trigger and enforce execution limits.
• Always retain reports and screenshots, including failed attempts.
• Make retries and agent recoveries visible rather than silently green.