From e83970252967aeb30555749d7fd9f801b2b9bc96 Mon Sep 17 00:00:00 2001 From: Will Zhao Date: Thu, 30 Jul 2026 17:39:08 -0700 Subject: [PATCH] docs: rewrite README and action description for the Marketplace listing The README renders in full on the GitHub Marketplace page, so it is the only surface in the listing that supports real links and real explanation. It previously opened with two malformed badges (image alt text set to a URL) and jumped straight into an inputs table, with no sentence saying what the action does and no link to shiplight.ai anywhere. - Add an intro, a quick start, a description of the pull request comment the action produces, worked examples for preview URLs, preflight gates, test context and async mode, and troubleshooting. - Pin examples to v2.0.1 (the current release) instead of v1. - Recommend the minimal permissions the action needs (contents: read, pull-requests: write) rather than write-all. - Fix the badges so they are links rather than images with URL alt text. - Route readers on the current platform (nova.shiplight.ai) to the CLI CI docs, since this action only drives Shiplight Cloud v1. - Replace the action.yml description, which was the Marketplace tagline and read "Use This GitHub Action to run tests on https://app.shiplight.ai". The URL was never a link there and pointed at the signed-in app. Verified: every URL in the README returns 200, prettier and markdownlint pass with the repo configs, and the documented inputs, outputs and comment behavior match src/main.ts and src/github/github.ts. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 255 +++++++++++++++++++++++++++++++++++++++++------------ action.yml | 4 +- 2 files changed, 202 insertions(+), 57 deletions(-) diff --git a/README.md b/README.md index 0c4458a..a166fdc 100644 --- a/README.md +++ b/README.md @@ -1,70 +1,213 @@ -# GitHub Action to Shiplight - -![https://github.com/ShiplightAI/github-action](https://img.shields.io/github/v/release/ShiplightAI/github-action) -![https://github.com/ShiplightAI/github-action](https://github.com/ShiplightAI/github-action/workflows/Continuous%20Integration/badge.svg) - -## Usage - -### Inputs - -| option | required | default | description | -| ---------------------- | -------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| api-token | true | | API token for Shiplight AI | -| test-suite-id | true | | Test suite ID (single ID or comma-separated list, e.g., `1` or `1,2,3`) | -| preflight-test-case-id | false | | Optional preflight test case ID. If set, the preflight test runs before the main test suites | -| environment-id | true | | Environment ID | -| environment-url | false | | Environment URL | -| github-comment | false | `true` | If enabled, the action will comment on the pull request with the test run results | -| github-token | false | `GITHUB_TOKEN` | Token used for leaving a comment on the pull request | -| async | false | `false` | If enabled, launches the tests but does not wait for them to finish. The action will always output success. GitHub comments will not function if this is set | -| commit-sha | false | | Commit SHA used to find the pull request for commenting | -| timeout-seconds | false | `86400` | Maximum time to wait for test runs to complete, in seconds | -| test-context | false | | Test context key-value pairs passed to the test run, one per line in `KEY=VALUE` format | - -**Note**: The `test-suite-id` parameter supports both single test suite IDs for -backward compatibility and comma-separated lists for running multiple test -suites in parallel. When multiple IDs are provided, all test suites run -simultaneously for improved efficiency. - -### Outputs - -| output | description | -| ---------------- | ----------------------------------------------------------- | -| success | Whether all test runs were successful | -| results | JSON string containing results for each test suite | -| run-id | Shiplight run ID | -| run-url | Shiplight run result URL | -| preflight-result | Preflight result (`Passed`, `Failed`, `Skipped`, `Pending`) | -| metadata | Trigger metadata JSON string | - -### Example - -```yml -name: Shiplight Test +# Shiplight Runner + +[![Release](https://img.shields.io/github/v/release/ShiplightAI/github-action)](https://github.com/ShiplightAI/github-action/releases) +[![Continuous Integration](https://github.com/ShiplightAI/github-action/workflows/Continuous%20Integration/badge.svg)](https://github.com/ShiplightAI/github-action/actions) + +Run your [Shiplight](https://www.shiplight.ai) end-to-end test suites from +GitHub Actions. The action starts a run, waits for it to finish, posts the +results as a pull request comment, and fails the job when a suite fails. + +Shiplight is AI-driven browser testing: tests are authored against your real +application, run in a real browser, and repair themselves when selectors move. +This action is the CI trigger for suites you have already set up. + +## Which Shiplight is this for? + +This action drives **Shiplight Cloud v1** at +[app.shiplight.ai](https://app.shiplight.ai), where test suites and environments +are configured in the web app and referenced here by ID. + +If you are on the current platform at +[nova.shiplight.ai](https://nova.shiplight.ai), where tests live in your +repository and run through the CLI, you do not need this action. See +[Running tests in CI](https://docs.shiplight.ai/local/ci/github-actions.html) +instead. + +## Quick start + +1. In Shiplight, go to **Settings → API Tokens** and create a token. +1. In your repository, go to **Settings → Secrets and variables → Actions** and + add it as a secret named `SHIPLIGHT_API_TOKEN`. +1. Note the ID of each test suite you want to run, and the environment ID from + **Settings → Environments**. +1. Add `.github/workflows/shiplight.yml`: + +```yaml +name: Shiplight on: pull_request: - branches: - - main + branches: [main] -# should set permissions to write-all for private repo -permissions: write-all +# Required so the action can comment on the pull request +permissions: + contents: read + pull-requests: write jobs: - test: - name: Test + e2e: + name: End-to-end tests runs-on: ubuntu-latest steps: - - name: Shiplight Test - uses: ShiplightAI/github-action@v1 + - name: Run Shiplight tests + uses: ShiplightAI/github-action@v2.0.1 with: api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }} - test-suite-id: YOUR_SHIPLIGHT_TEST_SUITE_ID # Single ID (backward compatible) - # test-suite-id: suite1,suite2,suite3 # Multiple IDs (new feature) - environment-id: YOUR_SHIPLIGHT_ENVIRONMENT_ID - commit-sha: YOUR_COMMIT_SHA + test-suite-id: 123 + environment-id: 1 +``` + +No checkout step is needed. The action calls the Shiplight API and never reads +your source. + +## What you get on the pull request + +The action posts one comment as soon as the run starts and edits that same +comment as the run progresses, so a pull request never collects a trail of +status updates. The comment carries: + +- A table of every suite in the run, with its result, pass count, and start and + end time in UTC +- An **Inspect** link per suite that opens the run in Shiplight +- A collapsed **Failed Tests** section listing each failing test case with the + step that broke and a link to its trace +- A **Preflight Gate** section, when a preflight test case is configured + +Each workflow step gets its own comment, keyed by environment and suite IDs, so +two Shiplight steps in one workflow do not overwrite each other. + +## Inputs + +| Input | Required | Default | Description | +| ------------------------ | -------- | -------------- | ---------------------------------------------------------------------------------------------------- | +| `api-token` | yes | | Shiplight API token. Store it as a repository secret. | +| `test-suite-id` | yes | | Test suite ID, or a comma-separated list (`1` or `1,2,3`). Multiple suites run in parallel. | +| `environment-id` | yes | | Environment ID the run targets. | +| `environment-url` | no | | Override the environment's configured URL. Use this for preview deployments. | +| `preflight-test-case-id` | no | | Test case to run first as a gate. The main suites run only if it passes. | +| `github-comment` | no | `true` | Post and update the pull request comment. | +| `github-token` | no | `GITHUB_TOKEN` | Token used to write the comment. | +| `async` | no | `false` | Start the run and return immediately. The job always succeeds and no comment is posted. | +| `commit-sha` | no | | Commit used to locate the pull request to comment on. Defaults to the commit that triggered the run. | +| `timeout-seconds` | no | `86400` | How long to wait for the run to finish. The job fails on timeout. | +| `test-context` | no | | Key-value pairs passed into the run, one `KEY=VALUE` per line. | + +`preflight-test-case-id` and `test-context` require v2.0.0 or later. + +## Outputs + +| Output | Description | +| ------------------ | ------------------------------------------------------------------------ | +| `success` | `true` when every suite passed | +| `results` | JSON array of per-suite results (`testSuiteID`, `name`, `result`, `url`) | +| `run-id` | Shiplight run ID | +| `run-url` | Link to the run in Shiplight | +| `preflight-result` | `Passed`, `Failed`, `Skipped`, or `Pending` | +| `metadata` | JSON of the commit, branch, actor, and workflow the run was tagged with | + +The action already fails the job when a suite fails, so you only need these to +do something extra: + +```yaml +- name: Run Shiplight tests + id: shiplight + uses: ShiplightAI/github-action@v2.0.1 + with: + api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }} + test-suite-id: 1,2 + environment-id: 1 + +- name: Notify on failure + if: steps.shiplight.outputs.success == 'false' + run: echo "Results: ${{ steps.shiplight.outputs.run-url }}" ``` -### License +## Common setups + +### Test a preview deployment + +Point the run at the URL your deploy step produced: + +```yaml +- name: Deploy preview + id: deploy + run: ./deploy.sh + +- name: Run Shiplight tests + uses: ShiplightAI/github-action@v2.0.1 + with: + api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }} + test-suite-id: 123,456 + environment-id: 1 + environment-url: ${{ steps.deploy.outputs.preview-url }} + timeout-seconds: 1800 +``` + +### Gate an expensive suite behind one fast test + +`preflight-test-case-id` runs a single test case first. If it fails, the main +suites are skipped, which keeps a broken deployment from burning a full run: + +```yaml +with: + api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }} + test-suite-id: 123,456 + environment-id: 1 + preflight-test-case-id: 789 +``` + +### Pass build context into the run + +```yaml +with: + api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }} + test-suite-id: 123 + environment-id: 1 + test-context: | + env=${{ vars.DEPLOY_ENV }} + branch=${{ github.ref_name }} + build-id=${{ github.run_id }} +``` + +### Fire and forget + +`async: true` starts the run and returns. Nothing is waited on, no comment is +posted, and the job always succeeds. Use it when the run is informational and +you do not want it holding a merge: + +```yaml +with: + api-token: ${{ secrets.SHIPLIGHT_API_TOKEN }} + test-suite-id: 123 + environment-id: 1 + async: true +``` + +## Troubleshooting + +**No comment appears on the pull request.** The job needs +`pull-requests: write`, or the broader `permissions: write-all`. Comments are +also skipped when `async: true` is set, and when the workflow was not triggered +by a pull request and no `commit-sha` was given to locate one. + +**Authentication failed.** Check that the secret is named exactly +`SHIPLIGHT_API_TOKEN` and that the token is still valid in **Settings → API +Tokens**. + +**The run times out.** `timeout-seconds` defaults to 24 hours. Lower it so CI +fails fast, or split a long suite into several that run in parallel by passing +comma-separated IDs. + +**A suite ID is rejected.** IDs must be numeric and comma-separated with no +spaces: `1,2,3`, not `1, 2, 3`. + +## Documentation + +- [GitHub Actions integration guide](https://docs.shiplight.ai/integrations/github-actions.html), + the full reference including Vercel deployment triggers and test accounts +- [Shiplight documentation](https://docs.shiplight.ai) +- [shiplight.ai](https://www.shiplight.ai) + +## License [MIT](./LICENSE) diff --git a/action.yml b/action.yml index d1cef1d..e49e25b 100644 --- a/action.yml +++ b/action.yml @@ -1,5 +1,7 @@ name: shiplight-runner -description: Use This GitHub Action to run tests on https://app.shiplight.ai +description: + Run Shiplight end-to-end tests in CI. AI-driven browser tests that self-heal, + with results commented on every pull request. author: Shiplight AI branding: