Run smoke tests against a built quarto (binary mode)

.claude/rules/testing/overview.md

@@ -28,6 +28,8 @@ QUARTO_TESTS_NO_CONFIG="true" ./run-tests.sh test.ts    # Linux/macOS
 $env:QUARTO_TESTS_NO_CONFIG=$true; .\run-tests.ps1      # Windows
 ```
 
+**Binary mode:** set `QUARTO_TEST_BIN` to a built quarto (extracted outside the checkout — the runner refuses the 99.9.9 dev sentinel) to spawn it instead of running the dev sources in-process; defaults to `smoke/` only (`unit/` and `integration/` are dev-only). See `tests/README.md` → "Binary mode"; architecture and design decisions in `llm-docs/built-version-testing-architecture.md`.
+
 ## Test Types
 
 | Type | Location | File Pattern | Details |
@@ -59,6 +61,7 @@ Managed via:
 | File | Purpose |
 |------|---------|
 | `test.ts` | Test infrastructure (`testQuartoCmd`, `unitTest`) |
+| `quarto-cmd.ts` | Quarto invocation dispatch (`runQuarto`; in-process dev vs `QUARTO_TEST_BIN` binary mode) |
 | `verify.ts` | Verification functions |
 | `utils.ts` | Path utilities (`docs()`, `outputForInput()`) |
 | `README.md` | Comprehensive documentation |

.claude/rules/testing/test-anti-patterns.md

@@ -6,6 +6,20 @@ paths:
 
 # Test Anti-Patterns
 
+## Don't: Rely on `execute()` Throwing to Fail a Test
+
+The harness catches every error thrown from a `TestDescriptor.execute` (including
+`runQuarto` failures and inline `assert`s) and converts it into a log record —
+the test then passes unless a **verifier** fails. A custom `execute` with
+`verify: []`, or with verifiers that never read the log, is silently green on
+failure. Rules: put assertions in `verify`, not in `execute`; and any test using
+`runQuarto(..., { throwOnFailure: false })` must include a log-reading verifier
+(`noErrors` at minimum) so command failures surface with their actual error.
+
+## Don't: Import `src/quarto.ts` in Tests
+
+A direct `import { quarto } from "src/quarto.ts"` bypasses binary mode (`QUARTO_TEST_BIN`) and always tests the dev sources. Invoke quarto through `testQuartoCmd()`/`runQuarto()` (`tests/quarto-cmd.ts` is the single dispatch point), and resolve subprocess spawns with `quartoDevCmd()` (`tests/utils.ts`).
+
 ## Don't: Modify Environment Variables
 
 `Deno.env.set()` modifies process-global state. Deno runs test files in parallel by default, so other tests can see modified values.

.claude/rules/testing/typescript-tests.md

@@ -22,7 +22,15 @@ TypeScript-based tests using Deno. Smoke tests render documents; unit tests veri
 
 ## Core Infrastructure
 
-Core test files (`test.ts`, `verify.ts`, `utils.ts`) are described in `.claude/rules/testing/overview.md` § Core Files.
+Core test files (`test.ts`, `quarto-cmd.ts`, `verify.ts`, `utils.ts`) are described in `.claude/rules/testing/overview.md` § Core Files.
+
+### Binary mode compatibility
+
+Tests must keep working when `QUARTO_TEST_BIN` targets a built quarto (binary mode) instead of the in-process dev sources:
+
+- Never `import { quarto } from "../../src/quarto.ts"` in tests — invoke quarto via `testQuartoCmd()`/`runQuarto()`.
+- Subprocess spawns of quarto: resolve the executable with `quartoDevCmd()` (`tests/utils.ts`, honors `QUARTO_TEST_BIN`) or `quartoDevBinCmd()` (`tests/quarto-cmd.ts`, pins the local dev build), and pass `quartoSpawnEnvOptions()` as spawn env options.
+- `TestContext.requiresDevQuarto: true` ignores a test in binary mode — rare escape hatch for tests exercising quarto internals in-process.
 
 ### Search for an existing verifier before writing one
 

.github/actions/build-dist-tarball/action.yml

@@ -0,0 +1,53 @@
+name: "Build Quarto dist tarball"
+description: |
+  Builds a Quarto distribution tarball from the current checkout:
+  configure.sh, quarto-bld prepare-dist --set-version, then a tar.gz of
+  package/pkg-working (top-level directory quarto-<version>) uploaded as a
+  workflow artifact. Shared by create-release.yml (make-tarball /
+  make-arm64-tarball) and test-smokes-built.yml.
+inputs:
+  version:
+    description: "Version to stamp into the distribution (--set-version)"
+    required: true
+  arch:
+    description: "Target architecture: amd64 or arm64"
+    required: false
+    default: "amd64"
+  tarball-name:
+    description: "File name of the produced .tar.gz"
+    required: true
+  artifact-name:
+    description: "Workflow artifact name to upload the tarball as"
+    required: true
+runs:
+  using: "composite"
+  steps:
+    - name: Configure
+      shell: bash
+      run: ./configure.sh
+
+    - name: Prepare Distribution
+      shell: bash
+      run: |
+        pushd package/src/
+        if [ "${{ inputs.arch }}" = "arm64" ]; then
+          ./quarto-bld prepare-dist --set-version ${{ inputs.version }} --arch aarch64 --log-level info
+        else
+          ./quarto-bld prepare-dist --set-version ${{ inputs.version }} --log-level info
+        fi
+        popd
+
+    - name: Make Tarball
+      shell: bash
+      run: |
+        pushd package/
+        mv pkg-working quarto-${{ inputs.version }}
+        tar --owner=root --group=root -czf "${{ inputs.tarball-name }}" quarto-${{ inputs.version }}
+        mv quarto-${{ inputs.version }} pkg-working
+        popd
+
+    - name: Upload Artifact
+      uses: actions/upload-artifact@v7
+      with:
+        name: ${{ inputs.artifact-name }}
+        path: ./package/${{ inputs.tarball-name }}

.github/workflows/create-release.yml

@@ -14,13 +14,20 @@ on:
         required: false
         type: boolean
         default: true
+      smoke-artifacts-only:
+        description: "Build only the artifacts consumed by Smoke Tests (Built Version) nightly mode: the linux amd64 tarball and the signed Windows zip. Skips source/arm64 tarballs, deb/rpm installers, and the macOS build. For cheap signed builds of a branch (dispatch with publish-release=false); incompatible with publishing."
+        required: false
+        type: boolean
+        default: false
 
 env:
   NFPM_VERSION: "2.43.1"
 
 concurrency:
   # make publishing release concurrent (but others trigger not)
-  group: building-releases-${{ inputs.publish-release && 'prerelease' || github.run_id }}
+  # smoke-artifacts-only runs never publish, so they must not queue in the
+  # shared 'prerelease' group behind/ahead of a real release
+  group: building-releases-${{ inputs.publish-release && !inputs.smoke-artifacts-only && 'prerelease' || github.run_id }}
 
 jobs:
   configure:
@@ -35,6 +42,16 @@ jobs:
       tag_pushed: ${{ steps.version_commit.outputs.tag_pushed }}
     if: github.event_name != 'schedule' || (github.event_name == 'schedule' && github.repository == 'quarto-dev/quarto-cli')
     steps:
+      # publish-release defaults to TRUE, so checking only the
+      # smoke-artifacts-only box would otherwise reach the version
+      # commit+tag step below and push an orphan tag (the publish-release
+      # job itself is guarded, but the tag push lives in this job)
+      - name: Fail on publish-release + smoke-artifacts-only
+        if: ${{ inputs.publish-release && inputs.smoke-artifacts-only }}
+        run: |
+          echo "::error::smoke-artifacts-only builds cannot publish - re-dispatch with publish-release=false"
+          exit 1
+
       - uses: actions/checkout@v6
         with:
           fetch-depth: 0
@@ -100,6 +117,7 @@ jobs:
           default_author: github_actions
 
   make-source-tarball:
+    if: ${{ !inputs.smoke-artifacts-only }}
     runs-on: ubuntu-latest
     needs: [configure]
     steps:
@@ -133,32 +151,16 @@ jobs:
         if: ${{ inputs.publish-release }}
         uses: ./.github/workflows/actions/prevent-rerun
 
-      - name: Configure
-        run: |
-          ./configure.sh
-
-      - name: Prepare Distribution
-        run: |
-          pushd package/src/
-          ./quarto-bld prepare-dist --set-version ${{needs.configure.outputs.version}} --log-level info
-          popd
-
-      - name: Make Tarball
-        run: |
-          pushd package/
-          mv pkg-working quarto-${{needs.configure.outputs.version}}
-          tar --owner=root --group=root -cvf  quarto-${{needs.configure.outputs.version}}-linux-amd64.tar quarto-${{needs.configure.outputs.version}}
-          gzip quarto-${{needs.configure.outputs.version}}-linux-amd64.tar
-          mv quarto-${{needs.configure.outputs.version}} pkg-working
-          popd
-
-      - name: Upload Artifact
-        uses: actions/upload-artifact@v7
+      - name: Build dist tarball
+        uses: ./.github/actions/build-dist-tarball
         with:
-          name: Deb Zip
-          path: ./package/quarto-${{needs.configure.outputs.version}}-linux-amd64.tar.gz
+          version: ${{ needs.configure.outputs.version }}
+          arch: amd64
+          tarball-name: quarto-${{ needs.configure.outputs.version }}-linux-amd64.tar.gz
+          artifact-name: Deb Zip
 
   make-arm64-tarball:
+    if: ${{ !inputs.smoke-artifacts-only }}
     runs-on: ubuntu-latest
     needs: [configure]
     steps:
@@ -170,30 +172,13 @@ jobs:
         if: ${{ inputs.publish-release }}
         uses: ./.github/workflows/actions/prevent-rerun
 
-      - name: Configure
-        run: |
-          ./configure.sh
-
-      - name: Prepare Distribution
-        run: |
-          pushd package/src/
-          ./quarto-bld prepare-dist --set-version ${{needs.configure.outputs.version}} --arch aarch64 --log-level info
-          popd
-
-      - name: Make Tarball
-        run: |
-          pushd package/
-          mv pkg-working quarto-${{needs.configure.outputs.version}}
-          tar --owner=root --group=root -cvf  quarto-${{needs.configure.outputs.version}}-linux-arm64.tar quarto-${{needs.configure.outputs.version}}
-          gzip quarto-${{needs.configure.outputs.version}}-linux-arm64.tar
-          mv quarto-${{needs.configure.outputs.version}} pkg-working
-          popd
-
-      - name: Upload Artifact
-        uses: actions/upload-artifact@v7
+      - name: Build dist tarball
+        uses: ./.github/actions/build-dist-tarball
         with:
-          name: Deb Arm64 Zip
-          path: ./package/quarto-${{needs.configure.outputs.version}}-linux-arm64.tar.gz
+          version: ${{ needs.configure.outputs.version }}
+          arch: arm64
+          tarball-name: quarto-${{ needs.configure.outputs.version }}-linux-arm64.tar.gz
+          artifact-name: Deb Arm64 Zip
 
   make-tarball-rhel:
     runs-on: ubuntu-latest
@@ -246,6 +231,7 @@ jobs:
           path: ./package/quarto-${{needs.configure.outputs.version}}-linux-rhel7-amd64.tar.gz
 
   make-installer-linux:
+    if: ${{ !inputs.smoke-artifacts-only }}
     runs-on: ubuntu-latest
     needs: [configure]
     strategy:
@@ -477,6 +463,7 @@ jobs:
           quarto --version
 
   make-installer-mac:
+    if: ${{ !inputs.smoke-artifacts-only }}
     runs-on: macos-latest
     needs: [configure]
     steps:
@@ -546,6 +533,7 @@ jobs:
           security delete-keychain build.keychain
 
   test-zip-mac:
+    if: ${{ !inputs.smoke-artifacts-only }}
     runs-on: macos-latest
     needs: [configure, make-installer-mac]
     steps:
@@ -585,7 +573,8 @@ jobs:
           quarto --version
 
   publish-release:
-    if: ${{ inputs.publish-release }}
+    # never publish from a partial (smoke-artifacts-only) build
+    if: ${{ inputs.publish-release && !inputs.smoke-artifacts-only }}
     runs-on: ubuntu-latest
     needs: [
         configure,