Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions .github/workflows/compiler-execution.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
name: Compiler execution acceptance
on:
push:
pull_request:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: compiler-execution-${{ github.ref }}
cancel-in-progress: true
jobs:
realistic-compilers:
name: Realistic compilers (${{ matrix.runtime }})
runs-on: ubuntu-24.04
timeout-minutes: 40
strategy:
fail-fast: false
matrix:
runtime: [release-0.4.2, current]
steps:
- name: Check out reviewed validation scenarios
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09
with:
path: validation
persist-credentials: false
- name: Check out independently identified runtime
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09
with:
path: runtime
ref: ${{ matrix.runtime == 'release-0.4.2' && 'eb9b1cd0c3e3fd1e70e397edeffb059073464f74' || github.sha }}
persist-credentials: false
- uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1
with:
python-version: '3.12'
- name: Install only controller validation dependencies
run: python -m pip install -e './runtime[dev]'
- name: Identify exact runtime source
shell: bash
run: |
set -euo pipefail
printf 'CFS_RUNTIME_SOURCE=%s\n' "$(git -C runtime rev-parse HEAD)" >> "$GITHUB_ENV"
- name: Build worker on disposable hosted runner
run: docker build --label org.opencontainers.image.revision="$CFS_RUNTIME_SOURCE" -t code-flow-worker:0.4.2 runtime/runner
- name: Run realistic programs and deliberate failure controls
working-directory: runtime
env:
PYTHONPATH: ${{ github.workspace }}/runtime:${{ github.workspace }}/validation
CFS_VALIDATION_LANE: ${{ matrix.runtime }}
run: >-
python ../validation/scripts/compiler_acceptance.py
--runtime-root .
--runtime-source "$CFS_RUNTIME_SOURCE"
--lane "$CFS_VALIDATION_LANE"
--output ../validation/artifacts/compiler-validation
- name: Test the actual published frozen 0.4.2 service
if: matrix.runtime == 'release-0.4.2'
working-directory: validation
run: python -m scripts.released_service_probe --output artifacts/released-service
- name: Retain a compact result table in the run summary
if: always()
shell: bash
run: |
if test -f validation/artifacts/compiler-validation/summary.md; then
cat validation/artifacts/compiler-validation/summary.md >> "$GITHUB_STEP_SUMMARY"
fi
- name: Save small synthetic reports, including failures
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
with:
name: compiler-validation-${{ matrix.runtime }}
path: |
validation/artifacts/compiler-validation/results.json
validation/artifacts/compiler-validation/summary.md
validation/artifacts/released-service/*.json
if-no-files-found: error
retention-days: 3
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ Download OS-specific installers/portable packages from [Releases](https://github
4. Review findings and add a candidate revision. In **Evidence**, filter by file/category and select a diagnostic or missing-coverage line. The inspector verifies the retained source fingerprint before displaying it. Browse nearby lines or open a matching current draft; changed drafts are never overwritten. Baseline expectations and existing tests cannot be removed. Compare results under the same environment and adjustable runtime/memory/readability/footprint preferences.
5. Download escaped HTML, machine-readable JSON and a source/test/harness/report ZIP. Reopen saved revisions and replay them.

## Hosted compiler acceptance

The [realistic compiler suite](docs/compiler-execution.md) covers all twelve executable adapters with 72 independently authored input/output cases, deliberate compiler/runtime/wrong-answer controls, exact unused/uncovered anchors, and encrypted private-report replay. Its release lane identifies the published 0.4.2 source and tests the actual published Linux frozen service; a second lane tests the current checkout. Read the [dated observations and limits](docs/reports/hosted-compilers-2026-11-10.md) before interpreting a green run. This hosted validation does not install, repair or reset Docker on an end user's PC.

## Features and limits

**Located diagnostics:** static review and execution evidence list original filenames and local line locations, with snapshot hashes. Python checks include auxiliary-file syntax, possibly unused imports/locals/functions and directly unreachable statements. Every other supported execution language now has native unused-code candidates and actual language-specific coverage. UI/HTML show missed source lines, genuine branch arcs or native branch IDs/counts, and explicit partial/unknown states. SQL measures completed statement starts, Bash command-entry lines and assembly instruction-backed debug lines—not universal statement/branch coverage. Unsupported locations and wrong-answer failures without source references stay unlocated. See the [language-by-language tools and limits](docs/polyglot-locations.md), [file/line contract](docs/located-diagnostics.md) and [validation](docs/validation-polyglot.md). No tool guarantees discovery or root-cause localization of every bug.
Expand Down
Loading
Loading