Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified atp/CoreTrace-GUI_ATP.xlsx
Binary file not shown.
Binary file modified atp/CoreTrace-GUI_ATP_appendices.zip
Binary file not shown.
67 changes: 30 additions & 37 deletions atp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,30 @@ What to upload on the Innovation platform:
| File | Where it goes |
|---|---|
| `CoreTrace-GUI_ATP.xlsx` | the ATP Excel |
| `CoreTrace-GUI_ATP_appendices.zip` (130 KB) | the appendices archive (limit: 200 MB) |
| `CoreTrace-GUI_ATP_appendices.zip` | the appendices archive (limit: 200 MB) |

The workbook is built from `ATP_template.xlsx` and keeps its two tabs:

- **Info & Access** — download links, the Docker environment, the test data set
with its verified expected results, and the notes a tester needs.
- **Test Plan** — 56 scenarios (`F1`…`F56`, plus `F3b`; `F24` retired) with Feature ID, name, category,
- **Test Plan** — 24 scenarios (`F1`…`F24`), all run in the Docker environment
so the plan is reproducible on any OS, with Feature ID, name, category,
objective, prerequisites, numbered steps, expected result and development
status. The **Feedback** column is intentionally empty: the tester fills it.

## Scope

Only **coretrace-gui v5.1.0** is under test. The analysis binary is a
prerequisite, shipped inside the release artifacts — testers never build
anything.
Only **CoreTrace Desktop v6.0.0-beta.3** (the Tauri rewrite) is under test, from
its published Linux package. Local analysis uses **ctrace v0.85.0**, installed
in the Docker image. `F23` and `F24` (sign-in, cloud analysis) need a CoreTrace
account; connected repositories, dashboards and automatic updates are out of
scope.

## Regenerating the workbook

The scenarios live in `scripts/gen_atp.py` next to this README; re-run it with
`python gen_atp.py` after editing, then rebuild the archive:
`python gen_atp.py` after editing (pass the path of `ATP_template.xlsx` to
rebuild from the blank template; by default it rewrites the existing workbook), then rebuild the archive:

```sh
cd atp && python scripts/make_appendices_zip.py
Expand All @@ -39,36 +43,25 @@ honours the mode at all.

## Known gaps to close before the next test cycle

Shipped in v5.1.0:
Found while running the plan against v6.0.0-beta.3 in Docker:

1. **macOS artifact.** ~~The `macos-latest` job is commented out.~~ v5.1.0
publishes an unsigned `.dmg` and `.zip` for Apple Silicon. `F3` is now a
native install with the Gatekeeper steps spelled out.
2. **Ctrl-only shortcuts.** ~~`e.metaKey` was never tested.~~ Shortcuts follow
the platform, and `F53` covers Cmd on macOS. Tab navigation stays on Ctrl
everywhere, because Cmd+Tab is the macOS application switcher.
3. **Format Document was a no-op** for C/C++ — dropped, and `F24` is retired
(the numbering keeps the gap so every cross-reference stays valid).
1. **Folder analysis shows no result** (`F18`). *Nouvelle analyse* reports
"6 fichier(s) analysé(s) sur 6" and "Résultats 0": the merged report filed
SARIF results under a key the interface reads in another shape. Fixed in
`src-tauri/src/analysis.rs` and `desktop/model.ts`; `F18` stays *In progress*
until a release carries the fix.
2. **In-file search does not open** (`F22`). The editor was built without
Monaco's find module. Fixed in `desktop/features/WorkspaceEditor.tsx`; `F22`
stays *In progress* until a release carries the fix.
3. **ctrace v0.74.0 is unreadable.** It writes its own JSON instead of SARIF
("Format de rapport non reconnu"). The image uses v0.85.0 and
`docs/releases/v6.0.0-beta.3.md` now says so; the text of the published
GitHub release still has to be edited by hand.
4. **No test account for `F23` and `F24`.** They were run with a team member's
own account. Testers need a dedicated CoreTrace account with CTU; put its
access in the `CoreTrace account` row of `gen_atp.py`. A cloud run of the
data set was quoted 4 832 CTU and billed 3 204.

Still open:

4. **No Intel macOS artifact.** The x64 job targeted `macos-13`, which GitHub
has retired; a job pinned to a retired label queues forever instead of
failing, so v5.1.0 shipped without it. The matrix now uses `macos-15-intel`,
but that fix is not in the v5.1.0 tag. Until an Intel build ships, `F3b`
routes Intel Macs through Docker.
5. **The analyser binary is Linux-only.** `bin/ctrace` is an x86-64 ELF, so on
macOS the app runs but cannot analyse — `F54` pins the message it shows
instead. A Darwin build of ctrace is the real fix; until then, run `F28`
to `F38` on Linux.
6. **Third-party analysers are unreachable.** `--invoke=cppcheck` resolves to
the hard-coded path `/opt/homebrew/bin/cppcheck` and silently produces no
output. Only `ctrace_stack_analyzer` is exercised by the plan.
7. ~~**The Docker image has not been built end to end here.**~~ It now has
been, from the delivered archive rather than the working copy: unzip into a
clean directory, run `build.sh` and `run.sh` from those files, and confirm
the app window on the virtual display with
`docker exec <container> xwininfo -root -tree -display :99`. Re-run that
after any change under `appendices/docker/`.
8. **Windows is not in the plan.** The WSL detection and setup flow
(`main.js`) has no scenario, because the delivery targets Linux and macOS.
When a release carries fixes 1 and 2, bump `CORETRACE_VERSION` in
`appendices/docker/Dockerfile` and `build.sh`, re-run `F18` and `F22`, and mark
them *Done*.
160 changes: 45 additions & 115 deletions atp/appendices/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# CoreTrace GUI — acceptance test appendices
# CoreTrace Desktop — acceptance test appendices

Everything a tester needs to run the test plan without help from the team.
No compilation is required: the tests run against the **published binaries** of
release **v5.1.0**.
No compilation is required: the tests run against the **published package** of
release **v6.0.0-beta.3**. The interface is in French; the test plan quotes its
labels as displayed.

```
appendices/
├── README.md ← this file
├── docker/ ← reproducible Linux environment (also used on Intel Macs)
├── docker/ ← reproducible test environment, runs on any OS with Docker
│ ├── Dockerfile
│ ├── entrypoint.sh
│ ├── build.sh
Expand All @@ -20,152 +21,81 @@ appendices/
├── DS04_infeasible_path.c
├── DS05_recursion.c
├── DS06_cpp_array_oob.cpp
├── DS07_invalid.txt
└── DS08_large_file.c
└── DS07_invalid.txt
```

## Download links

| Item | Link |
|---|---|
| Release under test (v5.1.0) | https://github.com/CoreTrace/coretrace-gui/releases/tag/v5.1.0 |
| Linux AppImage | https://github.com/CoreTrace/coretrace-gui/releases/download/v5.1.0/CtraceGUI-5.1.0.AppImage |
| Windows installer | https://github.com/CoreTrace/coretrace-gui/releases/download/v5.1.0/CtraceGUI-Setup-5.1.0.exe |
| macOS, Apple Silicon (DMG) | https://github.com/CoreTrace/coretrace-gui/releases/download/v5.1.0/CtraceGUI-5.1.0-arm64.dmg |
| macOS, Apple Silicon (ZIP) | https://github.com/CoreTrace/coretrace-gui/releases/download/v5.1.0/CtraceGUI-5.1.0-arm64-mac.zip |
| macOS, Intel | not published in v5.1.0 — use the Docker setup |
| Release under test (v6.0.0-beta.3) | https://github.com/CoreTrace/coretrace-gui/releases/tag/v6.0.0-beta.3 |
| Analyser installed in the image (ctrace v0.85.0) | https://github.com/CoreTrace/coretrace/releases/tag/v0.85.0 |
| Source code | https://github.com/CoreTrace/coretrace-gui |
| API documentation | https://coretrace.github.io/coretrace-gui/ |

## Setup A — Linux, through Docker (recommended, and the reference environment)
## Setup — Docker (Linux, macOS or Windows)

Requires Docker and about 3 GB of free disk space. The image downloads the
AppImage, extracts it, and serves the running application over noVNC, so the
only thing you need locally is a browser.
The whole test plan runs in this environment, whatever your operating system.
Requires Docker and about 3 GB of free disk space (on Windows, run the scripts
from WSL or Git Bash; on a Mac, give Docker Desktop at least 6 GB of RAM). The
image installs the published Linux package and the ctrace analyser, and serves
the running application over noVNC, so the only thing you need locally is a
browser.

```sh
cd appendices/docker
./build.sh # ~5 min, downloads the AppImage once
./build.sh # ~5 min the first time
./run.sh # mounts ../datasets at /home/tester/workspace
```

Then open <http://localhost:6080/>. The page connects on its own — there is no
Connect button to find, and the CoreTrace GUI window fills the canvas. Inside
the app, the test data set is at `/home/tester/workspace`.
Connect button to find — and shows the CoreTrace window. Inside the app, the
test data set is at `/home/tester/workspace`.

Stop the container with `Ctrl+C` in the terminal running `run.sh`.
Stop the container with `Ctrl+C` in the terminal running `run.sh`, or by closing
the application window. Nothing is kept between two runs of the container: the
analyser has to be selected again (scenario **F9**).

### File pickers

The application opens the system's file picker to choose a folder or the
analyser. Browse with the mouse (`Home`, then `workspace`), or type a path
starting with `/`: the first `/` opens the path field. Browser shortcuts such as
Ctrl+L act on your own browser, not on the picker.

### If the browser shows nothing

The container prints a lot at startup — xkbcomp keysym warnings, a D-Bus
connection error, `Xlib: extension "DPMS" missing`. All three are normal in a
headless container and none of them stops the application. Judge the container
by these instead:
The container prints a lot at startup — xkbcomp keysym warnings,
`Xlib: extension "DPMS" missing`. They are normal in a headless container and do
not stop the application. Judge the container by these instead:

```sh
docker exec coretrace-gui-atp xwininfo -root -tree -display :99 | grep "CTrace GUI"
docker exec coretrace-gui-atp xwininfo -root -tree -display :99 | grep '"CoreTrace"'
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:6080/
```

The first must list a `1200x800` window, the second must print `200`. If both
The first must list a `1440x940` window, the second must print `200`. If both
hold, the application is running and the problem is the browser view: open
<http://localhost:6080/vnc_lite.html>, or hard-reload the page (Ctrl+Shift+R) in
case an older non-connecting version of it was cached.

## Setup B — Linux, native AppImage

**Install clang 20 first.** The analyser compiles the source under test to LLVM
IR before analysing it, and needs a toolchain matching the LLVM it links
(`libclang-cpp.so.20.1`). Ubuntu 24.04 ships clang 18, which is not enough:
it fails with `'stddef.h' file not found`.

This matters more than it looks. **Without clang 20 the analyser does not
report an error — it returns "0 diagnostics" for every file.** `F29` (clean
file, no diagnostic) would then appear to pass while nothing is being analysed
at all, and `F28`, `F30`, `F32` and `F33` would fail for a reason no message
explains. Scenario **F27b** exists to catch exactly that, and must be run
before the analysis scenarios.

```sh
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key \
| sudo gpg --dearmor -o /usr/share/keyrings/llvm.gpg
echo "deb [signed-by=/usr/share/keyrings/llvm.gpg] http://apt.llvm.org/noble/ llvm-toolchain-noble-20 main" \
| sudo tee /etc/apt/sources.list.d/llvm20.list
sudo apt-get update && sudo apt-get install -y clang-20
sudo ln -sf /usr/bin/clang-20 /usr/bin/clang
clang --version # must report 20.x
```

Then run the application:

```sh
wget https://github.com/CoreTrace/coretrace-gui/releases/download/v5.1.0/CtraceGUI-5.1.0.AppImage
chmod +x CtraceGUI-5.1.0.AppImage
./CtraceGUI-5.1.0.AppImage
```

If your distribution has no FUSE 2 runtime, run
`./CtraceGUI-5.1.0.AppImage --appimage-extract` and start `squashfs-root/AppRun`
instead.
<http://localhost:6080/vnc_lite.html>, or hard-reload the page (Ctrl+Shift+R).

> The Docker environment in Setup A already contains clang 20, which is the
> main reason to prefer it: the analysis scenarios are reproducible there
> without touching your machine's toolchain.
## The analyser

## Setup C — macOS, Apple Silicon (native)

Download
[CtraceGUI-5.1.0-arm64.dmg](https://github.com/CoreTrace/coretrace-gui/releases/download/v5.1.0/CtraceGUI-5.1.0-arm64.dmg),
open it, and drag **CtraceGUI** to Applications.

The build is **unsigned** — signing needs a paid Apple Developer account — so
macOS quarantines it and refuses the first launch. That refusal is expected and
is part of scenario **F3**; it is not a defect. Get past it one of two ways,
depending on what macOS tells you:

- *"cannot be opened because the developer cannot be verified"* →
**right-click** (or Control-click) the app in Applications and choose
**Open**, then click **Open** in the dialog. Double-clicking will not work:
only the Open menu item offers the override.
- *"is damaged and can't be opened"* → that dialog has no Open button, so clear
the quarantine flag instead:

```sh
xattr -cr /Applications/CtraceGUI.app
```

Either way, macOS remembers the decision and every later launch is a normal
double-click.

> On Ventura and later the app may instead appear under **System Settings →
> Privacy & Security**; click **Open Anyway** there and authenticate.

**Analysis does not run on macOS.** The bundled `ctrace` is a Linux binary, so
**Run Analysis** reports that explicitly (scenario **F54**) instead of failing
silently. Everything else — editor, explorer, search, terminal, assistant,
session restore — works. Run the analysis scenarios **F28 to F38** on Linux.

## Setup D — macOS, Intel

v5.1.0 publishes no x64 artifact, so Intel Macs go through the Docker
environment (scenario **F3b**):

```sh
cd appendices/docker
./build.sh
./run.sh
```
CoreTrace Desktop ships no analyser: local analysis uses a `ctrace` program
chosen once in *Paramètres*. The image provides **ctrace v0.85.0** at
`/opt/coretrace/bin/ctrace`, with clang 20, which it needs to compile the file
under test.

then open <http://localhost:6080/>. Allocate at least 6 GB of RAM to
Docker Desktop (Settings → Resources).
- The release notes of v6.0.0-beta.3 point to ctrace v0.74.0. That version
writes a report the application cannot read (`Format de rapport non reconnu`),
which is why the image uses v0.85.0.
- Only the stack analyser built into ctrace runs. cppcheck, tscancode, ikos and
flawfinder are not installed; *Paramètres* lists them as missing.

## Reporting an anomaly

Fill the **Feedback** column of the test plan with, in this order:

1. `PASS` or `FAIL`.
2. Setup used (`Docker/Linux`, `Native/Linux`, `Native/macOS-AppleSilicon`,
`Docker/macOS-Intel`).
2. Host operating system (`Linux`, `macOS`, `Windows`).
3. For a `FAIL`: what you observed instead of the expected result, and the
severity — **blocking** (the feature is unusable or the app crashes),
**major** (wrong result produced), **minor** (cosmetic or wording).
Expand Down
2 changes: 1 addition & 1 deletion atp/appendices/datasets/DS02_clean.c
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

void copy_name(const char *name)
{
char buf[16];
char buf[16] = {0};
snprintf(buf, sizeof(buf), "%s", name);
printf("%s\n", buf);
}
Expand Down
Loading
Loading