feat(stack): add a strict mode to known-failures

`stack known-failures` gains a `--strict` flag.
The Model Context Protocol (MCP) tool `eb_stack_known_failures` gains a matching `strict: true` property.
With the flag, the command prints the same report.
It then exits 1 when a warning remains.
A pre-bot script or continuous integration (CI) job can now fail on a known failure that has no remedy yet.
In strict mode, the MCP tool returns a tool error whose text lists those warnings.
Remedied hits never count.
A plan with every hit remedied exits 0.
Without `--strict`, the behavior and output stay byte-for-byte the same.

A new helper, `KnownFailuresReport::strict_error`, builds one line per remaining warning.
Each line names the easyconfig and the architecture.
The command-line and MCP layers make the strict decision with that helper.
`run_known_failures` and its stdout stay unchanged.

The command-line reference, the arch-known-failures how-to, the eessi-extend skill and the changelog document the flag.
`llms-full.txt` now matches the docs again.

Author: HaoZeke

CHANGELOG.md

@@ -387,6 +387,12 @@ All notable changes to this project are documented here.
 
 ### Added
 
+- `stack known-failures` and the MCP tool `eb_stack_known_failures` take a
+  `--strict` flag (`strict: true`). It prints the same report, then exits 1
+  (returns a tool error that lists the warnings) when a warning remains, so a
+  pre-bot script or CI job can fail on a known failure with no remedy in
+  place. Remedied hits never count, so a plan whose every hit is remedied
+  exits 0. Without it the behavior and output are unchanged.
 - `stack bot-classify` and the MCP tool `eb_stack_bot_classify` read EESSI
   bot comments, `_bot_job<id>.result` files and job output, and classify
   each bot job as a real failure (with the campaign build-finding class),

docs/orgmode/howto/arch-known-failures.org

@@ -93,6 +93,24 @@ so a GROMACS without CUDA is not warned about, and ~min_easybuild =
 entries, ~in EESSI:~ names the hook that already applies the remedy, such
 as ~pre_single_extension_orjson~ for the a64fx orjson test.
 
+* Failing a pre-bot check on a remaining warning
+
+By default a warning is advice and the command exits 0. Pass ~--strict~
+to exit 1 while any warning remains, so a pre-bot script or CI job fails
+on a known failure that has no remedy in place yet:
+
+#+begin_src bash
+eb-stack stack known-failures --strict \
+  --arch aarch64/a64fx \
+  --easystack tests/arch_known_failures_fixtures/easystacks/1829/eessi-2026.06-eb-5.4.1-2026.1.yml \
+  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs
+#+end_src
+
+The stdout is the same report as without ~--strict~; the error on stderr
+names each remaining warning by easyconfig and target, here
+~orjson-3.11.8-GCCcore-15.2.0.eb on aarch64/a64fx~. A remedied hit never
+counts, so the merged 1670 easystack above exits 0 under ~--strict~.
+
 * Choosing architectures
 
 ~--arch~ repeats and takes a target or a family. ~--arch aarch64~ meets

docs/orgmode/reference/cli.org

@@ -15,7 +15,7 @@ are deterministic bundle artifacts.
 | 1 | The work was attempted and failed: a missing or unreadable file, a document that does not parse, a check that found errors, a network fetch that failed, a doctor probe, or a campaign whose build failed |
 | 2 | The command line was rejected before that work: a missing or unknown flag (clap), or a flag value or combination the command does not accept |
 
-A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails ~recipe lint~ or ~recipe check~ is status 1: the files were read and the check failed. ~stack from-commit~ is status 1 when any pin is ~released~, and status 0 when every pin is ~newer~ or ~unknown~ or the files name no pin. A file name with no ~eb-X.Y.Z~, and no ~--easybuild-version~, is status 2. ~stack arch~ is status 1 when an entry is unsupported or flagged, or when ~--probe-sources~ reports ~unreachable~ or ~arch-missing~ sources. ~stack files~ is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are ~no_url~ or ~unchecked~. ~stack generation~ is status 1 on any finding, and status 2 when an easystack path names no EESSI version and ~--eessi-version~ is absent. ~stack known-failures~ is status 0 when it prints warnings: they are advice for the pull request, not a refusal. ~stack oom~ is status 0 when it warns: the warnings are advice, not a failed check. ~stack bot-classify~ is status 0 for every class, and status 2 when a path is a directory. ~stack batch~ is status 1 when the plan includes, or depends on, an entry the architecture cannot build.
+A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails ~recipe lint~ or ~recipe check~ is status 1: the files were read and the check failed. ~stack from-commit~ is status 1 when any pin is ~released~, and status 0 when every pin is ~newer~ or ~unknown~ or the files name no pin. A file name with no ~eb-X.Y.Z~, and no ~--easybuild-version~, is status 2. ~stack arch~ is status 1 when an entry is unsupported or flagged, or when ~--probe-sources~ reports ~unreachable~ or ~arch-missing~ sources. ~stack files~ is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are ~no_url~ or ~unchecked~. ~stack generation~ is status 1 on any finding, and status 2 when an easystack path names no EESSI version and ~--eessi-version~ is absent. ~stack known-failures~ is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with ~--strict~ it is status 1 when a warning remains, and status 0 when every hit is remedied. ~stack oom~ is status 0 when it warns: the warnings are advice, not a failed check. ~stack bot-classify~ is status 0 for every class, and status 2 when a path is a directory. ~stack batch~ is status 1 when the plan includes, or depends on, an entry the architecture cannot build.
 
 ~--help~ for every command has an example and a description of every flag. A default that the command applies is printed in that help. A directory that receives several artifacts is ~--out-dir~. A single output file is ~--out~ (~--lock-out~ for the lock ~stack solve~ writes). ~--installed~, ~--easyconfigs~, ~--filter-deps~, and ~--generated-at~ use those names on every command that takes them.
 
@@ -840,6 +840,19 @@ easyconfig that already contains it, turns the line into one
 ~remedied:~ line. ~known=~ counts warnings and ~remedied=~ counts
 those. The command exits 0 either way.
 
+~--strict~ exits 1 when a warning remains after the report, so a
+pre-bot script or CI job can fail on a known failure that has no remedy
+in place. The stdout is the same report; the error on stderr names each
+remaining warning by easyconfig and target. Remedied hits never count,
+so a plan whose every hit is remedied exits 0.
+
+#+begin_src bash
+eb-stack stack known-failures --strict \
+  --arch aarch64/a64fx \
+  --easystack tests/arch_known_failures_fixtures/easystacks/1829/eessi-2026.06-eb-5.4.1-2026.1.yml \
+  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs
+#+end_src
+
 ~--arch~ takes a target (~aarch64/a64fx~) or a family (~aarch64~), and
 repeats. A family meets each of its targets. No ~--arch~ checks every
 architecture in the table. ~--easyconfigs~ is the robot tree for the

llms-full.txt

@@ -3857,6 +3857,24 @@ so a GROMACS without CUDA is not warned about, and ~min_easybuild =
 entries, `in EESSI:` names the hook that already applies the remedy, such
 as `pre_single_extension_orjson` for the a64fx orjson test.
 
+# Failing a pre-bot check on a remaining warning
+
+By default a warning is advice and the command exits 0. Pass `--strict`
+to exit 1 while any warning remains, so a pre-bot script or CI job fails
+on a known failure that has no remedy in place yet:
+
+```bash
+eb-stack stack known-failures --strict \
+  --arch aarch64/a64fx \
+  --easystack tests/arch_known_failures_fixtures/easystacks/1829/eessi-2026.06-eb-5.4.1-2026.1.yml \
+  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs
+```
+
+The stdout is the same report as without `--strict`; the error on stderr
+names each remaining warning by easyconfig and target, here
+`orjson-3.11.8-GCCcore-15.2.0.eb on aarch64/a64fx`. A remedied hit never
+counts, so the merged 1670 easystack above exits 0 under `--strict`.
+
 # Choosing architectures
 
 `--arch` repeats and takes a target or a family. `--arch aarch64` meets
@@ -4302,7 +4320,7 @@ are deterministic bundle artifacts.
 | 1 | The work was attempted and failed: a missing or unreadable file, a document that does not parse, a check that found errors, a network fetch that failed, a doctor probe, or a campaign whose build failed |
 | 2 | The command line was rejected before that work: a missing or unknown flag (clap), or a flag value or combination the command does not accept |
 
-A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails `recipe lint` or `recipe check` is status 1: the files were read and the check failed. `stack from-commit` is status 1 when any pin is `released`, and status 0 when every pin is `newer` or `unknown` or the files name no pin. A file name with no `eb-X.Y.Z`, and no `--easybuild-version`, is status 2. `stack arch` is status 1 when an entry is unsupported or flagged, or when `--probe-sources` reports `unreachable` or `arch-missing` sources. `stack files` is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are `no_url` or `unchecked`. `stack generation` is status 1 on any finding, and status 2 when an easystack path names no EESSI version and `--eessi-version` is absent. `stack known-failures` is status 0 when it prints warnings: they are advice for the pull request, not a refusal. `stack oom` is status 0 when it warns: the warnings are advice, not a failed check. `stack bot-classify` is status 0 for every class, and status 2 when a path is a directory. `stack batch` is status 1 when the plan includes, or depends on, an entry the architecture cannot build.
+A rejected value names the flag and the value. A missing file names the flag and the path. Status 2 is the same status clap already uses. A recipe that fails `recipe lint` or `recipe check` is status 1: the files were read and the check failed. `stack from-commit` is status 1 when any pin is `released`, and status 0 when every pin is `newer` or `unknown` or the files name no pin. A file name with no `eb-X.Y.Z`, and no `--easybuild-version`, is status 2. `stack arch` is status 1 when an entry is unsupported or flagged, or when `--probe-sources` reports `unreachable` or `arch-missing` sources. `stack files` is status 1 when a patch or an easyconfig is missing, and status 0 when the only lines are `no_url` or `unchecked`. `stack generation` is status 1 on any finding, and status 2 when an easystack path names no EESSI version and `--eessi-version` is absent. `stack known-failures` is status 0 when it prints warnings: they are advice for the pull request, not a refusal; with `--strict` it is status 1 when a warning remains, and status 0 when every hit is remedied. `stack oom` is status 0 when it warns: the warnings are advice, not a failed check. `stack bot-classify` is status 0 for every class, and status 2 when a path is a directory. `stack batch` is status 1 when the plan includes, or depends on, an entry the architecture cannot build.
 
 `--help` for every command has an example and a description of every flag. A default that the command applies is printed in that help. A directory that receives several artifacts is `--out-dir`. A single output file is `--out` (`--lock-out` for the lock `stack solve` writes). `--installed`, `--easyconfigs`, `--filter-deps`, and `--generated-at` use those names on every command that takes them.
 
@@ -5127,6 +5145,19 @@ easyconfig that already contains it, turns the line into one
 `remedied:` line. `known`` counts warnings and `remedied`` counts
 those. The command exits 0 either way.
 
+`--strict` exits 1 when a warning remains after the report, so a
+pre-bot script or CI job can fail on a known failure that has no remedy
+in place. The stdout is the same report; the error on stderr names each
+remaining warning by easyconfig and target. Remedied hits never count,
+so a plan whose every hit is remedied exits 0.
+
+```bash
+eb-stack stack known-failures --strict \
+  --arch aarch64/a64fx \
+  --easystack tests/arch_known_failures_fixtures/easystacks/1829/eessi-2026.06-eb-5.4.1-2026.1.yml \
+  --easyconfigs tests/arch_known_failures_fixtures/easyconfigs
+```
+
 `--arch` takes a target (`aarch64/a64fx`) or a family (`aarch64`), and
 repeats. A family meets each of its targets. No `--arch` checks every
 architecture in the table. `--easyconfigs` is the robot tree for the

skills/eessi-extend/SKILL.md

@@ -311,7 +311,9 @@ line, an `ARCH` conditional, or a pin), and the EESSI hook that applies
 it, if one does. Put the remedy in the easyconfig, or the warning in the
 pull request description, before the bot runs. Do not answer a known
 failure with a rerun. A `remedied:` line means a pin or the easyconfig
-already carries the fix. The command exits 0. The table is
+already carries the fix. The command exits 0; add `--strict` to exit 1
+when a warning remains (remedied hits never count), so a pre-bot script
+or CI job fails on a known failure with no remedy in place. The table is
 `data/arch-known-failures.toml`, and the how-to is
 `docs/orgmode/howto/arch-known-failures.org`.
 

src/arch_known_failures.rs

@@ -388,6 +388,22 @@ pub struct KnownHit {
 }
 
 impl KnownHit {
+    /// A one-line summary naming the easyconfig, the architectures, and the
+    /// phase, for the `--strict` error text.
+    pub fn warning_summary(&self) -> String {
+        let mut subject = self.file.clone();
+        if let Some(extension) = &self.extension {
+            subject.push_str(&format!(" (extension {extension})"));
+        }
+        let arches = self
+            .arches
+            .iter()
+            .map(|arch| arch_text(arch))
+            .collect::<Vec<_>>()
+            .join(", ");
+        format!("{subject} on {arches}: known {}", self.phase.label())
+    }
+
     /// The lines the command prints for this hit, without a final newline.
     pub fn message(&self) -> String {
         let mut subject = self.file.clone();
@@ -836,6 +852,14 @@ impl KnownFailuresReport {
     pub fn warnings(&self) -> impl Iterator<Item = &KnownHit> {
         self.hits.iter().filter(|hit| hit.remedied.is_none())
     }
+
+    /// The `--strict` error: one line per remaining warning, each naming the
+    /// easyconfig, the architectures, and the phase. `None` when every hit is
+    /// remedied, so `--strict` exits 0.
+    pub fn strict_error(&self) -> Option<String> {
+        let lines: Vec<String> = self.warnings().map(KnownHit::warning_summary).collect();
+        (!lines.is_empty()).then(|| lines.join("\n"))
+    }
 }
 
 /// Warn about known per-architecture failures a plan includes or depends on.

src/main.rs

@@ -1271,9 +1271,11 @@ enum StackCommand {
     /// remedy as easyconfig text, and the EESSI hook that applies it, if
     /// one does. A from-commit or include-easyblocks-from-commit pin that
     /// carries the fix, or an easyconfig that already has it, is reported
-    /// as remedied. The command exits 0 either way.
+    /// as remedied. The command exits 0 either way, unless `--strict` is
+    /// given, when a remaining warning makes it exit 1. Remedied hits never
+    /// count against `--strict`.
     #[command(
-        after_long_help = "Examples:\n  eb-stack stack known-failures --arch aarch64/a64fx --easystack eessi-2026.06-eb-5.4.1-2026.1.yml --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs\n  eb-stack stack known-failures --easystack eessi-2025.06-eb-5.4.0-2025b.yml --installed installed-zen4.txt --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs"
+        after_long_help = "Examples:\n  eb-stack stack known-failures --arch aarch64/a64fx --easystack eessi-2026.06-eb-5.4.1-2026.1.yml --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs\n  eb-stack stack known-failures --easystack eessi-2025.06-eb-5.4.0-2025b.yml --installed installed-zen4.txt --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs\n  eb-stack stack known-failures --strict --arch aarch64/a64fx --easystack eessi-2026.06-eb-5.4.1-2026.1.yml --easyconfigs easybuild-easyconfigs/easybuild/easyconfigs"
     )]
     KnownFailures {
         /// Target, `aarch64/a64fx`, or family, `aarch64`. Repeatable.
@@ -1302,6 +1304,10 @@ enum StackCommand {
         /// easystack file name.
         #[arg(long)]
         easybuild_version: Option<String>,
+        /// Exit 1 when a warning remains after the report. Remedied hits
+        /// never count. Without it the command exits 0 either way.
+        #[arg(long)]
+        strict: bool,
     },
     /// Warn about plan entries likely to run out of memory in a bot job.
     ///
@@ -2896,6 +2902,7 @@ fn run_stack(command: StackCommand) -> Result<()> {
             known_failures,
             eessi_version,
             easybuild_version,
+            strict,
         } => {
             let report = eb_stack::arch_known_failures::run_known_failures(
                 &eb_stack::arch_known_failures::KnownFailuresRequest {
@@ -2909,6 +2916,11 @@ fn run_stack(command: StackCommand) -> Result<()> {
                 },
             )?;
             print!("{}", report.stdout);
+            if strict {
+                if let Some(error) = report.strict_error() {
+                    bail!("{error}");
+                }
+            }
             Ok(())
         }
         StackCommand::Oom {

src/mcp.rs

@@ -471,7 +471,7 @@ fn tool_catalog() -> Vec<Value> {
         ),
         tool_with_optional(
             "eb_stack_known_failures",
-            "Warn about known per-architecture failures a plan includes or depends on, as `stack known-failures` does. Each warning names the easyconfig, the architectures, the symptom, the established easyconfig remedy as text, and the EESSI hook that applies it. A pin or easyconfig that already carries the fix is reported as remedied. Warnings are not a tool error.",
+            "Warn about known per-architecture failures a plan includes or depends on, as `stack known-failures` does. Each warning names the easyconfig, the architectures, the symptom, the established easyconfig remedy as text, and the EESSI hook that applies it. A pin or easyconfig that already carries the fix is reported as remedied. Warnings are not a tool error, except with strict: true, when a remaining warning returns a tool error whose text lists the warnings.",
             &[("easystack", "array_min1")],
             &[
                 ("arch", "array"),
@@ -480,6 +480,7 @@ fn tool_catalog() -> Vec<Value> {
                 ("known_failures", "string"),
                 ("eessi_version", "string"),
                 ("easybuild_version", "string"),
+                ("strict", "boolean"),
             ],
         ),
         tool_with_optional(
@@ -3211,6 +3212,11 @@ fn stack_known_failures_tool(arguments: &Value) -> Result<Value, String> {
         },
     )
     .map_err(workflow_error)?;
+    if optional_bool(arguments, "strict") {
+        if let Some(error) = report.strict_error() {
+            return Err(error);
+        }
+    }
     Ok(json!({
         "text": report.stdout,
         "hits": report.hits,