From b3ceff25cb0844716e14297a2cbbc48849d45d5a Mon Sep 17 00:00:00 2001 From: Abdulwahab Isam Date: Sun, 15 Nov 2026 10:00:42 +0300 Subject: [PATCH 1/2] docs(rf-core): add failure fingerprints to debug Add a fingerprint step and a rule-out checklist to rf-debug-expert. The agent names each failure group by a stable signature and checks the test before it reports an app or environment cause. The bundle and the agent variants are regenerated from the plugin. --- bundles/rf-agentskills/agents/rf-debug-expert.md | 16 ++++++++++++++++ plugins/rf-core/agents/rf-debug-expert.md | 16 ++++++++++++++++ .../variants/codex/agents/rf-debug-expert.toml | 16 ++++++++++++++++ .../variants/opencode/agents/rf-debug-expert.md | 16 ++++++++++++++++ 4 files changed, 64 insertions(+) diff --git a/bundles/rf-agentskills/agents/rf-debug-expert.md b/bundles/rf-agentskills/agents/rf-debug-expert.md index 6526463..d6a181a 100644 --- a/bundles/rf-agentskills/agents/rf-debug-expert.md +++ b/bundles/rf-agentskills/agents/rf-debug-expert.md @@ -28,6 +28,8 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. + ### 3. Classify each cause | Failure pattern | Category | Typical cause | @@ -68,6 +70,19 @@ whole directory with `--break-on-failed-test` instead of one longname. Retry loops and longer timeouts hide the cause; recommend them only as a documented temporary measure after the root cause is named. +### 6. Rule out the test before reporting an app or environment cause + +Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. + +- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. +- **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. +- **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data. +- **Long runs.** A 401 late in a long run usually means an expired token. Renew the token and rerun. +- **Absent is not null.** A missing key fails a null check. Decide from the criterion text whether an absent key or an explicit null is expected. +- **Correlation is not cause.** One crash during a test is a lead. Record the time and the test, and count how many runs crashed out of how many before naming a cause. +- **Trace the expected value.** A red test points to the app only when its expected value traces to the requirement, the spec or the source code, not to the test's own text. + ## Routing | Work | Skill | @@ -87,6 +102,7 @@ One block per root cause: ``` FAILURE: [longname(s)] +FINGERPRINT: [signature from section 2] CATEGORY: [Locator | Timing | Assertion | Environment | Keyword resolution | Library | Driver | Flakiness] ROOT CAUSE: [one-sentence explanation] EVIDENCE: [error message, log line or debugger value] diff --git a/plugins/rf-core/agents/rf-debug-expert.md b/plugins/rf-core/agents/rf-debug-expert.md index 6526463..d6a181a 100644 --- a/plugins/rf-core/agents/rf-debug-expert.md +++ b/plugins/rf-core/agents/rf-debug-expert.md @@ -28,6 +28,8 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. + ### 3. Classify each cause | Failure pattern | Category | Typical cause | @@ -68,6 +70,19 @@ whole directory with `--break-on-failed-test` instead of one longname. Retry loops and longer timeouts hide the cause; recommend them only as a documented temporary measure after the root cause is named. +### 6. Rule out the test before reporting an app or environment cause + +Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. + +- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. +- **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. +- **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data. +- **Long runs.** A 401 late in a long run usually means an expired token. Renew the token and rerun. +- **Absent is not null.** A missing key fails a null check. Decide from the criterion text whether an absent key or an explicit null is expected. +- **Correlation is not cause.** One crash during a test is a lead. Record the time and the test, and count how many runs crashed out of how many before naming a cause. +- **Trace the expected value.** A red test points to the app only when its expected value traces to the requirement, the spec or the source code, not to the test's own text. + ## Routing | Work | Skill | @@ -87,6 +102,7 @@ One block per root cause: ``` FAILURE: [longname(s)] +FINGERPRINT: [signature from section 2] CATEGORY: [Locator | Timing | Assertion | Environment | Keyword resolution | Library | Driver | Flakiness] ROOT CAUSE: [one-sentence explanation] EVIDENCE: [error message, log line or debugger value] diff --git a/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml b/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml index f8ddccd..77b3e8d 100644 --- a/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml +++ b/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml @@ -26,6 +26,8 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. + ### 3. Classify each cause | Failure pattern | Category | Typical cause | @@ -66,6 +68,19 @@ whole directory with `--break-on-failed-test` instead of one longname. Retry loops and longer timeouts hide the cause; recommend them only as a documented temporary measure after the root cause is named. +### 6. Rule out the test before reporting an app or environment cause + +Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. + +- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. +- **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. +- **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data. +- **Long runs.** A 401 late in a long run usually means an expired token. Renew the token and rerun. +- **Absent is not null.** A missing key fails a null check. Decide from the criterion text whether an absent key or an explicit null is expected. +- **Correlation is not cause.** One crash during a test is a lead. Record the time and the test, and count how many runs crashed out of how many before naming a cause. +- **Trace the expected value.** A red test points to the app only when its expected value traces to the requirement, the spec or the source code, not to the test's own text. + ## Routing | Work | Skill | @@ -85,6 +100,7 @@ One block per root cause: ``` FAILURE: [longname(s)] +FINGERPRINT: [signature from section 2] CATEGORY: [Locator | Timing | Assertion | Environment | Keyword resolution | Library | Driver | Flakiness] ROOT CAUSE: [one-sentence explanation] EVIDENCE: [error message, log line or debugger value] diff --git a/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md b/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md index 0292040..4f3d795 100644 --- a/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md +++ b/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md @@ -31,6 +31,8 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. + ### 3. Classify each cause | Failure pattern | Category | Typical cause | @@ -71,6 +73,19 @@ whole directory with `--break-on-failed-test` instead of one longname. Retry loops and longer timeouts hide the cause; recommend them only as a documented temporary measure after the root cause is named. +### 6. Rule out the test before reporting an app or environment cause + +Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. + +- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. +- **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. +- **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data. +- **Long runs.** A 401 late in a long run usually means an expired token. Renew the token and rerun. +- **Absent is not null.** A missing key fails a null check. Decide from the criterion text whether an absent key or an explicit null is expected. +- **Correlation is not cause.** One crash during a test is a lead. Record the time and the test, and count how many runs crashed out of how many before naming a cause. +- **Trace the expected value.** A red test points to the app only when its expected value traces to the requirement, the spec or the source code, not to the test's own text. + ## Routing | Work | Skill | @@ -90,6 +105,7 @@ One block per root cause: ``` FAILURE: [longname(s)] +FINGERPRINT: [signature from section 2] CATEGORY: [Locator | Timing | Assertion | Environment | Keyword resolution | Library | Driver | Flakiness] ROOT CAUSE: [one-sentence explanation] EVIDENCE: [error message, log line or debugger value] From b1e34a833e69b8c346955d86d4fc5189a43b0002 Mon Sep 17 00:00:00 2001 From: Abdulwahab Isam Date: Sun, 15 Nov 2026 10:49:59 +0300 Subject: [PATCH 2/2] docs: name RequestsLibrary order, place fingerprint Section 6 now says RequestsLibrary puts the actual value first and that other libraries order it differently. Section 2 now says the fingerprint goes in the bug title or BUGS.md entry. Same text in all four copies. Checked: cmp of plugin and bundle copies is identical; grep finds no bare python scripts/ command. scripts/check-drift.sh could not run: Git Bash python is the Windows Store alias (log ../drift.log). --- bundles/rf-agentskills/agents/rf-debug-expert.md | 4 ++-- plugins/rf-core/agents/rf-debug-expert.md | 4 ++-- plugins/rf-core/variants/codex/agents/rf-debug-expert.toml | 4 ++-- plugins/rf-core/variants/opencode/agents/rf-debug-expert.md | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/bundles/rf-agentskills/agents/rf-debug-expert.md b/bundles/rf-agentskills/agents/rf-debug-expert.md index d6a181a..b3968fe 100644 --- a/bundles/rf-agentskills/agents/rf-debug-expert.md +++ b/bundles/rf-agentskills/agents/rf-debug-expert.md @@ -28,7 +28,7 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. -Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. Put the fingerprint in the bug title or the BUGS.md entry, not only in a file name. ### 3. Classify each cause @@ -74,7 +74,7 @@ temporary measure after the root cause is named. Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. -- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Read the comparison in the right order.** RequestsLibrary puts the actual value first: in `Expected status: 400 != 200` the server answered 400 and the test expected 200. Other libraries order it differently, so check the library before reading the message. It does not show that validation stopped working. - **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. - **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. - **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data. diff --git a/plugins/rf-core/agents/rf-debug-expert.md b/plugins/rf-core/agents/rf-debug-expert.md index d6a181a..b3968fe 100644 --- a/plugins/rf-core/agents/rf-debug-expert.md +++ b/plugins/rf-core/agents/rf-debug-expert.md @@ -28,7 +28,7 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. -Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. Put the fingerprint in the bug title or the BUGS.md entry, not only in a file name. ### 3. Classify each cause @@ -74,7 +74,7 @@ temporary measure after the root cause is named. Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. -- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Read the comparison in the right order.** RequestsLibrary puts the actual value first: in `Expected status: 400 != 200` the server answered 400 and the test expected 200. Other libraries order it differently, so check the library before reading the message. It does not show that validation stopped working. - **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. - **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. - **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data. diff --git a/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml b/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml index 77b3e8d..45e80f8 100644 --- a/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml +++ b/plugins/rf-core/variants/codex/agents/rf-debug-expert.toml @@ -26,7 +26,7 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. -Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. Put the fingerprint in the bug title or the BUGS.md entry, not only in a file name. ### 3. Classify each cause @@ -72,7 +72,7 @@ temporary measure after the root cause is named. Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. -- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Read the comparison in the right order.** RequestsLibrary puts the actual value first: in `Expected status: 400 != 200` the server answered 400 and the test expected 200. Other libraries order it differently, so check the library before reading the message. It does not show that validation stopped working. - **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. - **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. - **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data. diff --git a/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md b/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md index 4f3d795..85d3061 100644 --- a/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md +++ b/plugins/rf-core/variants/opencode/agents/rf-debug-expert.md @@ -31,7 +31,7 @@ applies the fix and re-runs the tests. Failures that share a message, a keyword or a suite are one cause until proven otherwise (`robotcode results stats --by suite --failed`). Diagnose each cause once. -Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. +Give each group a fingerprint: take the error message, keep the status codes and counts, replace concrete ids with a placeholder, and drop hosts, query strings and timestamps. Keep endpoint paths as templates (`/orders/{id}`, not `/orders/42`). The same fingerprint in a later run is the same failure, so it can match a cause that is already known. Put the fingerprint in the bug title or the BUGS.md entry, not only in a file name. ### 3. Classify each cause @@ -77,7 +77,7 @@ temporary measure after the root cause is named. Check these before a cause is reported as an app or environment defect. Each one is a common way to blame the system for a problem in the test. -- **Read the comparison in the right order.** In `Expected status: 400 != 200` the server answered 400 and the test expected 200. It does not show that validation stopped working. +- **Read the comparison in the right order.** RequestsLibrary puts the actual value first: in `Expected status: 400 != 200` the server answered 400 and the test expected 200. Other libraries order it differently, so check the library before reading the message. It does not show that validation stopped working. - **Find the step that failed.** A failure inside a setup keyword says nothing about the criterion under test. Open the failing keyword chain and name the step. - **Outage first.** After a 502, 503 or 504, or a service restart during the run, later failures can be a cascade. Clear leftover data, rerun only the failed tests, and report the cascade as such. - **Shared state inside one test.** Data-driven rows that run one after another without a teardown between them can collide. Give each row its own data.