diff --git a/bundles/rf-agentskills/skills/rf-requests/SKILL.md b/bundles/rf-agentskills/skills/rf-requests/SKILL.md index 2cd3625..658c755 100644 --- a/bundles/rf-agentskills/skills/rf-requests/SKILL.md +++ b/bundles/rf-agentskills/skills/rf-requests/SKILL.md @@ -96,6 +96,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (` | references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` | | references/authentication.md | basic, digest, NTLM, client-certificate or token authentication | | references/troubleshooting.md | a request fails with an error you don't recognise | +| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty | Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`. diff --git a/bundles/rf-agentskills/skills/rf-requests/references/loose-assertions.md b/bundles/rf-agentskills/skills/rf-requests/references/loose-assertions.md new file mode 100644 index 0000000..091ce03 --- /dev/null +++ b/bundles/rf-agentskills/skills/rf-requests/references/loose-assertions.md @@ -0,0 +1,20 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` | +| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks | +| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent | +| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture | +| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error | + +## Rules + +- Give each check a `msg=` that names the field, so a failure says what was wrong. +- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status. diff --git a/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md b/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md index 66b5312..4347e99 100644 --- a/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md +++ b/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`, | references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` | | references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates | | references/troubleshooting.md | A request or assertion fails with an error you don't understand | +| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key | Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations). diff --git a/bundles/rf-agentskills/skills/rf-restinstance/references/loose-assertions.md b/bundles/rf-agentskills/skills/rf-restinstance/references/loose-assertions.md new file mode 100644 index 0000000..05ba5e0 --- /dev/null +++ b/bundles/rf-agentskills/skills/rf-restinstance/references/loose-assertions.md @@ -0,0 +1,18 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range | +| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets | +| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks | + +## Rules + +- Use the exact expected value for each field the test cares about. +- Use types and constraints for shape only, never as the only check of a value. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status. diff --git a/plugins/rf-api/skills/rf-requests/SKILL.md b/plugins/rf-api/skills/rf-requests/SKILL.md index 2cd3625..658c755 100644 --- a/plugins/rf-api/skills/rf-requests/SKILL.md +++ b/plugins/rf-api/skills/rf-requests/SKILL.md @@ -96,6 +96,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (` | references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` | | references/authentication.md | basic, digest, NTLM, client-certificate or token authentication | | references/troubleshooting.md | a request fails with an error you don't recognise | +| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty | Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`. diff --git a/plugins/rf-api/skills/rf-requests/references/loose-assertions.md b/plugins/rf-api/skills/rf-requests/references/loose-assertions.md new file mode 100644 index 0000000..091ce03 --- /dev/null +++ b/plugins/rf-api/skills/rf-requests/references/loose-assertions.md @@ -0,0 +1,20 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` | +| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks | +| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent | +| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture | +| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error | + +## Rules + +- Give each check a `msg=` that names the field, so a failure says what was wrong. +- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status. diff --git a/plugins/rf-api/skills/rf-restinstance/SKILL.md b/plugins/rf-api/skills/rf-restinstance/SKILL.md index 66b5312..4347e99 100644 --- a/plugins/rf-api/skills/rf-restinstance/SKILL.md +++ b/plugins/rf-api/skills/rf-restinstance/SKILL.md @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`, | references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` | | references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates | | references/troubleshooting.md | A request or assertion fails with an error you don't understand | +| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key | Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations). diff --git a/plugins/rf-api/skills/rf-restinstance/references/loose-assertions.md b/plugins/rf-api/skills/rf-restinstance/references/loose-assertions.md new file mode 100644 index 0000000..05ba5e0 --- /dev/null +++ b/plugins/rf-api/skills/rf-restinstance/references/loose-assertions.md @@ -0,0 +1,18 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range | +| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets | +| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks | + +## Rules + +- Use the exact expected value for each field the test cares about. +- Use types and constraints for shape only, never as the only check of a value. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status.