diff --git a/bundles/rf-agentskills/skills/rf-requests/SKILL.md b/bundles/rf-agentskills/skills/rf-requests/SKILL.md index 2cd3625..5ff9f70 100644 --- a/bundles/rf-agentskills/skills/rf-requests/SKILL.md +++ b/bundles/rf-agentskills/skills/rf-requests/SKILL.md @@ -64,6 +64,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (` | Assert body or headers | `Should Be Equal ${resp.json()}[id] ${1}`, `${resp.headers}[Content-Type]` | references/response-validation.md | | Basic / digest / NTLM / client cert | `Create Session … auth=`, `Create Digest Session`, `Create Ntlm Session`, `Create Client Cert Session` | references/authentication.md | | Bearer token for all later calls | `Update Session api headers=${auth}` | references/authentication.md | +| Wait for an async job to finish | bounded WHILE poll, no Sleep | references/async-jobs.md | | Unexpected error message | — | references/troubleshooting.md | ## Agent workflow @@ -96,6 +97,8 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (` | references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` | | references/authentication.md | basic, digest, NTLM, client-certificate or token authentication | | references/troubleshooting.md | a request fails with an error you don't recognise | +| references/async-jobs.md | a job or another async operation must finish before the test checks its result | +| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty | Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`. diff --git a/bundles/rf-agentskills/skills/rf-requests/references/async-jobs.md b/bundles/rf-agentskills/skills/rf-requests/references/async-jobs.md new file mode 100644 index 0000000..0932883 --- /dev/null +++ b/bundles/rf-agentskills/skills/rf-requests/references/async-jobs.md @@ -0,0 +1,42 @@ +# Wait for an async job + +Use this when an endpoint starts a job (an export, an import, a batch run, a provisioning step) and answers before the work is done. A status endpoint tells you when the job finished. + +## Poll with a bounded WHILE loop + +Do not use one long `Sleep` to wait. A fixed pause is too short on a slow run and wastes time on a fast one. Poll the status endpoint in a `WHILE` loop with a time-based `limit`, pause briefly between polls, and fail with a message when the limit is reached. Put the loop in a keyword; the test only calls it: + +```robotframework +*** Test Cases *** +Export Finishes + ${res}= POST On Session api /exports json=${body} + Status Should Be 202 ${res} + ${id}= Set Variable ${res.json()}[id] + ${state}= Wait For Export To Finish ${id} + Should Be Equal ${state} done msg=Export ${id} ended as ${state} + +*** Keywords *** +Wait For Export To Finish + [Arguments] ${id} + ${state}= Set Variable queued + WHILE $state not in ['done', 'failed'] limit=60s on_limit_message=Export ${id} not finished after 60s + Sleep 2s + ${res}= GET On Session api /exports/${id} + Status Should Be 200 ${res} + ${state}= Set Variable ${res.json()}[status] + END + RETURN ${state} +``` + +`$state` (no braces) passes the variable itself to the Python expression, so a value with quotes in it cannot break the condition. + +## Rules + +- **Set the limit from the measured run time.** `limit=60s` is a time limit; a bare number such as `limit=60` counts iterations instead. Measure how long a normal run takes, then allow a little more. A large limit turns a stuck job into a long wait. +- **Pause between polls.** A short `Sleep` inside the loop (1 to 5 seconds) keeps the loop from hammering the endpoint. If the API offers a long-poll or wait option, use it instead of the `Sleep`. +- **Control flow lives in keywords.** Keep the `WHILE` loop in a keyword such as `Wait For Export To Finish`, not in the test body. +- **Check the effect, not only the state.** When the job reports `done`, read the record or file it produced and assert its fields. A status alone is a loose check. +- **One loop per job.** Do not wrap `Wait Until Keyword Succeeds` around a keyword that already polls; the timeouts multiply. +- **Clean up.** Delete what the job created in the test teardown, by its exact id. + +`WHILE` and its `limit` option exist since Robot Framework 5.0; `on_limit_message` needs Robot Framework 6.1 or later. diff --git a/bundles/rf-agentskills/skills/rf-requests/references/loose-assertions.md b/bundles/rf-agentskills/skills/rf-requests/references/loose-assertions.md new file mode 100644 index 0000000..091ce03 --- /dev/null +++ b/bundles/rf-agentskills/skills/rf-requests/references/loose-assertions.md @@ -0,0 +1,20 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` | +| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks | +| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent | +| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture | +| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error | + +## Rules + +- Give each check a `msg=` that names the field, so a failure says what was wrong. +- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status. diff --git a/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md b/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md index 66b5312..4347e99 100644 --- a/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md +++ b/bundles/rf-agentskills/skills/rf-restinstance/SKILL.md @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`, | references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` | | references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates | | references/troubleshooting.md | A request or assertion fails with an error you don't understand | +| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key | Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations). diff --git a/bundles/rf-agentskills/skills/rf-restinstance/references/loose-assertions.md b/bundles/rf-agentskills/skills/rf-restinstance/references/loose-assertions.md new file mode 100644 index 0000000..05ba5e0 --- /dev/null +++ b/bundles/rf-agentskills/skills/rf-restinstance/references/loose-assertions.md @@ -0,0 +1,18 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range | +| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets | +| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks | + +## Rules + +- Use the exact expected value for each field the test cares about. +- Use types and constraints for shape only, never as the only check of a value. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status. diff --git a/plugins/rf-api/skills/rf-requests/SKILL.md b/plugins/rf-api/skills/rf-requests/SKILL.md index 2cd3625..5ff9f70 100644 --- a/plugins/rf-api/skills/rf-requests/SKILL.md +++ b/plugins/rf-api/skills/rf-requests/SKILL.md @@ -64,6 +64,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (` | Assert body or headers | `Should Be Equal ${resp.json()}[id] ${1}`, `${resp.headers}[Content-Type]` | references/response-validation.md | | Basic / digest / NTLM / client cert | `Create Session … auth=`, `Create Digest Session`, `Create Ntlm Session`, `Create Client Cert Session` | references/authentication.md | | Bearer token for all later calls | `Update Session api headers=${auth}` | references/authentication.md | +| Wait for an async job to finish | bounded WHILE poll, no Sleep | references/async-jobs.md | | Unexpected error message | — | references/troubleshooting.md | ## Agent workflow @@ -96,6 +97,8 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (` | references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` | | references/authentication.md | basic, digest, NTLM, client-certificate or token authentication | | references/troubleshooting.md | a request fails with an error you don't recognise | +| references/async-jobs.md | a job or another async operation must finish before the test checks its result | +| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty | Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`. diff --git a/plugins/rf-api/skills/rf-requests/references/async-jobs.md b/plugins/rf-api/skills/rf-requests/references/async-jobs.md new file mode 100644 index 0000000..0932883 --- /dev/null +++ b/plugins/rf-api/skills/rf-requests/references/async-jobs.md @@ -0,0 +1,42 @@ +# Wait for an async job + +Use this when an endpoint starts a job (an export, an import, a batch run, a provisioning step) and answers before the work is done. A status endpoint tells you when the job finished. + +## Poll with a bounded WHILE loop + +Do not use one long `Sleep` to wait. A fixed pause is too short on a slow run and wastes time on a fast one. Poll the status endpoint in a `WHILE` loop with a time-based `limit`, pause briefly between polls, and fail with a message when the limit is reached. Put the loop in a keyword; the test only calls it: + +```robotframework +*** Test Cases *** +Export Finishes + ${res}= POST On Session api /exports json=${body} + Status Should Be 202 ${res} + ${id}= Set Variable ${res.json()}[id] + ${state}= Wait For Export To Finish ${id} + Should Be Equal ${state} done msg=Export ${id} ended as ${state} + +*** Keywords *** +Wait For Export To Finish + [Arguments] ${id} + ${state}= Set Variable queued + WHILE $state not in ['done', 'failed'] limit=60s on_limit_message=Export ${id} not finished after 60s + Sleep 2s + ${res}= GET On Session api /exports/${id} + Status Should Be 200 ${res} + ${state}= Set Variable ${res.json()}[status] + END + RETURN ${state} +``` + +`$state` (no braces) passes the variable itself to the Python expression, so a value with quotes in it cannot break the condition. + +## Rules + +- **Set the limit from the measured run time.** `limit=60s` is a time limit; a bare number such as `limit=60` counts iterations instead. Measure how long a normal run takes, then allow a little more. A large limit turns a stuck job into a long wait. +- **Pause between polls.** A short `Sleep` inside the loop (1 to 5 seconds) keeps the loop from hammering the endpoint. If the API offers a long-poll or wait option, use it instead of the `Sleep`. +- **Control flow lives in keywords.** Keep the `WHILE` loop in a keyword such as `Wait For Export To Finish`, not in the test body. +- **Check the effect, not only the state.** When the job reports `done`, read the record or file it produced and assert its fields. A status alone is a loose check. +- **One loop per job.** Do not wrap `Wait Until Keyword Succeeds` around a keyword that already polls; the timeouts multiply. +- **Clean up.** Delete what the job created in the test teardown, by its exact id. + +`WHILE` and its `limit` option exist since Robot Framework 5.0; `on_limit_message` needs Robot Framework 6.1 or later. diff --git a/plugins/rf-api/skills/rf-requests/references/loose-assertions.md b/plugins/rf-api/skills/rf-requests/references/loose-assertions.md new file mode 100644 index 0000000..091ce03 --- /dev/null +++ b/plugins/rf-api/skills/rf-requests/references/loose-assertions.md @@ -0,0 +1,20 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` | +| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks | +| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent | +| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture | +| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error | + +## Rules + +- Give each check a `msg=` that names the field, so a failure says what was wrong. +- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status. diff --git a/plugins/rf-api/skills/rf-restinstance/SKILL.md b/plugins/rf-api/skills/rf-restinstance/SKILL.md index 66b5312..4347e99 100644 --- a/plugins/rf-api/skills/rf-restinstance/SKILL.md +++ b/plugins/rf-api/skills/rf-restinstance/SKILL.md @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`, | references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` | | references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates | | references/troubleshooting.md | A request or assertion fails with an error you don't understand | +| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key | Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations). diff --git a/plugins/rf-api/skills/rf-restinstance/references/loose-assertions.md b/plugins/rf-api/skills/rf-restinstance/references/loose-assertions.md new file mode 100644 index 0000000..05ba5e0 --- /dev/null +++ b/plugins/rf-api/skills/rf-restinstance/references/loose-assertions.md @@ -0,0 +1,18 @@ +# Loose assertions + +A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response. + +## Patterns to avoid + +| Loose check | Why it passes on a broken endpoint | Assert this instead | +|---|---|---| +| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range | +| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets | +| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks | + +## Rules + +- Use the exact expected value for each field the test cares about. +- Use types and constraints for shape only, never as the only check of a value. +- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value. +- After a write, check the stored result with a read-back, not with the write's own status.