Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions bundles/rf-agentskills/skills/rf-requests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (`
| Assert body or headers | `Should Be Equal ${resp.json()}[id] ${1}`, `${resp.headers}[Content-Type]` | references/response-validation.md |
| Basic / digest / NTLM / client cert | `Create Session … auth=`, `Create Digest Session`, `Create Ntlm Session`, `Create Client Cert Session` | references/authentication.md |
| Bearer token for all later calls | `Update Session api headers=${auth}` | references/authentication.md |
| Wait for an async job to finish | bounded WHILE poll, no Sleep | references/async-jobs.md |
| Unexpected error message | — | references/troubleshooting.md |

## Agent workflow
Expand Down Expand Up @@ -96,6 +97,8 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (`
| references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` |
| references/authentication.md | basic, digest, NTLM, client-certificate or token authentication |
| references/troubleshooting.md | a request fails with an error you don't recognise |
| references/async-jobs.md | a job or another async operation must finish before the test checks its result |
| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty |

Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Wait for an async job

Use this when an endpoint starts a job (an export, an import, a batch run, a provisioning step) and answers before the work is done. A status endpoint tells you when the job finished.

## Poll with a bounded WHILE loop

Do not use one long `Sleep` to wait. A fixed pause is too short on a slow run and wastes time on a fast one. Poll the status endpoint in a `WHILE` loop with a time-based `limit`, pause briefly between polls, and fail with a message when the limit is reached. Put the loop in a keyword; the test only calls it:

```robotframework
*** Test Cases ***
Export Finishes
${res}= POST On Session api /exports json=${body}
Status Should Be 202 ${res}
${id}= Set Variable ${res.json()}[id]
${state}= Wait For Export To Finish ${id}
Should Be Equal ${state} done msg=Export ${id} ended as ${state}

*** Keywords ***
Wait For Export To Finish
[Arguments] ${id}
${state}= Set Variable queued
WHILE $state not in ['done', 'failed'] limit=60s on_limit_message=Export ${id} not finished after 60s
Sleep 2s
${res}= GET On Session api /exports/${id}
Status Should Be 200 ${res}
${state}= Set Variable ${res.json()}[status]
END
RETURN ${state}
```

`$state` (no braces) passes the variable itself to the Python expression, so a value with quotes in it cannot break the condition.

## Rules

- **Set the limit from the measured run time.** `limit=60s` is a time limit; a bare number such as `limit=60` counts iterations instead. Measure how long a normal run takes, then allow a little more. A large limit turns a stuck job into a long wait.
- **Pause between polls.** A short `Sleep` inside the loop (1 to 5 seconds) keeps the loop from hammering the endpoint. If the API offers a long-poll or wait option, use it instead of the `Sleep`.
- **Control flow lives in keywords.** Keep the `WHILE` loop in a keyword such as `Wait For Export To Finish`, not in the test body.
- **Check the effect, not only the state.** When the job reports `done`, read the record or file it produced and assert its fields. A status alone is a loose check.
- **One loop per job.** Do not wrap `Wait Until Keyword Succeeds` around a keyword that already polls; the timeouts multiply.
- **Clean up.** Delete what the job created in the test teardown, by its exact id.

`WHILE` and its `limit` option exist since Robot Framework 5.0; `on_limit_message` needs Robot Framework 6.1 or later.
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` |
| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks |
| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent |
| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture |
| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error |

## Rules

- Give each check a `msg=` that names the field, so a failure says what was wrong.
- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
1 change: 1 addition & 0 deletions bundles/rf-agentskills/skills/rf-restinstance/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`,
| references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` |
| references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates |
| references/troubleshooting.md | A request or assertion fails with an error you don't understand |
| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key |

Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations).

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range |
| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets |
| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks |

## Rules

- Use the exact expected value for each field the test cares about.
- Use types and constraints for shape only, never as the only check of a value.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
3 changes: 3 additions & 0 deletions plugins/rf-api/skills/rf-requests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (`
| Assert body or headers | `Should Be Equal ${resp.json()}[id] ${1}`, `${resp.headers}[Content-Type]` | references/response-validation.md |
| Basic / digest / NTLM / client cert | `Create Session … auth=`, `Create Digest Session`, `Create Ntlm Session`, `Create Client Cert Session` | references/authentication.md |
| Bearer token for all later calls | `Update Session api headers=${auth}` | references/authentication.md |
| Wait for an async job to finish | bounded WHILE poll, no Sleep | references/async-jobs.md |
| Unexpected error message | — | references/troubleshooting.md |

## Agent workflow
Expand Down Expand Up @@ -96,6 +97,8 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (`
| references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` |
| references/authentication.md | basic, digest, NTLM, client-certificate or token authentication |
| references/troubleshooting.md | a request fails with an error you don't recognise |
| references/async-jobs.md | a job or another async operation must finish before the test checks its result |
| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty |

Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`.

Expand Down
42 changes: 42 additions & 0 deletions plugins/rf-api/skills/rf-requests/references/async-jobs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Wait for an async job

Use this when an endpoint starts a job (an export, an import, a batch run, a provisioning step) and answers before the work is done. A status endpoint tells you when the job finished.

## Poll with a bounded WHILE loop

Do not use one long `Sleep` to wait. A fixed pause is too short on a slow run and wastes time on a fast one. Poll the status endpoint in a `WHILE` loop with a time-based `limit`, pause briefly between polls, and fail with a message when the limit is reached. Put the loop in a keyword; the test only calls it:

```robotframework
*** Test Cases ***
Export Finishes
${res}= POST On Session api /exports json=${body}
Status Should Be 202 ${res}
${id}= Set Variable ${res.json()}[id]
${state}= Wait For Export To Finish ${id}
Should Be Equal ${state} done msg=Export ${id} ended as ${state}

*** Keywords ***
Wait For Export To Finish
[Arguments] ${id}
${state}= Set Variable queued
WHILE $state not in ['done', 'failed'] limit=60s on_limit_message=Export ${id} not finished after 60s
Sleep 2s
${res}= GET On Session api /exports/${id}
Status Should Be 200 ${res}
${state}= Set Variable ${res.json()}[status]
END
RETURN ${state}
```

`$state` (no braces) passes the variable itself to the Python expression, so a value with quotes in it cannot break the condition.

## Rules

- **Set the limit from the measured run time.** `limit=60s` is a time limit; a bare number such as `limit=60` counts iterations instead. Measure how long a normal run takes, then allow a little more. A large limit turns a stuck job into a long wait.
- **Pause between polls.** A short `Sleep` inside the loop (1 to 5 seconds) keeps the loop from hammering the endpoint. If the API offers a long-poll or wait option, use it instead of the `Sleep`.
- **Control flow lives in keywords.** Keep the `WHILE` loop in a keyword such as `Wait For Export To Finish`, not in the test body.
- **Check the effect, not only the state.** When the job reports `done`, read the record or file it produced and assert its fields. A status alone is a loose check.
- **One loop per job.** Do not wrap `Wait Until Keyword Succeeds` around a keyword that already polls; the timeouts multiply.
- **Clean up.** Delete what the job created in the test teardown, by its exact id.

`WHILE` and its `limit` option exist since Robot Framework 5.0; `on_limit_message` needs Robot Framework 6.1 or later.
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` |
| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks |
| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent |
| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture |
| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error |

## Rules

- Give each check a `msg=` that names the field, so a failure says what was wrong.
- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
1 change: 1 addition & 0 deletions plugins/rf-api/skills/rf-restinstance/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`,
| references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` |
| references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates |
| references/troubleshooting.md | A request or assertion fails with an error you don't understand |
| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key |

Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations).

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range |
| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets |
| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks |

## Rules

- Use the exact expected value for each field the test cares about.
- Use types and constraints for shape only, never as the only check of a value.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
Loading