Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions bundles/rf-agentskills/skills/rf-requests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (`
| references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` |
| references/authentication.md | basic, digest, NTLM, client-certificate or token authentication |
| references/troubleshooting.md | a request fails with an error you don't recognise |
| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty |

Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` |
| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks |
| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent |
| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture |
| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error |

## Rules

- Give each check a `msg=` that names the field, so a failure says what was wrong.
- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
1 change: 1 addition & 0 deletions bundles/rf-agentskills/skills/rf-restinstance/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`,
| references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` |
| references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates |
| references/troubleshooting.md | A request or assertion fails with an error you don't understand |
| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key |

Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations).

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range |
| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets |
| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks |

## Rules

- Use the exact expected value for each field the test cares about.
- Use types and constraints for shape only, never as the only check of a value.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
1 change: 1 addition & 0 deletions plugins/rf-api/skills/rf-requests/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ Escape hatch: for one or two calls to full URLs, use the sessionless keywords (`
| references/response-validation.md | asserting JSON bodies, lists, headers and cookies of a `requests.Response` |
| references/authentication.md | basic, digest, NTLM, client-certificate or token authentication |
| references/troubleshooting.md | a request fails with an error you don't recognise |
| references/loose-assertions.md | choosing what a test asserts, or reviewing a test that checks only the status or that a value is not empty |

Examples in `assets/examples/`: `basic-crud.robot`, `session-handling.robot`, `authentication-patterns.robot`, `file-upload.robot`.

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Should Not Be Empty ${resp.json()}` | any body with one key passes, an error object included | the field and value the contract names: `Should Be Equal ${resp.json()}[status] active msg=status` |
| `Should Be True ${resp.ok}` | `ok` is true for every status below 400, 3xx included | `Status Should Be 200 ${resp}` and the field checks |
| A status check alone after a write | the server can accept the request and store something else | a read-back: `GET On Session` the new record and compare the fields you sent |
| `Should Be Equal ${x} ${x}` | compares a value with itself | the expected value from the spec, the requirement or a fixture |
| A helper that turns a non-200 response into an empty result | the caller sees nothing to check and the test passes | let the status check raise, or pass `expected_status` on the one call that expects the error |

## Rules

- Give each check a `msg=` that names the field, so a failure says what was wrong.
- Assert the exact value the test sets up or the spec defines. Use a type or schema check for shape only.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
1 change: 1 addition & 0 deletions plugins/rf-api/skills/rf-restinstance/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ Paths start at the last instance: `response status`, `response body user name`,
| references/schema-validation.md | Using schema files, learning schemas with `Output Schema`, expectations, or an OpenAPI `spec=` |
| references/authentication.md | Adding tokens, API keys, basic/digest auth or client certificates |
| references/troubleshooting.md | A request or assertion fails with an error you don't understand |
| references/loose-assertions.md | choosing what a test asserts, or reviewing checks that test only a status, a type or a key |

Examples in `assets/examples/`: `basic-rest.robot` (requests, paths, chaining, negative status), `schema-validation.robot` (learned schema, schema file, expectations).

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Loose assertions

A check that cannot fail gives a green run against a broken API. Each pattern below passes when the endpoint returns the wrong data, so do not use one as the only check on a response.

## Patterns to avoid

| Loose check | Why it passes on a broken endpoint | Assert this instead |
|---|---|---|
| `Object response body required=["id"]` alone | it proves the key exists, not its value | `String response body name Alice` for the value, `Integer response body id minimum=1` for the range |
| `Integer response body id` alone | any integer passes | the value the test expects, or the bound the contract sets |
| `Integer response status 200` alone after a write | the server can accept the request and store something else | a `GET` of the new record, then the field checks |

## Rules

- Use the exact expected value for each field the test cares about.
- Use types and constraints for shape only, never as the only check of a value.
- Prove a new check can fail: change the expected value once, run the test, confirm it turns red, then restore the value.
- After a write, check the stored result with a read-back, not with the write's own status.
Loading