Skip to content

arazzo: support compound simple criteria and null operands - #694

Open
Shaun1 wants to merge 1 commit into
pb33f:mainfrom
ifc7:arazzo-criteria
Open

Shaun1 wants to merge 1 commit into
pb33f:mainfrom
ifc7:arazzo-criteria

Conversation

@Shaun1

@Shaun1 Shaun1 commented Nov 5, 2026 •

Copy link
Copy Markdown

Summary

Brings simple Criterion Object evaluation in line with the Arazzo 1.0.1 operators table, and aims for performance parity with main.

Changes

&&, || and (). Simple conditions only supported a single left <op> right comparison. They are now parsed with a small recursive-descent parser: || binds looser than &&, which binds looser than comparisons. This makes conditions like this one possible:

condition: >-
  $statusCode == 204 ||
  ($statusCode == 200 && $response.body#/challengeName == 'NEW_PASSWORD_REQUIRED')

Malformed conditions (unterminated strings, unbalanced parentheses, misplaced operators) are errors.

Runtime expressions end at whitespace, as with SplitSimpleCondition on main, because JSON pointers and header names can contain operator characters ($response.body#/a>b == 1). A trailing ) the expression doesn't balance itself closes an open group, so ($response.body#/flag) works. Whitespace matching is now ASCII-only, which fixes operands containing characters like Å and à being split.

Missing and null values. A value legitimately absent from a request or response (header, query/path parameter, body pointer segment, or the body itself) evaluates to null instead of failing the condition, so $statusCode == 204 || $response.body#/flag passes on an empty 204. null is falsy, equals only null, and makes <, <=, >, >= false. Previously it compared as the string "<nil>", so $response.body#/count > 0 passed on a 204. Unknown steps, workflows, inputs, outputs and components are still errors.

Compile once per engine. Each condition is compiled to a tree and cached in criterionCaches (reset by Engine.ClearCaches()). Literals are converted at compile time. Runtime expressions are evaluated against the current context on every call, so cached trees hold no runtime values.

BenchmarkSimpleCriterion vs main:

Condition main This PR
true 7.4 ns, 0 allocs 6 ns, 0 allocs
$statusCode == 200 80 ns, 2 allocs 43 ns, 0 allocs
$method == 'POST' 91 ns, 4 allocs 60 ns, 1 alloc
$statusCode == 200 && ($method == 'POST' || $method == 'PUT') unsupported 95 ns, 1 allocs

Behaviour notes

  • $statusCode==200 without spaces is one operand, as on main.
  • Trailing unmatched ) characters close open groups, one per open group. Any that are left over stay in the pointer.
  • Syntax errors are now reported before runtime errors from earlier operands.

Out of scope

! (logical not), [] and . as operators, and the null literal.

Tests

New tests in arazzo/criterion_test.go cover operator precedence and grouping, missing and null values, syntax errors, operator characters and parentheses in JSON pointers, non-ASCII operands, and the compiled-condition cache. go test ./... passes.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (88615e4) to head (0d6ed02).

Additional details and impacted files
@@            Coverage Diff             @@
##              main      #694    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files          315       315            
  Lines        39758     40001   +243     
==========================================
+ Hits         39758     40001   +243     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Shaun1

Shaun1 commented Nov 5, 2026

Copy link
Copy Markdown
Author

Not sure if you are accepting external PRs or what the procedure is, but I'm using your lib in an arazzo tool I'm building and want to see if I can get these fixes for issues I ran into merged upstream. This should bring it closer in line with the full arazzo spec.

I see there is already a PR for handling 1/0 properly so I can rebase and modify if that gets pushed.

Thanks for the work you do on this library, love it.

@Shaun1
Shaun1 marked this pull request as draft November 5, 2026 04:37
@Shaun1
Shaun1 marked this pull request as ready for review November 10, 2026 03:01
@Shaun1 Shaun1 changed the title arazzo: support compound simple criteria, fix numeric 0/1 literals, and tolerate missing response values arazzo: support compound simple criteria, null operands, and tolerate missing response values Nov 10, 2026

@daveshanley daveshanley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this Shaun, and thanks for the kind words. Compound criteria have been a gap for a while, so this is very welcome. The parser is clean, the precedence is right, and the null rules are spot on.

A few things need sorting before it can go in. There are two tokenizer regressions (operators in JSON pointers, and UTF-8), a performance hit, and the output change, which I'd like to split out. Details inline.

Also, the description still talks about the 0/1 change and its tests. That fix (#690) already landed on main, so worth a tidy up.

Comment thread arazzo/criterion.go
raw string
}

func evaluateBooleanExpr(condition string, exprCtx *expression.Context, caches *criterionCaches) (bool, error) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tokenizes and parses the condition on every single evaluation. I benchmarked it against main (alternating runs, n=12):

$statusCode == 200               72ns -> 181ns  (+151%)   2 -> 3 allocs
$response.body#/status == 'ok'  114ns -> 252ns  (+121%)   5 -> 6 allocs
true                              6ns ->  55ns  (+816%)   0 -> 1 allocs

About 76% of that is the tokenizer, mostly slice growth. We can't be dropping performance in libopenapi, so this needs to get back to parity.

The engine already keeps criterionCaches for regex and JSONPath. Compile the condition once into a small tree, cache it there, and evaluation becomes a cheap walk. Short-circuiting && and || falls out of that for free.

@Shaun1 Shaun1 Nov 12, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added cacheing into cachedSimpleCondition trees and short-circuiting of && and ||. Also added back a skip check for lone boolean values to keep parity with main there.

Not sure if there is a benchmark test I missed that you are referring to, but I added a benchmark test to cover these cases. Looks like it performs similarly to main now, if not slightly faster in some cases.

Comment thread arazzo/criterion.go Outdated
var tokens []boolToken
i := 0
for i < len(input) {
for i < len(input) && unicode.IsSpace(rune(input[i])) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rune(input[i]) looks at one byte at a time, and unicode.IsSpace counts 0xA0 and 0x85 as whitespace. Those are UTF-8 continuation bytes, so characters like à get cut in half:

$response.body#/voilà == 1   -> invalid UTF-8      (true on main)
$response.body#/s == voilà   -> false, no error    (true on main)

The second one is the nasty one. A silent wrong answer.

Either decode with utf8.DecodeRuneInString, or only treat ASCII whitespace as a separator. Same fix applies to the operand loops further down.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated to only treat ASCII whitespace as separator.

Comment thread arazzo/criterion.go
return nil, fmt.Errorf("unterminated string in simple condition %q", input)
}
i++
} else if rest[0] == '$' {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ends a runtime expression at the first operator, paren or space, so any of those inside a JSON pointer breaks the whole condition. Both of these work on main and fail here:

$response.body#/limits/>=minimum != true   -> unexpected token "!="
$response.body#/price(usd) == 5            -> unexpected token "("

The first one is lifted straight from the SplitSimpleCondition tests, and RFC 6901 allows all of these characters in a pointer. Main ends a $ operand at whitespace. I'd keep that rule, and only peel off a trailing ) while a group is open.

Also, this branch and the else below it are identical, so one of them can go.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverted to main behavior and added regression tests to cover.

Comment thread arazzo/criterion.go Outdated
}

// isOptionalRuntimeValueError must only be called with a non-nil error.
func isOptionalRuntimeValueError(err error) bool {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Matching on error message text is going to bite us. Someone rewords an error in expression and the runtime quietly changes behaviour here.

It also misses a few cases that are just as "missing" as the ones listed:

{"items": []}   $response.body#/items/0/id   -> array index 0 out of bounds
{"data": null}  $response.body#/data/id      -> cannot traverse into scalar node
no headers sent $request.header.X            -> no request headers available

The last one is inconsistent. If the step sent any other header, the same expression comes back as null. The engine leaves that map nil when a step sends no headers, so this happens for real.

I'd add a sentinel error in expression (like ErrLocalSymbolNotFound), wrap it with %w everywhere a value isn't there, and check it here with errors.Is.

And because || always evaluates both sides, $statusCode == 200 || $response.body#/items/0/id == 'x' errors on an empty list, even though the left side is true. I'd short-circuit.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes text error matching is dumb, I should have caught that. One option Claude suggested was determining optional runtime value status off of the existing ExpressionType rather than introduce a new sentinel error var. I implemented this. Thoughts?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, I prefer this to a sentinel. Every evaluation error for these types depends on the data in the request or response, so treating them all as null is safe. And there's no message text to drift.

One gap: the payload and message types are missing. $response.payload#/x still errors when it's absent, while $response.body#/x comes back null. I'd add RequestPayload, ResponsePayload and the Message* types to the switch.

Comment thread arazzo/step.go Outdated
value, err := e.evaluateStringValue(expressionValue, exprCtx)
if err != nil {
// Missing body/header values are valid on empty responses (e.g. HTTP 204).
if isOptionalRuntimeValueError(err) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd pull this out of the PR. A missing output now becomes nil and the workflow carries on.

I ran a two-step workflow. create gets a 204 and has output id: $response.body#/id. get uses $steps.create.outputs.id as a path param.

  • main: create fails with no response body available. Clear and early.
  • this PR: get is sent with userId: nil (which fmt.Sprint renders as "<nil>") and the workflow reports success.

The null rules make sense inside criteria, and the spec backs them there. Outputs are a different story. The spec doesn't ask for this, and failing fast is the safer default. If your tool needs it, put it behind an EngineConfig option.

Same goes for populateWorkflowOutputs below.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed this change. I'll try to handle it in the tool & address it later if I can't.

@Shaun1
Shaun1 marked this pull request as draft November 11, 2026 22:14
@Shaun1
Shaun1 force-pushed the arazzo-criteria branch 2 times, most recently from aace49d to fbf0505 Compare November 11, 2026 22:44
@Shaun1 Shaun1 changed the title arazzo: support compound simple criteria, null operands, and tolerate missing response values arazzo: support compound simple criteria and null operands Nov 11, 2026
@Shaun1
Shaun1 force-pushed the arazzo-criteria branch 6 times, most recently from 5c63bd7 to e1414f6 Compare November 12, 2026 06:51
The Arazzo Criterion Object operators table lists && (and), || (or) and
() (logical grouping) alongside the comparison operators, but simple
conditions were split on the first comparison operator found, so only a
single `left <op> right` comparison could be expressed.

Grammar

Simple conditions are tokenized and parsed by a small recursive-descent
parser. || binds looser than &&, which binds looser than the comparison
operators (==, !=, <, <=, >, >=). Comparisons do not chain, and a
parenthesised group is a boolean operand, so `($flag) == true` coerces
the group before comparing. Malformed conditions (unterminated strings,
unbalanced parentheses, misplaced or trailing operators) are errors.

Tokenizing runtime expressions

JSON pointers, header names and parameter names may contain operator
characters, so a runtime expression is one operand that runs to the next
whitespace, matching expression.SplitSimpleCondition on main:
`$response.body#/a>b == 1` compares the value at /a>b. A trailing ')'
that the expression does not balance itself closes an open group, so
`($response.body#/flag)` works and `#/f(x)` stays intact. Consequences:

- `$statusCode==200` without spaces is a single operand, as on main.
- Inside a group, a pointer token ending in an unbalanced ')' cannot be
  written; that ')' always closes the group.

Whitespace is matched as ASCII only. The tokenizer works on bytes, and
unicode.IsSpace(rune(b)) misread the UTF-8 continuation bytes 0x85 and
0xA0 (as in "Å" and "à") as whitespace, splitting those operands.

Compile once, cache per engine

Each condition is compiled once into an evaluation tree and cached in a
new criterionCaches.simple map, keyed by the trimmed condition. Syntax
errors are cached too, as the regex and JSONPath caches already do, and
Engine.ClearCaches() resets the map with the rest. Literal operands are
converted at compile time. Runtime expressions keep their raw text and
are evaluated against the current context through the engine's
expression cache, so a cached tree never holds runtime values.
EvaluateCriterion without an engine compiles on every call.

Because conditions compile before evaluating, a syntax error is now
reported ahead of a runtime error from an earlier operand.

https://spec.openapis.org/arazzo/v1.0.1.html#operators

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Shaun1
Shaun1 marked this pull request as ready for review November 12, 2026 06:54
@Shaun1
Shaun1 requested a review from daveshanley November 12, 2026 06:55

@daveshanley daveshanley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work Shaun, this is a big step up. All five threads are sorted, and the engine path went from 2.5x slower than main to roughly twice as fast:

$statusCode == 200               73ns -> 37ns   2 -> 0 allocs
$response.body#/status == 'ok'  121ns -> 72ns   5 -> 2 allocs

Short-circuiting works, the cached tree holds up across different contexts, and missing outputs fail fast again.

One blocker left, which is the uncached path. Details inline, plus a couple of small things.

Comment thread arazzo/criterion.go

// compileSimpleCondition returns the compiled tree for condition, caching both
// successful compiles and syntax errors when caches is non-nil.
func compileSimpleCondition(condition string, caches *criterionCaches) (*compiledCondition, error) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the last blocker. EvaluateCriterion is public and has no cache, so every call compiles the condition from scratch. Against main (alternating runs, n=12):

$statusCode == 200               64ns -> 210ns  (+226%)   2 -> 7 allocs
$response.body#/status == 'ok'  145ns -> 307ns  (+112%)   5 -> 10 allocs

The profile puts all seven allocs on compiling. The token slice grows three times, then there are three nodes and the compiledCondition.

Size the token slice up front and allocate the nodes from a single slice. That should cut most of them. Measure it against main's numbers and we're there.

Comment thread arazzo/criterion.go
func newOperandNode(raw string) *conditionNode {
op := strings.TrimSpace(raw)
if strings.HasPrefix(op, "$") {
return &conditionNode{kind: condOperand, raw: op, isExpr: true}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With short-circuiting, a typo in a branch that doesn't run is never caught:

$statusCode == 200 || $respnse.body#/x == 1   -> true, no error

Parse the runtime expression here, once, and keep the parsed expression.Expression on the node. Typos then fail at compile time in every branch, and value() skips the parse lookup on every evaluation.

Unknown steps still need the context, so those stay runtime errors.

Comment thread arazzo/criterion.go
return &conditionNode{kind: condOperand, raw: op, isExpr: true}
}
// Literals never touch the expression context, so they convert once here.
literal, _ := evaluateSimpleOperand(op, nil, nil)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

evaluateSimpleOperand only ever gets literals with no context now. Its $ and empty-string branches are only reachable from coverage_test.go, and the error gets thrown away here.

A small simpleLiteral(op string) any helper would be cleaner, and the tests for the dead branches can go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants