Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #694 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 315 315
Lines 39758 40001 +243
==========================================
+ Hits 39758 40001 +243
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Not sure if you are accepting external PRs or what the procedure is, but I'm using your lib in an arazzo tool I'm building and want to see if I can get these fixes for issues I ran into merged upstream. This should bring it closer in line with the full arazzo spec. I see there is already a PR for handling 1/0 properly so I can rebase and modify if that gets pushed. Thanks for the work you do on this library, love it. |
b492e05 to
46f2680
Compare
46f2680 to
6e7c78c
Compare
daveshanley
left a comment
There was a problem hiding this comment.
Thanks for this Shaun, and thanks for the kind words. Compound criteria have been a gap for a while, so this is very welcome. The parser is clean, the precedence is right, and the null rules are spot on.
A few things need sorting before it can go in. There are two tokenizer regressions (operators in JSON pointers, and UTF-8), a performance hit, and the output change, which I'd like to split out. Details inline.
Also, the description still talks about the 0/1 change and its tests. That fix (#690) already landed on main, so worth a tidy up.
| raw string | ||
| } | ||
|
|
||
| func evaluateBooleanExpr(condition string, exprCtx *expression.Context, caches *criterionCaches) (bool, error) { |
There was a problem hiding this comment.
This tokenizes and parses the condition on every single evaluation. I benchmarked it against main (alternating runs, n=12):
$statusCode == 200 72ns -> 181ns (+151%) 2 -> 3 allocs
$response.body#/status == 'ok' 114ns -> 252ns (+121%) 5 -> 6 allocs
true 6ns -> 55ns (+816%) 0 -> 1 allocs
About 76% of that is the tokenizer, mostly slice growth. We can't be dropping performance in libopenapi, so this needs to get back to parity.
The engine already keeps criterionCaches for regex and JSONPath. Compile the condition once into a small tree, cache it there, and evaluation becomes a cheap walk. Short-circuiting && and || falls out of that for free.
There was a problem hiding this comment.
added cacheing into cachedSimpleCondition trees and short-circuiting of && and ||. Also added back a skip check for lone boolean values to keep parity with main there.
Not sure if there is a benchmark test I missed that you are referring to, but I added a benchmark test to cover these cases. Looks like it performs similarly to main now, if not slightly faster in some cases.
| var tokens []boolToken | ||
| i := 0 | ||
| for i < len(input) { | ||
| for i < len(input) && unicode.IsSpace(rune(input[i])) { |
There was a problem hiding this comment.
rune(input[i]) looks at one byte at a time, and unicode.IsSpace counts 0xA0 and 0x85 as whitespace. Those are UTF-8 continuation bytes, so characters like à get cut in half:
$response.body#/voilà == 1 -> invalid UTF-8 (true on main)
$response.body#/s == voilà -> false, no error (true on main)
The second one is the nasty one. A silent wrong answer.
Either decode with utf8.DecodeRuneInString, or only treat ASCII whitespace as a separator. Same fix applies to the operand loops further down.
There was a problem hiding this comment.
updated to only treat ASCII whitespace as separator.
| return nil, fmt.Errorf("unterminated string in simple condition %q", input) | ||
| } | ||
| i++ | ||
| } else if rest[0] == '$' { |
There was a problem hiding this comment.
This ends a runtime expression at the first operator, paren or space, so any of those inside a JSON pointer breaks the whole condition. Both of these work on main and fail here:
$response.body#/limits/>=minimum != true -> unexpected token "!="
$response.body#/price(usd) == 5 -> unexpected token "("
The first one is lifted straight from the SplitSimpleCondition tests, and RFC 6901 allows all of these characters in a pointer. Main ends a $ operand at whitespace. I'd keep that rule, and only peel off a trailing ) while a group is open.
Also, this branch and the else below it are identical, so one of them can go.
There was a problem hiding this comment.
reverted to main behavior and added regression tests to cover.
| } | ||
|
|
||
| // isOptionalRuntimeValueError must only be called with a non-nil error. | ||
| func isOptionalRuntimeValueError(err error) bool { |
There was a problem hiding this comment.
Matching on error message text is going to bite us. Someone rewords an error in expression and the runtime quietly changes behaviour here.
It also misses a few cases that are just as "missing" as the ones listed:
{"items": []} $response.body#/items/0/id -> array index 0 out of bounds
{"data": null} $response.body#/data/id -> cannot traverse into scalar node
no headers sent $request.header.X -> no request headers available
The last one is inconsistent. If the step sent any other header, the same expression comes back as null. The engine leaves that map nil when a step sends no headers, so this happens for real.
I'd add a sentinel error in expression (like ErrLocalSymbolNotFound), wrap it with %w everywhere a value isn't there, and check it here with errors.Is.
And because || always evaluates both sides, $statusCode == 200 || $response.body#/items/0/id == 'x' errors on an empty list, even though the left side is true. I'd short-circuit.
There was a problem hiding this comment.
yes text error matching is dumb, I should have caught that. One option Claude suggested was determining optional runtime value status off of the existing ExpressionType rather than introduce a new sentinel error var. I implemented this. Thoughts?
There was a problem hiding this comment.
Good call, I prefer this to a sentinel. Every evaluation error for these types depends on the data in the request or response, so treating them all as null is safe. And there's no message text to drift.
One gap: the payload and message types are missing. $response.payload#/x still errors when it's absent, while $response.body#/x comes back null. I'd add RequestPayload, ResponsePayload and the Message* types to the switch.
| value, err := e.evaluateStringValue(expressionValue, exprCtx) | ||
| if err != nil { | ||
| // Missing body/header values are valid on empty responses (e.g. HTTP 204). | ||
| if isOptionalRuntimeValueError(err) { |
There was a problem hiding this comment.
I'd pull this out of the PR. A missing output now becomes nil and the workflow carries on.
I ran a two-step workflow. create gets a 204 and has output id: $response.body#/id. get uses $steps.create.outputs.id as a path param.
- main:
createfails withno response body available. Clear and early. - this PR:
getis sent withuserId: nil(whichfmt.Sprintrenders as"<nil>") and the workflow reports success.
The null rules make sense inside criteria, and the spec backs them there. Outputs are a different story. The spec doesn't ask for this, and failing fast is the safer default. If your tool needs it, put it behind an EngineConfig option.
Same goes for populateWorkflowOutputs below.
There was a problem hiding this comment.
removed this change. I'll try to handle it in the tool & address it later if I can't.
aace49d to
fbf0505
Compare
5c63bd7 to
e1414f6
Compare
The Arazzo Criterion Object operators table lists && (and), || (or) and () (logical grouping) alongside the comparison operators, but simple conditions were split on the first comparison operator found, so only a single `left <op> right` comparison could be expressed. Grammar Simple conditions are tokenized and parsed by a small recursive-descent parser. || binds looser than &&, which binds looser than the comparison operators (==, !=, <, <=, >, >=). Comparisons do not chain, and a parenthesised group is a boolean operand, so `($flag) == true` coerces the group before comparing. Malformed conditions (unterminated strings, unbalanced parentheses, misplaced or trailing operators) are errors. Tokenizing runtime expressions JSON pointers, header names and parameter names may contain operator characters, so a runtime expression is one operand that runs to the next whitespace, matching expression.SplitSimpleCondition on main: `$response.body#/a>b == 1` compares the value at /a>b. A trailing ')' that the expression does not balance itself closes an open group, so `($response.body#/flag)` works and `#/f(x)` stays intact. Consequences: - `$statusCode==200` without spaces is a single operand, as on main. - Inside a group, a pointer token ending in an unbalanced ')' cannot be written; that ')' always closes the group. Whitespace is matched as ASCII only. The tokenizer works on bytes, and unicode.IsSpace(rune(b)) misread the UTF-8 continuation bytes 0x85 and 0xA0 (as in "Å" and "à") as whitespace, splitting those operands. Compile once, cache per engine Each condition is compiled once into an evaluation tree and cached in a new criterionCaches.simple map, keyed by the trimmed condition. Syntax errors are cached too, as the regex and JSONPath caches already do, and Engine.ClearCaches() resets the map with the rest. Literal operands are converted at compile time. Runtime expressions keep their raw text and are evaluated against the current context through the engine's expression cache, so a cached tree never holds runtime values. EvaluateCriterion without an engine compiles on every call. Because conditions compile before evaluating, a syntax error is now reported ahead of a runtime error from an earlier operand. https://spec.openapis.org/arazzo/v1.0.1.html#operators Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e1414f6 to
0d6ed02
Compare
daveshanley
left a comment
There was a problem hiding this comment.
Nice work Shaun, this is a big step up. All five threads are sorted, and the engine path went from 2.5x slower than main to roughly twice as fast:
$statusCode == 200 73ns -> 37ns 2 -> 0 allocs
$response.body#/status == 'ok' 121ns -> 72ns 5 -> 2 allocs
Short-circuiting works, the cached tree holds up across different contexts, and missing outputs fail fast again.
One blocker left, which is the uncached path. Details inline, plus a couple of small things.
|
|
||
| // compileSimpleCondition returns the compiled tree for condition, caching both | ||
| // successful compiles and syntax errors when caches is non-nil. | ||
| func compileSimpleCondition(condition string, caches *criterionCaches) (*compiledCondition, error) { |
There was a problem hiding this comment.
This is the last blocker. EvaluateCriterion is public and has no cache, so every call compiles the condition from scratch. Against main (alternating runs, n=12):
$statusCode == 200 64ns -> 210ns (+226%) 2 -> 7 allocs
$response.body#/status == 'ok' 145ns -> 307ns (+112%) 5 -> 10 allocs
The profile puts all seven allocs on compiling. The token slice grows three times, then there are three nodes and the compiledCondition.
Size the token slice up front and allocate the nodes from a single slice. That should cut most of them. Measure it against main's numbers and we're there.
| func newOperandNode(raw string) *conditionNode { | ||
| op := strings.TrimSpace(raw) | ||
| if strings.HasPrefix(op, "$") { | ||
| return &conditionNode{kind: condOperand, raw: op, isExpr: true} |
There was a problem hiding this comment.
With short-circuiting, a typo in a branch that doesn't run is never caught:
$statusCode == 200 || $respnse.body#/x == 1 -> true, no error
Parse the runtime expression here, once, and keep the parsed expression.Expression on the node. Typos then fail at compile time in every branch, and value() skips the parse lookup on every evaluation.
Unknown steps still need the context, so those stay runtime errors.
| return &conditionNode{kind: condOperand, raw: op, isExpr: true} | ||
| } | ||
| // Literals never touch the expression context, so they convert once here. | ||
| literal, _ := evaluateSimpleOperand(op, nil, nil) |
There was a problem hiding this comment.
evaluateSimpleOperand only ever gets literals with no context now. Its $ and empty-string branches are only reachable from coverage_test.go, and the error gets thrown away here.
A small simpleLiteral(op string) any helper would be cleaner, and the tests for the dead branches can go.
Summary
Brings
simpleCriterion Object evaluation in line with the Arazzo 1.0.1 operators table, and aims for performance parity withmain.Changes
&&,||and(). Simple conditions only supported a singleleft <op> rightcomparison. They are now parsed with a small recursive-descent parser:||binds looser than&&, which binds looser than comparisons. This makes conditions like this one possible:Malformed conditions (unterminated strings, unbalanced parentheses, misplaced operators) are errors.
Runtime expressions end at whitespace, as with
SplitSimpleConditiononmain, because JSON pointers and header names can contain operator characters ($response.body#/a>b == 1). A trailing)the expression doesn't balance itself closes an open group, so($response.body#/flag)works. Whitespace matching is now ASCII-only, which fixes operands containing characters likeÅandàbeing split.Missing and null values. A value legitimately absent from a request or response (header, query/path parameter, body pointer segment, or the body itself) evaluates to
nullinstead of failing the condition, so$statusCode == 204 || $response.body#/flagpasses on an empty 204.nullis falsy, equals onlynull, and makes<,<=,>,>=false. Previously it compared as the string"<nil>", so$response.body#/count > 0passed on a 204. Unknown steps, workflows, inputs, outputs and components are still errors.Compile once per engine. Each condition is compiled to a tree and cached in
criterionCaches(reset byEngine.ClearCaches()). Literals are converted at compile time. Runtime expressions are evaluated against the current context on every call, so cached trees hold no runtime values.BenchmarkSimpleCriterionvsmain:maintrue$statusCode == 200$method == 'POST'$statusCode == 200 && ($method == 'POST' || $method == 'PUT')Behaviour notes
$statusCode==200without spaces is one operand, as onmain.)characters close open groups, one per open group. Any that are left over stay in the pointer.Out of scope
!(logical not),[]and.as operators, and thenullliteral.Tests
New tests in
arazzo/criterion_test.gocover operator precedence and grouping, missing and null values, syntax errors, operator characters and parentheses in JSON pointers, non-ASCII operands, and the compiled-condition cache.go test ./...passes.🤖 Generated with Claude Code