Conversation
Webhook events for the observability side: evaluation threshold breaches, issue detection findings, and trace errors, reusing the existing webhook delivery system. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| "feedback_name": "correctness", | ||
| "threshold": 0.7, | ||
| "direction": "below", | ||
| "window_size": 10, |
There was a problem hiding this comment.
I think trace signal should be aggregated into metrics with fixed unit (e.g. 1 minute average) before being emitted as webhook events. Using the raw event count for aggregation is not practical - target duration becomes highly dynamic depends on the trace traffic. At the same time, implementing it as a part of webhook API complicates the interface (e.g. no other events require that level of complexity).
We have some WIP design for this aggregation for alerting purpose, so probably combine it with this RFC.
There was a problem hiding this comment.
Yes, agreed — fixed time units make a lot more sense. Since you already have a WIP design for the aggregation side, what would be the best way to combine the two? Happy to keep this PR open and update it to line up with your design once it's shareable — is there a rough timeline for when that might be?
|
|
||
| | Event | Fires from | | ||
| |---|---| | ||
| | `trace_feedback.threshold_breached` | The online scoring job (automatic evaluations), immediately after it writes a judge score — server-internal, not a public API endpoint. Scores logged manually through the feedback API do not trigger it. | |
There was a problem hiding this comment.
I might miss some context about the full agent improvement proposal, but this doesn't seem to be necessary for building that flow in MLflow? Scheduled issue detection job can be the trigger for the whole workflow.
There was a problem hiding this comment.
Yes — it could run detection itself and read the results directly. It could also build on these events, so runs started from the UI or API feed it the same way. In my opinion the webhooks are worth having either way, since customers can build their own tooling on top of them (Slack, CI, their own pipelines) even without the improvement workflow — but what do you think?
Per review feedback: count-based windows behave differently for busy and quiet agents; fixed time units are predictable. Also notes the in-progress aggregation-for-alerting design as the likely home for the rule layer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Splitting #38 into two standalone RFCs per the review feedback. This is the first: webhook events for the observability side — evaluation scores crossing a threshold, issue detection creating or updating an issue, and traces completing with errors — reusing the webhook delivery system MLflow already ships. Useful on its own.
Follows the template with the journeys-first model: Detailed design, Drawbacks, Alternatives, and Adoption strategy are TBD until the journeys are settled. Review feedback from #38 is incorporated.