feat: Add consolidation, comparison, report modules and orchestrator wiring - #315
Merged
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughThe behavioral pipeline now compares consolidated agents, builds typed evaluation reports, serializes them to ChangesBehavioral evaluation reporting
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant BehavioralOrchestrator
participant RunDataLoader
participant Consolidator
participant AgentComparator
participant ReportFile
BehavioralOrchestrator->>RunDataLoader: load successful run data
RunDataLoader-->>BehavioralOrchestrator: return run data
BehavioralOrchestrator->>Consolidator: consolidate runs by agent
Consolidator-->>BehavioralOrchestrator: return consolidated results
BehavioralOrchestrator->>AgentComparator: compare agents
AgentComparator-->>BehavioralOrchestrator: return comparison result
BehavioralOrchestrator->>ReportFile: save eval_report.json
ReportFile-->>BehavioralOrchestrator: return output path
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lightspeed_evaluation/pipeline/behavioral/orchestrator.py`:
- Around line 431-432: Replace the broad Exception handler in the
report-generation flow with the project-specific error type used for expected
report-generation failures, and remove the inline pylint suppression. Keep the
warning log for that expected error while allowing unexpected programming errors
to propagate.
- Around line 396-434: The orchestration path around _build_and_save_report and
run lacks integration coverage. Add pytest tests using the mocker fixture that
verify successful RunResult values flow through load_run_data, consolidate,
compare_agents, build_report, and save_report, plus a failure-path test
confirming report-generation errors are caught and run() still returns its
evaluation results.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 2c4b3387-46bf-4950-801c-8282adfb9e1e
📒 Files selected for processing (5)
src/lightspeed_evaluation/pipeline/behavioral/comparison.pysrc/lightspeed_evaluation/pipeline/behavioral/orchestrator.pysrc/lightspeed_evaluation/pipeline/behavioral/report.pytests/unit/pipeline/behavioral/test_comparison.pytests/unit/pipeline/behavioral/test_report.py
asamal4
force-pushed
the
nxm-consolidation-wiring
branch
2 times, most recently
from
August 10, 2026 06:57
ed0ce0e to
68c2bea
Compare
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
asamal4
force-pushed
the
nxm-consolidation-wiring
branch
from
August 10, 2026 14:47
68c2bea to
c1d2ffb
Compare
asamal4
changed the title
asamal4
changed the title
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lightspeed_evaluation/pipeline/behavioral/consolidation.py`:
- Line 205: Clarify the count semantics in _build_by_metric: since scores
contains per-run mean scores, do not present len(scores) as a raw scored-result
count. Either rename the field to indicate it counts contributing runs or expose
a separate raw-score count, while preserving the aggregated mean calculation.
- Around line 64-77: Update the documentation for the behavioral report payload
in README.md and the relevant docs/ files to explicitly describe
total_agent_tokens, total_judge_tokens, total_embedding_tokens, and the
run-level agent_tokens_mean and judge_tokens_mean fields emitted by
consolidate().
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: b0223947-6866-439a-bf31-d3708b31c641
📒 Files selected for processing (3)
src/lightspeed_evaluation/pipeline/behavioral/consolidation.pysrc/lightspeed_evaluation/pipeline/behavioral/orchestrator.pytests/unit/pipeline/behavioral/test_orchestrator.py
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/unit/pipeline/behavioral/test_orchestrator.py
- src/lightspeed_evaluation/pipeline/behavioral/orchestrator.py
asamal4
force-pushed
the
nxm-consolidation-wiring
branch
from
August 10, 2026 18:40
c1d2ffb to
55b5742
Compare
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Action performedReview finished.
|
This was referenced Aug 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Initial wiring of NxM consolidation and comparison reporting.
More statistical comparison will be done in follow up PRs with reporting enhancements
Type of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit
Summary by CodeRabbit