feat: add --learn mode for expected state capture - #894
Open
ChristopherJHart wants to merge 6 commits into
Open
ChristopherJHart wants to merge 6 commits into
ChristopherJHart wants to merge 6 commits into
Conversation
Implements the framework for a two-phase operational test workflow: 1. Learn: `nac-test --learn` captures live state as baseline YAML 2. Verify: `nac-test -d ./learned_state` compares against baseline New components: - --learn CLI flag (propagated via NAC_TEST_LEARN env var) - LearningModeMixin: contract for tests that support learning mode (override capture_learned_state() to define what to capture) - learned_state utilities: save/load YAML files compatible with the existing -d merge mechanism (nac_yaml.merge_dict) - _run_learning_mode() in NACTestBase: branches on env var in run_verification_async(), calls capture method, writes output Tests opt in by inheriting LearningModeMixin and implementing capture_learned_state(). Tests without it are skipped gracefully in learn mode. Default (--learn not passed) behavior is unchanged. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…Meta compat PyATS's TestableMeta metaclass expects all methods on test classes to have a .source attribute set during class __init__. Without inheriting from aetest.Testcase, the mixin's methods lack this attribute, causing AttributeError at class definition time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instead of writing `--- {}` (confusing), empty captures now produce a
file with `_learned_state_empty: "reason"`. This distinguishes "learning
ran but found nothing" from "learning was never run for this device."
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Learned state files are meant to be version-controlled, not ephemeral.
Changed default output from {output_dir}/learned_state/ to
{cwd}/learned_state/ (repo root). Users pass -d learned_state/ during
verify mode to merge the captured baseline into the data model.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Provides a consistent, testable namespace identifier for each learning
test's output within the merged data model. Convention: "{type}.{Class}"
(e.g., "api.VerifyOMPRoutes", "d2d.VerifyBGPLearnedRoutes").
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ChristopherJHart
force-pushed
the
feat/learn-mode
branch
from
August 15, 2026 23:28
8425ffa to
dd57d9f
Compare
Rich/Typer adds ANSI escape codes to --help output on Linux but not Windows, causing the plain string assertion to fail in CI. Strip escape codes before checking for --learn flag presence. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collaborator
|
I would like to review the target architecture of this pre-/post comparison with the team. There are existing pre-/post-robot checks for ACI which are triggered by |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--learnCLI flag that switches test execution from verification to baseline capture modeLearningModeMixin— a contract class that D2D/API tests inherit to define what operational state to capturelearned_state/in the working directory, loadable vianac-test -d ./learned_statefor verification runsLEARNED_STATE_KEYclass variable for consistent namespacing of learned state within the merged data modelMotivation
Operational tests (NRFU) need a two-phase workflow:
nac-test --learn)nac-test -d ./learned_state)This enables "expected state testing" without hardcoding expected values — the network itself defines what's normal.
Changes
cli/main.py--learnflag (+env varNAC_TEST_LEARN)combined_orchestrator.pypyats_core/orchestrator.pypyats_core/common/base_test.py_run_learning_mode()integration in test lifecyclepyats_core/common/learning_mode_mixin.pyLearningModeMixinwithcapture_learned_state()contractutils/learned_state.pytests/unit/test_learning_mode.pyTest plan
tests/unit/test_learning_mode.py)--learnis not passed🤖 Generated with Claude Code
🤖 AI Generation Metadata