On-policy context distillation combines the best of both worlds -- dense per-token supervision from distillation with the distribution-matching properties of reinforcement learning -- making it the most sample-efficient approach for continual learning from sparse human feedback. The core insight, validated across multiple independent research groups in 2025-2026, is that off-policy methods (standard SFT) catastrophically forget prior knowledge because forward KL divergence aggressively shifts probability mass toward new data, while on-policy distillation using reverse KL is inherently biased toward KL-minimal solutions that preserve existing capabilities. For a coding CLI agent with approve/deny/edit feedback, the optimal strategy is a two-phase approach: off-policy SFT for cold-start initialization, then on-policy self-distillation (SDFT) for continual learning, where edited code serves as in-context demonstrations for a self-teacher. This approach achieves 9-30x compute savings over pure RL and dramatically reduces catastrophic forgetting compared to continued SFT.
Context distillation is a specific form of knowledge distillation where the teacher and student are the same model -- the teacher receives additional context (system prompts, few-shot examples, chain-of-thought scratchpads) that the student does not. The goal is to "bake" the behavior induced by that context directly into model parameters, so the model acts as if it had the prompt at inference time without actually needing it.
Askell et al. (2021) at Anthropic introduced this technique for alignment, using a 4,600-word HHH prompt containing 14 fictional conversations. They fine-tuned the model to minimize forward KL divergence between the prompted model's distribution and the distilled model's predictions, using data sampled from the pre-training distribution. Snell et al. (2022) at UC Berkeley then generalized context distillation as a universal learning method, demonstrating it could internalize three types of context: abstract task instructions, step-by-step reasoning chains, and concrete training examples. Their key result: context distillation outperformed direct gradient descent by 9% on the SPIDER Text-to-SQL benchmark for few-shot adaptation.
Both of these foundational works used off-policy training -- the training data came from the pre-training distribution or teacher-generated outputs, not from the student's own generations. This seemed fine initially, but a critical flaw emerged as researchers attempted to use context distillation for continual learning.
The fundamental difference between on-policy and off-policy context distillation lies in where the training data comes from and what loss function is minimized:
Off-policy context distillation trains the student on sequences generated by the teacher (or from a fixed dataset). The loss is typically forward KL divergence (equivalent to standard cross-entropy/negative log-likelihood). This is a mode-covering objective: the student is exponentially penalized for assigning low probability to any completion in the training data. When fine-tuning on new data, the model aggressively shifts probability toward that data, often destroying previously learned modes.
On-policy context distillation trains the student on its own generated sequences, using the teacher's token-level probabilities as supervision. The loss is typically reverse KL divergence -- a mode-seeking objective where assigning near-zero probability to some completion simply prevents it from being sampled, with no exponential penalty. Agarwal et al. (2023) at Google DeepMind formalized this in their Generalized Knowledge Distillation (GKD) framework, framing autoregressive distillation as an imitation learning problem. Their key insight: off-policy training creates compounding errors analogous to the covariate shift problem in behavioral cloning (Ross et al., 2011).
The theoretical foundation for why this matters for continual learning comes from Shenfeld and Pari's "RL's Razor" (2025): forward KL divergence between the fine-tuned and base model predicts forgetting with R^2 ~ 0.96 -- stronger than reverse KL (0.93), total variation (0.80), or any weight-space metric. On-policy methods are implicitly biased toward KL-minimal solutions among all solutions that accomplish the new task, while SFT can converge to distributions arbitrarily far from the base model.
Two distinct but related methods have emerged as the on-policy evolution of context distillation, both published in January 2026:
SDFT (Self-Distillation Fine-Tuning) by Shenfeld, Damani, and Guestrin is the most direct descendant. The same model serves dual roles: as teacher when conditioned on both the task input and an expert demonstration (via in-context learning), and as student when conditioned on the task input alone. The student generates its own trajectories, and the loss minimizes reverse KL against the demonstration-conditioned teacher. A critical implementation detail: the teacher is parameterized as an exponential moving average (EMA) of the student -- using the frozen base model fails because it cannot track learning progress, while using the raw student creates training instabilities. In sequential learning experiments (Science Q&A -> Tool Use -> Medical), SDFT enables skill accumulation without the catastrophic performance regression that plagues standard SFT.
SDPO (Self-Distillation Policy Optimization) by Hubotter et al. extends context distillation into an RL framework with rich environmental feedback. Instead of expert demonstrations, the "privileged context" is textual feedback from the environment -- runtime errors, compiler logs, judge evaluations. The model conditioned on this feedback becomes the self-teacher, and its corrected next-token predictions are distilled into the student policy via KL divergence. SDPO provides dense, token-level credit assignment by comparing student and teacher distributions at each position. On chemistry reasoning tasks, SDPO reached GRPO's accuracy 10x faster with reasoning traces up to 7x shorter.
The relationship to context distillation is direct: both SDPO and SDFT use the same model as teacher and student, with the teacher receiving privileged context (demonstrations or feedback) that the student lacks. The critical innovation is making the training loop on-policy rather than off-policy.
The sample efficiency evidence is striking and consistent across multiple independent research groups:
| Method | Sampling | Signal density | Relative compute | Forgetting risk |
|---|---|---|---|---|
| Off-policy SFT | Teacher-generated | Dense (per-token) | 1x (baseline) | High |
| Reinforcement learning (GRPO) | Student-generated | Sparse (per-episode) | 8-16x | Low |
| On-policy distillation | Student-generated | Dense (per-token) | 2-3x | Low |
On-policy distillation occupies a unique position: it provides O(N) bits of information per episode (where N is the token count) compared to RL's O(1) bits from sparse scalar rewards, while maintaining the distribution-matching properties that prevent catastrophic forgetting. Concrete results:
- Qwen3 technical report: on-policy distillation achieved 74.4% on AIME'24 in 1,800 GPU hours versus RL's 67.6% in 17,920 GPU hours -- roughly 10x cheaper for superior performance
- GKD: using only 5% of the training data, on-policy distillation outperformed off-policy methods trained on the entire dataset
- OPSD (Zhao et al., 2026): on-policy self-distillation with 2,000-token generations matched GRPO performance at 16,000 tokens -- an 8x token efficiency gain
For a coding CLI agent with approve/deny/edit feedback, the research converges on a clear two-phase strategy:
Phase 1 -- Off-policy initialization (cold start). Use standard SFT on publicly available coding datasets and context distillation to bake in CLI-specific behavioral patterns from crafted few-shot prompts. Forward KL (cross-entropy) is appropriate here because it converges faster and adds "support" for new token patterns. This establishes baseline coding capability.
Phase 2 -- On-policy self-distillation (continual learning). Switch to SDFT-style on-policy distillation using reverse KL. The feedback signals map naturally to this framework:
- Edits are the strongest signal: the edited code becomes the expert demonstration, and the model conditioned on (prompt + edited code) serves as teacher for the model conditioned on (prompt alone)
- Denials with corrections provide natural language feedback that conditions the self-teacher
- Approvals are implicit positive signals (no gradient step needed)
The practical training loop for each batch of accumulated feedback: (1) sample on-policy trajectories from the student on buffered prompts, (2) construct the self-teacher by conditioning the model on the demonstration/feedback via in-context learning, (3) compute per-token reverse KL between student and teacher distributions, and (4) update LoRA adapters using importance-sampled policy gradients.
The loss function question has a more nuanced answer than the literature initially suggested. Wu et al. (COLING 2025) showed that forward and reverse KL share the same optimization objective and converge to identical solutions given sufficient training. The practical difference emerges under limited training budgets: forward KL focuses on the head of distributions (high-probability tokens) and converges faster, while reverse KL focuses on the tail and is more robust to noisy signals.
For the two-phase architecture:
- Forward KL during Phase 1 adds broad support for coding tokens and converges quickly
- Reverse KL during Phase 2 performs mode-seeking within the initialization's support, learning specific coding patterns from user feedback without destabilizing existing capabilities
For the teacher parameterization, the EMA approach is essential. Using a frozen base model as teacher fails because it cannot track the student's learning progress. Using the current student directly creates training instabilities. An EMA teacher with update rate 0.05 provides the right balance -- slow enough for stable supervision, responsive enough to track skill acquisition.
continualcode implements Phase 2 (on-policy self-distillation) on top of Tinker's training API:
SDPOConfigdefines the paper-aligned 3-slot reprompt templatebuild_teacher_messages()constructs the self-teacher by appending feedback + solution demonstrations to the student's conversation contextsdpo_train_step()computes per-token KL advantages and trains via importance-sampled policy gradientsContinualSDPOSessionmanages the interactive deny-with-correction loopauto_train.pyruns the benchmark training loop with multi-rollout GRPO + SDPO
See ARCHITECTURAL.md for implementation details, Tinker API constraints, and the exact feature alignment with the SDPO reference implementation.
- Askell et al. (2021). "A General Language Assistant as a Laboratory for Alignment." Anthropic.
- Snell et al. (2022). "Learning by Distilling Context." UC Berkeley.
- Agarwal et al. (2023). "GKD: Generalized Knowledge Distillation for Auto-Regressive Sequence Models." Google DeepMind.
- Gu et al. (2023). "MiniLLM: Knowledge Distillation of Large Language Models." ICLR 2024.
- Ross et al. (2011). "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning." AISTATS.
- Wu et al. (2025). "Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models." COLING.
- Shenfeld and Pari (2025). "RL's Razor: Reinforcement Learning is the KL-Optimal Continual Learner."
- Shenfeld, Damani, and Guestrin (2026). "Self-Distillation Fine-Tuning (SDFT)."
- Hubotter et al. (2026). "Self-Distillation Policy Optimization (SDPO)." https://self-distillation.github.io/SDPO
- Zhao et al. (2026). "On-Policy Self-Distillation (OPSD)."
- Qwen Team (2025). "Qwen3 Technical Report."