Skip to content

Correct isolated Level 0 into a realistic GPT-2-BPE nanoGPT baseline - #116

Draft
charlesmartin14 wants to merge 20 commits into
mainfrom
agent/correct-realistic-level0-baseline
Draft

Correct isolated Level 0 into a realistic GPT-2-BPE nanoGPT baseline#116
charlesmartin14 wants to merge 20 commits into
mainfrom
agent/correct-realistic-level0-baseline

Conversation

@charlesmartin14

Copy link
Copy Markdown
Member

Purpose

Replace the isolated 82K-parameter byte-level diagnostic with the realistic Level 0 AdamW baseline that the experiment was intended to provide, while keeping all changes confined to level_0_baseline/.

Corrected protocol

  • pinned FineWeb-Edu sample-10BT source;
  • GPT-2 BPE tokenization stored as uint16 IDs;
  • document-disjoint 16M/1M/1M train, validation, and test splits;
  • 4 transformer blocks, 4 attention heads, width 128, context 256;
  • AdamW with decoupled weight decay, 100-step warmup, cosine decay, gradient clipping, and optional layerwise learning-rate decay;
  • 4 microbatches × 8 accumulation steps, or 8,192 tokens per optimizer update;
  • independent training RNG and fixed train/validation/test probes;
  • test evaluation only for the final and validation-selected checkpoints;
  • deterministic non-randomized WeightWatcher alpha measurements over only the 24 transformer block matrices;
  • atomic prepared-data writes, hashes, compatibility checks, completion metadata, and refreshed notebooks.

Validation

The exact source archive was validated locally with:

  • Python compilation;
  • shell syntax checks;
  • 9 passed in the isolated regression suite;
  • a two-step end-to-end CPU training smoke run producing metrics, checkpoints, selected/final test metrics, and run_complete.json;
  • notebook schema validation.

A same-repository workflow applies that exact validated tree to this branch and runs the focused regression suite before the PR is marked ready. Full repository CI will then be required before merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant