docs: add TODO files for remaining ISC migration work

TODO.rababa/:
01-fix-deep-equivalence-diffs — 13 remaining cosmetic/edge-case differences
02-investigate-imp-failing-maps — 2 maps Ruby DSL can't parse (ISC can)
03-fix-isc-spec-failures — bundler/Ruby 3.4 incompatibility workaround
04-commit-isc-to-maps-repo — push 289 .isc files to interscript/maps
05-performance-large-cjk-maps — Parslet PEG backtracking on 40k+ line maps

TODO.secryst/:
01-typescript-runtime-parity — port ISC parser to TS (user requirement)
02-is1-specification-compilation — compile Metanorma spec document
03-isc-compiler — compile .isc to executable Ruby/JS for performance
04-ruby-dsl-array-keys-bug — DSL silently drops url/notes fields
05-isc-runtime-integration — bridge ISC document hash to Node objects

Author: ronaldtse

TODO.rababa/01-fix-deep-equivalence-diffs.md

@@ -0,0 +1,39 @@
+# 01 — Fix remaining 13 deep equivalence differences
+
+## Priority: HIGH
+
+## Current State
+- 274/289 deep equivalent
+- 2 IMP-fail (ISC parses, Ruby DSL can't — ISC is strictly more capable)
+- 13 differ (cosmetic / edge cases)
+
+## Remaining Differences
+
+### Description whitespace normalization (5 maps)
+- `alalc-kor-Hang-Latn-1997`: description quoted value spans multiple lines
+- `gki-bel-Cyrl-Latn-2000`: description has relative indentation preserved by DSL
+- `var-pra-Deva-Latn-iast-1912`, `var-san-Deva-Latn-iast-1912`: description content truncation
+
+**Fix:** The `normalize_heredoc` in `DocumentBuilder` strips ALL leading whitespace.
+The DSL strips only the COMMON indent (YAML dedent). Need a proper dedent
+algorithm that:
+1. Finds the minimum indent across all non-blank lines
+2. Strips only that amount, preserving relative indentation
+3. Handles the first line specially (grammar consumed its indent after `{`)
+
+### Codemod edge cases (6 maps)
+- `alalc-tir-Ethi-Latn-1997/2011`: description includes `implementation_notes:` text
+- `bgnpcgn-fas-Arab-Latn-1956`: `TODO: Add tests from PDF` treated as metadata field
+- `mext-jpn-Hrkt-Latn-1954`: metadata name has CJK text, description mismatch
+- `odni-ara/fas/prs-Arab-Latn-2004`: description `[]` or `notes:` text leaking
+
+**Fix:** Audit each .imp file's metadata block structure and extend the codemod
+to handle the specific patterns. Most are multi-line description values where
+the codemod's handler chain misidentifies the field boundaries.
+
+### Rule count (2 maps)
+- `din-san-Deva-Latn-33904-2018`: imp=155 isc=154 (off by 1)
+- `var-ara-Arab-Arab-rababa`: imp=1 isc=0 (rababa directive — expected, not a bug)
+
+**Fix for din-san:** Compare the stage body item-by-item between .imp and .isc
+to find the missing rule. Likely a `run` or `deep`/`compose` directive not counted.

TODO.rababa/02-investigate-imp-failing-maps.md

@@ -0,0 +1,29 @@
+# 02 — Investigate 2 IMP-failing maps
+
+## Priority: MEDIUM
+
+## Current State
+Two maps fail on the Ruby DSL side but parse correctly on the ISC side:
+- `bgnpcgn-tuk-Cyrl-Latn-1979`
+- `bgnpcgn-tuk-Cyrl-Latn-1993`
+
+ISC captures: 1 stage, 21 tests each. Ruby DSL raises on parse.
+
+## Investigation Needed
+
+1. Open each `.imp` file and identify the syntax that breaks the Ruby DSL.
+2. Check whether the ISC parser's extracted data matches what the .imp intends.
+3. If the ISC parser is correct (likely — it parsed successfully), the Ruby DSL
+   has a bug. File an issue against the Ruby DSL.
+
+## Files to Examine
+- `/Users/mulgogi/src/interscript/maps/maps/bgnpcgn-tuk-Cyrl-Latn-1979.imp`
+- `/Users/mulgogi/src/interscript/maps/maps/bgnpcgn-tuk-Cyrl-Latn-1993.imp`
+- `/tmp/isc-verify/bgnpcgn-tuk-Cyrl-Latn-1979.isc`
+- `/tmp/isc-verify/bgnpcgn-tuk-Cyrl-Latn-1993.isc`
+
+## Likely Root Cause
+The Ruby DSL uses `instance_exec` to parse the metadata block. Tukmen (tuk) maps
+may have metadata fields with characters or syntax that the DSL's metadata
+parser can't handle (e.g., Turkmen-specific characters, unusual date formats,
+or specific field names not in `STANDARD_STRING_KEYS`).

TODO.rababa/03-fix-isc-spec-failures.md

@@ -0,0 +1,34 @@
+# 03 — Debug and fix ISC spec failures (bundler workaround)
+
+## Priority: HIGH
+
+## Current State
+- 73 ISC specs written, 45 failing due to Ruby 3.4 + bundler incompatibility.
+- The project's `spec/spec_helper.rb` requires `bundler/setup` which raises
+  `DidYouMean::SPELL_CHECKERS` NameError on Ruby 3.4.8.
+- ISC specs have their own `spec/interscript/isc/spec_helper.rb` that avoids
+  bundler, but `rspec` loads the project `.rspec` file which points to the main
+  helper.
+
+## Workaround
+Run ISC specs with:
+```bash
+rspec --options /dev/null --no-profile \
+  --require ./spec/interscript/isc/spec_helper.rb \
+  spec/interscript/isc/
+```
+
+## Remaining Issues
+1. Some specs use `parser.parse(...)` but the parser returns a hash tree, not
+   an object. Assert on `tree[:system][:body]` being an Array.
+2. Transform specs construct Parslet trees manually — the shape may not match
+   what the parser produces. Verify by parsing a minimal .isc and inspecting
+   the tree.
+3. The `system_code` in the parser output is a `Parslet::Slice`, not a String.
+   Call `.to_s` in assertions.
+
+## Fix Steps
+1. Fix the bundler issue globally (upgrade bundler or pin Ruby version).
+2. Update spec assertions to match actual parser output shapes.
+3. Add a `Rakefile` target for ISC specs: `rake spec:isc`.
+4. Run in CI with `--tag isc` to isolate from the main suite.

TODO.rababa/04-commit-isc-to-maps-repo.md

@@ -0,0 +1,23 @@
+# 04 — Commit .isc files to the maps repo
+
+## Priority: HIGH
+
+## Current State
+- All 289 .isc files generated in `/tmp/isc-verify/` and `/Users/mulgogi/src/interscript/maps/maps/`.
+- The maps repo (`interscript/maps`) has 289 untracked .isc files.
+- The interscript-ruby repo (`feat/isc-parser-codemod` branch) has the codemod
+  and grammar but not the .isc files.
+
+## Steps
+1. In the maps repo, create a branch: `feat/isc-maps`.
+2. Stage all .isc files: `git add maps/*.isc` (explicit, not `-A`).
+3. Verify: `git diff --cached --name-only | grep -c '.isc'` should be 289.
+4. Commit with message: `feat: add ISC-format maps for all 289 systems`.
+5. Push and open a PR against `interscript/maps`.
+
+## Considerations
+- The .isc files are GENERATED from .imp via the codemod. Consider adding a
+  CI check that re-runs the codemod and verifies no drift.
+- The maps repo may have its own CI (CodeQL, lint). Check before pushing.
+- Coordinate with the user before pushing — this is a large change to a
+  shared repo.

TODO.rababa/05-performance-large-cjk-maps.md

@@ -0,0 +1,41 @@
+# 05 — Performance optimization for large CJK maps
+
+## Priority: MEDIUM
+
+## Current State
+6 maps take 15-38 seconds to parse due to Parslet PEG backtracking:
+
+| Map | Lines | Parse Time |
+|-----|-------|------------|
+| var-kor-Kore-Hang-2013 | ~30k | 38.4s |
+| lshk-yue-Hani-Latn-jyutping-1993 | ~20k | 29.4s |
+| hk-yue-Hani-Latn-1888 | ~20k | 23.2s |
+| acadsin-zho-Hani-Latn-2002 | ~15k | 22.7s |
+| var-zho-Hani-Latn-wd-1979 | ~43k | 19.7s |
+| sac-zho-Hans-Latn-1979 | ~26k | 15.1s |
+
+## Root Cause
+The `alias_arg` rule adds `zero_width_primitive.absent?` lookahead, which is
+evaluated for every `any()` call. With 5000+ `any()` calls in var-zho, the
+overhead compounds.
+
+## Optimization Options
+
+### Option A: Memoize keyword/primitive lookaheads (quick win)
+Cache the result of `keyword.absent?` and `zero_width_primitive.absent?` per
+position. Parslet doesn't support this natively, but a wrapper atom could.
+
+### Option B: Switch to a faster parser (medium effort)
+Replace Parslet with:
+- **Tree-sitter**: Compile a grammar for ISC, get C-speed parsing.
+- **Racc (yacc)**: Generate an LALR parser — no backtracking.
+- **Hand-written recursive descent**: Fastest but most maintenance.
+
+### Option C: Pre-compile .isc to Ruby AST (long-term)
+Instead of parsing .isc at runtime, compile it to a Ruby file at build time.
+The compiled file constructs `Interscript::Node` objects directly.
+
+## Recommendation
+Option A for immediate relief (target: <5s per file).
+Option C for the long term — ISC becomes a source format, compiled to Ruby
+or JS at distribution time.

TODO.secryst/01-typescript-runtime-parity.md

@@ -0,0 +1,33 @@
+# 01 — TypeScript runtime parity
+
+## Priority: HIGH
+
+## Current State
+- ISC parser exists only in Ruby (Parslet-based).
+- The user explicitly requires: "Both Ruby and TS must be first class."
+- The TS runtime has no ISC parser — it still uses the old .imp format.
+
+## Scope
+1. Port the ISC grammar to TypeScript using a PEG parser library:
+   - ** Peggy.js** (formerly peggy) — mature PEG parser generator for JS/TS
+   - **tree-sitter-grammar** — if going the C-speed route
+   - **Hand-written parser** — following the Ruby grammar's structure
+
+2. Port the DocumentBuilder equivalent (tree → typed object model).
+3. Port the codemod (`.imp` → `.isc`) — likely in TS or as a Ruby-generated tool.
+4. Ensure the TS runtime can LOAD `.isc` files and produce the same
+   transliteration output as Ruby.
+
+## Architecture
+```
+packages/
+  isc-parser/          # TS ISC parser (Peggy grammar)
+  isc-document-builder/ # tree → typed model
+  isc-codemod/         # .imp → .isc converter
+```
+
+## Verification
+- Cross-validate: Ruby and TS parsers produce identical document hashes for
+  all 289 maps.
+- Integration test: run transliteration on test cases, compare Ruby vs TS
+  output character-by-character.

TODO.secryst/02-is1-specification-compilation.md

@@ -0,0 +1,29 @@
+# 02 — IS 1 specification compilation
+
+## Priority: MEDIUM
+
+## Current State
+- The IS 1 specification exists as a Metanorma AsciiDoc file at
+  `spec/isc/document.adoc`.
+- It has not been compiled to HTML/PDF/XML yet.
+- The spec describes the ISC format formally but may be out of date with
+  recent grammar changes.
+
+## Steps
+1. Review `spec/isc/document.adoc` against the current grammar:
+   - Verify all grammar rules are documented.
+   - Update the metadata, tests, stages, and items sections.
+   - Add the escaped-brace syntax (`\{`, `\}`) for raw text blocks.
+   - Document the `separate separator` and `decompose` directives.
+
+2. Compile the spec:
+   ```bash
+   bundle exec metanorma spec/isc/document.adoc
+   ```
+
+3. Publish the compiled HTML/PDF to the interscript.org website.
+
+## Annexes to Add
+- **Migration Annex**: step-by-step guide for converting .imp → .isc.
+- **Grammar Reference**: complete BNF/PEG grammar extracted from the Ruby code.
+- **Examples**: real-world ISC snippets from the 289 maps.

TODO.secryst/03-isc-compiler.md

@@ -0,0 +1,40 @@
+# 03 — ISC compiler: compile .isc to executable Ruby/JS
+
+## Priority: MEDIUM
+
+## Current State
+- ISC files are parsed at runtime by the Parslet parser.
+- For large CJK maps (40k+ lines), parsing takes 20-38 seconds.
+- The Ruby DSL (.imp) is compiled to `Interscript::Node` objects via
+  `instance_exec` — also not fast, but cached.
+
+## Proposal
+Build a compiler that transforms `.isc` source into an executable artifact:
+
+### Ruby target
+Compile `.isc` → `.rb` that constructs `Interscript::Node` objects directly:
+```ruby
+# Generated from foo.isc
+Interscript::Node::Document.new.tap do |doc|
+  doc.metadata = Interscript::Node::MetaData.new(...)
+  doc.stages[:main] = Interscript::Node::Stage.new(...)
+end
+```
+
+### JavaScript target
+Compile `.isc` → `.js` that constructs equivalent JS objects.
+
+### Distribution
+- Ship compiled `.rb`/`.js` files alongside (or instead of) `.isc` source.
+- The `.isc` source is for humans; the compiled artifact is for runtime.
+- A `rake compile` task generates all artifacts from `.isc` sources.
+
+## Benefits
+1. **Performance**: No parser overhead at runtime — load a `.rb` file.
+2. **Validation**: Compilation catches errors at build time, not runtime.
+3. **Distribution**: Compiled files are deterministic and cacheable.
+
+## Implementation
+- New class: `Interscript::Isc::Compiler`
+- Methods: `compile_to_ruby(tree)`, `compile_to_javascript(tree)`
+- Integrates with existing `Interscript::Compiler::Ruby` and `::Javascript`.

TODO.secryst/04-ruby-dsl-array-keys-bug.md

@@ -0,0 +1,56 @@
+# 04 — Ruby DSL STANDARD_ARRAY_KEYS bug fix
+
+## Priority: LOW (affects Ruby DSL only, ISC already correct)
+
+## Current State
+The Ruby DSL's `lib/interscript/dsl/metadata.rb` defines:
+
+```ruby
+STANDARD_ARRAY_KEYS = %i[notes implementation_notes original_notes url]
+
+STANDARD_ARRAY_KEYS.each do |sym|
+  define_method sym do |stuff|
+    stuff = Array(stuff)
+    stuff.map do |i|
+      case i
+      when String
+        i
+      else
+        warn "[#{@map_name}] Metadata key #{sym} expects all Array elements to be String"
+        i.inspect
+      end
+    end
+    # BUG: the processed array is never stored in @node!
+  end
+end
+```
+
+The method processes `stuff` but **never assigns the result to `@node[sym]`**.
+This means `notes`, `implementation_notes`, `original_notes`, and `url` are
+parsed from `.imp` files but silently discarded by the Ruby DSL.
+
+## Impact
+- Deep equivalence checker shows `imp=nil` for these fields across ALL maps.
+- The ISC parser correctly stores them — ISC is strictly more capable.
+- Users relying on Ruby DSL `metadata.data[:url]` get `nil`.
+
+## Fix
+```ruby
+STANDARD_ARRAY_KEYS.each do |sym|
+  define_method sym do |stuff|
+    @node[sym] = Array(stuff).map do |i|
+      case i
+      when String then i
+      else
+        warn "[#{@map_name}] Metadata key #{sym} expects String, got #{i.class}"
+        i.inspect
+      end
+    end
+  end
+end
+```
+
+## Verification
+After fix, re-run `exe/verify_isc_deep` — the metadata comparison for
+`url`, `notes`, `implementation_notes`, and `original_notes` should pass
+for all 289 maps.

TODO.secryst/05-isc-runtime-integration.md

@@ -0,0 +1,42 @@
+# 05 — ISC integration with existing Interscript runtime
+
+## Priority: HIGH
+
+## Current State
+- The ISC parser produces a document hash (metadata, tests, stages, aliases).
+- The existing Interscript runtime uses `Interscript::Node::*` objects.
+- There is NO bridge between ISC document hash and Interscript::Node objects.
+- Users cannot call `Interscript.transliterate("foo.isc", "hello")` yet.
+
+## Required Bridge
+Add a method to convert ISC document hash → `Interscript::Node::Document`:
+
+```ruby
+class Interscript::Isc::NodeAdapter
+  def self.to_interscript_node(isc_doc)
+    Interscript::Node::Document.new.tap do |doc|
+      doc.metadata = build_metadata(isc_doc[:metadata])
+      doc.tests = build_tests(isc_doc[:tests])
+      doc.stages = build_stages(isc_doc[:stages])
+      doc.aliases = build_aliases(isc_doc[:aliases])
+    end
+  end
+end
+```
+
+Then update `Interscript.load_map` to detect `.isc` extension and route
+through the ISC parser + adapter instead of the Ruby DSL.
+
+## Files to Create/Modify
+- `lib/interscript/isc/node_adapter.rb` (new)
+- `lib/interscript.rb` — update `load_map` to support `.isc`
+- `lib/interscript/path.rb` — resolve `.isc` files in the maps path
+
+## Verification
+```ruby
+# Should work identically:
+Interscript.transliterate("alalc-amh-Ethi-Latn-1997", "ሀለሐ")  # .imp
+Interscript.transliterate("alalc-amh-Ethi-Latn-1997.isc", "ሀለሐ")  # .isc
+```
+
+Both should produce the same output.