feat(core,cli,playwright,spec): resolve definitive engine, isolation, and schema defects

[CodeinScrubs]

Summary

Resolves the remaining engine defects, schema synchronization discrepancies, isolation edge cases, and CLI/testing enhancements documented in plan_bidilens_definitive_bugs_and_enhancements.md.

Note: This PR is stacked on top of PR #169 (feat/engine-chatgpt-breakdown-hardening). When #169 merges into main, GitHub will automatically retarget this PR to main (or it can be rebased cleanly).

---

Key Fixes & Enhancements

1. Core Engine & Analysis

• Custom Security Scanner Integration (analyzeBlock):
  • Added securityScanner?: (text: string) => BidiSecurityReport | BidiSecurityFinding[] to AnalyzeBlockOptions.
  • Supports both full BidiSecurityReport objects and direct BidiSecurityFinding[] arrays, returned on result.warnings.
• Inherited Direction Propagation (planInlineIsolation, analyzeBlock):
  • Propagates options.inheritedDirection in AnalyzeBlockOptions into planInlineIsolation.
  • Ensures container-level inherited direction (options.inheritedDirection ?? blockDirection) is evaluated during intervention checks (needsBidiIntervention), avoiding redundant or missing isolation wrappers when embedding within opposing-direction DOM contexts.

2. Technical Token & Boundary Detection

• Persian/Arabic Numbers, Decimal Momayyez & Percentages:
  • Added NUMBER_TOKEN, CURRENCY_PERCENT_TOKEN, and NUMBER_RANGE_TOKEN regular expressions.
  • Recognizes Persian thousands separator (?????????), Persian Momayyez decimal point (??????), Arabic percent sign (???, 50%), currency prefixes/suffixes ($50), and number ranges (10-20, 2020-2024, 10?20, ?????, ?????).
• Multiline Display Math:
  • Extended addMathRanges in @bidilens/core to recognize multiline $$...$$ and \[...\] math blocks while preserving single-pass linear time $O(N)$ scanning with delimiter boundary tracking.
• Path Token Boundaries:
  • Hardened file path matching to handle Windows drive paths (C:\...), Windows/POSIX relative paths (.\..., ..\..., ./..., ../...), and home paths (~/..., ~\...) without misclassifying LaTeX escapes (\$x\$) or Unix root paths (/usr/local/bin).

3. UAX #9 Character Classification

• Added CharacterClassification interface and getCharacterClassification(codePoint: number) in @bidilens/core.
• Overloaded classifyCharacter to support both codePoint: number (returning detailed CharacterClassification) and character: string (returning Direction), properly identifying:
  • AN (Arabic Number, isWeak: true)
  • EN (European Number, isWeak: true)
  • NSM (Non-Spacing Mark, isWeak: true, isMark: true)
  • BN (Boundary Neutral e.g. ZWNJ/ZWJ/SHY, isWeak: true)
  • B (Paragraph Separator e.g. LF/CR/NEL/PS, isNeutral: true)
  • S (Segment Separator e.g. TAB/US, isNeutral: true)
  • WS (Whitespace, isNeutral: true)
  • ON (Other Neutral, isNeutral: true)

4. Security & Confusable Whitelist

• Whitelisted standard SI metric units (?m, ?s, ?g, ?L, ?mol, ?V, ?A, ?F, ?W, ?H) in scanBidiSecurity supporting both Greek small letter mu (U+03BC ?) and micro sign (U+00B5 ?).
• Metric units containing micro signs preceded by numbers are no longer flagged as homoglyph/confusable security findings.
• Added confusable scanning support (scanConfusables: true) with accurate code point source offsets.

5. Specification Schema Synchronization

• Updated packages/spec/schemas/common.schema.json:
  • Added bidiSourceRange to directionEvidence.
  • Added bidiSourceRange and excluded to inlineIsolation.

6. CLI .gitignore Support

• Added .gitignore discovery and hierarchy traversal to @bidilens/cli (collectFiles and loadGitignore).
• Ignores files and directories matching .gitignore patterns during recursive audits and security scans across target directories.

7. Playwright Visual Coordinate Testing

• Exported expectTextOrder(target, selectorOrOrder, expectedOrder?, direction?) from @bidilens/playwright.
• Asserts that text fragments appear in correct visual horizontal sequence (left-to-right increasing X coordinates for LTR, decreasing X for RTL) using actual rendered geometry from getBoundingClientRect().
• Supports both Locator targets and Page targets with root/body fallback.

8. Conformance Corpus Synchronization

• Updated scripts/generate-corpus.ts policy overrides for Arabic and Persian percent (???, ???) and large thousands-separated numbers (??????).
• Synchronized all 932 conformance fixtures across root corpus/cases.json, @bidilens/cli packaged cases, Kotlin Android fixtures, and Swift Apple fixtures.

---

Verification Record

• pnpm run build: All 12 packages compiled cleanly via tsc, tsup, vite, and esbuild.
• pnpm run typecheck: Passed cleanly with zero TypeScript errors across the entire monorepo (tsc --noEmit).
• pnpm run lint: ESLint flat config passed with zero warnings or errors.
• pnpm test: 24 test files, 652 tests passed, 0 failures (including 23 tests in packages/core/src/hardening.test.ts, plus new tests in CLI and Playwright).
• pnpm run corpus:check: All 932 fixtures passed (0 failures), Android and Apple representations reproducible.
• pnpm run check: Full CI validation gate passed with exit code 0 (unicode:check, typecheck, lint, packages:depth, test:coverage, corpus:check, docs:check, build, action:check).
• pnpm run release:check: Validated distribution budgets, exports, tarballs, integration guides, and packed examples across all 12 packages on a clean worktree.