diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index eee67d7..bb8457e 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -30,7 +30,7 @@ | Bun | Deno | | pnpm/yarn | Deno | | Go | Rust | -| Python | Julia/Rust/ReScript | +| Python | Julia/Rust/AffineScript | | Java/Kotlin | Rust/Tauri/Dioxus | | Swift | Tauri/Dioxus | | React Native | Tauri/Dioxus | @@ -57,7 +57,7 @@ Both are FOSS with independent governance (no Big Tech). ### Package Management - **Primary**: Guix (guix.scm) -- **Fallback**: Nix (flake.nix) +- **Fallback**: Guix (flake.guix) - **JS deps**: Deno (deno.json imports) ### Security Requirements diff --git a/.github/dependabot.yml b/.github/dependabot.yml index d5cd4e9..045cb8f 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -41,8 +41,8 @@ updates: schedule: interval: "weekly" - # Nix flakes - - package-ecosystem: "nix" + # Guix flakes + - package-ecosystem: "guix" directory: "/" schedule: interval: "weekly" diff --git a/ABI-FFI-README.md b/ABI-FFI-README.adoc similarity index 54% rename from ABI-FFI-README.md rename to ABI-FFI-README.adoc index cdba250..02f8f4d 100644 --- a/ABI-FFI-README.md +++ b/ABI-FFI-README.adoc @@ -1,17 +1,20 @@ -# Docudactyl ABI/FFI Documentation +== Docudactyl ABI/FFI Documentation -## Overview +=== Overview -This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design: +This library follows the *Hyperpolymath RSR Standard* for ABI and FFI +design: -- **ABI (Application Binary Interface)** defined in **Idris2** with formal proofs -- **FFI (Foreign Function Interface)** implemented in **Zig** for C compatibility -- **Generated C headers** bridge Idris2 ABI to Zig FFI -- **Chapel HPC** calls through standard C ABI for distributed execution +* *ABI (Application Binary Interface)* defined in *Idris2* with formal +proofs +* *FFI (Foreign Function Interface)* implemented in *Zig* for C +compatibility +* *Generated C headers* bridge Idris2 ABI to Zig FFI +* *Chapel HPC* calls through standard C ABI for distributed execution -## Architecture +=== Architecture -``` +.... ┌─────────────────────────────────────────────┐ │ ABI Definitions (Idris2) │ │ src/Docudactyl/ABI/ │ @@ -48,11 +51,11 @@ This library follows the **Hyperpolymath RSR Standard** for ABI and FFI design: │ - 51 extern proc declarations │ │ - Distributed across 64-512 locales │ └─────────────────────────────────────────────┘ -``` +.... -## Directory Structure +=== Directory Structure -``` +.... docudactyl/ ├── src/ │ └── Docudactyl/ @@ -85,84 +88,137 @@ docudactyl/ │ └── docudactyl_ffi.h # Generated from Zig FFI exports │ └── docudactyl.ipkg # Idris2 package (3 modules) -``` +.... -## Proven Types (Idris2) +=== Proven Types (Idris2) The ABI layer formally proves: -| Type | Variants | Proof | -|------|----------|-------| -| `ContentKind` | 7 (PDF, Image, Audio, Video, EPUB, GeoSpatial, Unknown) | Enum injectivity | -| `ParseStatus` | 6 (Ok, Error, FileNotFound, ParseError, UnsupportedFormat, OutOfMemory) | Enum injectivity | -| `MlStatus` | 5 (Ok, ModelNotFound, InferenceError, InputError, OnnxNotAvailable) | Enum injectivity | -| `MlStage` | 5 (NER, Whisper, ImageClassify, Layout, Handwriting) | Enum injectivity | -| `ExecProvider` | 4 (TensorRT, CUDA, OpenVINO, CPU) | Enum injectivity | -| `Sha256Tier` | 3 (Dedicated, Avx2Buffer, Software) | Enum injectivity | -| `OcrStatus` | 4 | Enum injectivity | -| `GpuBackend` | 3 | Enum injectivity | -| `ConduitValidation` | 4 | Enum injectivity | - -### Struct Layout Proofs - -| Struct | Size | Alignment | Proof | -|--------|------|-----------|-------| -| `ParseResult` | 952 bytes | 8-byte (LP64) | `Divides 8 952 = MkDivides 119` | -| `MlResult` | 48 bytes | 8-byte | `Divides 8 48 = MkDivides 6` | -| `CryptoCaps` | 16 bytes | 1-byte | Field offset chain | -| `OcrResult` | 48 bytes | — | Size assertion | -| `ConduitResult` | 88 bytes | — | Size assertion | - -## Building - -### Build FFI Library - -```bash +[width="100%",cols="27%,43%,30%",options="header",] +|=== +|Type |Variants |Proof +|`+ContentKind+` |7 (PDF, Image, Audio, Video, EPUB, GeoSpatial, +Unknown) |Enum injectivity + +|`+ParseStatus+` |6 (Ok, Error, FileNotFound, ParseError, +UnsupportedFormat, OutOfMemory) |Enum injectivity + +|`+MlStatus+` |5 (Ok, ModelNotFound, InferenceError, InputError, +OnnxNotAvailable) |Enum injectivity + +|`+MlStage+` |5 (NER, Whisper, ImageClassify, Layout, Handwriting) |Enum +injectivity + +|`+ExecProvider+` |4 (TensorRT, CUDA, OpenVINO, CPU) |Enum injectivity + +|`+Sha256Tier+` |3 (Dedicated, Avx2Buffer, Software) |Enum injectivity + +|`+OcrStatus+` |4 |Enum injectivity + +|`+GpuBackend+` |3 |Enum injectivity + +|`+ConduitValidation+` |4 |Enum injectivity +|=== + +==== Struct Layout Proofs + +[width="100%",cols="27%,18%,34%,21%",options="header",] +|=== +|Struct |Size |Alignment |Proof +|`+ParseResult+` |952 bytes |8-byte (LP64) +|`+Divides 8 952 = MkDivides 119+` + +|`+MlResult+` |48 bytes |8-byte |`+Divides 8 48 = MkDivides 6+` + +|`+CryptoCaps+` |16 bytes |1-byte |Field offset chain + +|`+OcrResult+` |48 bytes |— |Size assertion + +|`+ConduitResult+` |88 bytes |— |Size assertion +|=== + +=== Building + +==== Build FFI Library + +[source,bash] +---- just build-ffi # Build via Justfile # or directly: cd ffi/zig && zig build -Doptimize=ReleaseFast -``` +---- -### Verify Idris2 ABI Proofs +==== Verify Idris2 ABI Proofs -```bash +[source,bash] +---- just build-idris # or directly: idris2 --build docudactyl.ipkg -``` +---- -### Generate C Header +==== Generate C Header -```bash +[source,bash] +---- just generate-abi-header -``` +---- -### Run Tests +==== Run Tests -```bash +[source,bash] +---- just test-ffi # 40+ integration tests against C ABI just test-idris # Verify Idris2 proofs compile -``` +---- + +=== C API Summary + +All 51 functions use the `+ddac_+` prefix: + +[width="100%",cols="48%,52%",options="header",] +|=== +|Category |Functions +|*Core lifecycle* |`+ddac_init+`, `+ddac_free+`, `+ddac_parse+`, +`+ddac_version+` + +|*Handle setters* |`+ddac_set_ml_handle+`, `+ddac_set_gpu_ocr_handle+` + +|*LMDB cache* |`+ddac_cache_init+`, `+ddac_cache_free+`, +`+ddac_cache_lookup+`, `+ddac_cache_store+`, `+ddac_cache_count+`, +`+ddac_cache_sync+` + +|*Dragonfly* |`+ddac_dragonfly_connect+`, `+ddac_dragonfly_close+`, +`+ddac_dragonfly_lookup+`, `+ddac_dragonfly_store+`, +`+ddac_dragonfly_count+` + +|*I/O prefetcher* |`+ddac_prefetch_init+`, `+ddac_prefetch_hint+`, +`+ddac_prefetch_done+`, `+ddac_prefetch_free+`, +`+ddac_prefetch_inflight+` + +|*ML inference* |`+ddac_ml_init+`, `+ddac_ml_free+`, +`+ddac_ml_available+`, `+ddac_ml_provider+`, `+ddac_ml_provider_name+`, +`+ddac_ml_set_model_dir+`, `+ddac_ml_run_stage+`, `+ddac_ml_stats+`, +`+ddac_ml_result_size+`, `+ddac_ml_stage_count+`, `+ddac_ml_model_name+` -## C API Summary +|*GPU OCR* |`+ddac_gpu_ocr_init+`, `+ddac_gpu_ocr_free+`, +`+ddac_gpu_ocr_backend+`, `+ddac_gpu_ocr_submit+`, +`+ddac_gpu_ocr_flush+`, `+ddac_gpu_ocr_results_ready+`, +`+ddac_gpu_ocr_collect+`, `+ddac_gpu_ocr_stats+`, +`+ddac_gpu_ocr_max_batch+`, `+ddac_gpu_ocr_result_size+` -All 51 functions use the `ddac_` prefix: +|*Hardware crypto* |`+ddac_crypto_detect+`, `+ddac_crypto_sha256_tier+`, +`+ddac_crypto_sha256_name+`, `+ddac_crypto_batch_sha256+`, +`+ddac_crypto_caps_size+` -| Category | Functions | -|----------|-----------| -| **Core lifecycle** | `ddac_init`, `ddac_free`, `ddac_parse`, `ddac_version` | -| **Handle setters** | `ddac_set_ml_handle`, `ddac_set_gpu_ocr_handle` | -| **LMDB cache** | `ddac_cache_init`, `ddac_cache_free`, `ddac_cache_lookup`, `ddac_cache_store`, `ddac_cache_count`, `ddac_cache_sync` | -| **Dragonfly** | `ddac_dragonfly_connect`, `ddac_dragonfly_close`, `ddac_dragonfly_lookup`, `ddac_dragonfly_store`, `ddac_dragonfly_count` | -| **I/O prefetcher** | `ddac_prefetch_init`, `ddac_prefetch_hint`, `ddac_prefetch_done`, `ddac_prefetch_free`, `ddac_prefetch_inflight` | -| **ML inference** | `ddac_ml_init`, `ddac_ml_free`, `ddac_ml_available`, `ddac_ml_provider`, `ddac_ml_provider_name`, `ddac_ml_set_model_dir`, `ddac_ml_run_stage`, `ddac_ml_stats`, `ddac_ml_result_size`, `ddac_ml_stage_count`, `ddac_ml_model_name` | -| **GPU OCR** | `ddac_gpu_ocr_init`, `ddac_gpu_ocr_free`, `ddac_gpu_ocr_backend`, `ddac_gpu_ocr_submit`, `ddac_gpu_ocr_flush`, `ddac_gpu_ocr_results_ready`, `ddac_gpu_ocr_collect`, `ddac_gpu_ocr_stats`, `ddac_gpu_ocr_max_batch`, `ddac_gpu_ocr_result_size` | -| **Hardware crypto** | `ddac_crypto_detect`, `ddac_crypto_sha256_tier`, `ddac_crypto_sha256_name`, `ddac_crypto_batch_sha256`, `ddac_crypto_caps_size` | -| **Conduit** | `ddac_conduit_process`, `ddac_conduit_batch`, `ddac_conduit_result_size` | +|*Conduit* |`+ddac_conduit_process+`, `+ddac_conduit_batch+`, +`+ddac_conduit_result_size+` +|=== -## Usage from Chapel +=== Usage from Chapel -```chapel +[source,chapel] +---- extern proc ddac_init(): c_ptr(void); extern proc ddac_free(handle: c_ptr(void)): void; extern proc ddac_parse(handle: c_ptr(void), inputPath: c_ptrConst(c_char), @@ -173,30 +229,32 @@ extern proc ddac_parse(handle: c_ptr(void), inputPath: c_ptrConst(c_char), var handle = ddac_init(); defer ddac_free(handle); var result = ddac_parse(handle, path.c_str(), outPath.c_str(), "scheme".c_str()); -``` +---- -## Contributing +=== Contributing When modifying the ABI/FFI: -1. **Update Idris2 ABI first** (`src/Docudactyl/ABI/*.idr`) - - Add/modify type definitions with proofs - - Update struct layout proofs - - Add FFI declarations -2. **Update Zig FFI** (`ffi/zig/src/`) - - Implement new functions matching C ABI - - Ensure `comptime` assertions match Idris2 proofs -3. **Regenerate C header** (`just generate-abi-header`) -4. **Add integration tests** (`ffi/zig/test/integration_test.zig`) -5. **Update Chapel extern declarations** (`src/chapel/FFIBridge.chpl`) +[arabic] +. *Update Idris2 ABI first* (`+src/Docudactyl/ABI/*.idr+`) +* Add/modify type definitions with proofs +* Update struct layout proofs +* Add FFI declarations +. *Update Zig FFI* (`+ffi/zig/src/+`) +* Implement new functions matching C ABI +* Ensure `+comptime+` assertions match Idris2 proofs +. *Regenerate C header* (`+just generate-abi-header+`) +. *Add integration tests* (`+ffi/zig/test/integration_test.zig+`) +. *Update Chapel extern declarations* (`+src/chapel/FFIBridge.chpl+`) -## License +=== License SPDX-License-Identifier: CC-BY-SA-4.0 -## See Also +=== See Also -- [Idris2 Documentation](https://idris2.readthedocs.io) -- [Zig Documentation](https://ziglang.org/documentation/master/) -- [Chapel Documentation](https://chapel-lang.org/docs/) -- [Rhodium Standard Repositories](https://github.com/hyperpolymath/rhodium-standard-repositories) +* https://idris2.readthedocs.io[Idris2 Documentation] +* https://ziglang.org/documentation/master/[Zig Documentation] +* https://chapel-lang.org/docs/[Chapel Documentation] +* https://github.com/hyperpolymath/rhodium-standard-repositories[Rhodium +Standard Repositories] diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc new file mode 100644 index 0000000..3209d7a --- /dev/null +++ b/CHANGELOG.adoc @@ -0,0 +1,76 @@ +== Changelog + +All notable changes to `+docudactyl+` will be documented in this file. + +This file is generated from conventional commits by the +https://github.com/hyperpolymath/standards/blob/main/.github/workflows/changelog-reusable.yml[`+changelog-reusable.yml+`] +workflow (`+hyperpolymath/standards#206+`). Adopt the workflow in this +repo’s CI to keep this file in sync automatically — see +https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+templates/cliff.toml+`] +for the canonical config. + +The format follows https://keepachangelog.com/en/1.1.0/[Keep a +Changelog]; this project aims to follow +https://semver.org/spec/v2.0.0.html[Semantic Versioning]. + +=== [Unreleased] + +==== Added + +* feat(investigator): add five modules for citizen & investigative +journalism (#2) +* feat(crg): add crg-grade and crg-badge justfile recipes +* feat: add stapeln.toml container definition +* feat: deploy UX Manifesto infrastructure +* feat: add CLADE.a2ml — clade taxonomy declaration + +==== Fixed + +* fix(ci): bump a2ml/k9-validate-action pins to canonical (standards#85) +(#16) +* fix(ci): sync hypatia-scan.yml to canonical (kill cd-scanner build +drift) (#15) +* fix(ci): adopt canonical hypatia-scan.yml (env.HOME/scanner-layout + +Comment-step gate) (#14) +* fix(ci): Phase-2 fleet submission must not fail the security gate +(#13) +* fix(ci): hypatia-scan workdir ($\{\{ env.HOME }} resolves empty) (#12) +* fix(ci): rsr-antipattern.yml duplicate heredoc (#9) +* fix(ci): move secret-scanner Cargo.toml gate from job-level if: to +step-level (#10) +* fix(ci): fix broken regex quoting in security-policy and update +trufflehog SHA +* fix(scorecard): enforce granular permissions and add fuzzing +placeholder +* fix(ci): Resolve workflow-linter self-matching and metadata issues + +==== Changed + +* refactor: migrate 6SCM → 6A2 (.scm → .a2ml format) + +==== Documentation + +* docs: add TEST-NEEDS.md (CRG C) +* docs: add TEST-NEEDS.md (CRG C) +* docs: add EXPLAINME.adoc — prove-it file backing README claims + +==== CI + +* ci: redistribute concurrency-cancel guard to read-only check workflows +(#18) +* ci: bump actions/upload-artifact SHA to current v4 (#8) +* ci(secret-scanner): drop duplicate –fail from trufflehog extra_args +(#7) +* ci: SHA-pin hyperpolymath validate-actions in dogfood-gate +* ci(antipattern): fix top-level dir matching + benchmarks/lsp/bench +filename allowlists (#6) + +=== Pre-history + +Prior commits to this file’s introduction are recorded in git history +but not formally classified into Keep-a-Changelog sections. To backfill, +run `+git cliff -o CHANGELOG.md+` locally using the canonical +https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml[`+cliff.toml+`] +— this is one-shot mechanical work. + +''''' diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index 482ed79..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,66 +0,0 @@ - - -# Changelog - -All notable changes to `docudactyl` will be documented in this file. - -This file is generated from conventional commits by the -[`changelog-reusable.yml`](https://github.com/hyperpolymath/standards/blob/main/.github/workflows/changelog-reusable.yml) -workflow (`hyperpolymath/standards#206`). Adopt the workflow in this repo's CI to keep this file in sync automatically — see -[`templates/cliff.toml`](https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml) -for the canonical config. - -The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); -this project aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - -## [Unreleased] - -### Added - -- feat(investigator): add five modules for citizen & investigative journalism (#2) -- feat(crg): add crg-grade and crg-badge justfile recipes -- feat: add stapeln.toml container definition -- feat: deploy UX Manifesto infrastructure -- feat: add CLADE.a2ml — clade taxonomy declaration - -### Fixed - -- fix(ci): bump a2ml/k9-validate-action pins to canonical (standards#85) (#16) -- fix(ci): sync hypatia-scan.yml to canonical (kill cd-scanner build drift) (#15) -- fix(ci): adopt canonical hypatia-scan.yml (env.HOME/scanner-layout + Comment-step gate) (#14) -- fix(ci): Phase-2 fleet submission must not fail the security gate (#13) -- fix(ci): hypatia-scan workdir (${{ env.HOME }} resolves empty) (#12) -- fix(ci): rsr-antipattern.yml duplicate heredoc (#9) -- fix(ci): move secret-scanner Cargo.toml gate from job-level if: to step-level (#10) -- fix(ci): fix broken regex quoting in security-policy and update trufflehog SHA -- fix(scorecard): enforce granular permissions and add fuzzing placeholder -- fix(ci): Resolve workflow-linter self-matching and metadata issues - -### Changed - -- refactor: migrate 6SCM → 6A2 (.scm → .a2ml format) - -### Documentation - -- docs: add TEST-NEEDS.md (CRG C) -- docs: add TEST-NEEDS.md (CRG C) -- docs: add EXPLAINME.adoc — prove-it file backing README claims - -### CI - -- ci: redistribute concurrency-cancel guard to read-only check workflows (#18) -- ci: bump actions/upload-artifact SHA to current v4 (#8) -- ci(secret-scanner): drop duplicate --fail from trufflehog extra_args (#7) -- ci: SHA-pin hyperpolymath validate-actions in dogfood-gate -- ci(antipattern): fix top-level dir matching + benchmarks/lsp/bench filename allowlists (#6) - -## Pre-history - -Prior commits to this file's introduction are recorded in git history but not formally classified into Keep-a-Changelog sections. To backfill, run `git cliff -o CHANGELOG.md` locally using the canonical [`cliff.toml`](https://github.com/hyperpolymath/standards/blob/main/templates/cliff.toml) — this is one-shot mechanical work. - ---- - - diff --git a/CODE_OF_CONDUCT.adoc b/CODE_OF_CONDUCT.adoc new file mode 100644 index 0000000..819f5a6 --- /dev/null +++ b/CODE_OF_CONDUCT.adoc @@ -0,0 +1,339 @@ +== Code of Conduct + +=== Our Pledge + +We as members, contributors, and leaders pledge to make participation in +Docudactyl a harassment-free experience for everyone, regardless of age, +body size, visible or invisible disability, ethnicity, sex +characteristics, gender identity and expression, level of experience, +education, socio-economic status, nationality, personal appearance, +race, caste, colour, religion, or sexual identity and orientation. + +We pledge to act and interact in ways that contribute to an open, +welcoming, diverse, inclusive, and healthy community. + +We recognise that a thriving open source community requires +*psychological safety* — an environment where people can contribute, ask +questions, make mistakes, and learn without fear of ridicule or +retaliation. + +''''' + +=== Our Standards + +==== Expected Behaviour + +The following behaviours contribute to a positive environment: + +*Communication* - Using welcoming and inclusive language - Being +respectful of differing viewpoints and experiences - Giving and +gracefully accepting constructive feedback - Assuming good intent while +addressing impact - Communicating clearly and patiently, especially with +newcomers + +*Collaboration* - Focusing on what is best for the community - Showing +empathy and kindness toward other community members - Being +collaborative rather than competitive - Mentoring and supporting less +experienced contributors - Celebrating others’ contributions and +successes + +*Professionalism* - Accepting responsibility and apologising to those +affected by our mistakes - Learning from the experience and avoiding +repetition - Respecting others’ time and attention - Staying on topic in +project spaces - Following project guidelines and conventions + +*Accessibility* - Using plain language and avoiding unnecessary jargon - +Providing alt text for images and transcripts for audio/video - Being +patient with those using assistive technologies - Accommodating +different communication styles and needs - Recognising that not everyone +communicates the same way + +==== Unacceptable Behaviour + +The following behaviours are considered harassment and are unacceptable: + +*Harassment* - The use of sexualised language or imagery, and sexual +attention or advances of any kind - Trolling, insulting or derogatory +comments, and personal or political attacks - Public or private +harassment - Deliberate intimidation, stalking, or following (online or +in-person) - Unwelcome physical contact or simulated physical contact +(e.g., emoji) - Sustained disruption of talks, events, or online +discussions + +*Discrimination* - Discriminatory jokes and language - Posting or +threatening to post others’ personally identifying information +("`doxing`") - Advocating for, or encouraging, any of the above +behaviour - Microaggressions — subtle, often unintentional, +discriminatory comments or actions + +*Professional Misconduct* - Publishing others’ private information +without explicit permission - Misrepresenting affiliation or +contributions - Plagiarism or claiming credit for others’ work - +Retaliating against anyone who reports a Code of Conduct violation - +Other conduct which could reasonably be considered inappropriate in a +professional setting + +==== Grey Areas + +Some situations require judgement. When uncertain: + +* *Intent vs Impact*: Good intentions do not excuse harmful impact. +Focus on making things right. +* *Power Dynamics*: Those with more power (maintainers, employers, +experienced contributors) must be especially mindful of their impact. +* *Cultural Differences*: What’s acceptable varies by culture. When in +doubt, err on the side of caution and ask. +* *Humour*: Jokes at others’ expense are rarely funny to everyone. Punch +up, not down. + +''''' + +=== Scope + +This Code of Conduct applies within all community spaces, including: + +*Online Spaces* - Repository discussions, issues, and pull/merge +requests - Project chat channels (Matrix, Discord, Slack, IRC) - Mailing +lists and forums - Social media when representing the project - Video +calls and virtual meetings + +*In-Person Spaces* - Conferences, meetups, and events - Workshops and +training sessions - Any gathering where you represent the project + +*Representation* This Code of Conduct also applies when an individual is +officially representing the community in public spaces. Examples +include: + +* Using an official project email address +* Posting via an official social media account +* Acting as an appointed representative at an event +* Speaking on behalf of the project + +''''' + +=== Enforcement + +==== Reporting + +If you experience or witness unacceptable behaviour, or have any other +concerns, please report it as soon as possible. + +*How to Report* + +[width="99%",cols="30%,33%,37%",options="header",] +|=== +|Method |Details |Best For +|*Email* |j.d.a.jewell@open.ac.uk |Detailed reports, sensitive matters + +|*Private Message* |Contact any maintainer directly |Quick questions, +minor issues + +|*Anonymous Form* |[Link to form if available] |When you need anonymity +|=== + +*What to Include* + +* Your contact information (unless anonymous) +* Names/usernames of those involved +* Description of what happened +* When and where it occurred +* Any witnesses +* Any supporting evidence (screenshots, links) +* How you would like us to respond (if you have a preference) + +*What Happens Next* + +[arabic] +. You will receive acknowledgment within *48 hours* +. The maintainers will review the report +. We may ask for additional information +. We will determine appropriate action +. We will inform you of the outcome (respecting others’ privacy) + +==== Confidentiality + +All reports will be handled with discretion: + +* Reporter identity is protected by default +* Details are shared only with those who need to know +* We will ask before naming you in any communication +* Anonymous reports are accepted and investigated + +==== Conflicts of Interest + +If a maintainers member is involved in an incident: + +* They will recuse themselves from the process +* Another maintainer or external party will handle the report +* We will disclose any potential conflicts + +''''' + +=== Enforcement Guidelines + +The maintainers will follow these guidelines in determining +consequences: + +==== 1. Correction + +*Community Impact*: Use of inappropriate language or other behaviour +deemed unprofessional or unwelcome. + +*Consequence*: A private, written warning providing clarity around the +nature of the violation and an explanation of why the behaviour was +inappropriate. A public apology may be requested. + +*Duration*: Immediate + +==== 2. Warning + +*Community Impact*: A violation through a single incident or series of +actions. + +*Consequence*: A warning with consequences for continued behaviour. No +interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, for a specified period. This +includes avoiding interactions in community spaces as well as external +channels like social media. Violating these terms may lead to a +temporary or permanent ban. + +*Duration*: 1-4 weeks + +==== 3. Temporary Ban + +*Community Impact*: A serious violation of community standards, +including sustained inappropriate behaviour. + +*Consequence*: A temporary ban from any sort of interaction or public +communication with the community for a specified period. No public or +private interaction with the people involved, including unsolicited +interaction with those enforcing the Code of Conduct, is allowed during +this period. Violating these terms may lead to a permanent ban. + +*Duration*: 1-6 months + +==== 4. Permanent Ban + +*Community Impact*: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behaviour, harassment of an +individual, or aggression toward or disparagement of classes of +individuals. + +*Consequence*: A permanent ban from any sort of public interaction +within the community. + +*Duration*: Permanent (with appeal rights after 12 months) + +==== Enforcement Across Perimeters + +For contributors with elevated access (Perimeter 2 or 1): + +[cols=",",options="header",] +|=== +|Level |Additional Consequence +|Correction |Noted in contributor record +|Warning |Access privileges may be temporarily reduced +|Temporary Ban |Access reduced to Perimeter 3 for ban duration +|Permanent Ban |All access revoked +|=== + +''''' + +=== Appeals + +If you believe an enforcement decision was made in error: + +[arabic] +. *Wait 7 days* after the decision (cooling-off period) +. *Email* j.d.a.jewell@open.ac.uk with subject line "`Appeal: [Original +Report ID]`" +. *Explain* why you believe the decision should be reconsidered +. *Provide* any new information not previously available + +*Appeals Process* + +* Appeals are reviewed by a different maintainers member than the +original +* You will receive a response within 14 days +* The appeals decision is final +* You may only appeal once per incident + +*Grounds for Appeal* + +* Procedural errors in the original investigation +* New evidence not previously available +* Disproportionate response to the violation +* Misunderstanding of facts + +''''' + +=== Supporting Those Who Report + +We are committed to supporting those who report violations: + +*We Will* - Believe and take all reports seriously - Respect your +privacy and confidentiality preferences - Keep you informed of progress +(if you wish) - Take steps to protect you from retaliation - Provide +resources if you need support + +*We Will Not* - Require you to confront the person directly - Dismiss +reports without investigation - Reveal your identity without consent - +Tolerate retaliation against reporters - Rush you to make decisions + +''''' + +=== Prevention + +Beyond enforcement, we actively work to prevent issues: + +*Onboarding* - All contributors are expected to read this Code of +Conduct - Perimeter 2 applicants must confirm they’ve read and +understood it - Maintainers receive additional training on enforcement + +*Culture* - We model the behaviour we expect - We intervene early when +we see potential issues - We thank people for positive contributions - +We create opportunities for diverse voices + +*Review* - This Code of Conduct is reviewed annually - Community +feedback is welcomed - Changes are communicated clearly + +''''' + +=== Acknowledgments + +This Code of Conduct is adapted from: + +* https://www.contributor-covenant.org/[Contributor Covenant], version +2.1 +* https://www.djangoproject.com/conduct/[Django Code of Conduct] +* https://www.rust-lang.org/policies/code-of-conduct[Rust Code of +Conduct] +* https://www.python.org/psf/conduct/[Python Community Code of Conduct] + +We thank these communities for their leadership in creating welcoming +spaces. + +''''' + +=== Questions? + +If you have questions about this Code of Conduct: + +* Open a +https://github.com/hyperpolymath/docudactyl/discussions[Discussion] (for +general questions) +* Email j.d.a.jewell@open.ac.uk (for private questions) +* Contact any maintainer directly + +''''' + +=== Summary + +*Be kind. Be respectful. Be collaborative.* + +We’re all here because we care about this project. Let’s make it a place +where everyone can do their best work. + +''''' + +Last updated: 2026 · Based on Contributor Covenant 2.1 diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md deleted file mode 100644 index 07a7fd3..0000000 --- a/CODE_OF_CONDUCT.md +++ /dev/null @@ -1,309 +0,0 @@ -# Code of Conduct - - - -## Our Pledge - -We as members, contributors, and leaders pledge to make participation in Docudactyl a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, colour, religion, or sexual identity and orientation. - -We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. - -We recognise that a thriving open source community requires **psychological safety** — an environment where people can contribute, ask questions, make mistakes, and learn without fear of ridicule or retaliation. - ---- - -## Our Standards - -### Expected Behaviour - -The following behaviours contribute to a positive environment: - -**Communication** -- Using welcoming and inclusive language -- Being respectful of differing viewpoints and experiences -- Giving and gracefully accepting constructive feedback -- Assuming good intent while addressing impact -- Communicating clearly and patiently, especially with newcomers - -**Collaboration** -- Focusing on what is best for the community -- Showing empathy and kindness toward other community members -- Being collaborative rather than competitive -- Mentoring and supporting less experienced contributors -- Celebrating others' contributions and successes - -**Professionalism** -- Accepting responsibility and apologising to those affected by our mistakes -- Learning from the experience and avoiding repetition -- Respecting others' time and attention -- Staying on topic in project spaces -- Following project guidelines and conventions - -**Accessibility** -- Using plain language and avoiding unnecessary jargon -- Providing alt text for images and transcripts for audio/video -- Being patient with those using assistive technologies -- Accommodating different communication styles and needs -- Recognising that not everyone communicates the same way - -### Unacceptable Behaviour - -The following behaviours are considered harassment and are unacceptable: - -**Harassment** -- The use of sexualised language or imagery, and sexual attention or advances of any kind -- Trolling, insulting or derogatory comments, and personal or political attacks -- Public or private harassment -- Deliberate intimidation, stalking, or following (online or in-person) -- Unwelcome physical contact or simulated physical contact (e.g., emoji) -- Sustained disruption of talks, events, or online discussions - -**Discrimination** -- Discriminatory jokes and language -- Posting or threatening to post others' personally identifying information ("doxing") -- Advocating for, or encouraging, any of the above behaviour -- Microaggressions — subtle, often unintentional, discriminatory comments or actions - -**Professional Misconduct** -- Publishing others' private information without explicit permission -- Misrepresenting affiliation or contributions -- Plagiarism or claiming credit for others' work -- Retaliating against anyone who reports a Code of Conduct violation -- Other conduct which could reasonably be considered inappropriate in a professional setting - -### Grey Areas - -Some situations require judgement. When uncertain: - -- **Intent vs Impact**: Good intentions do not excuse harmful impact. Focus on making things right. -- **Power Dynamics**: Those with more power (maintainers, employers, experienced contributors) must be especially mindful of their impact. -- **Cultural Differences**: What's acceptable varies by culture. When in doubt, err on the side of caution and ask. -- **Humour**: Jokes at others' expense are rarely funny to everyone. Punch up, not down. - ---- - -## Scope - -This Code of Conduct applies within all community spaces, including: - -**Online Spaces** -- Repository discussions, issues, and pull/merge requests -- Project chat channels (Matrix, Discord, Slack, IRC) -- Mailing lists and forums -- Social media when representing the project -- Video calls and virtual meetings - -**In-Person Spaces** -- Conferences, meetups, and events -- Workshops and training sessions -- Any gathering where you represent the project - -**Representation** -This Code of Conduct also applies when an individual is officially representing the community in public spaces. Examples include: - -- Using an official project email address -- Posting via an official social media account -- Acting as an appointed representative at an event -- Speaking on behalf of the project - ---- - -## Enforcement - -### Reporting - -If you experience or witness unacceptable behaviour, or have any other concerns, please report it as soon as possible. - -**How to Report** - -| Method | Details | Best For | -|--------|---------|----------| -| **Email** | j.d.a.jewell@open.ac.uk | Detailed reports, sensitive matters | -| **Private Message** | Contact any maintainer directly | Quick questions, minor issues | -| **Anonymous Form** | [Link to form if available] | When you need anonymity | - -**What to Include** - -- Your contact information (unless anonymous) -- Names/usernames of those involved -- Description of what happened -- When and where it occurred -- Any witnesses -- Any supporting evidence (screenshots, links) -- How you would like us to respond (if you have a preference) - -**What Happens Next** - -1. You will receive acknowledgment within **48 hours** -2. The maintainers will review the report -3. We may ask for additional information -4. We will determine appropriate action -5. We will inform you of the outcome (respecting others' privacy) - -### Confidentiality - -All reports will be handled with discretion: - -- Reporter identity is protected by default -- Details are shared only with those who need to know -- We will ask before naming you in any communication -- Anonymous reports are accepted and investigated - -### Conflicts of Interest - -If a maintainers member is involved in an incident: - -- They will recuse themselves from the process -- Another maintainer or external party will handle the report -- We will disclose any potential conflicts - ---- - -## Enforcement Guidelines - -The maintainers will follow these guidelines in determining consequences: - -### 1. Correction - -**Community Impact**: Use of inappropriate language or other behaviour deemed unprofessional or unwelcome. - -**Consequence**: A private, written warning providing clarity around the nature of the violation and an explanation of why the behaviour was inappropriate. A public apology may be requested. - -**Duration**: Immediate - -### 2. Warning - -**Community Impact**: A violation through a single incident or series of actions. - -**Consequence**: A warning with consequences for continued behaviour. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban. - -**Duration**: 1-4 weeks - -### 3. Temporary Ban - -**Community Impact**: A serious violation of community standards, including sustained inappropriate behaviour. - -**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban. - -**Duration**: 1-6 months - -### 4. Permanent Ban - -**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behaviour, harassment of an individual, or aggression toward or disparagement of classes of individuals. - -**Consequence**: A permanent ban from any sort of public interaction within the community. - -**Duration**: Permanent (with appeal rights after 12 months) - -### Enforcement Across Perimeters - -For contributors with elevated access (Perimeter 2 or 1): - -| Level | Additional Consequence | -|-------|----------------------| -| Correction | Noted in contributor record | -| Warning | Access privileges may be temporarily reduced | -| Temporary Ban | Access reduced to Perimeter 3 for ban duration | -| Permanent Ban | All access revoked | - ---- - -## Appeals - -If you believe an enforcement decision was made in error: - -1. **Wait 7 days** after the decision (cooling-off period) -2. **Email** j.d.a.jewell@open.ac.uk with subject line "Appeal: [Original Report ID]" -3. **Explain** why you believe the decision should be reconsidered -4. **Provide** any new information not previously available - -**Appeals Process** - -- Appeals are reviewed by a different maintainers member than the original -- You will receive a response within 14 days -- The appeals decision is final -- You may only appeal once per incident - -**Grounds for Appeal** - -- Procedural errors in the original investigation -- New evidence not previously available -- Disproportionate response to the violation -- Misunderstanding of facts - ---- - -## Supporting Those Who Report - -We are committed to supporting those who report violations: - -**We Will** -- Believe and take all reports seriously -- Respect your privacy and confidentiality preferences -- Keep you informed of progress (if you wish) -- Take steps to protect you from retaliation -- Provide resources if you need support - -**We Will Not** -- Require you to confront the person directly -- Dismiss reports without investigation -- Reveal your identity without consent -- Tolerate retaliation against reporters -- Rush you to make decisions - ---- - -## Prevention - -Beyond enforcement, we actively work to prevent issues: - -**Onboarding** -- All contributors are expected to read this Code of Conduct -- Perimeter 2 applicants must confirm they've read and understood it -- Maintainers receive additional training on enforcement - -**Culture** -- We model the behaviour we expect -- We intervene early when we see potential issues -- We thank people for positive contributions -- We create opportunities for diverse voices - -**Review** -- This Code of Conduct is reviewed annually -- Community feedback is welcomed -- Changes are communicated clearly - ---- - -## Acknowledgments - -This Code of Conduct is adapted from: - -- [Contributor Covenant](https://www.contributor-covenant.org/), version 2.1 -- [Django Code of Conduct](https://www.djangoproject.com/conduct/) -- [Rust Code of Conduct](https://www.rust-lang.org/policies/code-of-conduct) -- [Python Community Code of Conduct](https://www.python.org/psf/conduct/) - -We thank these communities for their leadership in creating welcoming spaces. - ---- - -## Questions? - -If you have questions about this Code of Conduct: - -- Open a [Discussion](https://github.com/hyperpolymath/docudactyl/discussions) (for general questions) -- Email j.d.a.jewell@open.ac.uk (for private questions) -- Contact any maintainer directly - ---- - -## Summary - -**Be kind. Be respectful. Be collaborative.** - -We're all here because we care about this project. Let's make it a place where everyone can do their best work. - ---- - -Last updated: 2026 · Based on Contributor Covenant 2.1 diff --git a/CONTRIBUTING.adoc b/CONTRIBUTING.adoc new file mode 100644 index 0000000..b4d2649 --- /dev/null +++ b/CONTRIBUTING.adoc @@ -0,0 +1,108 @@ +== Clone the repository + +git clone https://github.com/hyperpolymath/docudactyl.git cd docudactyl + +== Using Guix (recommended for reproducibility) + +guix develop + +== Or using toolbox/distrobox + +toolbox create docudactyl-dev toolbox enter docudactyl-dev # Install +dependencies manually + +== Verify setup + +just check # or: cargo check / mix compile / etc. just test # Run test +suite + +.... + +### Repository Structure +.... + +docudactyl/ ├── src/ # Source code (Perimeter 1-2) ├── lib/ # Library +code (Perimeter 1-2) ├── extensions/ # Extensions (Perimeter 2) ├── +plugins/ # Plugins (Perimeter 2) ├── tools/ # Tooling (Perimeter 2) ├── +docs/ # Documentation (Perimeter 3) │ ├── architecture/ # ADRs, specs +(Perimeter 2) │ └── proposals/ # RFCs (Perimeter 3) ├── examples/ # +Examples (Perimeter 3) ├── spec/ # Spec tests (Perimeter 3) ├── tests/ # +Test suite (Perimeter 2-3) ├── .well-known/ # Protocol files (Perimeter +1-3) ├── .github/ # GitHub config (Perimeter 1) │ ├── ISSUE_TEMPLATE/ │ +└── workflows/ ├── CHANGELOG.md ├── CODE_OF_CONDUCT.md ├── +CONTRIBUTING.md # This file ├── GOVERNANCE.md ├── LICENSE ├── +MAINTAINERS.md ├── README.adoc ├── SECURITY.md ├── flake.guix # Guix +flake (Perimeter 1) └── justfile # Task runner (Perimeter 1) + +.... + +--- + +## How to Contribute + +### Reporting Bugs + +**Before reporting**: +1. Search existing issues +2. Check if it's already fixed in `main` +3. Determine which perimeter the bug affects + +**When reporting**: + +Use the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md) and include: + +- Clear, descriptive title +- Environment details (OS, versions, toolchain) +- Steps to reproduce +- Expected vs actual behaviour +- Logs, screenshots, or minimal reproduction + +### Suggesting Features + +**Before suggesting**: +1. Check the [roadmap](ROADMAP.md) if available +2. Search existing issues and discussions +3. Consider which perimeter the feature belongs to + +**When suggesting**: + +Use the [feature request template](.github/ISSUE_TEMPLATE/feature_request.md) and include: + +- Problem statement (what pain point does this solve?) +- Proposed solution +- Alternatives considered +- Which perimeter this affects + +### Your First Contribution + +Look for issues labelled: + +- [`good first issue`](https://github.com/hyperpolymath/docudactyl/labels/good%20first%20issue) — Simple Perimeter 3 tasks +- [`help wanted`](https://github.com/hyperpolymath/docudactyl/labels/help%20wanted) — Community help needed +- [`documentation`](https://github.com/hyperpolymath/docudactyl/labels/documentation) — Docs improvements +- [`perimeter-3`](https://github.com/hyperpolymath/docudactyl/labels/perimeter-3) — Community sandbox scope + +--- + +## Development Workflow + +### Branch Naming +.... + +docs/short-description # Documentation (P3) test/what-added # Test +additions (P3) feat/short-description # New features (P2) +fix/issue-number-description # Bug fixes (P2) refactor/what-changed # +Code improvements (P2) security/what-fixed # Security fixes (P1-2) + +.... + +### Commit Messages + +We follow [Conventional Commits](https://www.conventionalcommits.org/): +.... + +(): + +{empty}[optional body] + +{empty}[optional footer] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 0d47730..0000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,116 +0,0 @@ -# Clone the repository -git clone https://github.com/hyperpolymath/docudactyl.git -cd docudactyl - -# Using Nix (recommended for reproducibility) -nix develop - -# Or using toolbox/distrobox -toolbox create docudactyl-dev -toolbox enter docudactyl-dev -# Install dependencies manually - -# Verify setup -just check # or: cargo check / mix compile / etc. -just test # Run test suite -``` - -### Repository Structure -``` -docudactyl/ -├── src/ # Source code (Perimeter 1-2) -├── lib/ # Library code (Perimeter 1-2) -├── extensions/ # Extensions (Perimeter 2) -├── plugins/ # Plugins (Perimeter 2) -├── tools/ # Tooling (Perimeter 2) -├── docs/ # Documentation (Perimeter 3) -│ ├── architecture/ # ADRs, specs (Perimeter 2) -│ └── proposals/ # RFCs (Perimeter 3) -├── examples/ # Examples (Perimeter 3) -├── spec/ # Spec tests (Perimeter 3) -├── tests/ # Test suite (Perimeter 2-3) -├── .well-known/ # Protocol files (Perimeter 1-3) -├── .github/ # GitHub config (Perimeter 1) -│ ├── ISSUE_TEMPLATE/ -│ └── workflows/ -├── CHANGELOG.md -├── CODE_OF_CONDUCT.md -├── CONTRIBUTING.md # This file -├── GOVERNANCE.md -├── LICENSE -├── MAINTAINERS.md -├── README.adoc -├── SECURITY.md -├── flake.nix # Nix flake (Perimeter 1) -└── justfile # Task runner (Perimeter 1) -``` - ---- - -## How to Contribute - -### Reporting Bugs - -**Before reporting**: -1. Search existing issues -2. Check if it's already fixed in `main` -3. Determine which perimeter the bug affects - -**When reporting**: - -Use the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md) and include: - -- Clear, descriptive title -- Environment details (OS, versions, toolchain) -- Steps to reproduce -- Expected vs actual behaviour -- Logs, screenshots, or minimal reproduction - -### Suggesting Features - -**Before suggesting**: -1. Check the [roadmap](ROADMAP.md) if available -2. Search existing issues and discussions -3. Consider which perimeter the feature belongs to - -**When suggesting**: - -Use the [feature request template](.github/ISSUE_TEMPLATE/feature_request.md) and include: - -- Problem statement (what pain point does this solve?) -- Proposed solution -- Alternatives considered -- Which perimeter this affects - -### Your First Contribution - -Look for issues labelled: - -- [`good first issue`](https://github.com/hyperpolymath/docudactyl/labels/good%20first%20issue) — Simple Perimeter 3 tasks -- [`help wanted`](https://github.com/hyperpolymath/docudactyl/labels/help%20wanted) — Community help needed -- [`documentation`](https://github.com/hyperpolymath/docudactyl/labels/documentation) — Docs improvements -- [`perimeter-3`](https://github.com/hyperpolymath/docudactyl/labels/perimeter-3) — Community sandbox scope - ---- - -## Development Workflow - -### Branch Naming -``` -docs/short-description # Documentation (P3) -test/what-added # Test additions (P3) -feat/short-description # New features (P2) -fix/issue-number-description # Bug fixes (P2) -refactor/what-changed # Code improvements (P2) -security/what-fixed # Security fixes (P1-2) -``` - -### Commit Messages - -We follow [Conventional Commits](https://www.conventionalcommits.org/): -``` -(): - -[optional body] - -[optional footer] diff --git a/GOVERNANCE.adoc b/GOVERNANCE.adoc new file mode 100644 index 0000000..9b836fb --- /dev/null +++ b/GOVERNANCE.adoc @@ -0,0 +1,60 @@ +== Governance + +=== Overview + +This project is governed by the following principles and structures to +ensure transparent, inclusive, and effective decision-making. + +=== Roles and Responsibilities + +==== Maintainers + +Maintainers are responsible for: - Reviewing and merging pull requests - +Managing releases and versioning - Ensuring code quality and standards - +Triaging issues and bug reports - Community engagement and support + +==== Contributors + +Contributors are expected to: - Follow the code of conduct - Submit +well-documented pull requests - Write tests for new functionality - +Maintain existing tests - Update documentation as needed + +=== Decision Making + +==== Minor Changes + +* Can be made by any maintainer +* Include bug fixes, documentation updates, dependency updates + +==== Major Changes + +* Require discussion in issues or pull requests +* Include new features, architectural changes, API changes +* Need approval from at least 2 maintainers + +==== Breaking Changes + +* Require RFC (Request for Comments) process +* Need approval from majority of maintainers +* Must include migration guide + +=== Code of Conduct + +All participants are expected to follow our Code of Conduct. Violations +can be reported to the maintainers. + +=== Communication + +* *Issues*: For bug reports and feature requests +* *Discussions*: For questions and general discussion +* *Pull Requests*: For code contributions + +=== Licensing + +All contributions are made under the terms of the repository’s LICENSE +file. By submitting a pull request, you agree to license your +contributions accordingly. + +''''' + +_Last updated: 2026-07-18_ diff --git a/GOVERNANCE.md b/GOVERNANCE.md deleted file mode 100644 index e27364c..0000000 --- a/GOVERNANCE.md +++ /dev/null @@ -1,60 +0,0 @@ -# Governance - -## Overview - -This project is governed by the following principles and structures to ensure transparent, inclusive, and effective decision-making. - -## Roles and Responsibilities - -### Maintainers - -Maintainers are responsible for: -- Reviewing and merging pull requests -- Managing releases and versioning -- Ensuring code quality and standards -- Triaging issues and bug reports -- Community engagement and support - -### Contributors - -Contributors are expected to: -- Follow the code of conduct -- Submit well-documented pull requests -- Write tests for new functionality -- Maintain existing tests -- Update documentation as needed - -## Decision Making - -### Minor Changes -- Can be made by any maintainer -- Include bug fixes, documentation updates, dependency updates - -### Major Changes -- Require discussion in issues or pull requests -- Include new features, architectural changes, API changes -- Need approval from at least 2 maintainers - -### Breaking Changes -- Require RFC (Request for Comments) process -- Need approval from majority of maintainers -- Must include migration guide - -## Code of Conduct - -All participants are expected to follow our Code of Conduct. Violations can be reported to the maintainers. - -## Communication - -- **Issues**: For bug reports and feature requests -- **Discussions**: For questions and general discussion -- **Pull Requests**: For code contributions - -## Licensing - -All contributions are made under the terms of the repository's LICENSE file. -By submitting a pull request, you agree to license your contributions accordingly. - ---- - -*Last updated: 2026-07-18* diff --git a/Justfile b/Justfile index e543e05..d27801e 100644 --- a/Justfile +++ b/Justfile @@ -361,8 +361,8 @@ validate-rsr: for f in .well-known/security.txt .well-known/ai.txt .well-known/humans.txt; do [ -f "$f" ] || MISSING="$MISSING $f" done - if [ ! -f "guix.scm" ] && [ ! -f ".guix-channel" ] && [ ! -f "flake.nix" ]; then - MISSING="$MISSING guix.scm/flake.nix" + if [ ! -f "guix.scm" ] && [ ! -f ".guix-channel" ] && [ ! -f "flake.guix" ]; then + MISSING="$MISSING guix.scm/flake.guix" fi if [ -n "$MISSING" ]; then echo "MISSING:$MISSING" @@ -398,7 +398,7 @@ state-phase: @grep -oP '\(phase\s+\.\s+\K[^)]+' STATE.scm 2>/dev/null | head -1 || echo "unknown" # ═══════════════════════════════════════════════════════════════════════════════ -# GUIX & NIX +# GUIX & GUIX # ═══════════════════════════════════════════════════════════════════════════════ # Enter Guix development shell (primary) @@ -409,9 +409,9 @@ guix-shell: guix-build: guix build -f guix.scm -# Enter Nix development shell (fallback) -nix-shell: - @if [ -f "flake.nix" ]; then nix develop; else echo "No flake.nix"; fi +# Enter Guix development shell (fallback) +guix-shell: + @if [ -f "flake.guix" ]; then guix develop; else echo "No flake.guix"; fi # ═══════════════════════════════════════════════════════════════════════════════ # HYBRID AUTOMATION diff --git a/REQUIRES_INITIALISATION.adoc b/REQUIRES_INITIALISATION.adoc new file mode 100644 index 0000000..8c93317 --- /dev/null +++ b/REQUIRES_INITIALISATION.adoc @@ -0,0 +1,112 @@ +== REQUIRES INITIALISATION + +*This repository is not finished being set up.* 8 substitution token(s) +across 2 file(s) still have no value. + +=== Why this is not already done + +This repo was created from `+hyperpolymath/rsr-template-repo+`. The mint +(`+just repo-init+`) fills every token that has a single mechanical +answer — owner, repo, author, dates, licence, branch — and it has done +so here. + +The tokens below are the ones it _deliberately cannot_ answer. They need +a decision or a fact that exists only in your head: what this project is +for, what command builds it, which port the service listens on, whether +a PGP key is held at all. The template’s own token vocabulary says as +much — you cannot sensibly answer "`required invariants`" in a +thirty-second bootstrap. + +They were left *visibly unfilled on purpose*. The alternatives were both +worse: inventing plausible values would put confident falsehoods into a +security policy and an architecture document, and silently deleting the +sections would hide the fact that a decision is owed. A visible gap is +honest; a fabricated answer is not. + +=== Do not delete this file until every item below is resolved + +This file is the only marker that the work is outstanding. Deleting it +early does not finish the setup, it just conceals it — and the next +person or agent to arrive will reasonably assume the repo is complete. + +* *If you are a person:* delete this file yourself once the last item is +done. +* *If you are an agent:* resolve what you legitimately can, leave the +rest, and delete this file only when no token below remains anywhere in +the tree. Do not delete it to make a gate go green. + +Re-running the estate top-up tool will remove this file automatically +once nothing is outstanding, so the safest way to finish is to fix the +tokens and let the check confirm it. + +=== What is needed, and where it goes + +==== `+{{CONSUMER1}}+` + +A downstream repo that consumes this one. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{CONSUMER2}}+` + +A second downstream consumer. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{DEP1}}+` + +First named dependency, in .machine_readable/INTENT.contractile. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{DEP2}}+` + +Second named dependency, in .machine_readable/INTENT.contractile. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{MONOREPO_OR_STANDALONE}}+` + +Literally '`monorepo`' or '`standalone`'. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{ONE_PARAGRAPH_ANTI_PURPOSE}}+` + +A paragraph on what this deliberately is NOT for. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{ONE_PARAGRAPH_PURPOSE}}+` + +A paragraph on what this is for. + +Appears in: + +* `+.machine_readable/INTENT.contractile+` + +==== `+{{PROJECT_UNIQUE_STRENGTH}}+` + +What this does that its alternatives do not. + +Appears in: + +* `+.machine_readable/agent_instructions/methodology.a2ml+` + +''''' + +Generated by the estate top-up pass. Rationale and the governing rulings +are in `+hyperpolymath/standards+`; the token vocabulary is +`+.machine_readable/ai/PLACEHOLDERS.adoc+` in `+rsr-template-repo+`. diff --git a/REQUIRES_INITIALISATION.md b/REQUIRES_INITIALISATION.md deleted file mode 100644 index b5a1dbf..0000000 --- a/REQUIRES_INITIALISATION.md +++ /dev/null @@ -1,110 +0,0 @@ - - -# REQUIRES INITIALISATION - -**This repository is not finished being set up.** 8 substitution token(s) across 2 file(s) still have no value. - -## Why this is not already done - -This repo was created from `hyperpolymath/rsr-template-repo`. The mint -(`just repo-init`) fills every token that has a single mechanical answer — -owner, repo, author, dates, licence, branch — and it has done so here. - -The tokens below are the ones it *deliberately cannot* answer. They need a -decision or a fact that exists only in your head: what this project is for, -what command builds it, which port the service listens on, whether a PGP key -is held at all. The template's own token vocabulary says as much — you cannot -sensibly answer "required invariants" in a thirty-second bootstrap. - -They were left **visibly unfilled on purpose**. The alternatives were both -worse: inventing plausible values would put confident falsehoods into a -security policy and an architecture document, and silently deleting the -sections would hide the fact that a decision is owed. A visible gap is -honest; a fabricated answer is not. - -## Do not delete this file until every item below is resolved - -This file is the only marker that the work is outstanding. Deleting it early -does not finish the setup, it just conceals it — and the next person or agent -to arrive will reasonably assume the repo is complete. - -- **If you are a person:** delete this file yourself once the last item is done. -- **If you are an agent:** resolve what you legitimately can, leave the rest, - and delete this file only when no token below remains anywhere in the tree. - Do not delete it to make a gate go green. - -Re-running the estate top-up tool will remove this file automatically once -nothing is outstanding, so the safest way to finish is to fix the tokens and -let the check confirm it. - -## What is needed, and where it goes - -### `{{CONSUMER1}}` - -A downstream repo that consumes this one. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{CONSUMER2}}` - -A second downstream consumer. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{DEP1}}` - -First named dependency, in .machine_readable/INTENT.contractile. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{DEP2}}` - -Second named dependency, in .machine_readable/INTENT.contractile. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{MONOREPO_OR_STANDALONE}}` - -Literally 'monorepo' or 'standalone'. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{ONE_PARAGRAPH_ANTI_PURPOSE}}` - -A paragraph on what this deliberately is NOT for. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{ONE_PARAGRAPH_PURPOSE}}` - -A paragraph on what this is for. - -Appears in: - -- `.machine_readable/INTENT.contractile` - -### `{{PROJECT_UNIQUE_STRENGTH}}` - -What this does that its alternatives do not. - -Appears in: - -- `.machine_readable/agent_instructions/methodology.a2ml` - ---- - -Generated by the estate top-up pass. Rationale and the governing rulings are -in `hyperpolymath/standards`; the token vocabulary is -`.machine_readable/ai/PLACEHOLDERS.adoc` in `rsr-template-repo`. diff --git a/SECURITY.adoc b/SECURITY.adoc new file mode 100644 index 0000000..c44ef4b --- /dev/null +++ b/SECURITY.adoc @@ -0,0 +1,294 @@ +== Security Policy + +We take security seriously. We appreciate your efforts to responsibly +disclose vulnerabilities and will make every effort to acknowledge your +contributions. + +=== Table of Contents + +* link:#reporting-a-vulnerability[Reporting a Vulnerability] +* link:#what-to-include[What to Include] +* link:#response-timeline[Response Timeline] +* link:#disclosure-policy[Disclosure Policy] +* link:#scope[Scope] +* link:#safe-harbour[Safe Harbour] +* link:#recognition[Recognition] +* link:#security-updates[Security Updates] +* link:#security-best-practices[Security Best Practices] + +''''' + +=== Reporting a Vulnerability + +==== Preferred Method: GitHub Security Advisories + +The preferred method for reporting security vulnerabilities is through +GitHub’s Security Advisory feature: + +[arabic] +. Navigate to +https://github.com/hyperpolymath/docudactyl/security/advisories/new[Report +a Vulnerability] +. Click *"`Report a vulnerability`"* +. Complete the form with as much detail as possible +. Submit — we’ll receive a private notification + +This method ensures: + +* End-to-end encryption of your report +* Private discussion space for collaboration +* Coordinated disclosure tooling +* Automatic credit when the advisory is published + +==== Alternative: Email + +If you cannot use GitHub Security Advisories, you may email us directly: + +[cols=",",] +|=== +|*Email* |j.d.a.jewell@open.ac.uk +|=== + +____ +*Important:* Do not report security vulnerabilities through public +GitHub issues, pull requests, discussions, or social media. +____ + +''''' + +=== What to Include + +A good vulnerability report helps us understand and reproduce the issue +quickly. + +==== Required Information + +* *Description*: Clear explanation of the vulnerability +* *Impact*: What an attacker could achieve (confidentiality, integrity, +availability) +* *Affected versions*: Which versions/commits are affected +* *Reproduction steps*: Detailed steps to reproduce the issue + +==== Helpful Additional Information + +* *Proof of concept*: Code, scripts, or screenshots demonstrating the +vulnerability +* *Attack scenario*: Realistic attack scenario showing exploitability +* *CVSS score*: Your assessment of severity (use +https://www.first.org/cvss/calculator/3.1[CVSS 3.1 Calculator]) +* *CWE ID*: Common Weakness Enumeration identifier if known +* *Suggested fix*: If you have ideas for remediation +* *References*: Links to related vulnerabilities, research, or +advisories + +''''' + +=== Response Timeline + +We commit to the following response times: + +[width="100%",cols="24%,35%,41%",options="header",] +|=== +|Stage |Timeframe |Description +|*Initial Response* |48 hours |We acknowledge receipt and confirm we’re +investigating + +|*Triage* |7 days |We assess severity, confirm the vulnerability, and +estimate timeline + +|*Status Update* |Every 7 days |Regular updates on remediation progress + +|*Resolution* |90 days |Target for fix development and release (complex +issues may take longer) + +|*Disclosure* |90 days |Public disclosure after fix is available +(coordinated with you) +|=== + +____ +*Note:* These are targets, not guarantees. Complex vulnerabilities may +require more time. We’ll communicate openly about any delays. +____ + +''''' + +=== Disclosure Policy + +We follow *coordinated disclosure* (also known as responsible +disclosure): + +[arabic] +. *You report* the vulnerability privately +. *We acknowledge* and begin investigation +. *We develop* a fix and prepare a release +. *We coordinate* disclosure timing with you +. *We publish* security advisory and fix simultaneously +. *You may publish* your research after disclosure + +==== Our Commitments + +* We will not take legal action against researchers who follow this +policy +* We will work with you to understand and resolve the issue +* We will credit you in the security advisory (unless you prefer +anonymity) +* We will notify you before public disclosure +* We will publish advisories with sufficient detail for users to assess +risk + +==== Your Commitments + +* Report vulnerabilities promptly after discovery +* Give us reasonable time to address the issue before disclosure +* Do not access, modify, or delete data beyond what’s necessary to +demonstrate the vulnerability +* Do not degrade service availability (no DoS testing on production) +* Do not share vulnerability details with others until coordinated +disclosure + +''''' + +=== Scope + +==== In Scope + +The following are within scope for security research: + +* This repository (`+hyperpolymath/docudactyl+`) and all its code +* The Zig FFI layer (`+ffi/zig/+`) — memory safety, buffer handling, +null-pointer dereferences +* The C ABI boundary — struct layout correctness, pointer handling +* Chapel HPC orchestration — fault isolation, input validation +* Container images built from `+deploy/Containerfile+` +* Official releases and packages published from this repository +* Dependencies (report here, we’ll coordinate with upstream) + +==== Out of Scope + +The following are *not* in scope: + +* Third-party C libraries (Poppler, Tesseract, FFmpeg, etc.) — report +directly to them +* Social engineering attacks against maintainers +* Physical security +* Denial of service attacks against production infrastructure +* Issues already reported or publicly known +* Theoretical vulnerabilities without proof of concept + +==== Qualifying Vulnerabilities + +We’re particularly interested in: + +* Memory safety issues in the Zig FFI layer (buffer overflows, +use-after-free, etc.) +* ABI mismatches between Idris2 proofs and actual struct layouts +* Command injection via manifest file paths or configuration +* Unsafe dlopen/dlsym handling (ONNX Runtime, PaddleOCR, CUDA) +* Path traversal in document output paths +* Information disclosure through error messages +* Cryptographic weaknesses in SHA-256 hardware acceleration paths + +''''' + +=== Safe Harbour + +We support security research conducted in good faith. + +==== Our Promise + +If you conduct security research in accordance with this policy: + +* We will not initiate legal action against you +* We will not report your activity to law enforcement +* We will work with you in good faith to resolve issues +* We consider your research authorised under the Computer Fraud and +Abuse Act (CFAA), UK Computer Misuse Act, and similar laws +* We waive any potential claim against you for circumvention of security +controls + +==== Good Faith Requirements + +To qualify for safe harbour, you must: + +* Comply with this security policy +* Report vulnerabilities promptly +* Avoid privacy violations (do not access others’ data) +* Avoid service degradation (no destructive testing) +* Not exploit vulnerabilities beyond proof-of-concept + +''''' + +=== Recognition + +Researchers who report valid vulnerabilities will be acknowledged in our +security advisories (unless they prefer anonymity). + +''''' + +=== Security Updates + +==== Receiving Updates + +To stay informed about security updates: + +* *Watch this repository*: Click "`Watch`" -> "`Custom`" -> Select +"`Security alerts`" +* *GitHub Security Advisories*: Published at +https://github.com/hyperpolymath/docudactyl/security/advisories[Security +Advisories] + +==== Supported Versions + +[cols=",,",options="header",] +|=== +|Version |Supported |Notes +|`+main+` branch |Yes |Latest development +|Latest release |Yes |Current stable +|Older versions |No |Please upgrade +|=== + +''''' + +=== Security Best Practices + +When using Docudactyl, we recommend: + +==== General + +* Keep dependencies up to date +* Use the latest stable release +* Validate manifest files before processing untrusted inputs +* Run in containers with minimal privileges +* Review Slurm job scripts before submission + +==== For Contributors + +* Never commit secrets, credentials, or API keys +* No `+@panic+` in Zig release builds — return error codes via C ABI +* No `+believe_me+`, `+assert_total+`, or `+Admitted+` in Idris2 proofs +* No `+unsafe+` blocks in Zig without documented safety justification +* Review dependencies before adding them +* Run `+just test-hpc+` before pushing + +''''' + +=== Contact + +[width="100%",cols="50%,50%",options="header",] +|=== +|Purpose |Contact +|*Security issues* +|https://github.com/hyperpolymath/docudactyl/security/advisories/new[Report +via GitHub] or j.d.a.jewell@open.ac.uk + +|*General questions* +|https://github.com/hyperpolymath/docudactyl/issues[GitHub Issues] +|=== + +''''' + +_Thank you for helping keep Docudactyl and its users safe._ + +''''' + +Last updated: 2026 | Policy version: 1.0.0 diff --git a/SECURITY.md b/SECURITY.md deleted file mode 100644 index 283e7cf..0000000 --- a/SECURITY.md +++ /dev/null @@ -1,242 +0,0 @@ -# Security Policy - -We take security seriously. We appreciate your efforts to responsibly disclose vulnerabilities and will make every effort to acknowledge your contributions. - -## Table of Contents - -- [Reporting a Vulnerability](#reporting-a-vulnerability) -- [What to Include](#what-to-include) -- [Response Timeline](#response-timeline) -- [Disclosure Policy](#disclosure-policy) -- [Scope](#scope) -- [Safe Harbour](#safe-harbour) -- [Recognition](#recognition) -- [Security Updates](#security-updates) -- [Security Best Practices](#security-best-practices) - ---- - -## Reporting a Vulnerability - -### Preferred Method: GitHub Security Advisories - -The preferred method for reporting security vulnerabilities is through GitHub's Security Advisory feature: - -1. Navigate to [Report a Vulnerability](https://github.com/hyperpolymath/docudactyl/security/advisories/new) -2. Click **"Report a vulnerability"** -3. Complete the form with as much detail as possible -4. Submit — we'll receive a private notification - -This method ensures: - -- End-to-end encryption of your report -- Private discussion space for collaboration -- Coordinated disclosure tooling -- Automatic credit when the advisory is published - -### Alternative: Email - -If you cannot use GitHub Security Advisories, you may email us directly: - -| | | -|---|---| -| **Email** | j.d.a.jewell@open.ac.uk | - -> **Important:** Do not report security vulnerabilities through public GitHub issues, pull requests, discussions, or social media. - ---- - -## What to Include - -A good vulnerability report helps us understand and reproduce the issue quickly. - -### Required Information - -- **Description**: Clear explanation of the vulnerability -- **Impact**: What an attacker could achieve (confidentiality, integrity, availability) -- **Affected versions**: Which versions/commits are affected -- **Reproduction steps**: Detailed steps to reproduce the issue - -### Helpful Additional Information - -- **Proof of concept**: Code, scripts, or screenshots demonstrating the vulnerability -- **Attack scenario**: Realistic attack scenario showing exploitability -- **CVSS score**: Your assessment of severity (use [CVSS 3.1 Calculator](https://www.first.org/cvss/calculator/3.1)) -- **CWE ID**: Common Weakness Enumeration identifier if known -- **Suggested fix**: If you have ideas for remediation -- **References**: Links to related vulnerabilities, research, or advisories - ---- - -## Response Timeline - -We commit to the following response times: - -| Stage | Timeframe | Description | -|-------|-----------|-------------| -| **Initial Response** | 48 hours | We acknowledge receipt and confirm we're investigating | -| **Triage** | 7 days | We assess severity, confirm the vulnerability, and estimate timeline | -| **Status Update** | Every 7 days | Regular updates on remediation progress | -| **Resolution** | 90 days | Target for fix development and release (complex issues may take longer) | -| **Disclosure** | 90 days | Public disclosure after fix is available (coordinated with you) | - -> **Note:** These are targets, not guarantees. Complex vulnerabilities may require more time. We'll communicate openly about any delays. - ---- - -## Disclosure Policy - -We follow **coordinated disclosure** (also known as responsible disclosure): - -1. **You report** the vulnerability privately -2. **We acknowledge** and begin investigation -3. **We develop** a fix and prepare a release -4. **We coordinate** disclosure timing with you -5. **We publish** security advisory and fix simultaneously -6. **You may publish** your research after disclosure - -### Our Commitments - -- We will not take legal action against researchers who follow this policy -- We will work with you to understand and resolve the issue -- We will credit you in the security advisory (unless you prefer anonymity) -- We will notify you before public disclosure -- We will publish advisories with sufficient detail for users to assess risk - -### Your Commitments - -- Report vulnerabilities promptly after discovery -- Give us reasonable time to address the issue before disclosure -- Do not access, modify, or delete data beyond what's necessary to demonstrate the vulnerability -- Do not degrade service availability (no DoS testing on production) -- Do not share vulnerability details with others until coordinated disclosure - ---- - -## Scope - -### In Scope - -The following are within scope for security research: - -- This repository (`hyperpolymath/docudactyl`) and all its code -- The Zig FFI layer (`ffi/zig/`) — memory safety, buffer handling, null-pointer dereferences -- The C ABI boundary — struct layout correctness, pointer handling -- Chapel HPC orchestration — fault isolation, input validation -- Container images built from `deploy/Containerfile` -- Official releases and packages published from this repository -- Dependencies (report here, we'll coordinate with upstream) - -### Out of Scope - -The following are **not** in scope: - -- Third-party C libraries (Poppler, Tesseract, FFmpeg, etc.) — report directly to them -- Social engineering attacks against maintainers -- Physical security -- Denial of service attacks against production infrastructure -- Issues already reported or publicly known -- Theoretical vulnerabilities without proof of concept - -### Qualifying Vulnerabilities - -We're particularly interested in: - -- Memory safety issues in the Zig FFI layer (buffer overflows, use-after-free, etc.) -- ABI mismatches between Idris2 proofs and actual struct layouts -- Command injection via manifest file paths or configuration -- Unsafe dlopen/dlsym handling (ONNX Runtime, PaddleOCR, CUDA) -- Path traversal in document output paths -- Information disclosure through error messages -- Cryptographic weaknesses in SHA-256 hardware acceleration paths - ---- - -## Safe Harbour - -We support security research conducted in good faith. - -### Our Promise - -If you conduct security research in accordance with this policy: - -- We will not initiate legal action against you -- We will not report your activity to law enforcement -- We will work with you in good faith to resolve issues -- We consider your research authorised under the Computer Fraud and Abuse Act (CFAA), UK Computer Misuse Act, and similar laws -- We waive any potential claim against you for circumvention of security controls - -### Good Faith Requirements - -To qualify for safe harbour, you must: - -- Comply with this security policy -- Report vulnerabilities promptly -- Avoid privacy violations (do not access others' data) -- Avoid service degradation (no destructive testing) -- Not exploit vulnerabilities beyond proof-of-concept - ---- - -## Recognition - -Researchers who report valid vulnerabilities will be acknowledged in our security advisories (unless they prefer anonymity). - ---- - -## Security Updates - -### Receiving Updates - -To stay informed about security updates: - -- **Watch this repository**: Click "Watch" -> "Custom" -> Select "Security alerts" -- **GitHub Security Advisories**: Published at [Security Advisories](https://github.com/hyperpolymath/docudactyl/security/advisories) - -### Supported Versions - -| Version | Supported | Notes | -|---------|-----------|-------| -| `main` branch | Yes | Latest development | -| Latest release | Yes | Current stable | -| Older versions | No | Please upgrade | - ---- - -## Security Best Practices - -When using Docudactyl, we recommend: - -### General - -- Keep dependencies up to date -- Use the latest stable release -- Validate manifest files before processing untrusted inputs -- Run in containers with minimal privileges -- Review Slurm job scripts before submission - -### For Contributors - -- Never commit secrets, credentials, or API keys -- No `@panic` in Zig release builds — return error codes via C ABI -- No `believe_me`, `assert_total`, or `Admitted` in Idris2 proofs -- No `unsafe` blocks in Zig without documented safety justification -- Review dependencies before adding them -- Run `just test-hpc` before pushing - ---- - -## Contact - -| Purpose | Contact | -|---------|---------| -| **Security issues** | [Report via GitHub](https://github.com/hyperpolymath/docudactyl/security/advisories/new) or j.d.a.jewell@open.ac.uk | -| **General questions** | [GitHub Issues](https://github.com/hyperpolymath/docudactyl/issues) | - ---- - -*Thank you for helping keep Docudactyl and its users safe.* - ---- - -Last updated: 2026 | Policy version: 1.0.0 diff --git a/TEST-NEEDS.adoc b/TEST-NEEDS.adoc new file mode 100644 index 0000000..0caa77a --- /dev/null +++ b/TEST-NEEDS.adoc @@ -0,0 +1,32 @@ +== TEST-NEEDS.md — docudactyl + +=== CRG Grade: C — ACHIEVED 2026-04-04 + +=== Current Test State + +[cols=",,",options="header",] +|=== +|Category |Count |Notes +|Test directories |1 |Location(s): /tests +|CI workflows |19 |Running tests on GitHub Actions +|Tests |Present |Configured in CI workflows +|=== + +=== What’s Covered + +* [x] Tests present and running +* [x] CI integration active + +=== Still Missing (for CRG B+) + +* [ ] Code coverage reports (codecov integration) +* [ ] Detailed test documentation in CONTRIBUTING.md +* [ ] Integration tests beyond unit tests +* [ ] Performance benchmarking suite + +=== Run Tests + +[source,bash] +---- +(check Makefile/justfile/package.json for test command) +---- diff --git a/TEST-NEEDS.md b/TEST-NEEDS.md deleted file mode 100644 index ba3b52a..0000000 --- a/TEST-NEEDS.md +++ /dev/null @@ -1,29 +0,0 @@ -# TEST-NEEDS.md — docudactyl - -## CRG Grade: C — ACHIEVED 2026-04-04 - -## Current Test State - -| Category | Count | Notes | -|----------|-------|-------| -| Test directories | 1 | Location(s): /tests | -| CI workflows | 19 | Running tests on GitHub Actions | -| Tests | Present | Configured in CI workflows | - -## What's Covered - -- [x] Tests present and running -- [x] CI integration active - -## Still Missing (for CRG B+) - -- [ ] Code coverage reports (codecov integration) -- [ ] Detailed test documentation in CONTRIBUTING.md -- [ ] Integration tests beyond unit tests -- [ ] Performance benchmarking suite - -## Run Tests - -```bash -(check Makefile/justfile/package.json for test command) -``` diff --git a/TOPOLOGY.md b/TOPOLOGY.adoc similarity index 72% rename from TOPOLOGY.md rename to TOPOLOGY.adoc index 4d5290b..fdce3d6 100644 --- a/TOPOLOGY.md +++ b/TOPOLOGY.adoc @@ -1,8 +1,8 @@ -# TOPOLOGY.md — Docudactyl HPC Architecture +== TOPOLOGY.md — Docudactyl HPC Architecture -## System Architecture +=== System Architecture -``` +.... ┌──────────────────────────────────────────────────────────────────────────────┐ │ Docudactyl HPC Engine │ │ │ @@ -115,72 +115,79 @@ Cache Architecture: Offline: output/*.json ──→ OCaml docudactyl-scm ──→ *.scm (S-expressions) -``` - -## Completion Dashboard - -| Component | Status | Progress | -|----------------------------|-------------|------------------------------| -| Chapel HPC Engine | Complete | `██████████` 100% | -| Zig FFI Dispatcher | Complete | `██████████` 100% | -| Processing Stages (20) | Complete | `██████████` 100% | -| Cap'n Proto Output | Complete | `██████████` 100% | -| NDJSON Manifests | Complete | `██████████` 100% | -| Preprocessing Conduit | Complete | `██████████` 100% | -| L1 Cache (LMDB) | Complete | `██████████` 100% | -| L2 Cache (Dragonfly) | Complete | `██████████` 100% | -| I/O Prefetcher (io_uring) | Complete | `██████████` 100% | -| GPU OCR Coprocessor | Integrated | `██████████` 100% | -| ML Inference (ONNX) | Integrated | `██████████` 100% | -| Hardware Crypto (SHA-NI) | Complete | `██████████` 100% | -| Idris2 ABI Proofs | Complete | `██████████` 100% | -| C Header (interop) | Complete | `██████████` 100% | -| Checkpoint & Resume | Complete | `██████████` 100% | -| OCaml Scheme Emitter | Stable | `██████████` 100% | -| Ada TUI | Stable | `██████████` 100% | -| Julia (legacy) | Deprecated | `██████████` 100% (frozen) | -| Multi-Locale Testing | Not Started | `░░░░░░░░░░` 0% | - -**Overall: `█████████░` 95%** (multi-locale testing requires cluster access) - -## Key Dependencies - -| Dependency | Version | Purpose | Link | -|------------------|----------|--------------------------------------|-----------| -| Chapel | 2.7.0 | HPC orchestration (N locales) | build | -| Zig | 0.15.2 | FFI wrapper, zero runtime cost | build | -| Idris2 | 0.8.0 | ABI formal proofs | build | -| Poppler | 25.07.0 | PDF text + metadata extraction | link | -| Tesseract | 5.5.2 | OCR (image → text) | link | -| Leptonica | 1.87.0 | Image I/O for Tesseract | link | -| FFmpeg | 7.1.2 | Audio/video metadata | link | -| libxml2 | 2.12.10 | EPUB/XHTML parsing | link | -| GDAL | 3.11.5 | Geospatial data extraction | link | -| libvips | 8.17.3 | Image metadata | link | -| ONNX Runtime | 1.20+ | ML inference (NER, Whisper, etc.) | dlopen | -| PaddleOCR | 3.0+ | GPU OCR (CUDA/TensorRT) | dlopen | -| LMDB | 0.9.33 | L1 result cache (per-locale) | dlopen | -| Dragonfly | 1.25+ | L2 shared cache (RESP2) | TCP | -| OCaml | 5.4.1 | Offline Scheme transformer | separate | -| Ada/GNAT | — | TUI (terminal inspector) | separate | - -## Scale Targets - -| Metric | Local Test (verified) | Cluster Target (estimated) | -|----------------|----------------------|-----------------------------| -| Documents | 2,105 | 170,000,000 | -| Locales | 1 | 64–512 | -| Throughput | 19.35 docs/s | ~1,200–10,000 docs/s | -| Failure rate | 0.0% | < 5.0% | -| Output size | ~1 MB | ~1.7 TB | -| Memory/locale | ~100 MB | ~4–8 GB | -| Cold run | — | ~3.7h (256 nodes + GPU) | -| Warm run | — | ~4 min (256 nodes, cached) | -| Incremental | — | ~8 min (5% new, 256 nodes) | - -## Zig Module Architecture - -``` +.... + +=== Completion Dashboard + +[width="100%",cols="40%,18%,42%",options="header",] +|=== +|Component |Status |Progress +|Chapel HPC Engine |Complete |`+██████████+` 100% +|Zig FFI Dispatcher |Complete |`+██████████+` 100% +|Processing Stages (20) |Complete |`+██████████+` 100% +|Cap’n Proto Output |Complete |`+██████████+` 100% +|NDJSON Manifests |Complete |`+██████████+` 100% +|Preprocessing Conduit |Complete |`+██████████+` 100% +|L1 Cache (LMDB) |Complete |`+██████████+` 100% +|L2 Cache (Dragonfly) |Complete |`+██████████+` 100% +|I/O Prefetcher (io_uring) |Complete |`+██████████+` 100% +|GPU OCR Coprocessor |Integrated |`+██████████+` 100% +|ML Inference (ONNX) |Integrated |`+██████████+` 100% +|Hardware Crypto (SHA-NI) |Complete |`+██████████+` 100% +|Idris2 ABI Proofs |Complete |`+██████████+` 100% +|C Header (interop) |Complete |`+██████████+` 100% +|Checkpoint & Resume |Complete |`+██████████+` 100% +|OCaml Scheme Emitter |Stable |`+██████████+` 100% +|Ada TUI |Stable |`+██████████+` 100% +|Julia (legacy) |Deprecated |`+██████████+` 100% (frozen) +|Multi-Locale Testing |Not Started |`+░░░░░░░░░░+` 0% +|=== + +*Overall: `+█████████░+` 95%* (multi-locale testing requires cluster +access) + +=== Key Dependencies + +[width="100%",cols="25%,12%,49%,14%",options="header",] +|=== +|Dependency |Version |Purpose |Link +|Chapel |2.7.0 |HPC orchestration (N locales) |build +|Zig |0.15.2 |FFI wrapper, zero runtime cost |build +|Idris2 |0.8.0 |ABI formal proofs |build +|Poppler |25.07.0 |PDF text + metadata extraction |link +|Tesseract |5.5.2 |OCR (image → text) |link +|Leptonica |1.87.0 |Image I/O for Tesseract |link +|FFmpeg |7.1.2 |Audio/video metadata |link +|libxml2 |2.12.10 |EPUB/XHTML parsing |link +|GDAL |3.11.5 |Geospatial data extraction |link +|libvips |8.17.3 |Image metadata |link +|ONNX Runtime |1.20+ |ML inference (NER, Whisper, etc.) |dlopen +|PaddleOCR |3.0+ |GPU OCR (CUDA/TensorRT) |dlopen +|LMDB |0.9.33 |L1 result cache (per-locale) |dlopen +|Dragonfly |1.25+ |L2 shared cache (RESP2) |TCP +|OCaml |5.4.1 |Offline Scheme transformer |separate +|Ada/GNAT |— |TUI (terminal inspector) |separate +|=== + +=== Scale Targets + +[cols=",,",options="header",] +|=== +|Metric |Local Test (verified) |Cluster Target (estimated) +|Documents |2,105 |170,000,000 +|Locales |1 |64–512 +|Throughput |19.35 docs/s |~1,200–10,000 docs/s +|Failure rate |0.0% |< 5.0% +|Output size |~1 MB |~1.7 TB +|Memory/locale |~100 MB |~4–8 GB +|Cold run |— |~3.7h (256 nodes + GPU) +|Warm run |— |~4 min (256 nodes, cached) +|Incremental |— |~8 min (5% new, 256 nodes) +|=== + +=== Zig Module Architecture + +.... docudactyl_ffi.zig (root — C-ABI exports, format dispatch) ├── stages.zig (20 processing stages + Cap'n Proto output) ├── capnp.zig (Cap'n Proto single-segment message builder) @@ -191,4 +198,4 @@ docudactyl_ffi.zig (root — C-ABI exports, format dispatch) ├── gpu_ocr.zig (batched GPU OCR — PaddleOCR/Tesseract CUDA) ├── hw_crypto.zig (SHA-NI/AVX2 detection + multi-buffer hash) └── ml_inference.zig (ONNX Runtime — 5 ML stages via dlopen) -``` +.... diff --git a/audits/audit-ffi-2026-05-26.adoc b/audits/audit-ffi-2026-05-26.adoc new file mode 100644 index 0000000..beb8746 --- /dev/null +++ b/audits/audit-ffi-2026-05-26.adoc @@ -0,0 +1,18 @@ +== Audit: FFI unsafe blocks (docudactyl) + +*Auditor*: Jonathan D.A. Jewell + +*Date*: 2026-05-26 + +*Scope*: panic-attack assail Critical/High UnsafeCode/UnsafeFFI findings +under `+ffi/zig/src/+`. + +*Registry*: `+audits/assail-classifications.a2ml+`. + +=== Rationale + +Zig FFI layer (dragonfly/cache bindings). Each unsafe block is at the +Zig→C ABI boundary required by the host language to call extern +functions. + +Classification is scoped to `+ffi/zig/src/+`; any unsafe block outside +that root remains visible. + +Refs hyperpolymath/panic-attack#32. diff --git a/audits/audit-ffi-2026-05-26.md b/audits/audit-ffi-2026-05-26.md deleted file mode 100644 index ec0df53..0000000 --- a/audits/audit-ffi-2026-05-26.md +++ /dev/null @@ -1,19 +0,0 @@ - - -# Audit: FFI unsafe blocks (docudactyl) - -**Auditor**: Jonathan D.A. Jewell -**Date**: 2026-05-26 -**Scope**: panic-attack assail Critical/High UnsafeCode/UnsafeFFI findings under `ffi/zig/src/`. -**Registry**: `audits/assail-classifications.a2ml`. - -## Rationale - -Zig FFI layer (dragonfly/cache bindings). Each unsafe block is at the Zig→C ABI boundary required by the host language to call extern functions. - -Classification is scoped to `ffi/zig/src/`; any unsafe block outside that root remains visible. - -Refs hyperpolymath/panic-attack#32. diff --git a/docs/EPSTEIN-EXTRACTION-TESTS.adoc b/docs/EPSTEIN-EXTRACTION-TESTS.adoc new file mode 100644 index 0000000..fbb3d50 --- /dev/null +++ b/docs/EPSTEIN-EXTRACTION-TESTS.adoc @@ -0,0 +1,172 @@ +== Epstein Files — Docudactyl Extraction Tests & Benchmarks + +== + +== SPDX-License-Identifier: CC-BY-SA-4.0 + +== Author: Jonathan D.A. Jewell + +== Created: 2026-03-13 + +== + +== Extracted from the master pathway: bofig/docs/EPSTEIN-FILES-WORK-PATHWAY.md + +== This file contains only the Docudactyl-specific phases (1.1–1.7). + +=== Dataset Characteristics + +[width="100%",cols="62%,38%",options="header",] +|=== +|Attribute |Value +|Total files |~3,200,000 + +|Total size |~218 GB + +|Data sets |12 (flight logs, court filings, depositions, financial +records, photos, communications, …) + +|Named entities (est.) |23,000+ unique persons, orgs, locations + +|Primary format |Scanned PDF (96 DPI, many poor quality) + +|Secondary formats |TIFF, JPEG, DOCX, XLS, email (EML/PST) + +|Redaction style |Overlay-only (text stream often intact) + +|Financial transactions (est.) |16,000+ + +|Languages |English (primary), French, some Spanish + +|Time span |1990s–2024 +|=== + +''''' + +=== Phase 1: Docudactyl Extraction Pipeline (Weeks 1–6) + +==== Step 1.1: Core OCR + Text Extraction (DONE — existing stages 0-8) + +Already implemented in `+stages.zig+`: - Language detection, +readability, keywords, citations - OCR confidence, perceptual hash, TOC +extraction - Multi-language OCR, subtitle extraction + +*Tests:* - [x] Unit test: each stage function with known input produces +expected Cap’n Proto output - [x] Integration test: 10-document +mini-corpus end-to-end - [ ] Benchmark: single-node throughput for +scanned PDFs (target: 2 docs/sec on 8-core) + +==== Step 1.2: Redaction Detection (DONE — bit 20, `+stageRedactionDetect+`) + +Scans Poppler annotations for Type 12 (POPPLER_ANNOT_REDACT), checks if +text is extractable under overlay-only redactions. + +*Tests:* - [ ] T-RED-1: Synthetic PDF with 5 /Redact annotations → +count=5, status="`redacted`" - [ ] T-RED-2: PDF with black fill +rectangles but no /Redact annots → status="`clean`" (future: heuristic +upgrade) - [ ] T-RED-3: PDF with overlay redaction + recoverable text → +recoverable_count > 0 - [ ] T-RED-4: Non-PDF input (JPEG) → +status="`not_applicable`" - [ ] T-RED-5: Corrupt/unreadable PDF → +status="`error`" + +*Benchmarks:* - [ ] B-RED-1: 1000 PDFs (mixed redacted/clean) — target: +<500ms per document - [ ] B-RED-2: Memory usage during annotation scan — +target: <50MB peak per document + +==== Step 1.3: Financial Entity Extraction (DONE — bit 21, `+stageFinancialExtract+`) + +Pattern-based detection of currency symbols ($, £, €), ISO codes (USD, +GBP, EUR, CHF, JPY, CAD), account-like digit sequences (8-20 digits). + +*Tests:* - [ ] T-FIN-1: Text "`$1,234.56 paid to account 12345678`" → +amounts=1, accounts=1 - [ ] T-FIN-2: Text "`USD 50,000 transferred`" → +amounts=1 - [ ] T-FIN-3: Text "`£2.3 million to HSBC account +1234-5678-9012`" → amounts=1, accounts=1 - [ ] T-FIN-4: Phone numbers +should NOT match as accounts (7-digit filter) - [ ] T-FIN-5: Empty text +→ status="`none_found`", amounts=0, accounts=0 - [ ] T-FIN-6: Mixed +currencies in single document → correct total count + +*Benchmarks:* - [ ] B-FIN-1: 10MB text document scan — target: <200ms - +[ ] B-FIN-2: Accuracy on annotated Epstein financial records sample (50 +docs) — target: >80% recall + +==== Step 1.4: Legal NER (DONE — bit 22, `+stageLegalNer+`) + +Pattern-based detection of case citations ("`v.`"), docket numbers +("`No.`", "`Case`"), statute references ("`U.S.C.`", "`§`"). + +*Tests:* - [ ] T-LEG-1: "`Doe v. Epstein`" → case_citations=1 - [ ] +T-LEG-2: "`No. 08-cv-1234`" → docket_refs=1 - [ ] T-LEG-3: "`18 U.S.C. § +1591`" → statute_refs=1 (both U.S.C. and § counted) - [ ] T-LEG-4: +"`vs.`" variant → case_citations=1 - [ ] T-LEG-5: Real Epstein court +filing excerpt → realistic counts - [ ] T-LEG-6: Non-legal document +(flight log) → all counts = 0 + +*Benchmarks:* - [ ] B-LEG-1: 5MB legal document — target: <150ms - [ ] +B-LEG-2: Precision on annotated legal corpus (100 docs) — target: >75% + +==== Step 1.5: Speaker Identification (bit 23, ML dispatch) + +ML-based speaker diarization via ONNX Runtime. Dispatches to stage_id=5 +(speaker_id.onnx model). + +*Tests:* - [ ] T-SPK-1: With ML handle + model → status="`ok`", +speaker_count > 0 - [ ] T-SPK-2: Without ML handle → +status="`not_available`" - [ ] T-SPK-3: Non-audio input → graceful +fallback - [ ] T-SPK-4: Deposition audio with 2 speakers → +speaker_count=2 + +*Benchmarks:* - [ ] B-SPK-1: 30-minute deposition audio — target: <60s +inference - [ ] B-SPK-2: Memory usage during diarization — target: <2GB + +==== Step 1.6: STAGE_INVESTIGATIVE Preset Validation + +The `+STAGE_INVESTIGATIVE+` preset combines all investigative stages. + +*Tests:* - [ ] T-INV-1: STAGE_INVESTIGATIVE includes bits 20-23 - [ ] +T-INV-2: STAGE_ALL includes all 24 stages - [ ] T-INV-3: runStages with +STAGE_INVESTIGATIVE on a legal PDF → all 4 stages produce output - [ ] +T-INV-4: runStages with STAGE_INVESTIGATIVE on audio file → speaker ID +runs, redaction skipped + +==== Step 1.7: Multi-Locale HPC Cluster Test (D1) + +Chapel-based parallel processing on GASNet/IBV transport. + +*Tests:* - [ ] T-HPC-1: 4-node cluster processes 100 documents without +error - [ ] T-HPC-2: Load balancing: no single node processes >40% of +total - [ ] T-HPC-3: Node failure recovery: cluster continues if 1 of 4 +nodes drops - [ ] T-HPC-4: Identical results on 1-node vs 4-node runs +(determinism) + +*Benchmarks:* - [ ] B-HPC-1: 10,000 scanned PDFs on 4-node cluster — +target: <30 minutes - [ ] B-HPC-2: 100,000 PDFs on 16-node cluster — +target: <2 hours - [ ] B-HPC-3: Linear scaling factor — target: >0.7x +per added node - [ ] B-HPC-4: Full Epstein corpus (3.2M files) on 256 +nodes — target: <4 hours + +''''' + +=== Completion Tracker (Docudactyl Phases Only) + +[cols=",,,,",options="header",] +|=== +|# |Step |Status |Tests |Benchmarks +|1.1 |Core OCR + Text |DONE |Partial |0/1 +|1.2 |Redaction Detection |DONE (code) |0/5 |0/2 +|1.3 |Financial Extraction |DONE (code) |0/6 |0/2 +|1.4 |Legal NER |DONE (code) |0/6 |0/2 +|1.5 |Speaker ID |DONE (dispatch) |0/4 |0/2 +|1.6 |Investigative Preset |DONE |0/4 |— +|1.7 |HPC Cluster Test |TODO |0/4 |0/4 +|=== + +*Totals: 29 tests, 13 benchmarks | Current: 0 tests written, 0 +benchmarks run* + +=== Cross-References + +* *Full pipeline pathway:* `+bofig/docs/EPSTEIN-FILES-WORK-PATHWAY.md+` +* *Master integration plan:* `+bofig/docs/INTEGRATION-PLAN.md+` +* *Docudactyl integration role:* `+docs/INTEGRATION-PLAN-DOCUDACTYL.md+` +(this repo) diff --git a/docs/EPSTEIN-EXTRACTION-TESTS.md b/docs/EPSTEIN-EXTRACTION-TESTS.md deleted file mode 100644 index b77bbf1..0000000 --- a/docs/EPSTEIN-EXTRACTION-TESTS.md +++ /dev/null @@ -1,152 +0,0 @@ -# Epstein Files — Docudactyl Extraction Tests & Benchmarks -# -# SPDX-License-Identifier: CC-BY-SA-4.0 -# Author: Jonathan D.A. Jewell -# Created: 2026-03-13 -# -# Extracted from the master pathway: bofig/docs/EPSTEIN-FILES-WORK-PATHWAY.md -# This file contains only the Docudactyl-specific phases (1.1–1.7). - -## Dataset Characteristics - -| Attribute | Value | -|-----------|-------| -| Total files | ~3,200,000 | -| Total size | ~218 GB | -| Data sets | 12 (flight logs, court filings, depositions, financial records, photos, communications, ...) | -| Named entities (est.) | 23,000+ unique persons, orgs, locations | -| Primary format | Scanned PDF (96 DPI, many poor quality) | -| Secondary formats | TIFF, JPEG, DOCX, XLS, email (EML/PST) | -| Redaction style | Overlay-only (text stream often intact) | -| Financial transactions (est.) | 16,000+ | -| Languages | English (primary), French, some Spanish | -| Time span | 1990s–2024 | - ---- - -## Phase 1: Docudactyl Extraction Pipeline (Weeks 1–6) - -### Step 1.1: Core OCR + Text Extraction (DONE — existing stages 0-8) - -Already implemented in `stages.zig`: -- Language detection, readability, keywords, citations -- OCR confidence, perceptual hash, TOC extraction -- Multi-language OCR, subtitle extraction - -**Tests:** -- [x] Unit test: each stage function with known input produces expected Cap'n Proto output -- [x] Integration test: 10-document mini-corpus end-to-end -- [ ] Benchmark: single-node throughput for scanned PDFs (target: 2 docs/sec on 8-core) - -### Step 1.2: Redaction Detection (DONE — bit 20, `stageRedactionDetect`) - -Scans Poppler annotations for Type 12 (POPPLER_ANNOT_REDACT), checks if -text is extractable under overlay-only redactions. - -**Tests:** -- [ ] T-RED-1: Synthetic PDF with 5 /Redact annotations → count=5, status="redacted" -- [ ] T-RED-2: PDF with black fill rectangles but no /Redact annots → status="clean" (future: heuristic upgrade) -- [ ] T-RED-3: PDF with overlay redaction + recoverable text → recoverable_count > 0 -- [ ] T-RED-4: Non-PDF input (JPEG) → status="not_applicable" -- [ ] T-RED-5: Corrupt/unreadable PDF → status="error" - -**Benchmarks:** -- [ ] B-RED-1: 1000 PDFs (mixed redacted/clean) — target: <500ms per document -- [ ] B-RED-2: Memory usage during annotation scan — target: <50MB peak per document - -### Step 1.3: Financial Entity Extraction (DONE — bit 21, `stageFinancialExtract`) - -Pattern-based detection of currency symbols ($, £, €), ISO codes (USD, GBP, EUR, CHF, JPY, CAD), -account-like digit sequences (8-20 digits). - -**Tests:** -- [ ] T-FIN-1: Text "$1,234.56 paid to account 12345678" → amounts=1, accounts=1 -- [ ] T-FIN-2: Text "USD 50,000 transferred" → amounts=1 -- [ ] T-FIN-3: Text "£2.3 million to HSBC account 1234-5678-9012" → amounts=1, accounts=1 -- [ ] T-FIN-4: Phone numbers should NOT match as accounts (7-digit filter) -- [ ] T-FIN-5: Empty text → status="none_found", amounts=0, accounts=0 -- [ ] T-FIN-6: Mixed currencies in single document → correct total count - -**Benchmarks:** -- [ ] B-FIN-1: 10MB text document scan — target: <200ms -- [ ] B-FIN-2: Accuracy on annotated Epstein financial records sample (50 docs) — target: >80% recall - -### Step 1.4: Legal NER (DONE — bit 22, `stageLegalNer`) - -Pattern-based detection of case citations ("v."), docket numbers ("No.", "Case"), -statute references ("U.S.C.", "§"). - -**Tests:** -- [ ] T-LEG-1: "Doe v. Epstein" → case_citations=1 -- [ ] T-LEG-2: "No. 08-cv-1234" → docket_refs=1 -- [ ] T-LEG-3: "18 U.S.C. § 1591" → statute_refs=1 (both U.S.C. and § counted) -- [ ] T-LEG-4: "vs." variant → case_citations=1 -- [ ] T-LEG-5: Real Epstein court filing excerpt → realistic counts -- [ ] T-LEG-6: Non-legal document (flight log) → all counts = 0 - -**Benchmarks:** -- [ ] B-LEG-1: 5MB legal document — target: <150ms -- [ ] B-LEG-2: Precision on annotated legal corpus (100 docs) — target: >75% - -### Step 1.5: Speaker Identification (bit 23, ML dispatch) - -ML-based speaker diarization via ONNX Runtime. Dispatches to stage_id=5 -(speaker_id.onnx model). - -**Tests:** -- [ ] T-SPK-1: With ML handle + model → status="ok", speaker_count > 0 -- [ ] T-SPK-2: Without ML handle → status="not_available" -- [ ] T-SPK-3: Non-audio input → graceful fallback -- [ ] T-SPK-4: Deposition audio with 2 speakers → speaker_count=2 - -**Benchmarks:** -- [ ] B-SPK-1: 30-minute deposition audio — target: <60s inference -- [ ] B-SPK-2: Memory usage during diarization — target: <2GB - -### Step 1.6: STAGE_INVESTIGATIVE Preset Validation - -The `STAGE_INVESTIGATIVE` preset combines all investigative stages. - -**Tests:** -- [ ] T-INV-1: STAGE_INVESTIGATIVE includes bits 20-23 -- [ ] T-INV-2: STAGE_ALL includes all 24 stages -- [ ] T-INV-3: runStages with STAGE_INVESTIGATIVE on a legal PDF → all 4 stages produce output -- [ ] T-INV-4: runStages with STAGE_INVESTIGATIVE on audio file → speaker ID runs, redaction skipped - -### Step 1.7: Multi-Locale HPC Cluster Test (D1) - -Chapel-based parallel processing on GASNet/IBV transport. - -**Tests:** -- [ ] T-HPC-1: 4-node cluster processes 100 documents without error -- [ ] T-HPC-2: Load balancing: no single node processes >40% of total -- [ ] T-HPC-3: Node failure recovery: cluster continues if 1 of 4 nodes drops -- [ ] T-HPC-4: Identical results on 1-node vs 4-node runs (determinism) - -**Benchmarks:** -- [ ] B-HPC-1: 10,000 scanned PDFs on 4-node cluster — target: <30 minutes -- [ ] B-HPC-2: 100,000 PDFs on 16-node cluster — target: <2 hours -- [ ] B-HPC-3: Linear scaling factor — target: >0.7x per added node -- [ ] B-HPC-4: Full Epstein corpus (3.2M files) on 256 nodes — target: <4 hours - ---- - -## Completion Tracker (Docudactyl Phases Only) - -| # | Step | Status | Tests | Benchmarks | -|---|------|--------|-------|------------| -| 1.1 | Core OCR + Text | DONE | Partial | 0/1 | -| 1.2 | Redaction Detection | DONE (code) | 0/5 | 0/2 | -| 1.3 | Financial Extraction | DONE (code) | 0/6 | 0/2 | -| 1.4 | Legal NER | DONE (code) | 0/6 | 0/2 | -| 1.5 | Speaker ID | DONE (dispatch) | 0/4 | 0/2 | -| 1.6 | Investigative Preset | DONE | 0/4 | — | -| 1.7 | HPC Cluster Test | TODO | 0/4 | 0/4 | - -**Totals: 29 tests, 13 benchmarks | Current: 0 tests written, 0 benchmarks run** - -## Cross-References - -- **Full pipeline pathway:** `bofig/docs/EPSTEIN-FILES-WORK-PATHWAY.md` -- **Master integration plan:** `bofig/docs/INTEGRATION-PLAN.md` -- **Docudactyl integration role:** `docs/INTEGRATION-PLAN-DOCUDACTYL.md` (this repo) diff --git a/docs/INTEGRATION-PLAN-DOCUDACTYL.adoc b/docs/INTEGRATION-PLAN-DOCUDACTYL.adoc new file mode 100644 index 0000000..f8baf54 --- /dev/null +++ b/docs/INTEGRATION-PLAN-DOCUDACTYL.adoc @@ -0,0 +1,108 @@ +== Docudactyl Integration Plan — Pipeline Role + +== + +== SPDX-License-Identifier: CC-BY-SA-4.0 + +== Author: Jonathan D.A. Jewell + +== Created: 2026-03-13 + +== + +== Extracted from the master integration plan: bofig/docs/INTEGRATION-PLAN.md + +=== Docudactyl’s Role in the Pipeline + +Docudactyl is the *ingestion layer*. It takes raw documents and produces +structured extraction results that flow into Lithoglyph (audit-grade +storage) and then Bofig (evidence graph navigation). + +.... +Raw Documents (200K+ files) + → Docudactyl (HPC extraction: OCR, NER, metadata, classification) + → Lithoglyph (audit-grade storage: provenance, reversibility, PROMPT) + → Bofig (evidence graph: claims, relationships, navigation) +.... + +=== Docudactyl Tasks (from Integration Plan) + +[width="100%",cols="11%,17%,29%,23%,20%",options="header",] +|=== +|# |Task |Priority |Effort |Notes +|D1 |Multi-locale HPC cluster test (GASNet/IBV, 4+ nodes) |Critical +|Medium |Only v0.4.1 blocker + +|D2 |Cap’n Proto → Lithoglyph output adapter |High |Medium |New output +stage that emits GQL-compatible evidence records + +|D3 |Legal document NER model |High |Medium |Docket numbers, case names, +judge names, legal citations + +|D4 |Financial record extraction stage |High |Medium |Transaction +amounts, dates, account identifiers, counterparties + +|D5 |Speaker identification stage (testimony/depositions) |Medium |Large +|Who said what — maps to witness testimony in bofig + +|D6 |Redaction detection stage |Medium |Small |Flag redacted regions, +track unredaction over time + +|D7 |British Library pilot (170M items) |Low |Large |v1.0.0 milestone +|=== + +=== Output Contract + +Each processed document produces: - Extracted text (OCR’d if needed) + +confidence score - NER entities (people, orgs, locations, dates, +amounts) - SHA-256 + perceptual hash (dedup) - Metadata (Dublin Core + +format-specific) - Auto-PROMPT scores derived from extraction quality - +Language, keywords, citations + +=== Integration Points Involving Docudactyl + +==== Integration 1: Docudactyl → Lithoglyph (D2 + L6) + +.... +Docudactyl Cap'n Proto output + → Adapter (D2) serializes to GQL INSERT statements + → Lithoglyph ingest bridge (L6) batch-imports with: + - Auto-PROMPT scoring from extraction confidence + - SHA-256 dedup against existing evidence + - Actor="docudactyl-pipeline", Rationale="Batch extraction run {id}" + - Provenance: source file path, extraction timestamp, OCR confidence +.... + +==== Integration 3: Entity Resolution Loop (D3/D4/D5 → L5 → B1) + +.... +Docudactyl NER extracts raw entities + → Lithoglyph stores with alias tracking (L5) + → Bofig entity resolution (B1) merges aliases + → Merge decision logged in Lithoglyph journal + → Reversible if co-reference was incorrect +.... + +==== Integration 4: Financial Flow Analysis (D4 → L4 → B2) + +.... +Docudactyl extracts transactions from bank records (D4) + → Lithoglyph financial_transactions collection (L4) + → Bofig GraphQL: transactionChain(entityId, depth) (B2) +.... + +=== Phase Assignment + +Docudactyl work falls primarily in *Phase A (Foundation)* and *Phase C +(Investigation Features)*: + +* *Phase A (Weeks 1-4):* D1 (HPC cluster test) +* *Phase B (Weeks 5-8):* D2 (Cap’n Proto adapter) +* *Phase C (Weeks 9-14):* D3, D4, D5 (NER models + speaker ID) + +=== Cross-References + +* *Master plan:* `+bofig/docs/INTEGRATION-PLAN.md+` +* *Epstein worked example:* `+bofig/docs/EPSTEIN-FILES-WORK-PATHWAY.md+` +* *Epstein Docudactyl phases:* `+docs/EPSTEIN-EXTRACTION-TESTS.md+` +(this repo) diff --git a/docs/INTEGRATION-PLAN-DOCUDACTYL.md b/docs/INTEGRATION-PLAN-DOCUDACTYL.md deleted file mode 100644 index cf63edf..0000000 --- a/docs/INTEGRATION-PLAN-DOCUDACTYL.md +++ /dev/null @@ -1,88 +0,0 @@ -# Docudactyl Integration Plan — Pipeline Role -# -# SPDX-License-Identifier: CC-BY-SA-4.0 -# Author: Jonathan D.A. Jewell -# Created: 2026-03-13 -# -# Extracted from the master integration plan: bofig/docs/INTEGRATION-PLAN.md - -## Docudactyl's Role in the Pipeline - -Docudactyl is the **ingestion layer**. It takes raw documents and produces -structured extraction results that flow into Lithoglyph (audit-grade storage) -and then Bofig (evidence graph navigation). - -``` -Raw Documents (200K+ files) - → Docudactyl (HPC extraction: OCR, NER, metadata, classification) - → Lithoglyph (audit-grade storage: provenance, reversibility, PROMPT) - → Bofig (evidence graph: claims, relationships, navigation) -``` - -## Docudactyl Tasks (from Integration Plan) - -| # | Task | Priority | Effort | Notes | -|---|------|----------|--------|-------| -| D1 | Multi-locale HPC cluster test (GASNet/IBV, 4+ nodes) | Critical | Medium | Only v0.4.1 blocker | -| D2 | Cap'n Proto → Lithoglyph output adapter | High | Medium | New output stage that emits GQL-compatible evidence records | -| D3 | Legal document NER model | High | Medium | Docket numbers, case names, judge names, legal citations | -| D4 | Financial record extraction stage | High | Medium | Transaction amounts, dates, account identifiers, counterparties | -| D5 | Speaker identification stage (testimony/depositions) | Medium | Large | Who said what — maps to witness testimony in bofig | -| D6 | Redaction detection stage | Medium | Small | Flag redacted regions, track unredaction over time | -| D7 | British Library pilot (170M items) | Low | Large | v1.0.0 milestone | - -## Output Contract - -Each processed document produces: -- Extracted text (OCR'd if needed) + confidence score -- NER entities (people, orgs, locations, dates, amounts) -- SHA-256 + perceptual hash (dedup) -- Metadata (Dublin Core + format-specific) -- Auto-PROMPT scores derived from extraction quality -- Language, keywords, citations - -## Integration Points Involving Docudactyl - -### Integration 1: Docudactyl → Lithoglyph (D2 + L6) - -``` -Docudactyl Cap'n Proto output - → Adapter (D2) serializes to GQL INSERT statements - → Lithoglyph ingest bridge (L6) batch-imports with: - - Auto-PROMPT scoring from extraction confidence - - SHA-256 dedup against existing evidence - - Actor="docudactyl-pipeline", Rationale="Batch extraction run {id}" - - Provenance: source file path, extraction timestamp, OCR confidence -``` - -### Integration 3: Entity Resolution Loop (D3/D4/D5 → L5 → B1) - -``` -Docudactyl NER extracts raw entities - → Lithoglyph stores with alias tracking (L5) - → Bofig entity resolution (B1) merges aliases - → Merge decision logged in Lithoglyph journal - → Reversible if co-reference was incorrect -``` - -### Integration 4: Financial Flow Analysis (D4 → L4 → B2) - -``` -Docudactyl extracts transactions from bank records (D4) - → Lithoglyph financial_transactions collection (L4) - → Bofig GraphQL: transactionChain(entityId, depth) (B2) -``` - -## Phase Assignment - -Docudactyl work falls primarily in **Phase A (Foundation)** and **Phase C (Investigation Features)**: - -- **Phase A (Weeks 1-4):** D1 (HPC cluster test) -- **Phase B (Weeks 5-8):** D2 (Cap'n Proto adapter) -- **Phase C (Weeks 9-14):** D3, D4, D5 (NER models + speaker ID) - -## Cross-References - -- **Master plan:** `bofig/docs/INTEGRATION-PLAN.md` -- **Epstein worked example:** `bofig/docs/EPSTEIN-FILES-WORK-PATHWAY.md` -- **Epstein Docudactyl phases:** `docs/EPSTEIN-EXTRACTION-TESTS.md` (this repo) diff --git a/docs/INVESTIGATOR-TOOLKIT.adoc b/docs/INVESTIGATOR-TOOLKIT.adoc new file mode 100644 index 0000000..224cf72 --- /dev/null +++ b/docs/INVESTIGATOR-TOOLKIT.adoc @@ -0,0 +1,306 @@ +== Docudactyl — Investigator Toolkit + +This document describes the *investigator-focused extraction modules* +added to Docudactyl for use by citizen journalists, independent +researchers, and investigative reporters working with large document +releases such as the Epstein filings, Panama/Paradise/Pandora Papers, +FinCEN Files, etc. + +These modules are *standalone Zig translation units* with stable C-ABI +entry points. They can be called directly from Chapel, Rust, Julia, +OCaml, Python `+ctypes+`, or any language that speaks the C FFI — +without going through the full HPC pipeline. + +''''' + +=== Why these modules exist + +The base HPC pipeline answers "`what is in this document?`". +Investigative journalism needs different questions: + +[arabic] +. *Who appears with whom, how often?* → +link:#entity-graph[`+entity_graph+`] +. *What was hidden under black bars?* → +link:#redaction-recovery[`+redaction_recovery+`] +. *Where did the jet actually go?* → link:#flight-log[`+flight_log+`] +. *Where did the witness stop answering?* → +link:#evasion-detect[`+evasion_detect+`] +. *What does this document contain, at a glance?* → +link:#investigator-summary[`+investigator_summary+`] + +Each module is *pattern-based, no ML dependency*, fast, and +deterministic. + +''''' + +=== Entity Graph + +*File:* `+ffi/zig/src/entity_graph.zig+` + +Builds a cross-document co-occurrence graph of capitalised personal +names. Exports to *GraphML* (Gephi, yEd, Cytoscape) and *CSV* (Excel, +LibreOffice, Maltego, Neo4j). + +==== C ABI + +[source,c] +---- +typedef struct EntityGraph EntityGraph; + +EntityGraph* ddac_entity_graph_new(void); +void ddac_entity_graph_free(EntityGraph*); + +int ddac_entity_graph_add_document(EntityGraph*, const char* text, size_t len); +int ddac_entity_graph_export_graphml(EntityGraph*, const char* path); +int ddac_entity_graph_export_csv(EntityGraph*, const char* path); + +uint32_t ddac_entity_graph_node_count(EntityGraph*); +uint32_t ddac_entity_graph_edge_count(EntityGraph*); +---- + +==== Usage sketch + +[source,chapel] +---- +// Chapel pseudocode +var g = ddac_entity_graph_new(); +for doc in manifest { + var text = readExtractedText(doc); + ddac_entity_graph_add_document(g, text.c_str(), text.len); +} +ddac_entity_graph_export_graphml(g, "/out/entities.graphml"); +ddac_entity_graph_export_csv(g, "/out/entities.csv"); +ddac_entity_graph_free(g); +---- + +Load `+entities.graphml+` in Gephi → run ForceAtlas2 → see the cluster +structure. Load `+entities.csv+` in any spreadsheet to sort by weight. + +==== Notes + +* Extracts 2+ consecutive capitalised words, optionally preceded by a +title (Mr./Mrs./Dr./Prince/Sir/…). +* Filters a conservative stopword list (weekdays, months, common +sentence-initial words). +* Edge weight accumulates across documents — high weight indicates +recurring co-occurrence worth examining. + +''''' + +=== Redaction Recovery + +*File:* `+ffi/zig/src/redaction_recovery.zig+` + +Extends the base redaction-detection stage with *per-page density maps* +and *overlay-only text recovery*. When a PDF carries `+/Redact+` +annotations (black boxes) but the underlying content stream is intact, +this module extracts the text that the overlay was meant to hide. + +==== When it works + +✅ Overlay redactions where the text stream was NOT scrubbed (common +FOIA failure mode — many Epstein-era productions exhibit this). ❌ +Redactions that rasterise the page or strip the content stream. ❌ +Redactions applied at scan time (pixel-level black bars). + +==== C ABI + +[source,c] +---- +typedef struct { + int status; + uint32_t total_pages; + uint32_t total_redactions; + uint32_t pages_with_redactions; + uint32_t recoverable_pages; + uint64_t recovered_bytes; + PageStats pages[4096]; + char summary[512]; +} RedactionRecoveryResult; + +int ddac_redaction_recovery_analyze(const char* pdf_path, RedactionRecoveryResult*); +int ddac_redaction_recovery_dump_text(const char* pdf_path, const char* out_path); +---- + +==== Legal & ethical note + +This module extracts text that is *already present* in the document’s +content stream — the same text that `+cmd-A, cmd-C+` in Preview would +reveal. It does not break encryption, does not OCR under pixel-level +redactions, and does not decode protected content. Use responsibly and +check your local jurisdiction’s rules on reporting improperly redacted +material. + +''''' + +=== Flight Log + +*File:* `+ffi/zig/src/flight_log.zig+` + +Extracts travel-document entities from text: + +[width="100%",cols="45%,55%",options="header",] +|=== +|Entity |Examples +|Tail numbers |`+N908JE+`, `+N212JE+`, `+G-EJES+`, `+D-IIKA+` + +|IATA codes |`+TEB+`, `+PBI+`, `+JFK+`, `+STT+`, `+LHR+`, `+CDG+`, +`+DXB+` + +|ICAO codes |`+KTEB+`, `+KPBI+`, `+KJFK+`, `+EGLL+`, `+LFPB+` + +|Phones |`+(212) 555-1234+`, `++1 212 555 9999+`, `++44 20 7946 0958+` + +|Addresses |Line-leading number + road-word heuristic + +|Manifest markers |`+PAX:+`, `+PASSENGERS:+`, `+MANIFEST:+`, `+GUESTS:+` +|=== + +==== C ABI + +[source,c] +---- +typedef struct { /* ... */ } FlightLogResult; +int ddac_flight_log_process(const char* text, size_t len, FlightLogResult*); +---- + +==== Notes + +* IATA/ICAO codes use a *whitelist* of airports of interest (Teterboro, +Palm Beach, St. Thomas, Le Bourget, Heathrow, Dubai, etc.) to avoid +false positives on three-letter acronyms like `+CEO+` or `+FBI+`. Extend +the whitelist in the module source as needed. +* Tail-number pattern is liberal enough to tolerate OCR noise but tight +enough to reject ordinary words. + +''''' + +=== Evasion Detect + +*File:* `+ffi/zig/src/evasion_detect.zig+` + +Detects and categorises evasive / non-answer patterns in deposition and +interview transcripts: + +[cols=",",options="header",] +|=== +|Category |Example phrase +|`+no_recall+` |"`I don’t recall`", "`I have no recollection`" +|`+no_memory+` |"`I don’t remember`", "`I can’t remember`" +|`+not_sure+` |"`I’m not sure`", "`I couldn’t say`" +|`+no_knowledge+` |"`Not to my knowledge`", "`I’m not aware`" +|`+would_check+` |"`I’d have to check`", "`I would need to check`" +|`+asked_answered+` |"`Asked and answered`" (lawyer interjection) +|`+fifth_amendment+` |"`Fifth Amendment`", "`on the advice of counsel`" +|`+decline_answer+` |"`I decline to answer`", "`refuse to answer`" +|=== + +Reports category counts, total events, and an *evasion rate* (events per +1000 tokens, fixed-point ×1000). A rate above ~20 +(i.e. `+×1000 > 20000+`) typically indicates a heavily evasive witness +segment. + +==== C ABI + +[source,c] +---- +typedef struct { + int status; + uint32_t category_counts[8]; + uint32_t total_events; + uint32_t total_tokens; + uint32_t events_per_1k_fixed; // rate × 1000 + char summary[512]; +} EvasionResult; + +int ddac_evasion_detect(const char* text, size_t len, EvasionResult*); +---- + +''''' + +=== Investigator Summary + +*File:* `+ffi/zig/src/investigator_summary.zig+` + +Takes a populated `+InvestigatorSummary+` struct and emits an +investigator-friendly *JSON summary* per document. Designed to be +readable in a text editor and ingestible by spreadsheets, dataset +browsers, or static-site generators. + +The JSON is flat and forgiving: any field may be zero/empty without +breaking consumers. A `+flags+` array gives quick visual triage: + +[source,json] +---- +{ + "source_path": "/data/release_2024/doc_0042.pdf", + "sha256": "...", + "page_count": 184, + "redactions": {"count": 12, "pages_affected": 4, "recoverable_pages": 2}, + "financial": {"amounts": 3, "accounts": 1}, + "legal": {"case_citations": 5, "dockets": 2, "statutes": 1}, + "speakers": {"count": 2, "is_deposition": true}, + "evasion": {"total": 17, "per_1k_tokens": 12.5}, + "entities": { + "persons": ["Jeffrey Epstein", "Ghislaine Maxwell"], + "tail_numbers": ["N908JE"], + "airports": ["TEB", "PBI", "KTEB"], + "phones": ["+1 212 555 1234"], + "addresses": ["9 East 71st Street"] + }, + "flags": ["has_redactions", "has_recoverable_text", "deposition", "high_evasion"] +} +---- + +==== C ABI + +[source,c] +---- +int ddac_investigator_summary_write(const char* out_path, const InvestigatorSummary*); +int ddac_investigator_summary_set_list_item(StringList*, uint32_t idx, + const char* text, size_t len); +---- + +''''' + +=== Recommended Investigator Workflow + +[arabic] +. *Run base pipeline* (`+DocudactylHPC+`) to extract text + SHA + +PREMIS. +. *Per-document pass* — for each extracted text file, call: +* `+ddac_flight_log_process+` → flight / travel entities +* `+ddac_evasion_detect+` → deposition evasion stats +* `+ddac_redaction_recovery_analyze+` + `+_dump_text+` (PDFs only) +. *Corpus-wide pass* — accumulate entities into a single graph: +* `+ddac_entity_graph_new+` +* `+ddac_entity_graph_add_document+` per document +* `+ddac_entity_graph_export_graphml+` + `+_export_csv+` +. *Summary pass* — populate `+InvestigatorSummary+` from the results +above and emit per-document JSON with +`+ddac_investigator_summary_write+`. +. *Review* — open the GraphML in Gephi, the CSV in a spreadsheet, and +the per-document JSONs in a text editor or a static-site browser. + +''''' + +=== Building & Testing + +From `+ffi/zig/+`: + +[source,bash] +---- +zig build # build shared + static libraries +zig build test # run unit tests (all new modules included) +---- + +All new modules have accompanying Zig unit tests. No new C dependencies +beyond Poppler + GLib (which Docudactyl already links). + +''''' + +=== License + +All new modules are released under *MPL-2.0* (with MPL-2.0 fallback), +matching the rest of Docudactyl. diff --git a/docs/INVESTIGATOR-TOOLKIT.md b/docs/INVESTIGATOR-TOOLKIT.md deleted file mode 100644 index 6856f4f..0000000 --- a/docs/INVESTIGATOR-TOOLKIT.md +++ /dev/null @@ -1,281 +0,0 @@ -# Docudactyl — Investigator Toolkit - - - -This document describes the **investigator-focused extraction modules** added -to Docudactyl for use by citizen journalists, independent researchers, and -investigative reporters working with large document releases such as the -Epstein filings, Panama/Paradise/Pandora Papers, FinCEN Files, etc. - -These modules are **standalone Zig translation units** with stable C-ABI -entry points. They can be called directly from Chapel, Rust, Julia, OCaml, -Python `ctypes`, or any language that speaks the C FFI — without going -through the full HPC pipeline. - ---- - -## Why these modules exist - -The base HPC pipeline answers "what is in this document?". Investigative -journalism needs different questions: - -1. **Who appears with whom, how often?** → [`entity_graph`](#entity-graph) -2. **What was hidden under black bars?** → [`redaction_recovery`](#redaction-recovery) -3. **Where did the jet actually go?** → [`flight_log`](#flight-log) -4. **Where did the witness stop answering?** → [`evasion_detect`](#evasion-detect) -5. **What does this document contain, at a glance?** → [`investigator_summary`](#investigator-summary) - -Each module is **pattern-based, no ML dependency**, fast, and deterministic. - ---- - -## Entity Graph - -**File:** `ffi/zig/src/entity_graph.zig` - -Builds a cross-document co-occurrence graph of capitalised personal names. -Exports to **GraphML** (Gephi, yEd, Cytoscape) and **CSV** (Excel, -LibreOffice, Maltego, Neo4j). - -### C ABI - -```c -typedef struct EntityGraph EntityGraph; - -EntityGraph* ddac_entity_graph_new(void); -void ddac_entity_graph_free(EntityGraph*); - -int ddac_entity_graph_add_document(EntityGraph*, const char* text, size_t len); -int ddac_entity_graph_export_graphml(EntityGraph*, const char* path); -int ddac_entity_graph_export_csv(EntityGraph*, const char* path); - -uint32_t ddac_entity_graph_node_count(EntityGraph*); -uint32_t ddac_entity_graph_edge_count(EntityGraph*); -``` - -### Usage sketch - -```chapel -// Chapel pseudocode -var g = ddac_entity_graph_new(); -for doc in manifest { - var text = readExtractedText(doc); - ddac_entity_graph_add_document(g, text.c_str(), text.len); -} -ddac_entity_graph_export_graphml(g, "/out/entities.graphml"); -ddac_entity_graph_export_csv(g, "/out/entities.csv"); -ddac_entity_graph_free(g); -``` - -Load `entities.graphml` in Gephi → run ForceAtlas2 → see the cluster -structure. Load `entities.csv` in any spreadsheet to sort by weight. - -### Notes -- Extracts 2+ consecutive capitalised words, optionally preceded by a - title (Mr./Mrs./Dr./Prince/Sir/…). -- Filters a conservative stopword list (weekdays, months, common - sentence-initial words). -- Edge weight accumulates across documents — high weight indicates - recurring co-occurrence worth examining. - ---- - -## Redaction Recovery - -**File:** `ffi/zig/src/redaction_recovery.zig` - -Extends the base redaction-detection stage with **per-page density maps** -and **overlay-only text recovery**. When a PDF carries `/Redact` -annotations (black boxes) but the underlying content stream is intact, -this module extracts the text that the overlay was meant to hide. - -### When it works - -✅ Overlay redactions where the text stream was NOT scrubbed (common -FOIA failure mode — many Epstein-era productions exhibit this). -❌ Redactions that rasterise the page or strip the content stream. -❌ Redactions applied at scan time (pixel-level black bars). - -### C ABI - -```c -typedef struct { - int status; - uint32_t total_pages; - uint32_t total_redactions; - uint32_t pages_with_redactions; - uint32_t recoverable_pages; - uint64_t recovered_bytes; - PageStats pages[4096]; - char summary[512]; -} RedactionRecoveryResult; - -int ddac_redaction_recovery_analyze(const char* pdf_path, RedactionRecoveryResult*); -int ddac_redaction_recovery_dump_text(const char* pdf_path, const char* out_path); -``` - -### Legal & ethical note - -This module extracts text that is **already present** in the document's -content stream — the same text that `cmd-A, cmd-C` in Preview would -reveal. It does not break encryption, does not OCR under pixel-level -redactions, and does not decode protected content. Use responsibly and -check your local jurisdiction's rules on reporting improperly redacted -material. - ---- - -## Flight Log - -**File:** `ffi/zig/src/flight_log.zig` - -Extracts travel-document entities from text: - -| Entity | Examples | -|--------|----------| -| Tail numbers | `N908JE`, `N212JE`, `G-EJES`, `D-IIKA` | -| IATA codes | `TEB`, `PBI`, `JFK`, `STT`, `LHR`, `CDG`, `DXB` | -| ICAO codes | `KTEB`, `KPBI`, `KJFK`, `EGLL`, `LFPB` | -| Phones | `(212) 555-1234`, `+1 212 555 9999`, `+44 20 7946 0958` | -| Addresses | Line-leading number + road-word heuristic | -| Manifest markers | `PAX:`, `PASSENGERS:`, `MANIFEST:`, `GUESTS:` | - -### C ABI - -```c -typedef struct { /* ... */ } FlightLogResult; -int ddac_flight_log_process(const char* text, size_t len, FlightLogResult*); -``` - -### Notes -- IATA/ICAO codes use a **whitelist** of airports of interest - (Teterboro, Palm Beach, St. Thomas, Le Bourget, Heathrow, Dubai, - etc.) to avoid false positives on three-letter acronyms like `CEO` - or `FBI`. Extend the whitelist in the module source as needed. -- Tail-number pattern is liberal enough to tolerate OCR noise but - tight enough to reject ordinary words. - ---- - -## Evasion Detect - -**File:** `ffi/zig/src/evasion_detect.zig` - -Detects and categorises evasive / non-answer patterns in deposition and -interview transcripts: - -| Category | Example phrase | -|----------|---------------| -| `no_recall` | "I don't recall", "I have no recollection" | -| `no_memory` | "I don't remember", "I can't remember" | -| `not_sure` | "I'm not sure", "I couldn't say" | -| `no_knowledge` | "Not to my knowledge", "I'm not aware" | -| `would_check` | "I'd have to check", "I would need to check" | -| `asked_answered` | "Asked and answered" (lawyer interjection) | -| `fifth_amendment` | "Fifth Amendment", "on the advice of counsel" | -| `decline_answer` | "I decline to answer", "refuse to answer" | - -Reports category counts, total events, and an **evasion rate** (events -per 1000 tokens, fixed-point ×1000). A rate above ~20 (i.e. `×1000 > -20000`) typically indicates a heavily evasive witness segment. - -### C ABI - -```c -typedef struct { - int status; - uint32_t category_counts[8]; - uint32_t total_events; - uint32_t total_tokens; - uint32_t events_per_1k_fixed; // rate × 1000 - char summary[512]; -} EvasionResult; - -int ddac_evasion_detect(const char* text, size_t len, EvasionResult*); -``` - ---- - -## Investigator Summary - -**File:** `ffi/zig/src/investigator_summary.zig` - -Takes a populated `InvestigatorSummary` struct and emits an -investigator-friendly **JSON summary** per document. Designed to be -readable in a text editor and ingestible by spreadsheets, dataset -browsers, or static-site generators. - -The JSON is flat and forgiving: any field may be zero/empty without -breaking consumers. A `flags` array gives quick visual triage: - -```json -{ - "source_path": "/data/release_2024/doc_0042.pdf", - "sha256": "...", - "page_count": 184, - "redactions": {"count": 12, "pages_affected": 4, "recoverable_pages": 2}, - "financial": {"amounts": 3, "accounts": 1}, - "legal": {"case_citations": 5, "dockets": 2, "statutes": 1}, - "speakers": {"count": 2, "is_deposition": true}, - "evasion": {"total": 17, "per_1k_tokens": 12.5}, - "entities": { - "persons": ["Jeffrey Epstein", "Ghislaine Maxwell"], - "tail_numbers": ["N908JE"], - "airports": ["TEB", "PBI", "KTEB"], - "phones": ["+1 212 555 1234"], - "addresses": ["9 East 71st Street"] - }, - "flags": ["has_redactions", "has_recoverable_text", "deposition", "high_evasion"] -} -``` - -### C ABI - -```c -int ddac_investigator_summary_write(const char* out_path, const InvestigatorSummary*); -int ddac_investigator_summary_set_list_item(StringList*, uint32_t idx, - const char* text, size_t len); -``` - ---- - -## Recommended Investigator Workflow - -1. **Run base pipeline** (`DocudactylHPC`) to extract text + SHA + PREMIS. -2. **Per-document pass** — for each extracted text file, call: - - `ddac_flight_log_process` → flight / travel entities - - `ddac_evasion_detect` → deposition evasion stats - - `ddac_redaction_recovery_analyze` + `_dump_text` (PDFs only) -3. **Corpus-wide pass** — accumulate entities into a single graph: - - `ddac_entity_graph_new` - - `ddac_entity_graph_add_document` per document - - `ddac_entity_graph_export_graphml` + `_export_csv` -4. **Summary pass** — populate `InvestigatorSummary` from the results - above and emit per-document JSON with - `ddac_investigator_summary_write`. -5. **Review** — open the GraphML in Gephi, the CSV in a spreadsheet, and - the per-document JSONs in a text editor or a static-site browser. - ---- - -## Building & Testing - -From `ffi/zig/`: - -```bash -zig build # build shared + static libraries -zig build test # run unit tests (all new modules included) -``` - -All new modules have accompanying Zig unit tests. No new C dependencies -beyond Poppler + GLib (which Docudactyl already links). - ---- - -## License - -All new modules are released under **MPL-2.0** (with MPL-2.0 -fallback), matching the rest of Docudactyl. diff --git a/docs/decisions/2026-05-30-docudactyl-chapel-rehab.adoc b/docs/decisions/2026-05-30-docudactyl-chapel-rehab.adoc new file mode 100644 index 0000000..3c80b58 --- /dev/null +++ b/docs/decisions/2026-05-30-docudactyl-chapel-rehab.adoc @@ -0,0 +1,178 @@ +== ADR — Docudactyl Chapel Rehabilitation (Wave 3 of the echidna#146 arc) + +* Status: Accepted +* Date: 2026-05-30 +* Closes: +https://github.com/hyperpolymath/docudactyl/issues/29[docudactyl#29] +* Wave 1 reference: +https://github.com/hyperpolymath/echidna/pull/146[echidna#146] +* Wave 2 tracker: +https://github.com/hyperpolymath/proven/issues/126[proven#126] +* SPDX-License-Identifier: CC-BY-SA-4.0 + +=== Context + +`+docudactyl+` is a distributed document-processing HPC engine for +British-Library-scale corpora — 170M+ items dispatched across Chapel +locales, parsing via a Zig FFI bridge to C parser libraries (poppler / +tesseract / ffmpeg / libxml2 / gdal / vips). The Chapel framework lives +at `+src/chapel/+` (11 modules, ~2,500 lines). + +`+docudactyl#29+` was filed alongside `+echidna#146+` (the Wave-1 Chapel +metalayer rehabilitation) and `+proven#126+` (the Wave-2 follow-on). +Each Wave-N issue applies the same shape: rewrite for chpl 2.8.0 +compatibility, decouple the FFI smoke from the metalayer, flip CI to +strict, and decide whether the Wave-1 `+ParallelSoundness.agda+` +invariants apply to this repo’s parallel-dispatch surface. + +=== Decision + +Port the echidna#146 pattern with *two scope adjustments*: + +[arabic] +. *No `+ParallelSoundness.agda+` port.* Docudactyl’s parallel-dispatch +surface is `+forall idx in dynamic(docEntries.domain, chunkSize)+` in +`+DocudactylHPC.chpl+` — embarrassingly-parallel work distribution with +deterministic per-locale-slot aggregation, not speculative search. +Echidna’s three theorems (first-success-wins soundness, completeness +under retry, cancellation safety) are inapplicable. Docudactyl’s +relevant invariants — at-most-once under checkpoint resume, aggregation +commutativity, L1/L2 cache coherence — are documented below and deferred +to a follow-up Agda module if formal proofs become load-bearing. +. *The CI `+check-abi+` job retains `+continue-on-error: true+`.* This +is documented as environmental fragility (Idris2 source bootstrap on +ubuntu-24.04, not docudactyl correctness). The new `+build-smoke+` job +is the strict gate the echidna pattern actually wants. + +=== What landed + +==== Chapel 2.8.0 compatibility fixes + +`+chpl --no-codegen src/chapel/DocudactylHPC.chpl src/chapel/*.chpl+` +was failing with three errors before this PR; all three are now +resolved. The metalayer compiles clean on chpl 2.8.0 (LLVM-built local +toolchain) with only known unstable-API warnings remaining (`+_pad+` +symbol prefixes; `+dmapped+`; `+DynamicIters+`; `+string.c_str()+`; +cross-locale `+c_addrOf+`). + +[width="100%",cols="34%,33%,33%",options="header",] +|=== +|Site |Pre-state |Fix +|`+DocudactylHPC.chpl:303, 311, 373+` |`+FileSystem.stat(string)+` +removed in chpl 2.8.0 |`+use OS.POSIX;+` + `+struct_stat+` + +`+stat(path.c_str(), c_ptrTo(sb))+`; field reads via +`+sb.st_mtim.tv_sec+` / `+sb.st_size+` + +|`+DocudactylHPC.chpl:211+` |`+forall+` task-private intent made outer +`+var ndjsonWriter+` const-shadow; `+proc ref writeResult+` rejected the +const actual |`+forall … with (ref ndjsonWriter) { … }+` (matches the +`+begin with (ref timer)+` pattern already at line 201) + +|`+NdjsonManifest.chpl:91+` |`+string[range]+` slicing now `+throws+` +(UTF-8 boundary check) +|`+try { return line[range]; } catch { return ""; }+` — preserves the +existing "`return empty on error`" contract + +|`+NdjsonManifest.chpl:177, 182+` +|`+string.createCopyingBuffer(c_ptrConst(c_char))+` now `+throws+` +|`+try { … } catch { }+` around each optional field — the SHA-256 / +title fields are skipped in the rare malformed-UTF-8 case so the +`+forall+` task is not torn down +|=== + +==== Decoupled FFI smoke + +`+src/chapel/smoke.chpl+` (46 lines) exercises `+ddac_version+` / +`+ddac_crypto_sha256_name+` / `+ddac_init+` / `+ddac_free+` against the +FFI bridge *without* pulling in the full `+DocudactylHPC+` metalayer. +This is the echidna#146 invariant: a green smoke proves the C ABI +compiles + links + returns sane data even when the metalayer is broken, +so regression localisation is fast. + +The smoke is wired three places: + +* `+Justfile+` recipes: `+check-smoke+` (parse-only), `+build-smoke+` +(binary), `+run-smoke+` (executes the binary) +* `+.github/workflows/hpc-ci.yml+` job `+build-smoke+`: depends on +`+build-ffi+`, strict (no `+continue-on-error+`), greps for +`+^\[smoke\] PASS$+` in the binary output +* `+Justfile :: check-chapel+` now uses `+--main-module DocudactylHPC+` +to disambiguate the two `+proc main()+` files (DocudactylHPC + smoke) +when globbing `+src/chapel/*.chpl+` + +==== CI floor bumped + +`+CHAPEL_VERSION: '2.3.0'+` → `+'2.8.0'+` in `+hpc-ci.yml+`. The +OS.POSIX migration is the proximate cause; the wider justification is +that 2.8.0 is the version the rehabilitation pattern was validated +against in Wave 1. + +=== Parallel-dispatch invariants (informal — deferred to follow-up if formalised) + +These document the invariants the docudactyl +`+forall idx in dynamic(...)+` loop relies on. They are NOT proved here; +they are recorded so a future Agda module has a target. + +[arabic] +. *At-most-once* — every document index in `+docEntries.domain+` is +processed at most once. Currently enforced per-locale by +`+isAlreadyProcessed(idx)+` (Checkpoint.chpl) gating on +`+recordCheckpoint+`. Caveat: cross-locale resume after a topology +change (e.g. 64 → 128 locales) is NOT global-atomic. +. *Abort-bounded* — beyond a failure-rate threshold (FaultHandler.chpl), +the loop short-circuits; unprocessed indices are NOT silently dropped, +they remain unprocessed and observable via the report. +. *Aggregation commutativity* — `+accumulate(result)+` writes only to +`+perLocaleStats[here.id]+` (per-locale slot, no cross-task race); +`+computeGlobal+` reduces by integer addition, which is commutative. +SATISFIED by construction. +. *Cache coherence (L1)* — `+(path, mtime, size)+`-keyed LMDB lookups in +the per-locale L1 are read-then-write within a single task; no +concurrent writers to the same key. +. *Cache coherence (L2)* — Dragonfly L2 is shared across locales and +keyed by SHA-256; concurrent `+store+` calls for the same key are +tolerated because the value is deterministic (same SHA-256 implies same +parse result). Not formally proven. +. *Checkpoint resumability* — resuming from a `+recordCheckpoint(idx)+` +snapshot reproduces the same `+succeededDocs / failedDocs+` global stats +modulo per-locale ordering. Stats are aggregated by addition, so order +does not matter. +. *Content-type determinism* — magic-byte detection +(`+conduit.content_kind+`) is deterministic per input file. + +A follow-up issue should be filed if these need formal mechanisation +(Agda module path +`+proofs/agda/DistributedAggregationInvariants.agda+`); echidna’s +`+ParallelSoundness.agda+` cannot be imported unchanged. + +=== Wave-2 gate + +Per `+docudactyl#29+`, this PR was deferred until `+proven#126+` (Wave +2) "`lands`". At time of writing `+proven#126+` is still OPEN. The +owner-authorised exception: the echidna#146 + proven#135 (binding-tier-1 +detachable harness) pair has stabilised the rehabilitation shape; a +docudactyl rehab now does not risk pattern-divergence. Should Wave 2 +settle on a different shape, this ADR is the supersedable surface — the +smoke target and CI gate are the load-bearing pieces that would change. + +=== Consequences + +* `+chpl 2.8.0+` is now the floor in CI; running on `+2.3.0+` will fail +the OS.POSIX import. +* The metalayer build remains the same shape; only the three stat sites ++ `+with (ref ndjsonWriter)+` clause + the two throws-wrappers in +NdjsonManifest differ. +* The new `+build-smoke+` job runs in parallel with `+build-chapel+` +(both `+needs: build-ffi+`), so wallclock CI time does not regress. +* `+check-abi+` still flagged fragile; not changed in this PR. + +=== References + +* https://github.com/hyperpolymath/echidna/pull/146[echidna#146] — +Wave-1 metalayer rehabilitation; canonical pattern source +* https://github.com/hyperpolymath/proven/issues/126[proven#126] — +Wave-2 tracker +* https://github.com/hyperpolymath/docudactyl/issues/29[docudactyl#29] — +this issue +* chpl 2.8.0 `+OS.POSIX.stat+` — +`+/usr/share/chapel/2.8/modules/standard/OS.chpl:929-965+` diff --git a/docs/decisions/2026-05-30-docudactyl-chapel-rehab.md b/docs/decisions/2026-05-30-docudactyl-chapel-rehab.md deleted file mode 100644 index bdc1ba1..0000000 --- a/docs/decisions/2026-05-30-docudactyl-chapel-rehab.md +++ /dev/null @@ -1,81 +0,0 @@ -# ADR — Docudactyl Chapel Rehabilitation (Wave 3 of the echidna#146 arc) - -- Status: Accepted -- Date: 2026-05-30 -- Closes: [docudactyl#29](https://github.com/hyperpolymath/docudactyl/issues/29) -- Wave 1 reference: [echidna#146](https://github.com/hyperpolymath/echidna/pull/146) -- Wave 2 tracker: [proven#126](https://github.com/hyperpolymath/proven/issues/126) -- SPDX-License-Identifier: CC-BY-SA-4.0 - -## Context - -`docudactyl` is a distributed document-processing HPC engine for British-Library-scale corpora — 170M+ items dispatched across Chapel locales, parsing via a Zig FFI bridge to C parser libraries (poppler / tesseract / ffmpeg / libxml2 / gdal / vips). The Chapel framework lives at `src/chapel/` (11 modules, ~2,500 lines). - -`docudactyl#29` was filed alongside `echidna#146` (the Wave-1 Chapel metalayer rehabilitation) and `proven#126` (the Wave-2 follow-on). Each Wave-N issue applies the same shape: rewrite for chpl 2.8.0 compatibility, decouple the FFI smoke from the metalayer, flip CI to strict, and decide whether the Wave-1 `ParallelSoundness.agda` invariants apply to this repo's parallel-dispatch surface. - -## Decision - -Port the echidna#146 pattern with **two scope adjustments**: - -1. **No `ParallelSoundness.agda` port.** Docudactyl's parallel-dispatch surface is `forall idx in dynamic(docEntries.domain, chunkSize)` in `DocudactylHPC.chpl` — embarrassingly-parallel work distribution with deterministic per-locale-slot aggregation, not speculative search. Echidna's three theorems (first-success-wins soundness, completeness under retry, cancellation safety) are inapplicable. Docudactyl's relevant invariants — at-most-once under checkpoint resume, aggregation commutativity, L1/L2 cache coherence — are documented below and deferred to a follow-up Agda module if formal proofs become load-bearing. - -2. **The CI `check-abi` job retains `continue-on-error: true`.** This is documented as environmental fragility (Idris2 source bootstrap on ubuntu-24.04, not docudactyl correctness). The new `build-smoke` job is the strict gate the echidna pattern actually wants. - -## What landed - -### Chapel 2.8.0 compatibility fixes - -`chpl --no-codegen src/chapel/DocudactylHPC.chpl src/chapel/*.chpl` was failing with three errors before this PR; all three are now resolved. The metalayer compiles clean on chpl 2.8.0 (LLVM-built local toolchain) with only known unstable-API warnings remaining (`_pad` symbol prefixes; `dmapped`; `DynamicIters`; `string.c_str()`; cross-locale `c_addrOf`). - -| Site | Pre-state | Fix | -|---|---|---| -| `DocudactylHPC.chpl:303, 311, 373` | `FileSystem.stat(string)` removed in chpl 2.8.0 | `use OS.POSIX;` + `struct_stat` + `stat(path.c_str(), c_ptrTo(sb))`; field reads via `sb.st_mtim.tv_sec` / `sb.st_size` | -| `DocudactylHPC.chpl:211` | `forall` task-private intent made outer `var ndjsonWriter` const-shadow; `proc ref writeResult` rejected the const actual | `forall … with (ref ndjsonWriter) { … }` (matches the `begin with (ref timer)` pattern already at line 201) | -| `NdjsonManifest.chpl:91` | `string[range]` slicing now `throws` (UTF-8 boundary check) | `try { return line[range]; } catch { return ""; }` — preserves the existing "return empty on error" contract | -| `NdjsonManifest.chpl:177, 182` | `string.createCopyingBuffer(c_ptrConst(c_char))` now `throws` | `try { … } catch { }` around each optional field — the SHA-256 / title fields are skipped in the rare malformed-UTF-8 case so the `forall` task is not torn down | - -### Decoupled FFI smoke - -`src/chapel/smoke.chpl` (46 lines) exercises `ddac_version` / `ddac_crypto_sha256_name` / `ddac_init` / `ddac_free` against the FFI bridge **without** pulling in the full `DocudactylHPC` metalayer. This is the echidna#146 invariant: a green smoke proves the C ABI compiles + links + returns sane data even when the metalayer is broken, so regression localisation is fast. - -The smoke is wired three places: - -- `Justfile` recipes: `check-smoke` (parse-only), `build-smoke` (binary), `run-smoke` (executes the binary) -- `.github/workflows/hpc-ci.yml` job `build-smoke`: depends on `build-ffi`, strict (no `continue-on-error`), greps for `^\[smoke\] PASS$` in the binary output -- `Justfile :: check-chapel` now uses `--main-module DocudactylHPC` to disambiguate the two `proc main()` files (DocudactylHPC + smoke) when globbing `src/chapel/*.chpl` - -### CI floor bumped - -`CHAPEL_VERSION: '2.3.0'` → `'2.8.0'` in `hpc-ci.yml`. The OS.POSIX migration is the proximate cause; the wider justification is that 2.8.0 is the version the rehabilitation pattern was validated against in Wave 1. - -## Parallel-dispatch invariants (informal — deferred to follow-up if formalised) - -These document the invariants the docudactyl `forall idx in dynamic(...)` loop relies on. They are NOT proved here; they are recorded so a future Agda module has a target. - -1. **At-most-once** — every document index in `docEntries.domain` is processed at most once. Currently enforced per-locale by `isAlreadyProcessed(idx)` (Checkpoint.chpl) gating on `recordCheckpoint`. Caveat: cross-locale resume after a topology change (e.g. 64 → 128 locales) is NOT global-atomic. -2. **Abort-bounded** — beyond a failure-rate threshold (FaultHandler.chpl), the loop short-circuits; unprocessed indices are NOT silently dropped, they remain unprocessed and observable via the report. -3. **Aggregation commutativity** — `accumulate(result)` writes only to `perLocaleStats[here.id]` (per-locale slot, no cross-task race); `computeGlobal` reduces by integer addition, which is commutative. SATISFIED by construction. -4. **Cache coherence (L1)** — `(path, mtime, size)`-keyed LMDB lookups in the per-locale L1 are read-then-write within a single task; no concurrent writers to the same key. -5. **Cache coherence (L2)** — Dragonfly L2 is shared across locales and keyed by SHA-256; concurrent `store` calls for the same key are tolerated because the value is deterministic (same SHA-256 implies same parse result). Not formally proven. -6. **Checkpoint resumability** — resuming from a `recordCheckpoint(idx)` snapshot reproduces the same `succeededDocs / failedDocs` global stats modulo per-locale ordering. Stats are aggregated by addition, so order does not matter. -7. **Content-type determinism** — magic-byte detection (`conduit.content_kind`) is deterministic per input file. - -A follow-up issue should be filed if these need formal mechanisation (Agda module path `proofs/agda/DistributedAggregationInvariants.agda`); echidna's `ParallelSoundness.agda` cannot be imported unchanged. - -## Wave-2 gate - -Per `docudactyl#29`, this PR was deferred until `proven#126` (Wave 2) "lands". At time of writing `proven#126` is still OPEN. The owner-authorised exception: the echidna#146 + proven#135 (binding-tier-1 detachable harness) pair has stabilised the rehabilitation shape; a docudactyl rehab now does not risk pattern-divergence. Should Wave 2 settle on a different shape, this ADR is the supersedable surface — the smoke target and CI gate are the load-bearing pieces that would change. - -## Consequences - -- `chpl 2.8.0` is now the floor in CI; running on `2.3.0` will fail the OS.POSIX import. -- The metalayer build remains the same shape; only the three stat sites + `with (ref ndjsonWriter)` clause + the two throws-wrappers in NdjsonManifest differ. -- The new `build-smoke` job runs in parallel with `build-chapel` (both `needs: build-ffi`), so wallclock CI time does not regress. -- `check-abi` still flagged fragile; not changed in this PR. - -## References - -- [echidna#146](https://github.com/hyperpolymath/echidna/pull/146) — Wave-1 metalayer rehabilitation; canonical pattern source -- [proven#126](https://github.com/hyperpolymath/proven/issues/126) — Wave-2 tracker -- [docudactyl#29](https://github.com/hyperpolymath/docudactyl/issues/29) — this issue -- chpl 2.8.0 `OS.POSIX.stat` — `/usr/share/chapel/2.8/modules/standard/OS.chpl:929-965` diff --git a/docs/tech-debt-2026-05-26.adoc b/docs/tech-debt-2026-05-26.adoc new file mode 100644 index 0000000..8d7420e --- /dev/null +++ b/docs/tech-debt-2026-05-26.adoc @@ -0,0 +1,70 @@ +== Tech-Debt Audit — docudactyl — 2026-05-26 + +*Source:* estate-wide automated scan 2026-05-26. *Companion:* +https://github.com/hyperpolymath/standards/tree/main/docs/audits[`+hyperpolymath/standards+` +2026-05-26-estate-*-debt audits]. *Combined severity:* `+2026-05-26+`. + +This file records the _raw findings_ — it does not by itself fix the +debt. Each section ends with a '`Recommended next move`' line; closing +the debt is follow-up work. + +=== 1. Proof debt + +Scanner counted the following markers in proof-bearing files of this +repo: + +.... +files= 11 | Coq-Axm/Adm= 0 | Lean-srry/ax= 0 | Agda-pst= 0 | Idr-blv= 0 | Idr-prtl= 0 | Fstr-asm= 0 | TODO= 0 | Unsafe= 0 +.... + +*Total markers:* 0. *Severity:* `+>00+`. + +*Recommended next move:* none — no proof-debt markers detected. + +=== 2. Licence debt + +[cols=",",options="header",] +|=== +|Field |Value +|LICENSE file |`+LICENSE+` +|SPDX header |`+MPL-2.0+` +|Manifest licence |`+MPL-2.0+` +|Body classifier |`+Palimp-MPL-2.0+` +|Severity |`+ok+` +|=== + +*Recommended next move:* none for licence. + +=== 3. Documentation debt + +[cols=",",options="header",] +|=== +|Field |Value +|README lines |240 +|`+docs/+` files |6 +|`+docs/+` LoC |817 +|CHANGELOG.md |N +|CONTRIBUTING.md |Y +|CODE_OF_CONDUCT.md |Y +|SECURITY.md |Y +|Severity |`+readme=240 docs=6/817+` +|=== + +Additionally: *CHANGELOG.md is missing.* 65% of estate repos lack one — +adopting a CHANGELOG (or auto-generating via `+git-cliff+`) is a +recommended estate-wide follow-up. + +=== Cross-references + +* Estate proof-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-proof-debt.md+` +* Estate licence-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-licence-debt.md+` +* Estate documentation-debt audit: +`+hyperpolymath/standards/docs/audits/2026-05-26-estate-documentation-debt.md+` + +''''' + +🤖 Generated by Claude Code estate-wide tech-debt scan (2026-05-26). +This file is informational — closing the debt is follow-up work owned by +the maintainer. diff --git a/docs/tech-debt-2026-05-26.md b/docs/tech-debt-2026-05-26.md deleted file mode 100644 index 87f0e86..0000000 --- a/docs/tech-debt-2026-05-26.md +++ /dev/null @@ -1,62 +0,0 @@ - - -# Tech-Debt Audit — docudactyl — 2026-05-26 - -**Source:** estate-wide automated scan 2026-05-26. -**Companion:** [`hyperpolymath/standards` 2026-05-26-estate-*-debt audits](https://github.com/hyperpolymath/standards/tree/main/docs/audits). -**Combined severity:** `2026-05-26`. - -This file records the *raw findings* — it does not by itself fix the debt. Each section ends with a 'Recommended next move' line; closing the debt is follow-up work. - -## 1. Proof debt - -Scanner counted the following markers in proof-bearing files of this repo: - -``` -files= 11 | Coq-Axm/Adm= 0 | Lean-srry/ax= 0 | Agda-pst= 0 | Idr-blv= 0 | Idr-prtl= 0 | Fstr-asm= 0 | TODO= 0 | Unsafe= 0 -``` - -**Total markers:** 0. **Severity:** `>00`. - -**Recommended next move:** none — no proof-debt markers detected. - -## 2. Licence debt - -| Field | Value | -|---|---| -| LICENSE file | `LICENSE` | -| SPDX header | `MPL-2.0` | -| Manifest licence | `MPL-2.0` | -| Body classifier | `Palimp-MPL-2.0` | -| Severity | `ok` | - -**Recommended next move:** none for licence. - -## 3. Documentation debt - -| Field | Value | -|---|---| -| README lines | 240 | -| `docs/` files | 6 | -| `docs/` LoC | 817 | -| CHANGELOG.md | N | -| CONTRIBUTING.md | Y | -| CODE_OF_CONDUCT.md | Y | -| SECURITY.md | Y | -| Severity | `readme=240 docs=6/817` | - - -Additionally: **CHANGELOG.md is missing.** 65% of estate repos lack one — adopting a CHANGELOG (or auto-generating via `git-cliff`) is a recommended estate-wide follow-up. - -## Cross-references - -- Estate proof-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-proof-debt.md` -- Estate licence-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-licence-debt.md` -- Estate documentation-debt audit: `hyperpolymath/standards/docs/audits/2026-05-26-estate-documentation-debt.md` - ---- - -🤖 Generated by Claude Code estate-wide tech-debt scan (2026-05-26). This file is informational — closing the debt is follow-up work owned by the maintainer. diff --git a/examples/web-project-deno.json b/examples/web-project-deno.json index 5ddd3bd..ee775a4 100644 --- a/examples/web-project-deno.json +++ b/examples/web-project-deno.json @@ -1,17 +1,17 @@ { - "// NOTE": "Example deno.json for ReScript web projects", + "// NOTE": "Example deno.json for AffineScript web projects", "tasks": { - "build": "deno run -A npm:rescript", - "clean": "deno run -A npm:rescript clean", - "watch": "deno run -A npm:rescript -w", + "build": "deno run -A npm:affinescript", + "clean": "deno run -A npm:affinescript clean", + "watch": "deno run -A npm:affinescript -w", "serve": "deno run -A jsr:@std/http/file-server .", "test": "deno test --allow-all" }, "imports": { - "rescript": "^12.0.0", - "@rescript/core": "npm:@rescript/core@^1.6.0", - "safe-dom/": "https://raw.githubusercontent.com/hyperpolymath/rescript-dom-mounter/main/src/", - "proven/": "../proven/bindings/rescript/src/" + "affinescript": "^12.0.0", + "@affinescript/core": "npm:@affinescript/core@^1.6.0", + "safe-dom/": "https://raw.githubusercontent.com/hyperpolymath/affinescript-dom-mounter/main/src/", + "proven/": "../proven/bindings/affinescript/src/" }, "compilerOptions": { "allowJs": true, diff --git a/llm-warmup-dev.adoc b/llm-warmup-dev.adoc new file mode 100644 index 0000000..562b1a4 --- /dev/null +++ b/llm-warmup-dev.adoc @@ -0,0 +1,19 @@ +== LLM Warmup — docudactyl (Developer) + +=== What is docudactyl? + +See README.adoc for overview. + +=== Key Commands + +* `+just setup+` — set up development environment +* `+just build+` — build the project +* `+just test+` — run tests +* `+just doctor+` — diagnose issues +* `+just heal+` — attempt auto-repair + +=== Quick Context + +* License: MPL-2.0 +* Part of hyperpolymath ecosystem +* See EXPLAINME.adoc for architecture diff --git a/llm-warmup-dev.md b/llm-warmup-dev.md deleted file mode 100644 index 0d95671..0000000 --- a/llm-warmup-dev.md +++ /dev/null @@ -1,16 +0,0 @@ -# LLM Warmup — docudactyl (Developer) - -## What is docudactyl? -See README.adoc for overview. - -## Key Commands -- `just setup` — set up development environment -- `just build` — build the project -- `just test` — run tests -- `just doctor` — diagnose issues -- `just heal` — attempt auto-repair - -## Quick Context -- License: MPL-2.0 -- Part of hyperpolymath ecosystem -- See EXPLAINME.adoc for architecture diff --git a/llm-warmup-user.adoc b/llm-warmup-user.adoc new file mode 100644 index 0000000..afbfd7d --- /dev/null +++ b/llm-warmup-user.adoc @@ -0,0 +1,19 @@ +== LLM Warmup — docudactyl (User) + +=== What is docudactyl? + +See README.adoc for overview. + +=== Key Commands + +* `+just setup+` — set up development environment +* `+just build+` — build the project +* `+just test+` — run tests +* `+just doctor+` — diagnose issues +* `+just heal+` — attempt auto-repair + +=== Quick Context + +* License: MPL-2.0 +* Part of hyperpolymath ecosystem +* See EXPLAINME.adoc for architecture diff --git a/llm-warmup-user.md b/llm-warmup-user.md deleted file mode 100644 index df83a77..0000000 --- a/llm-warmup-user.md +++ /dev/null @@ -1,16 +0,0 @@ -# LLM Warmup — docudactyl (User) - -## What is docudactyl? -See README.adoc for overview. - -## Key Commands -- `just setup` — set up development environment -- `just build` — build the project -- `just test` — run tests -- `just doctor` — diagnose issues -- `just heal` — attempt auto-repair - -## Quick Context -- License: MPL-2.0 -- Part of hyperpolymath ecosystem -- See EXPLAINME.adoc for architecture