Initial draft

.agents/skills/alpine.md

@@ -0,0 +1,162 @@
+# Alpine Skill
+
+Use this note before testing formascute on CURC Alpine.
+
+## Connection
+
+From the local machine, use the `ssh-alpine` zsh alias:
+
+```bash
+zsh -lic 'ssh-alpine "hostname; pwd"'
+```
+
+The alias should be defined in the user's shell configuration as an SSH command
+to `login.rc.colorado.edu` using that user's Alpine SSH key.
+
+`Persistence1` is not directly resolvable from the local machine. It is reachable
+from the Alpine login node:
+
+```bash
+zsh -lic 'ssh-alpine "ssh Persistence1 hostname"'
+```
+
+## Module Behavior
+
+The regular Alpine login node did not expose a usable Nextflow module during
+validation. `Persistence1` did expose Nextflow modules:
+
+- `nextflow/22.10.6`
+- `nextflow/23.04`
+- `nextflow/24.04.4`
+- `nextflow/25.10.2`
+
+The validated module is:
+
+```bash
+module load nextflow/25.10.2
+nextflow -version
+```
+
+On `Persistence1`, `/usr/bin/apptainer` and `/usr/bin/singularity` were both
+available and reported Apptainer 1.4.5.
+
+## Slurm Defaults
+
+Use these defaults for small synthetic CPU tests:
+
+- account: project allocation supplied by the user
+- partition: `acpu`
+- QoS: `cpu-normal`
+- submit host: `Persistence1`
+
+Slurm accepted `acpu` and `cpu-normal` and mapped them to the current CPU
+partition backing the older `amilan`/`normal` names.
+
+`Persistence1` is the selected submission location for this project. Treat it as
+known, not as an open question.
+
+## Validated Submission Path
+
+From a clone on Alpine shared scratch:
+
+```bash
+make check
+make preflight ACCOUNT=<allocation>
+make submit-dry-run EXPERIMENT=nf1_featurization_independent ITEMS=4 ACCOUNT=<allocation> SUBMIT_HOST=Persistence1
+make submit EXPERIMENT=nf1_featurization_independent ITEMS=4 ACCOUNT=<allocation> SUBMIT_HOST=Persistence1
+```
+
+The successful validation run used:
+
+```bash
+make submit EXPERIMENT=nf1_featurization_independent ITEMS=4 ACCOUNT=<allocation> SUBMIT_HOST=Persistence1 RUN_ID=remote-smoke-persistence
+```
+
+Observed result:
+
+- coordinator Slurm job completed
+- `completion_status.txt` reported `nextflow_exit_status: 0`
+- `validation.json` reported `"valid": true`
+- `trace.tsv` contained native Slurm job IDs
+- `slurm.tsv` was collected
+
+## Synthetic Experiment Findings
+
+Small synthetic runs completed successfully through `Persistence1`.
+
+Useful observed shape:
+
+- one-item minimal overhead: about 21 seconds
+- independent 16-item fan-out: about 51-52 seconds, 17 native Slurm jobs
+- batch size 2 with 16 items: about 42 seconds, 9 native Slurm jobs
+- batch size 4 with 16 items: about 41 seconds, 5 native Slurm jobs
+- `queueSize=4` with 16 independent items: about 1 minute
+- `submitRateLimit='4 / 1 min'`: about 4 minutes and should not be the first
+  tuning knob
+- queue wait was negligible in tiny tests, usually 0-1 seconds
+
+Interpretation:
+
+- Nextflow can submit, track, validate, and collect Slurm accounting on Alpine
+  when submitted through `Persistence1`.
+- Batching reduced Slurm job count and modestly improved elapsed time in tiny
+  synthetic tests.
+- Batch sizes 2-4 are the first batching range to consider.
+- Use `queueSize` before `submitRateLimit` for scheduler pressure control.
+- Keep CPU and GPU work separated into different profiles or runs.
+
+For NF1 orchestration, prefer a thin Nextflow layer around the existing Stage 3
+work list and feature scripts before rewriting processing code.
+
+## ZedProfiler Notes
+
+The `zp_synthetic_features` experiment completed successfully on Alpine through
+`Persistence1`:
+
+```bash
+make submit EXPERIMENT=zp_synthetic_features ITEMS=4 ACCOUNT=<allocation> SUBMIT_HOST=Persistence1 RUN_ID=zp-sim-01
+```
+
+Observed result:
+
+- coordinator job completed with Nextflow exit status `0`
+- validation passed for 4 expected items
+- 4 feature tasks plus 1 validation task were submitted
+- 16 simulated feature rows were emitted
+- feature tasks took about 1-3 seconds each
+- peak RSS was about 9-32 MB
+
+This confirms the orchestration shape for tiny CPU-first 3D feature work, but it
+does not validate real ZedProfiler imports or image I/O.
+
+The tested base Python environment on `Persistence1` is not suitable for real
+ZedProfiler:
+
+- Python 3.9.13
+- NumPy 1.21.5
+- Pandas 1.4.4
+- scikit-image 0.19.2
+- missing `mahotas`
+- missing `pyarrow`
+- missing `zedprofiler`
+
+Real ZedProfiler work needs a project-owned Python 3.11+ environment or an
+Apptainer image. For NF1, prefer grouping compatible ZedProfiler feature families
+by image-set when possible so each task loads image data once and writes validated
+outputs.
+
+## Important Failure Mode
+
+Submitting directly from the regular Alpine login node failed because the batch
+job could not load `nextflow/25.10.2`. Use `SUBMIT_HOST=Persistence1` unless the
+module environment changes.
+
+## CURC Questions To Preserve
+
+Ask CURC before scaling:
+
+- What `queueSize` range is recommended for Nextflow on Alpine?
+- Should small NF1-style jobs use moderate independent submission or batching?
+- Are Slurm job arrays recommended, or is Nextflow-managed submission preferred
+  for retries and accounting?
+- Should SAMMed3D GPU work be a separate Nextflow profile and submission path?

.github/workflows/run-checks.yml

@@ -0,0 +1,31 @@
+---
+name: checks
+
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+jobs:
+  prek:
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v7.0.1
+        with:
+          persist-credentials: false
+
+      - name: Install ShellCheck
+        run: sudo apt-get update && sudo apt-get install -y shellcheck
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v6.0.0
+        with:
+          node-version: "24"
+
+      - name: Run prek
+        uses: j178/prek-action@v3.0.0

.gitignore

@@ -181,6 +181,13 @@ dmypy.json
 # Cython debug symbols
 cython_debug/
 
+# Nextflow and FormaScute run outputs
+.nextflow/
+.nextflow.log*
+.formascute/
+work/
+results/
+
 # PyCharm
 #   JetBrains specific template is maintained in a separate JetBrains.gitignore that can
 #   be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore

.markdownlint-cli2.yaml

@@ -0,0 +1,8 @@
+config:
+  MD013: false
+  MD033: false
+globs:
+  - "*.md"
+  - ".agents/**/*.md"
+  - "docs/**/*.md"
+  - ".github/**/*.md"

.pre-commit-config.yaml

@@ -0,0 +1,8 @@
+repos:
+  - repo: local
+    hooks:
+      - id: formascute-check
+        name: formascute lint
+        entry: bash scripts/lint
+        language: system
+        pass_filenames: false

AGENTS.md

@@ -0,0 +1,5 @@
+# Agent Notes
+
+Use `.agents/skills/alpine.md` before working on Alpine validation or Slurm
+submission behavior. It records the connection and module-loading details already
+validated for this project.

CITATION.cff

@@ -0,0 +1,16 @@
+cff-version: 1.2.0
+message: "If you use formascute, please cite this repository."
+title: "formascute"
+abstract: >-
+  Small, reproducible characterization experiments for understanding workflow
+  execution on HPC systems before running production science.
+authors:
+  - name: "formascute contributors"
+license: "BSD-3-Clause"
+type: software
+keywords:
+  - high-performance computing
+  - workflow orchestration
+  - nextflow
+  - slurm
+  - reproducible research

CODE_OF_CONDUCT.md

@@ -0,0 +1,31 @@
+# Code Of Conduct
+
+This project follows a practical standard for respectful, research-focused
+collaboration.
+
+## Expected Behavior
+
+- Be direct, professional, and constructive.
+- Focus technical discussion on evidence, reproducibility, and clear tradeoffs.
+- Assume collaborators are working in good faith.
+- Make uncertainty explicit when interpreting HPC behavior.
+- Respect privacy and operational constraints around shared computing systems.
+
+## Unacceptable Behavior
+
+- Harassment, intimidation, discriminatory language, or personal attacks.
+- Sharing private credentials, SSH keys, tokens, or user-specific access details.
+- Publishing sensitive HPC account, path, allocation, or dataset information.
+- Misrepresenting benchmark-like results as production performance guarantees.
+- Pressuring contributors to run unsafe or inappropriate workloads on shared
+  systems.
+
+## Reporting
+
+Report concerns through the maintainers or the repository issue tracker. If the
+concern involves private access details, credentials, or sensitive system
+information, do not post those details publicly.
+
+Maintainers should respond by clarifying the concern, preserving privacy, and
+taking proportionate action to keep the project useful and safe for
+collaboration.

CONTRIBUTING.md

@@ -0,0 +1,60 @@
+# Contributing
+
+formascute is intentionally small. Contributions should preserve that shape:
+simple entry points, reproducible experiments, explicit validation, and clear
+provenance.
+
+## Development Checks
+
+Run these before opening a pull request:
+
+```bash
+make check
+make lint
+```
+
+If `prek` or `pre-commit` is installed, also run:
+
+```bash
+make prek
+```
+
+`make check` verifies that the repository is ready to clone and use on Alpine.
+`make lint` covers shell syntax, ShellCheck when available, Markdown, and
+best-effort Nextflow config checks.
+
+## Contribution Guidelines
+
+- Keep experiments small and bounded.
+- Do not commit real HPC usernames, private paths, allocation names, SSH key
+  paths, production data paths, or Slurm job identifiers.
+- Use placeholders such as `<allocation>`, `<username>`, and `<repo-url>` in
+  documentation.
+- Do not use production NF1 data in this repository.
+- Add or update documentation when a command, assumption, or Alpine observation
+  changes.
+- Prefer Make targets and `bin/formascute` entry points over one-off scripts.
+- Keep CPU and GPU assumptions separate.
+
+## Experiment Criteria
+
+Every experiment should:
+
+- change one thing relative to the baseline
+- run quickly at small scale
+- validate correctness
+- write outputs under `results/<run-id>/`
+- preserve enough provenance to interpret the run later
+- document remaining uncertainty
+
+## Pull Requests
+
+Pull requests should describe:
+
+- what changed
+- how it was validated
+- whether Alpine behavior or assumptions changed
+- any follow-up questions for HPC support
+
+Do not include generated `results/` outputs unless there is a specific reason to
+review a tiny artifact.