SLURM job folders with per-attempt subfolders; SLURM results marked in History

CHANGELOG.md

@@ -9,6 +9,18 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
 
 ### Added
 
+- **SLURM job folders with per-attempt subfolders (M-JOBDIRS)** — each cluster
+  job gets a readable folder (`<formula>_<calc>_<method>_<basis>`, or the
+  optional **Job name** on the Calculate tab) that doubles as the SLURM job
+  name. Every run of its `submit.slurm` creates a new `attempt-NN_job<id>/`
+  subfolder, so **Resubmit** (new, Cluster Jobs tab) and a hand-run
+  `sbatch submit.slurm` never overwrite an earlier attempt. The job root is
+  configurable in **System Settings** (or `QUANTUI_STAGING_DIR`), for
+  clusters with small home quotas.
+- **SLURM results marked in History** — entries show
+  `🖥 SLURM <job id>·a<attempt>` and the result card has a "Ran on" row with
+  the job folder. Hand-run attempts are added to History on the next Cluster
+  Jobs refresh; failed attempts stay in the job folder only.
 - **PyFock geometry and analysis phase** — native-Windows PBE/def2 geometry
   optimization now uses PyFock's ASE calculator with analytical density-fitted
   gradients. Single-point and final optimized-geometry results retain orbital
@@ -21,6 +33,13 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht
   parity gate against PySCF. Unsupported hybrids, ions, open-shell systems,
   solvent, checkpoints, GPU, and orbital analysis are rejected before compute.
 
+### Fixed
+
+- Reconnecting to a finished SLURM job no longer saves a second copy of its
+  result to History.
+- Removed the unused `QUANTUI_RESULTS_DIR` export from generated SLURM
+  scripts (it only created an empty `results/` folder per job).
+
 ## [0.8.2] - 2026-08-30
 
 ### Added

apptainer/slurm/README.md

@@ -9,7 +9,37 @@ Reference scripts for operator-driven NCShare / Apptainer batch jobs.
 | `quantui-batch.sbatch` | Reference `sbatch` template (partition must be set) |
 | `quantui-gpu-test.sbatch` | GPU smoke test (operator use) |
 
-QuantUI generates per-job scripts automatically under `~/.quantui/staging/<request_id>/submit.slurm` when students submit from the UI.
+QuantUI generates per-job scripts automatically when students submit from the UI.
+
+## Job folders
+
+Each submission gets its own folder under the job root (default
+`~/.quantui/staging`; set it in **System Settings → SLURM job folder** or with
+`QUANTUI_STAGING_DIR`):
+
+```
+<job root>/H2O_opt_B3LYP_def2-SVP/        # job name = SLURM --job-name
+    request.json   submit.slurm
+    slurm-812345.out  slurm-812345.err   # SLURM's own output, one pair per job
+    attempt-01_job812345/                # first run
+        live.log  progress.json  ...
+    attempt-02_job812399/                # a rerun, never overwrites attempt-01
+        result.json  orbitals.npz  H2O_opt_B3LYP_def2-SVP.molden  ...
+    latest -> attempt-02_job812399
+```
+
+- The job name defaults to `<formula>_<calc>_<method>_<basis>`; the optional
+  **Job name** field on the Calculate tab (or `quantui submit --job-name`)
+  overrides it. A name already in use gets `_2`, `_3`, ....
+- `submit.slurm` creates a new `attempt-NN_job<SLURM id>/` folder every time it
+  runs, so both **Resubmit** on the Cluster Jobs tab and a hand-run
+  `sbatch submit.slurm` from the job folder start a fresh attempt. Earlier
+  attempts are never overwritten, and checkpoints let a rerun resume.
+- Every successful attempt appears in **History**, marked `🖥 SLURM <job id>·a<attempt>`,
+  including hand-run ones (picked up on the next Cluster Jobs refresh).
+  Failed attempts stay in the job folder only.
+- Jobs submitted before per-job folders existed keep their
+  `~/.quantui/staging/<request_id>/` layout and cannot be resubmitted in place.
 
 Status polling uses **`squeue`** for active jobs and **`sacct`** for terminal state, exit code, and cancel confirmation. Cluster Jobs **Remove** clears terminal registry rows without deleting staging logs.
 
@@ -26,6 +56,7 @@ See the [NCShare SLURM batch runbook](https://github.com/The-Schultz-Lab/QuantUI
 | `QUANTUI_SLURM_CANCEL_CONFIRM_S` | `30` | Seconds to wait for `scancel` confirmation via `sacct` |
 | `QUANTUI_SLURM_PARTITION` | `common` | Default `#SBATCH` partition |
 | `QUANTUI_BATCH_IMAGE` | `~/quantui-gpu.sif` | Apptainer image for batch worker |
+| `QUANTUI_STAGING_DIR` | *(unset — System Settings value, else `~/.quantui/staging`)* | Job folder root. Overrides the System Settings value and locks that field. A root outside `$HOME` is bound into Apptainer automatically. |
 
 ## Operator runbook
 
@@ -36,7 +67,11 @@ See the planning repo runbook:
 ## Worker entrypoint
 
 ```bash
-python -m quantui.backends.worker --request /path/to/staging/request.json
+python -m quantui.backends.worker --request /path/to/job/request.json \
+    --attempt-dir /path/to/job/attempt-01_job812345
 ```
 
+`submit.slurm` passes `--attempt-dir` itself. Without it, outputs go next to
+`request.json` (the pre-job-folder layout).
+
 Supported calc types: `single_point`, `geometry_opt`, `frequency`, `tddft`, `nmr`, `pes_scan`, `reorganization_energy`.

quantui/app.py

@@ -406,6 +406,9 @@
 from quantui.app_runflow import (
     update_scan_widgets as _run_update_scan_widgets,
 )
+from quantui.app_slurm import (
+    on_slurm_job_root_changed as _slurm_on_job_root_changed,
+)
 from quantui.app_slurm import (
     on_slurm_jobs_cancel_clicked as _slurm_on_jobs_cancel_clicked,
 )
@@ -415,6 +418,9 @@
 from quantui.app_slurm import (
     on_slurm_jobs_remove_clicked as _slurm_on_jobs_remove_clicked,
 )
+from quantui.app_slurm import (
+    on_slurm_jobs_resubmit_clicked as _slurm_on_jobs_resubmit_clicked,
+)
 from quantui.app_slurm import (
     on_slurm_jobs_view_clicked as _slurm_on_jobs_view_clicked,
 )
@@ -568,6 +574,7 @@
 from quantui.app_xyz_input import (
     on_xyz_fill_table as _xyz_on_fill_table,
 )
+from quantui.backends import cluster_config as _cluster_cfg
 from quantui.backends.dispatch import is_slurm_available
 from quantui.cancellation import CalcCancelled as _CalcCancelled
 
@@ -1440,6 +1447,10 @@ class QuantUIApp:
         density_fit_enabled_cb: Any
         freq_parallel_enabled_cb: Any
         execution_backend_dd: Any
+        slurm_job_root_txt: Any
+        _slurm_job_name_txt: Any
+        _slurm_job_name_row: Any
+        slurm_job_root_note: Any
         quantum_engine_dd: Any
         quantum_engine_note: Any
         engine_capability_html: Any
@@ -1491,6 +1502,7 @@ class QuantUIApp:
         _slurm_jobs_view_btn: Any
         _slurm_jobs_cancel_btn: Any
         _slurm_jobs_remove_btn: Any
+        _slurm_jobs_resubmit_btn: Any
         _slurm_jobs_status_html: Any
         slurm_jobs_tab_panel: Any
         _slurm_jobs_tab_index: int | None
@@ -2124,6 +2136,9 @@ def _build_status_panel(self) -> None:
             execution_backend=self._user_settings.compute.execution_backend,
             slurm_available=is_slurm_available(),
             quantum_engine=self._user_settings.compute.quantum_engine,
+            slurm_job_root=self._user_settings.compute.slurm_job_root,
+            slurm_job_root_env_locked=_cluster_cfg.staging_root_env_configured(),
+            slurm_job_root_effective=str(_cluster_cfg.default_staging_root()),
         )
 
     # ── Welcome header ────────────────────────────────────────────────────
@@ -2464,6 +2479,9 @@ def _sync_root_tab_layout(self) -> None:
         self._slurm_jobs_tab_index = (
             order.index("slurm_jobs") if "slurm_jobs" in order else None
         )
+        self._slurm_job_name_row.layout.display = (
+            "flex" if "slurm_jobs" in order else "none"
+        )
         self._root_tab_order_cache = list(order)
 
         if "slurm_jobs" in order:
@@ -2537,6 +2555,12 @@ def _wire_callbacks(self) -> None:
         self.execution_backend_dd.observe(
             self._safe_cb(self._on_execution_backend_changed), names="value"
         )
+        self.slurm_job_root_txt.observe(
+            self._safe_cb(
+                lambda change: _slurm_on_job_root_changed(self, change["new"])
+            ),
+            names="value",
+        )
         self.quantum_engine_dd.observe(
             self._safe_cb(self._on_quantum_engine_changed), names="value"
         )
@@ -2650,6 +2674,9 @@ def _wire_callbacks(self) -> None:
         self._slurm_jobs_remove_btn.on_click(
             self._safe_cb(lambda _btn: _slurm_on_jobs_remove_clicked(self, _btn))
         )
+        self._slurm_jobs_resubmit_btn.on_click(
+            self._safe_cb(lambda _btn: _slurm_on_jobs_resubmit_clicked(self, _btn))
+        )
         self.cancel_btn.on_click(self._safe_cb(self._on_cancel))
         self.basis_fix_btn.on_click(self._safe_cb(self._on_basis_fix))
         self.charge_mult_suggest_btn.on_click(

quantui/app_builders.py

@@ -128,6 +128,9 @@ def build_status_panel(
     execution_backend: str = "local",
     slurm_available: bool = False,
     quantum_engine: str = "auto",
+    slurm_job_root: str = "",
+    slurm_job_root_env_locked: bool = False,
+    slurm_job_root_effective: str = "",
 ) -> None:
     """Build the Status tab panel."""
     cores, mem_gb = get_session_resources_fn()
@@ -407,6 +410,44 @@ def _render_status(gpu_state: Any) -> str:
             "Use the Cluster Jobs tab to monitor and cancel runs.</div>"
         )
 
+    # SLURM job folder root (M-JOBDIRS JD.1). Built unconditionally so the
+    # app can wire it; only shown when SLURM is available.
+    app.slurm_job_root_txt = widgets.Text(
+        value=slurm_job_root_effective if slurm_job_root_env_locked else slurm_job_root,
+        placeholder="~/.quantui/staging (default)",
+        continuous_update=False,
+        disabled=slurm_job_root_env_locked,
+        layout=layout_fn(width="420px"),
+    )
+    if slurm_job_root_env_locked:
+        _root_note_text = (
+            "Set by <code>QUANTUI_STAGING_DIR</code> in the container or "
+            "session environment."
+        )
+    else:
+        _root_note_text = (
+            "Each cluster job gets its own folder here, with one subfolder per "
+            "attempt. Leave blank for the default; on clusters with a small "
+            "home quota, use a scratch or project folder. Applies to new "
+            "submissions."
+        )
+    app.slurm_job_root_note = widgets.HTML(
+        f'<div style="font-size:11px;color:{_theme.css.TEXT_SUBTLE};margin:2px 0 0 0">'
+        f"{_root_note_text}</div>"
+    )
+    slurm_root_rows: list[Any] = []
+    if slurm_available:
+        slurm_root_rows = [
+            widgets.HTML(
+                f'<div style="font-size:12px;color:{_theme.css.TEXT_SLATE_DARK};'
+                'margin-top:12px;margin-bottom:0px">SLURM job folder '
+                f'<span style="color:{_theme.css.TEXT_SUBTLE};font-size:11px">'
+                "(persists across launches)</span></div>"
+            ),
+            app.slurm_job_root_txt,
+            app.slurm_job_root_note,
+        ]
+
     fp_settings_rows: list[Any] = [
         fp_toggle_label,
         app.freq_parallel_enabled_cb,
@@ -431,6 +472,7 @@ def _render_status(gpu_state: Any) -> str:
             exec_backend_label,
             app.execution_backend_dd,
             exec_backend_note,
+            *slurm_root_rows,
         ],
         layout=layout_fn(margin="0 0 8px 0"),
     )
@@ -1485,6 +1527,31 @@ def build_shared_widgets(
     # native phase (first gradient / Hessian) visibly advances.
     app._run_elapsed_lbl = widgets.HTML(value="")
 
+    # Optional job folder / SLURM job name (M-JOBDIRS JD.2). Shown only in
+    # SLURM mode; blank uses <formula>_<calc>_<method>_<basis>.
+    app._slurm_job_name_txt = widgets.Text(
+        value="",
+        placeholder="default: <formula>_<calc>_<method>_<basis>",
+        description="Job name:",
+        style={"description_width": "80px"},
+        layout=layout_fn(width="460px"),
+        tooltip=(
+            "Names this job's folder and its SLURM job. A name already in use "
+            "gets _2, _3, ... appended."
+        ),
+    )
+    app._slurm_job_name_row = widgets.VBox(
+        [
+            app._slurm_job_name_txt,
+            widgets.HTML(
+                f'<div style="font-size:11px;color:{_theme.css.TEXT_SUBTLE};'
+                'margin:0 0 6px 84px">Optional. Letters, digits, - and _ are kept; '
+                "other characters become _.</div>"
+            ),
+        ],
+        layout=layout_fn(display="none"),
+    )
+
     app._slurm_job_banner = widgets.HTML(value="", layout=layout_fn(display="none"))
     app._slurm_reconnect_btn = widgets.Button(
         description="View SLURM progress",
@@ -1994,6 +2061,7 @@ def build_run_section(app: Any, *, layout_fn: Any) -> None:
             app.perf_estimate_html,
             app._resume_notice_html,
             app._resume_cb,
+            app._slurm_job_name_row,
             app._slurm_job_banner,
             widgets.HBox(
                 [app._slurm_reconnect_btn],
@@ -3456,6 +3524,15 @@ def build_slurm_jobs_tab(app: Any, *, layout_fn: Any) -> None:
         layout=layout_fn(width="120px"),
         tooltip="Cancel the selected active cluster job",
     )
+    app._slurm_jobs_resubmit_btn = widgets.Button(
+        description="Resubmit",
+        icon="redo",
+        layout=layout_fn(width="120px"),
+        tooltip=(
+            "Run a finished or failed job again in the same job folder, as a "
+            "new attempt. Earlier attempts' files are kept."
+        ),
+    )
     app._slurm_jobs_remove_btn = widgets.Button(
         description="Remove",
         icon="trash",
@@ -3465,7 +3542,7 @@ def build_slurm_jobs_tab(app: Any, *, layout_fn: Any) -> None:
     app._slurm_jobs_status_html = widgets.HTML(
         value=(
             f'<span style="font-size:12px;color:{_theme.css.TEXT_SUBTLE}">'
-            "Select a job and use View progress or Cancel.</span>"
+            "Select a job and use View progress, Cancel, or Resubmit.</span>"
         )
     )
 
@@ -3485,6 +3562,7 @@ def build_slurm_jobs_tab(app: Any, *, layout_fn: Any) -> None:
                 [
                     app._slurm_jobs_view_btn,
                     app._slurm_jobs_cancel_btn,
+                    app._slurm_jobs_resubmit_btn,
                     app._slurm_jobs_remove_btn,
                 ],
                 layout=layout_fn(gap="8px", margin="6px 0"),

quantui/app_formatters.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import html
 from pathlib import Path
 from typing import Any, Optional
 
@@ -815,6 +816,49 @@ def format_reorg_result(r: Any) -> str:
     )
 
 
+def slurm_history_marker(data: dict[str, Any]) -> str:
+    """Short History-list marker for a SLURM result, e.g. ``🖥 SLURM 812345·a2 ``.
+
+    Empty for local results and for results saved before SLURM provenance
+    was recorded (M-JOBDIRS JD.6).
+    """
+    if data.get("execution_backend") != "slurm":
+        return ""
+    info = data.get("slurm") or {}
+    job_id = info.get("job_id")
+    attempt = info.get("attempt")
+    text = "🖥 SLURM"
+    if job_id:
+        text += f" {job_id}"
+        if attempt:
+            text += f"·a{attempt}"
+    return text + " "
+
+
+def _slurm_provenance_row(data: dict[str, Any]) -> str:
+    """History-card row naming the SLURM job, attempt and job folder."""
+    if data.get("execution_backend") != "slurm":
+        return ""
+    info = data.get("slurm") or {}
+    parts = ["SLURM batch"]
+    if info.get("job_id"):
+        parts.append(f"job {html.escape(str(info['job_id']))}")
+    if info.get("attempt"):
+        parts.append(f"attempt {int(info['attempt'])}")
+    value = " · ".join(parts)
+    folder = info.get("attempt_dir") or info.get("job_dir")
+    if folder:
+        value += (
+            f'<br><span style="font-family:monospace;font-size:12px;'
+            f'color:{_theme.css.TEXT_MUTED_LIGHT}">{html.escape(str(folder))}</span>'
+        )
+    return (
+        f'<tr><td style="padding:3px 18px 3px 0;color:{_theme.css.TEXT_LABEL};'
+        f'vertical-align:top">Ran on</td>'
+        f'<td style="color:{_theme.css.TEXT_HEADING}">{value}</td></tr>'
+    )
+
+
 def format_past_result(data: dict[str, Any], result_dir: Optional[Path] = None) -> str:
     """Format a saved result.json payload as an HTML result card."""
     import base64 as _b64
@@ -912,6 +956,7 @@ def format_past_result(data: dict[str, Any], result_dir: Optional[Path] = None)
     # Shared 'extra' rows (correlation breakdown / solvent / device / dipole /
     # Mulliken) — same builder as the live card so the two never drift.
     _extra = _result_extra_rows(lambda k, d=None: data.get(k, d))
+    _extra += _slurm_provenance_row(data)
 
     # Embed thumbnail if saved
     _thumb_html = ""