Clone a task's submodules from the repo's own checkout

AGENTS.md

@@ -48,9 +48,10 @@ src/panopticon/
                    # agent pane's own oom_score_adj, `nice` for shell tasks, and the cgroup-flag
                    # strip the runner degrades through on a daemon that refuses them); clones.py =
                    # per-repo clone cache; spawn.py = spawn-prep (clone --local the per-task
-                   # checkout, mounted rw at /workspace; point origin at the forge, then init any
+                   # checkout, mounted rw at /workspace; point origin at the forge, then fill in any
                    # submodules — in that order, since relative .gitmodules URLs resolve against
-                   # origin); spawner.py = the spawn loop (claim an unclaimed task → spawn its
+                   # origin — hardlink-cloning them out of the repo's own checkout on this host when
+                   # git_url names one, ADR 0011 §1c); spawner.py = the spawn loop (claim an unclaimed task → spawn its
                    # container; prefills claude's input box with the task memo on a first spawn);
                    # prefill.py = the detached input-box prefill
                    # poller (mirrors cloude-cade: pipe-pane watch for ESC[?2004h → paste-buffer the
@@ -243,9 +244,14 @@ on every PR (the same commands the Makefile wraps).
 - `tests/test_spawn.py` — spawn-prep (ADR 0011): unit tests pin the `clone --local` of the
   per-task checkout and the idempotency gate (skips when the checkout already exists), plus the
   **submodule** init — emitted after the `origin` repoint, gated on `submodule status` reporting an
-  uninitialized (`-`) submodule so it retries but never touches an initialized one; a `skipif`
-  integration test fills in a real submodule and then **moves** the checkout, pinning that the
-  recorded links stay relative (the ADR 0011 mounts-anywhere property).
+  uninitialized (`-`) submodule so it retries but never touches an initialized one — and the
+  **donor hydration** (ADR 0011 §1c): the per-level init/url-override/update/sync when the repo has
+  a checkout on this host, and every fallback to the plain fetch (no donor, donor gone, hydration
+  raised, a submodule still uninitialized). A `skipif` integration test fills in a real submodule
+  and then **moves** the checkout, pinning that the recorded links stay relative (the ADR 0011
+  mounts-anywhere property); another hydrates a nested submodule from a source repo whose
+  submodules' own repos have been moved away, pinning that the objects are hardlinked from it and
+  that `sync` leaves no donor path behind.
 - `tests/container/test_cli_base.py` — the agent-CLI seam, including the **resume fallback** (ADR
   0014 §4b): with the process runner and clock injected (no CLI is ever started), a resumed launch
   that exits non-zero fast quarantines the session and relaunches — recovering an older healthy

docs/design/BACKLOG.md

@@ -81,13 +81,13 @@ in the ADRs; this file is for the smaller stuff that doesn't have a home there y
   `docker buildx bake` (target inheritance, which maps cleanly onto base→workflow→repo). Not
   needed now; the fragment approach is the minimal thing that works. _(Slice 6, P3)_
 
-- [ ] **Share submodule objects with the cache clone** — spawn-prep initializes a task's
-  submodules from their forge-resolved URLs (ADR 0011 §1b), so every task pays a full submodule
-  fetch. Sharing the repo cache's object store would avoid it, but the obvious lever
-  (`submodule.alternateLocation=superproject`) derives the alternate from the superproject's
-  `origin`, which spawn deliberately repoints at the forge before initializing — so it needs the
-  per-submodule alternate paths (`<cache>/.git/modules/<name>`) passed explicitly, and the cache
-  clone made submodule-aware. Only worth it for repos with large submodules. _(P3)_
+- [ ] **Share submodule objects for forge-hosted repos** — a task whose repo has a checkout on
+  this host now hardlink-clones its submodules out of it (ADR 0011 §1c), but a repo whose `git_url`
+  is a hosted forge has no local donor and still pays a full submodule fetch per task. Making the
+  repo's **cache** clone submodule-aware would give those repos a donor too. The obvious lever
+  (`submodule.alternateLocation=superproject`) isn't it: the alternate is derived from the
+  superproject's `origin`, which spawn deliberately repoints at the forge — so it would be the same
+  path-keyed hydration, pointed at the cache. Only worth it for repos with large submodules. _(P3)_
 
 ## Tracked elsewhere (pointers, do not duplicate)
 

docs/design/decisions/0011-provisioning-per-task-clone.md

@@ -66,11 +66,41 @@ The "mounts at any container path" property survives, because `submodule update`
 submodule's gitdir pointer (`gitdir: ../../.git/modules/<name>`) and its `core.worktree`
 **relatively** — nothing absolute to mirror, same as the superproject.
 
-The repo's **cache** clone stays submodule-free: the per-task clone fetches submodules from their
-forge-resolved URLs, so cache-side submodule checkouts would be disk and time for nothing. The cost
-is one submodule fetch per task; sharing the cache's object store instead
-(`submodule.alternateLocation=superproject`) computes the alternate from the superproject's *origin*,
-which is exactly what we repoint at the forge — so it needs more than a flag. Backlogged.
+### 1c. Submodules come from the repo's own checkout when there is one
+
+Fetching each submodule from its URL is paid by **every** task on the repo, and for a large
+submodule it dominates spawn-prep — while the superproject itself is nearly free (`clone --local`
+hardlinks the cache's objects). When the repo's `git_url` names a checkout on this host (the
+local-git flow), that checkout already holds every submodule's objects, so spawn-prep clones them
+**from it** — a local clone, hardlinked object store, no network (`hydrate_submodules`).
+
+The donor is matched **by path**, never by URL: the donor resolved its relative `.gitmodules` URLs
+against *its own* `origin` while the per-task clone resolves them against `git_url`, so the same
+submodule can legitimately have two different URLs. Per superproject level:
+
+1. `submodule init` — git resolves the declared URLs into `submodule.<name>.url`;
+2. for each submodule the donor has checked out, that resolved URL is overwritten with the donor's
+   path (one the donor lacks keeps its own URL and is simply fetched);
+3. `submodule update` for **this level only** — a nested submodule's URL cannot be resolved, let
+   alone redirected, before its parent exists;
+4. recurse into each submodule against the matching donor level.
+
+Then, once at the top, `submodule sync --recursive` restores the canonical URLs — in the config
+*and* in each submodule's own `origin` — so no host path from the donor reaches the container, and
+the agent fetches and pushes where it should.
+
+It is **only** an optimisation, never a precondition: if there's no local checkout, if hydration
+raises, or if it leaves any submodule uninitialized (a donor behind the recorded commit), spawn-prep
+falls back to the plain `submodule update --init --recursive` above. Hardlinks make the task's
+objects independent of a later `git gc` in the donor — the same property `clone --local` already
+relies on for the superproject — and a cross-filesystem donor degrades to a copy, still local.
+
+The repo's **cache** clone stays submodule-free: it is not the donor, and cache-side submodule
+checkouts would be disk and time for nothing. A repo whose `git_url` is a hosted forge has no local
+donor and still pays one submodule fetch per task; making the cache clone submodule-aware would
+close that too (backlogged) — `submodule.alternateLocation=superproject` isn't the lever, since it
+computes the alternate from the superproject's *origin*, which is exactly what we repoint at the
+forge.
 
 ### 2. Provisioning = branch whatever's there
 

src/panopticon/core/git.py

@@ -18,11 +18,48 @@
 import subprocess
 from collections.abc import Sequence
 from dataclasses import dataclass
+from pathlib import Path
 from typing import Protocol
 
 #: Feature-branch namespace (PARITY §8/§14, renamed from cloude-cade's ``cloude/``).
 BRANCH_PREFIX = "panopticon"
 
+#: URL schemes that mean a networked (hosted-forge) remote rather than a local path.
+_FORGE_SCHEMES = ("https://", "http://", "ssh://", "git://", "ftp://", "ftps://")
+
+
+def is_forge_url(git_url: str) -> bool:
+    """True when ``git_url`` names a hosted-forge remote (network push/PR/CI), not a local path.
+
+    Recognizes URL-scheme remotes (``https://…``, ``ssh://…``, …) and scp-like ``user@host:path``
+    remotes; treats a bare filesystem path or a ``file://`` URL as local-only.
+    """
+    url = git_url.strip()
+    if url.lower().startswith("file://"):
+        return False
+    if url.lower().startswith(_FORGE_SCHEMES):
+        return True
+    # scp-like syntax: user@host:path — an '@' and a ':' before any '/'. A Windows drive path
+    # (``C:\…``) has the ':' but no '@', so it stays local.
+    at, colon, slash = url.find("@"), url.find(":"), url.find("/")
+    return at != -1 and colon > at and (slash == -1 or colon < slash)
+
+
+def local_repo_path(git_url: str) -> str | None:
+    """The filesystem path ``git_url`` names, or ``None`` when it names a networked remote.
+
+    The counterpart of :func:`is_forge_url`: a bare path or a ``file://`` URL is somewhere on this
+    host — which is what makes panopticon's host-side push, and cloning a task's submodules from
+    the repo's own checkout (:func:`panopticon.sessionservice.spawn.hydrate_submodules`), possible
+    at all.
+    """
+    if is_forge_url(git_url):
+        return None
+    url = git_url.strip()
+    if url.lower().startswith("file://"):
+        url = url[len("file://") :]
+    return str(Path(url).expanduser()) if url else None
+
 
 class GitError(RuntimeError):
     """A ``git`` command that exited non-zero, carrying its ``stderr``.
@@ -87,6 +124,27 @@ def parse_submodule_status(output: str) -> dict[str, str]:
     return states
 
 
+def parse_submodule_paths(output: str) -> dict[str, str]:
+    """Parse ``git config --get-regexp`` output into ``{submodule name: path}``.
+
+    Each line is ``submodule.<name>.path <path>`` — read from ``.gitmodules`` rather than from
+    ``git submodule status`` because it answers *before* ``submodule init`` has run and for
+    submodules that aren't checked out. The name is whatever sits between the ``submodule.``
+    prefix and the ``.path`` suffix (it usually **is** the path, dots and slashes included), and
+    the value is the rest of the line, so a path containing spaces survives. Lines that don't
+    match the shape are skipped.
+    """
+    paths: dict[str, str] = {}
+    for line in output.splitlines():
+        key, _, value = line.partition(" ")
+        if not value or not key.startswith("submodule.") or not key.endswith(".path"):
+            continue
+        name = key[len("submodule.") : -len(".path")]
+        if name:
+            paths[name] = value
+    return paths
+
+
 @dataclass(frozen=True)
 class Worktree:
     """A created worktree: its branch and on-disk path."""
@@ -164,8 +222,59 @@ def submodule_status(self, *, repo_path: str) -> dict[str, str]:
         out = self._run(["git", "-C", repo_path, "submodule", "status", "--recursive"])
         return parse_submodule_status(out)
 
-    def update_submodules(self, *, repo_path: str) -> None:
-        """``git -C <repo> submodule update --init --recursive`` — fill in the submodule checkouts.
+    def submodule_paths(self, *, repo_path: str) -> dict[str, str]:
+        """The submodules this repo *declares* — ``{name: path}``, empty when there are none.
+
+        Reads ``.gitmodules`` (``git config --file``), not ``git submodule status``: the caller
+        overriding a submodule's URL (:meth:`set_submodule_url`) needs the **name** git keys that
+        config on, and needs it for a submodule that isn't checked out yet. Only this level's
+        submodules — a nested one is declared in its own superproject's ``.gitmodules``, so the
+        caller recurses. ``check=False`` because ``git config`` exits non-zero when the file (or a
+        match) is absent, which is just "no submodules".
+        """
+        out = self._run(
+            [
+                "git",
+                "-C",
+                repo_path,
+                "config",
+                "--file",
+                ".gitmodules",
+                "--get-regexp",
+                "^submodule\\..*\\.path$",
+            ],
+            check=False,
+        )
+        return parse_submodule_paths(out)
+
+    def init_submodules(self, *, repo_path: str) -> None:
+        """``git -C <repo> submodule init`` — resolve each declared URL into ``submodule.<n>.url``.
+
+        Separated from ``update`` so a caller can *override* the resolved URL in between (the
+        donor hydration in ``sessionservice.spawn``); ``update --init`` would clone before the
+        override could land.
+        """
+        self._run(["git", "-C", repo_path, "submodule", "init"])
+
+    def set_submodule_url(self, *, repo_path: str, name: str, url: str) -> None:
+        """``git -C <repo> config submodule.<name>.url <url>`` — where ``update`` clones from.
+
+        The config value wins over ``.gitmodules`` until :meth:`sync_submodules` restores it.
+        """
+        self._run(["git", "-C", repo_path, "config", f"submodule.{name}.url", url])
+
+    def sync_submodules(self, *, repo_path: str) -> None:
+        """``git -C <repo> submodule sync --recursive`` — restore the canonical submodule URLs.
+
+        Rewrites every ``submodule.<name>.url`` from ``.gitmodules`` (resolved against the
+        superproject's ``origin``) **and** repoints each checked-out submodule's own
+        ``remote.origin.url`` at it — so a temporary local-donor override leaves nothing of the
+        host's paths behind in the checkout the container gets.
+        """
+        self._run(["git", "-C", repo_path, "submodule", "sync", "--recursive"])
+
+    def update_submodules(self, *, repo_path: str, recursive: bool = True) -> None:
+        """``git -C <repo> submodule update --init [--recursive]`` — fill in the submodule checkouts.
 
         ``protocol.file.allow=always`` is **required**, not cosmetic: since git 2.38 a submodule
         whose resolved URL is a local path is refused (``transport 'file' not allowed``,
@@ -178,20 +287,24 @@ def update_submodules(self, *, repo_path: str) -> None:
         Submodule URLs are resolved against the superproject's ``remote.origin.url`` *here*, so the
         caller must point ``origin`` at the forge first (:meth:`set_origin`) — resolving a relative
         URL against the cache path would look for the submodule next to the cache clone.
+
+        ``recursive=False`` updates **this level only**: a nested submodule's URL can't be resolved
+        (or overridden) before its parent exists, so the donor hydration walks the tree a level at
+        a time instead.
         """
-        self._run(
-            [
-                "git",
-                "-C",
-                repo_path,
-                "-c",
-                "protocol.file.allow=always",
-                "submodule",
-                "update",
-                "--init",
-                "--recursive",
-            ]
-        )
+        args = [
+            "git",
+            "-C",
+            repo_path,
+            "-c",
+            "protocol.file.allow=always",
+            "submodule",
+            "update",
+            "--init",
+        ]
+        if recursive:
+            args.append("--recursive")
+        self._run(args)
 
     def push(self, *, repo_path: str, remote: str, branch: str) -> None:
         """``git -C <repo> push <remote> <branch>`` — send one branch, as-is (never forced).