From 21abdf10290905a052507f6675d08f798c631103 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Fri, 4 Sep 2026 12:29:38 +0200 Subject: [PATCH] =?UTF-8?q?docs(substantiate):=20driver=20launch=20is=20re?= =?UTF-8?q?try-wrapped=20=E2=80=94=20preemptions=20self-heal=20(3=20GKD=20?= =?UTF-8?q?preemptions=20observed)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- TODO.substantiate/CONTINUE.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/TODO.substantiate/CONTINUE.md b/TODO.substantiate/CONTINUE.md index ba2434a..4b71180 100644 --- a/TODO.substantiate/CONTINUE.md +++ b/TODO.substantiate/CONTINUE.md @@ -18,9 +18,12 @@ lands each step). Derive everything else from live state, not memory: interscript-ml): if no run dir `rababa_arabic_distill_small/ run-012-r7-muon-gkd` exists on the `rababa-checkpoints` volume AND fewer than two ephemeral GPU apps are running (`modal app list`), - launch: `cd /Users/mulgogi/src/interscript/ml-qwen-feat && nohup - modal run --detach src/gpu/modal_distill.py::main --spec - ara-diac-small-2-gkd > /Users/mulgogi/gkd_distill.log 2>&1 &`. If the + launch (retry-wrapped — preemptions self-heal from checkpoints, + bounded at 20 attempts): `cd /Users/mulgogi/src/interscript/ml-qwen-feat + && nohup bash -c 'for i in $(seq 1 20); do modal run --detach + src/gpu/modal_distill.py::main --spec ara-diac-small-2-gkd && break; + echo "[retry $i]"; sleep 120; done' > + /Users/mulgogi/gkd_distill.log 2>&1 &`. If the run dir exists with a best/ checkpoint but no final_eval.json (training complete, eval unchained), run `::evaluate_der --spec-id ara-diac-small-2-gkd` the same way