From c634553cfc3432e0704ef8fb2c0f8779a296dd54 Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 14 Oct 2026 10:08:03 -0500 Subject: [PATCH 01/63] 2026-usrse: duct poster mockup at 42x36 in, with logos HTML mockup of the US-RSE'26 con-duct poster, sized for a 42 x 36 in fabric print: @page sets the page size and a print stylesheet zooms the 2158 x 1850 px layout onto it, so headless chromium --print-to-pdf gives one page. Title row: author block, title with the con-duct logo inline, then the Dartmouth D-Pine, CON and ReproNim logos and QR placeholders. Assets include the con-duct overhead and fMRIPrep plots. Co-Authored-By: Claude Code 2.1.299 / Claude Opus 5.5 --- .../duct-in-duct-overhead.png | 1 + .../duct-in-duct-overhead.svg | 1841 +++++++++++++++++ .../fmriprep-ds000001-sub-01-plot.png | 1 + .../fmriprep-ds000001-sub-01-plot.svg | 1 + .../duct-poster-assets/logos/D-Pine_RGB.svg | 11 + .../duct-poster-assets/logos/D-Pine_Rev.svg | 11 + .../logos/con_logo_blue_big_tight.svg | 1 + .../logos/con_logo_white.svg | 127 ++ .../duct-poster-assets/logos/duct_logo.svg | 1 + 2026-usrse/duct-poster-mockup.html | 316 +++ 10 files changed, 2311 insertions(+) create mode 100644 2026-usrse/duct-poster-assets/duct-in-duct-overhead.png create mode 100644 2026-usrse/duct-poster-assets/duct-in-duct-overhead.svg create mode 100644 2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.png create mode 100644 2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.svg create mode 100644 2026-usrse/duct-poster-assets/logos/D-Pine_RGB.svg create mode 100644 2026-usrse/duct-poster-assets/logos/D-Pine_Rev.svg create mode 100644 2026-usrse/duct-poster-assets/logos/con_logo_blue_big_tight.svg create mode 100644 2026-usrse/duct-poster-assets/logos/con_logo_white.svg create mode 100644 2026-usrse/duct-poster-assets/logos/duct_logo.svg create mode 100644 2026-usrse/duct-poster-mockup.html diff --git a/2026-usrse/duct-poster-assets/duct-in-duct-overhead.png b/2026-usrse/duct-poster-assets/duct-in-duct-overhead.png new file mode 100644 index 00000000..78f1649e --- /dev/null +++ b/2026-usrse/duct-poster-assets/duct-in-duct-overhead.png @@ -0,0 +1 @@ +/annex/objects/MD5E-s100302--41106443b8cf6c83f4354cfa7bf3c09a.png diff --git a/2026-usrse/duct-poster-assets/duct-in-duct-overhead.svg b/2026-usrse/duct-poster-assets/duct-in-duct-overhead.svg new file mode 100644 index 00000000..fa6ba041 --- /dev/null +++ b/2026-usrse/duct-poster-assets/duct-in-duct-overhead.svg @@ -0,0 +1,1841 @@ + + + + + + + + 2026-09-18T10:19:54.607406 + image/svg+xml + + + Matplotlib v3.11.2, https://matplotlib.org/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.png b/2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.png new file mode 100644 index 00000000..fc54a312 --- /dev/null +++ b/2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.png @@ -0,0 +1 @@ +/annex/objects/MD5E-s119888--796c192c098f401b9a83d406217c0167.png diff --git a/2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.svg b/2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.svg new file mode 100644 index 00000000..94d37a85 --- /dev/null +++ b/2026-usrse/duct-poster-assets/fmriprep-ds000001-sub-01-plot.svg @@ -0,0 +1 @@ +/annex/objects/MD5E-s438596--45d61c1dd27f978a3a25fb550be197d3.svg diff --git a/2026-usrse/duct-poster-assets/logos/D-Pine_RGB.svg b/2026-usrse/duct-poster-assets/logos/D-Pine_RGB.svg new file mode 100644 index 00000000..ac6d4c3b --- /dev/null +++ b/2026-usrse/duct-poster-assets/logos/D-Pine_RGB.svg @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/2026-usrse/duct-poster-assets/logos/D-Pine_Rev.svg b/2026-usrse/duct-poster-assets/logos/D-Pine_Rev.svg new file mode 100644 index 00000000..7341889b --- /dev/null +++ b/2026-usrse/duct-poster-assets/logos/D-Pine_Rev.svg @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/2026-usrse/duct-poster-assets/logos/con_logo_blue_big_tight.svg b/2026-usrse/duct-poster-assets/logos/con_logo_blue_big_tight.svg new file mode 100644 index 00000000..3a023d9d --- /dev/null +++ b/2026-usrse/duct-poster-assets/logos/con_logo_blue_big_tight.svg @@ -0,0 +1 @@ +/annex/objects/MD5E-s125807--60c9b67a52c2afdf00c9ceccea01e6ef.svg diff --git a/2026-usrse/duct-poster-assets/logos/con_logo_white.svg b/2026-usrse/duct-poster-assets/logos/con_logo_white.svg new file mode 100644 index 00000000..04907bcc --- /dev/null +++ b/2026-usrse/duct-poster-assets/logos/con_logo_white.svg @@ -0,0 +1,127 @@ + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + diff --git a/2026-usrse/duct-poster-assets/logos/duct_logo.svg b/2026-usrse/duct-poster-assets/logos/duct_logo.svg new file mode 100644 index 00000000..9f330545 --- /dev/null +++ b/2026-usrse/duct-poster-assets/logos/duct_logo.svg @@ -0,0 +1 @@ +/annex/objects/MD5E-s716776--0a66b516151438af418da744424d7f46.svg diff --git a/2026-usrse/duct-poster-mockup.html b/2026-usrse/duct-poster-mockup.html new file mode 100644 index 00000000..0737dc74 --- /dev/null +++ b/2026-usrse/duct-poster-mockup.html @@ -0,0 +1,316 @@ + + + + +con-duct poster mockup (draft) + + + +
DRAFT MOCKUP · 42×36 in landscape · S-numbers refer to duct-poster-strawmen.md
+
+
+ +
+

Austin Macdonald
Cody C. Baker
John A. Lee
Yaroslav O. Halchenko

+
+

Agent-friendly Provenance Capture with con-duct

+

Center for Open Neuroscience, Dartmouth College  ·  RRID:SCR_025436

+
+
+ Dartmouth + Center for Open Neuroscience + ReproNim +
+
QR
github.com/con/duct
QR
Zenodo entry
+
+ +
+ +
+
+

S1Output nobody kept

+

Humans and agents now run commands side by side. The terminal output is the evidence, and it is either bloating an agent's context window or gone past the scrollback.

+
what the agent sawwhat one nebari deploy printed: 3,227 lines
+

Wrap it: duct <cmd> instead of <cmd>. The command runs unchanged and leaves four files.

+
+
$ duct nebari deploy …
→ +
duct
→ +
stdoutstderrinfo.jsonusage.jsonl
+
+
Scope. Cheap one-offs do not need this; agents are good at those. duct earns its keep when execution is expensive, slow, or dangerous, where “just try again” is not an option and every attempt should be reviewable.
+
+ +
+

S6Where it sits

+ + + + + + + + + +
teeGNU timeductworkflow managers
Snakemake, Nextflow		experiment trackers
MLflow, W&B
wrap any command, no code changes✓✓✓✗ rewrite as a workflow✗ add logging calls to your code
works anywhere, no scheduler✓◐ GNU only◐ Linux, macOS✓✓
stdout/stderr kept✓ merged✗✓✓ per task◐
exit code, invocation, environment✗◐ exit code✓✓◐
peak memory, CPU, whole tree✗◐ one process✓ sampled✓◐
time series, not just a peak✗✗✓◐◐
queryable across runs✗ grep✗✓◐ trace file★
+

✓ yes · ◐ partial · ✗ no · ★ richer than duct. Same one-word change: tee keeps the words, time one number, duct both and the rest.

+

When the others are better: many steps with declared inputs and outputs (a workflow manager knows the graph, reruns only what changed); comparing runs by parameters and metrics (a tracker keeps them with the artifacts, side by side).

+
+ +
+

S7Adopted inside tools

+

Tools wrap their own runs so users get records without asking, and one con-duct ls spans all of them.

+
logo row: ReproNim/containers · ReproStim · babs (PennLINC/babs#413, draft) · external: MRIQC logs at cerebra.fz-juelich.de
+

What it took: a hard dependency on con-duct and a choice of output prefix so the record sits next to the result. No API; the wrapped command is unchanged.

+ 
<output dir>/logs/<sub>/acq-{datetime}+<arrayid>_<taskid>_   # babs
+
+
+ + +
+
+

S3An agent reads a human's records and acts: the k8s upgrade

+

A Kubernetes 1.33 upgrade on AWS with many components we do not own. Day 1, staging: 9 deploys, 8 failures, each root-caused with an agent reading the duct record, until a runbook existed. Day 2, production: 10 deploys, 9 failures, most already predicted by the runbook. Every run: datalad run "duct -p logs/… nebari deploy …".

+ + staging · 2026-03-26 + production · 2026-03-27 + + + + +

Bar length = wall clock time (from info.json); red = nonzero exit, green = success. Cause text from the diagnosis note committed beside each record. Deploy wall time: 1.8 h + 1.5 h. Records on disk: 2.3 MB.

+
+
+
Clean boundary. The deployment repo is full of secrets. The agent got the logs/ directory and the nebari source, never the repo: everything the command printed, none of the credentials that produced it.
+ 
$ con-duct ls \
+    -e "exit_code != 0 and wall_clock_time > 600"
+# 6 deploys burned >10 min, then failed
+
+
+
simulated: mirrors the real session, condensed
+
user
deploy 09 failed, look at the log
+
agent
con-duct ls -e "'09_deploy' in prefix" → exit 1, 285 s.
stderr tail: nebari-bot creation 503 after 10 attempts. Keycloak pod pending 50 min; main container waiting on image pull secret extcrcreds, which a later stage creates.
Recommend: re-run. The secret exists on the second pass.
+
+
+
+ +
+

S11What one record looks like: con-duct plot

+ 
$ con-duct plot acq-2026.09.09T00.09.47+64119356_1_info.json \
+    -o plot.png --cpu ps-cpu-timepoint
+ con-duct plot of one fMRIPrep run +

One c5 record, unedited: fMRIPrep 25.2.5 minimal, ds000001 sub-01, run by babs under SLURM (8 CPUs, 40 GB). Wall 5.0 h, peak RSS 5.9 GB, exit 0. CPU is the time-point estimate (--cpu ps-cpu-timepoint).

+
+
+ + +
+
+

S2An agent captures its own runs, chooses later what enters context

+

On our cluster every job's app call is duct-wrapped (by babs; whether the agent types duct or the tool has it built in is a surface difference). 28 datasets × 2 apps = 56 records. Later: “which cells failed, and why?”

+ 
$ con-duct ls -e "exit_code != 0" -F command exit_code wall_clock_time
+mriqc     ds002685   1    7 min
+fmriprep  ds002685   1   12 min
+mriqc     ds004884   1   29 min
+mriqc     ds005752   1   18 min   ← numbers TBC from typhon
+
Records on disk: 56. Read into context: 4 stderr tails. Nothing re-run.
+
+ +
+

S4A human audits the agent: bug replication

+

The agent writes a reproducer and runs it under duct, numbering each step in the tag:

+ 
$ duct -p logs/1_ -m "1: reproduce user's bug"                      ./repro.sh
+$ duct -p logs/2_ -m "2: simplified reproducer to test hypothesis"  ./repro.sh
+$ duct -p logs/3_ -m "3: verify fix"                                ./repro.sh
+

The agent reports “fixed”. You do not take its word:

+ 
$ con-duct ls -F prefix message exit_code logs/*info.json
+logs/1_  1: reproduce user's bug                      1
+logs/2_  2: simplified reproducer to test hypothesis  1
+logs/3_  3: verify fix                                0
+

Each row has the exact command and full stderr behind it. Auditing is reading three tagged records, not scrolling a transcript. (simulated; to be staged on a real duct bug)

+
+ +
+

S5Second act: the record as provenance

+

DataLad. datalad run "duct <cmd>" commits command, logs, and the resulting diff together:

+
[DATALAD RUNCMD] excerpt from a real non-nebari datalad run "duct …" (TBD; or one sentence, no excerpt)
+

HPC. fMRIPrep minimal on 26 datasets: wall 3.7–8.2 h, peak RSS median 12.5 GiB, tail to 39.4 GiB; the 40 G grant fit 25 of 26. The outlier (64 BOLD runs) timed out at 24 h, OOM'd at 40 G, finished at 80 G in 43 h, duct peak 52.9 GB.

+

Caveat: sacct MaxRSS sits 5–8 GB above duct's peak on the same jobs (page cache under cgroup v2).

+
+
+
+ +
+
+

S8Overhead, measured: duct wrapped around duct

+ + + + + +
pids monitoredduct CPU, % of one coreduct RSSrecord bytes / report
11.6 ± 0.237 MB0.6 KB
161.6 ± 0.537 MB3.4 KB
2562.2 ± 0.938 MB48 KB
+

Outer duct watching an inner duct that watches N idle processes, 5 min per phase, 1 s sampling. CPU is the python process as the outer duct saw it (phase mean; ± is the ps rounding bound), flat to 16 pids, about 2 % at 64 and 256: consistent with a slow rise, not a measured slope. Whole run by time -v, including duct's own ps calls: 3.3 % of one core; duct's sampler misses those few-ms children itself. Laptop i7-1280P, hybrid cores. Records: datalad run in ~/datasets/duct-in-duct.

+
+
+

S9Honesty box

+
    +
  • Output is written to disk: do not wrap commands that print secrets.
    • +
  • Programs that check isatty() take their non-interactive path (README has workarounds).
    • +
  • Sub-second commands leave four files and no samples. POSIX only; ps numbers are approximate; cgroup measurement in progress (con/duct#438).
    • +
  • A process that lives a few ms is invisible to a 1 s sampler, duct's own ps included: the overhead panel's time -v total is nearly twice what duct saw of itself.
    • +
  • Resource peaks paid off on SLURM; for agentic work the value has been the logs and con-duct ls. Wrap the runs you cannot afford to repeat.
    • +
+
+
+

S10Try it

+ 
pip install con-duct      # or conda-forge
+duct <your command>
+con-duct ls
+

Live demo at the poster: con-duct ls over the real records above.

+
QR
+
+
+ +
+
+ + + + From 95691e371bdfdcacbce7c12bb85dde1848edbd04 Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 14 Oct 2026 10:08:03 -0500 Subject: [PATCH 02/63] 2026-usrse: QR codes for the duct poster title row Two QR codes replace the placeholders: the con/duct repository and the abstract's Zenodo DOI (10.5281/zenodo.22833931). Generated as SVG with `segno --scale 10 --border 0 --error H `; both decode to their URL from the printed PDF. Co-Authored-By: Claude Code 2.1.299 / Claude Opus 5.5 --- 2026-usrse/duct-poster-assets/qr/qr-github-con-duct.svg | 2 ++ 2026-usrse/duct-poster-assets/qr/qr-zenodo-abstract.svg | 2 ++ 2026-usrse/duct-poster-mockup.html | 9 +++++++-- 3 files changed, 11 insertions(+), 2 deletions(-) create mode 100644 2026-usrse/duct-poster-assets/qr/qr-github-con-duct.svg create mode 100644 2026-usrse/duct-poster-assets/qr/qr-zenodo-abstract.svg diff --git a/2026-usrse/duct-poster-assets/qr/qr-github-con-duct.svg b/2026-usrse/duct-poster-assets/qr/qr-github-con-duct.svg new file mode 100644 index 00000000..0edbc2c3 --- /dev/null +++ b/2026-usrse/duct-poster-assets/qr/qr-github-con-duct.svg @@ -0,0 +1,2 @@ + + diff --git a/2026-usrse/duct-poster-assets/qr/qr-zenodo-abstract.svg b/2026-usrse/duct-poster-assets/qr/qr-zenodo-abstract.svg new file mode 100644 index 00000000..02d33343 --- /dev/null +++ b/2026-usrse/duct-poster-assets/qr/qr-zenodo-abstract.svg @@ -0,0 +1,2 @@ + + diff --git a/2026-usrse/duct-poster-mockup.html b/2026-usrse/duct-poster-mockup.html index 0737dc74..ccd1ac84 100644 --- a/2026-usrse/duct-poster-mockup.html +++ b/2026-usrse/duct-poster-mockup.html @@ -27,7 +27,9 @@ header h1 code { font-family: var(--mono); color: var(--accent); } header h1 .duct-logo { height: 64px; vertical-align: middle; margin: 0 6px 0 2px; } header .affil { margin: 4px 0 0; font-size: 18px; color: var(--muted); } - header .qrs { display: flex; gap: 12px; } + header .qrs { display: flex; gap: 16px; } + header .qrs figure { margin: 0; text-align: center; font-size: 12px; color: var(--muted); } + header .qrs img { display: block; width: 92px; height: 92px; margin-bottom: 3px; } .qr { width: 90px; height: 90px; border: 2px dashed var(--rule); display: grid; place-items: center; font-size: 11px; color: var(--muted); text-align: center; } main { display: grid; grid-template-columns: 1fr 1.55fr 1fr; gap: 14px; min-height: 0; } @@ -114,7 +116,10 @@

Agent-friendly Provenance Capture with ReproNim -
QR
github.com/con/duct
QR
Zenodo entry
+
+
QR: https://github.com/con/ductcode
+
QR: https://doi.org/10.5281/zenodo.22833931abstract
+
From b878f48ae4300f79b76f90311dd6d10bb91aee57 Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 14 Oct 2026 10:08:04 -0500 Subject: [PATCH 03/63] 2026-usrse: REUSE markers for the duct poster's borrowed logos The D-Pine joins the third-party logos marker. The con-duct logo is a bitmap trace of an image of unrecorded origin (con/duct#207), so it gets the NOASSERTION mixed marker rather than the catch-all. The ReproNim logo is the already-annotated pics/borrowed/repronim_logo.svg, and the CON logos stay under the catch-all, being our own. `reuse lint` passes. Co-Authored-By: Claude Code 2.1.299 / Claude Opus 5.5 --- REUSE.toml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/REUSE.toml b/REUSE.toml index 7e97e6fb..e46a74bd 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -169,6 +169,15 @@ precedence = "aggregate" SPDX-FileCopyrightText = "NOASSERTION" SPDX-License-Identifier = "LicenseRef-ThirdPartyMixed" +# The con-duct logo: an Inkscape bitmap trace of the image in the duct blog +# post, adopted in con/duct#207. Who made that source image is not recorded, +# so the trace cannot be offered under our terms alone. +[[annotations]] +path = "2026-usrse/duct-poster-assets/logos/duct_logo.svg" +precedence = "aggregate" +SPDX-FileCopyrightText = "NOASSERTION" +SPDX-License-Identifier = "LicenseRef-ThirdPartyMixed" + # === 3. Third-party logos and trademarks =================================== @@ -210,6 +219,9 @@ path = [ "pics/openneuro.svg", "pics/repronim-logo-vertical.svg", "pics/twitter.png", + # Dartmouth's D-Pine, converted from the EPS in Dartmouth's brand kit for + # the US-RSE'26 duct poster. Dartmouth's guidelines forbid altering it. + "2026-usrse/duct-poster-assets/logos/D-Pine_*.svg", ] precedence = "aggregate" SPDX-FileCopyrightText = "NOASSERTION" From 41813ae0fb4705462120318dbeed23c0e0a9e8f0 Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 14 Oct 2026 14:17:52 -0500 Subject: [PATCH 04/63] 2026-usrse: restructure the duct poster into story-led panels Middle column: what con-duct is (capture, search, plot). Left: Try it with a real hello-world run, You drive / the agent drives with simulated chats, and the plot. Right: DataLad with a real datalad run commit, adopted inside tools (BABS, ReproNim/containers, ReproStim), comparison table. Footer: overhead and limitations. The k8s case study is cut and nebari is gone from the poster. Co-Authored-By: Claude Code 2.1.299 / Claude Opus 5.5 --- 2026-usrse/duct-poster-mockup.html | 288 +++++++++++++++-------------- 1 file changed, 150 insertions(+), 138 deletions(-) diff --git a/2026-usrse/duct-poster-mockup.html b/2026-usrse/duct-poster-mockup.html index ccd1ac84..2e7238a7 100644 --- a/2026-usrse/duct-poster-mockup.html +++ b/2026-usrse/duct-poster-mockup.html @@ -32,11 +32,13 @@ header .qrs img { display: block; width: 92px; height: 92px; margin-bottom: 3px; } .qr { width: 90px; height: 90px; border: 2px dashed var(--rule); display: grid; place-items: center; font-size: 11px; color: var(--muted); text-align: center; } - main { display: grid; grid-template-columns: 1fr 1.55fr 1fr; gap: 14px; min-height: 0; } + main { display: grid; grid-template-columns: minmax(0,1fr) minmax(0,1.55fr) minmax(0,1fr); gap: 14px; min-height: 0; } .col { display: flex; flex-direction: column; gap: 14px; min-height: 0; } section { border: 1px solid var(--rule); border-radius: 6px; padding: 12px 14px; position: relative; min-height: 0; overflow: hidden; } section > h2 { margin: 0 0 6px; font-size: 20px; color: var(--accent); } - section > h2 .tag { font: 600 11px/1 var(--mono); color: #fff; background: var(--accent); border-radius: 3px; padding: 3px 5px; margin-right: 6px; vertical-align: middle; } + section h3 { margin: 12px 0 4px; padding-top: 8px; border-top: 1px solid var(--rule); font-size: 17px; color: var(--accent); } + section h2 + h3 { margin-top: 4px; padding-top: 0; border-top: 0; } + .lede { font-weight: 700; } section p, section li { font-size: 14px; line-height: 1.32; margin: 4px 0; } .grow { flex: 1; } pre, code { font-family: var(--mono); } @@ -47,7 +49,6 @@ .vis b { color: var(--ink); } .caption { font-size: 12px; color: var(--muted); margin-top: 4px; } .callout { background: var(--accent-bg); border-left: 3px solid var(--accent); padding: 6px 10px; font-size: 14px; margin: 6px 0; } - .honest { background: var(--warn-bg); } table { border-collapse: collapse; font-size: 13px; width: 100%; } th, td { border-bottom: 1px solid var(--rule); padding: 3px 6px; text-align: left; } th { color: var(--muted); font-weight: 600; } @@ -62,6 +63,8 @@ /* mechanism diagram */ .flow { display: flex; align-items: center; gap: 8px; font-family: var(--mono); font-size: 13px; margin: 8px 0; } .flow .box { border: 1.5px solid var(--ink); border-radius: 4px; padding: 6px 8px; background: #fff; } + .flow .outs { display: flex; flex-direction: column; gap: 6px; } + .flow .term { border: 1.5px dashed var(--muted); border-radius: 4px; padding: 3px 6px; font-size: 11.5px; color: var(--muted); } .flow .files { display: grid; grid-template-columns: 1fr 1fr; gap: 3px; } .flow .files span { border: 1px solid var(--accent); color: var(--accent); border-radius: 3px; padding: 2px 5px; font-size: 11.5px; } .flow .arrow { font-size: 20px; color: var(--muted); } @@ -79,13 +82,7 @@ .chat .a { background: var(--accent-bg); } .chat .who { font: 600 10.5px/1 var(--mono); color: var(--muted); margin-bottom: 3px; } - /* timeline svg */ - svg text { font-family: system-ui, sans-serif; } - svg .cause { font-size: 14px; fill: var(--ink); } - svg .dur { font-size: 12px; fill: var(--muted); font-family: var(--mono); } - svg .h { font-size: 14px; font-weight: 700; fill: var(--ink); } - - footer { display: grid; grid-template-columns: 1.2fr 1fr .8fr; gap: 14px; min-height: 0; } + footer { display: grid; grid-template-columns: 1.2fr 1fr; gap: 14px; min-height: 0; } footer section { padding: 10px 14px; } footer p, footer li { font-size: 12.5px; } footer ul { margin: 2px 0 0 16px; padding: 0; } @@ -101,7 +98,7 @@ -
DRAFT MOCKUP · 42×36 in landscape · S-numbers refer to duct-poster-strawmen.md
+
DRAFT MOCKUP · 42×36 in landscape
@@ -126,120 +123,180 @@

Agent-friendly Provenance Capture with
-

S1Output nobody kept

-

Humans and agents now run commands side by side. The terminal output is the evidence, and it is either bloating an agent's context window or gone past the scrollback.

-
what the agent sawwhat one nebari deploy printed: 3,227 lines
-

Wrap it: duct <cmd> instead of <cmd>. The command runs unchanged and leaves four files.

-
-
$ duct nebari deploy …
→ -
duct
→ -
stdoutstderrinfo.jsonusage.jsonl
-
-
Scope. Cheap one-offs do not need this; agents are good at those. duct earns its keep when execution is expensive, slow, or dangerous, where “just try again” is not an option and every attempt should be reviewable.
+

Try it

+ 
$ pip install con-duct      # or conda
+$ duct bash -c 'echo "hello world" && sleep 2'
+2026-10-14T13:47:50-0500 [INFO    ] con-duct: duct 0.23.0 is executing 'bash -c echo "hello world" && sleep 2'...
+2026-10-14T13:47:50-0500 [INFO    ] con-duct: Log files will be written to .duct/logs/2026.10.14T13.47.50-1205327_
+hello world
+2026-10-14T13:47:52-0500 [INFO    ] con-duct: Summary:
+Exit Code: 0
+Command: bash -c echo "hello world" && sleep 2
+Log files location: .duct/logs/2026.10.14T13.47.50-1205327_
+Wall Clock Time: 2.002 sec
+Memory Peak Usage (RSS): 2.4 MB
+Memory Average Usage (RSS): 2.4 MB
+Memory Peak Percentage: 0.00%
+Memory Average Percentage: 0.00%
+CPU Peak Usage: 0.00%
+Average CPU Usage: 0.00%
-

S6Where it sits

- - - - - - - - - -
teeGNU timeductworkflow managers
Snakemake, Nextflow		experiment trackers
MLflow, W&B
wrap any command, no code changes✓✓✓✗ rewrite as a workflow✗ add logging calls to your code
works anywhere, no scheduler✓◐ GNU only◐ Linux, macOS✓✓
stdout/stderr kept✓ merged✗✓✓ per task◐
exit code, invocation, environment✗◐ exit code✓✓◐
peak memory, CPU, whole tree✗◐ one process✓ sampled✓◐
time series, not just a peak✗✗✓◐◐
queryable across runs✗ grep✗✓◐ trace file★
-

✓ yes · ◐ partial · ✗ no · ★ richer than duct. Same one-word change: tee keeps the words, time one number, duct both and the rest.

-

When the others are better: many steps with declared inputs and outputs (a workflow manager knows the graph, reruns only what changed); comparing runs by parameters and metrics (a tracker keeps them with the artifacts, side by side).

+

You drive, the agent reads the record

+

Stay in charge, without copy/pasting terminal output into a chat.

+

You run the commands. When one fails, the agent reads the record instead of your scrollback.

+ 
$ duct singularity run …
+
+
simulated
+
user
it didn't work
+
claude
I used con-duct ls to find the run you mean. The exit code was 1 after 38 min, the problem was ...
+
-

S7Adopted inside tools

-

Tools wrap their own runs so users get records without asking, and one con-duct ls spans all of them.

-
logo row: ReproNim/containers · ReproStim · babs (PennLINC/babs#413, draft) · external: MRIQC logs at cerebra.fz-juelich.de
-

What it took: a hard dependency on con-duct and a choice of output prefix so the record sits next to the result. No API; the wrapped command is unchanged.

- 
<output dir>/logs/<sub>/acq-{datetime}+<arrayid>_<taskid>_   # babs
+

The agent drives, you read the summary

+

The agent writes a reproducer and runs it under duct, numbering each step in the tag:

+ 
$ duct -p logs/1_ -m "1: reproduce user's bug"                      ./repro.sh
+$ duct -p logs/2_ -m "2: simplified reproducer to test hypothesis"  ./repro.sh
+$ duct -p logs/3_ -m "3: verify fix"                                ./repro.sh
+

The agent reports “fixed”. You do not take its word:

+ 
$ con-duct ls -F prefix message exit_code logs/*info.json
+logs/1_  1: reproduce user's bug                      1
+logs/2_  2: simplified reproducer to test hypothesis  1
+logs/3_  3: verify fix                                0
+

Each row has the exact command and full stderr behind it. Auditing is reading three tagged records, not scrolling a transcript. (simulated; to be staged on a real duct bug)

+
+ +
+

Plot resource usage

+ 
$ con-duct plot <prefix>info.json -o plot.png --cpu ps-cpu-timepoint
+ con-duct plot of one fMRIPrep run

-
-

S3An agent reads a human's records and acts: the k8s upgrade

-

A Kubernetes 1.33 upgrade on AWS with many components we do not own. Day 1, staging: 9 deploys, 8 failures, each root-caused with an agent reading the duct record, until a runbook existed. Day 2, production: 10 deploys, 9 failures, most already predicted by the runbook. Every run: datalad run "duct -p logs/… nebari deploy …".

- - staging · 2026-03-26 - production · 2026-03-27 - - - - -

Bar length = wall clock time (from info.json); red = nonzero exit, green = success. Cause text from the diagnosis note committed beside each record. Deploy wall time: 1.8 h + 1.5 h. Records on disk: 2.3 MB.

-
+
+

What is con-duct?

+

con-duct is a lightweight command-line wrapper that records what a command printed and the resources it used, for any command, with no code changes. Its companion commands let people and agents find those records later, query them, and plot them.

+ +

Capture now, ask questions later

+

Wrap it: duct <cmd> instead of <cmd>. The command runs unchanged, its output still reaches your terminal (like tee), and it leaves four files.

+
+
$ duct my-fancy-command-or-script
→ +
+
your terminal, as before
+
stdoutstderrinfo.jsonusage.jsonl
+
+
+
    +
  • stdout, stderr: everything the command printed, in full.
    • +
  • info.json: run metadata and summary, e.g. command, exit code, wall time, peak memory and CPU.
    • +
  • usage.jsonl: resource use over time, one line per report: memory and CPU for each process in the tree, and their totals.
    • +
+

Resource usage is recorded too. duct samples the whole process tree every second and writes a report every minute: peak and average memory and CPU, per process and in total.

+

Record everything, read it later if helpful. Humans and agents now run commands side by side. Without a record, the terminal output is either bloating an agent's context window or gone past the scrollback.

+
Active ContextAvailable Context
+
+
duct becomes especially valuable when execution is expensive, slow, or dangerous, where “just try again” is not an option and every attempt should be reviewable.
+ +

Search and retrieve duct records

+

Every record is a small JSON file. con-duct ls reads them all and answers questions across runs, machines, and months, for you or for an agent.

+
-
Clean boundary. The deployment repo is full of secrets. The agent got the logs/ directory and the nebari source, never the repo: everything the command printed, none of the credentials that produced it.
- 
$ con-duct ls \
-    -e "exit_code != 0 and wall_clock_time > 600"
-# 6 deploys burned >10 min, then failed
+

Which jobs failed, and how long did they burn first?

+ 
$ con-duct ls -e "exit_code != 0" \
+    -F command exit_code wall_clock_time
+mriqc     ds002685   1    7 min
+fmriprep  ds002685   1   12 min
+mriqc     ds004884   1   29 min
+mriqc     ds005752   1   18 min   ← TBC
+

28 datasets × 2 apps = 56 records on our cluster. Read into context: 4 stderr tails. Nothing re-run.

-
-
simulated: mirrors the real session, condensed
-
user
deploy 09 failed, look at the log
-
agent
con-duct ls -e "'09_deploy' in prefix" → exit 1, 285 s.
stderr tail: nebari-bot creation 503 after 10 attempts. Keycloak pod pending 50 min; main container waiting on image pull secret extcrcreds, which a later stage creates.
Recommend: re-run. The secret exists on the second pass.
+
+

Which runs needed more than 32 GB, and where?

+ 
$ con-duct ls -e "peak_rss > 32e9" \
+    -F hostname peak_rss wall_clock_time
+← example TBD
+

Structured output for scripts and agents:

+ 
$ con-duct ls -f json logs/ | jq …
-
+
    +
  • Filter with any Python expression over the 40+ fields (-e); pick columns (-F); sort by any field (--sort-by).
    • +
  • Output as a table, JSON, or YAML (-f).
    • +
  • No database: point it at directories, or let it find records by your output prefix.
    • +
+
graphic idea: fan-in. Left: record folders from different places (laptop, cluster jobs, CI, an agent's session), each a stack of the four files. Arrows converge on one con-duct ls -e … funnel. Right: a short table of only the matching rows, one row highlighted with its stderr tail pulled out beside it.
-
-

S11What one record looks like: con-duct plot

- 
$ con-duct plot acq-2026.09.09T00.09.47+64119356_1_info.json \
-    -o plot.png --cpu ps-cpu-timepoint
- con-duct plot of one fMRIPrep run -

One c5 record, unedited: fMRIPrep 25.2.5 minimal, ds000001 sub-01, run by babs under SLURM (8 CPUs, 40 GB). Wall 5.0 h, peak RSS 5.9 GB, exit 0. CPU is the time-point estimate (--cpu ps-cpu-timepoint).

-

S2An agent captures its own runs, chooses later what enters context

-

On our cluster every job's app call is duct-wrapped (by babs; whether the agent types duct or the tool has it built in is a surface difference). 28 datasets × 2 apps = 56 records. Later: “which cells failed, and why?”

- 
$ con-duct ls -e "exit_code != 0" -F command exit_code wall_clock_time
-mriqc     ds002685   1    7 min
-fmriprep  ds002685   1   12 min
-mriqc     ds004884   1   29 min
-mriqc     ds005752   1   18 min   ← numbers TBC from typhon
-
Records on disk: 56. Read into context: 4 stderr tails. Nothing re-run.
+

DataLad and provenance DataLad

+

datalad run "duct <cmd>" commits the command, the duct record, and the resulting diff together:

+ 
$ datalad run -m "Say hello" "duct bash -c 'echo hello world > hello.txt'"
+$ git show --name-only
+commit fae1995dc37fa7925e04bbf1eca5b3cc62332ef9
+Author: Austin Macdonald <austin@dartmouth.edu>
+Date:   Wed Oct 14 13:50:38 2026 -0500
+
+    [DATALAD RUNCMD] Say hello
+
+    === Do not change lines below ===
+    {
+     "chain": [],
+     "cmd": "duct bash -c 'echo hello world > hello.txt'",
+     "dsid": "3329b331-5517-4336-abe3-f335ab4ef64c",
+     "exit": 0,
+     "extra_inputs": [],
+     "inputs": [],
+     "outputs": [],
+     "pwd": "."
+    }
+    ^^^ Do not change lines above ^^^
+
+.duct/logs/2026.10.14T13.50.38-1207637_info.json
+.duct/logs/2026.10.14T13.50.38-1207637_stderr
+.duct/logs/2026.10.14T13.50.38-1207637_stdout
+.duct/logs/2026.10.14T13.50.38-1207637_usage.jsonl
+hello.txt
-

S4A human audits the agent: bug replication

-

The agent writes a reproducer and runs it under duct, numbering each step in the tag:

- 
$ duct -p logs/1_ -m "1: reproduce user's bug"                      ./repro.sh
-$ duct -p logs/2_ -m "2: simplified reproducer to test hypothesis"  ./repro.sh
-$ duct -p logs/3_ -m "3: verify fix"                                ./repro.sh
-

The agent reports “fixed”. You do not take its word:

- 
$ con-duct ls -F prefix message exit_code logs/*info.json
-logs/1_  1: reproduce user's bug                      1
-logs/2_  2: simplified reproducer to test hypothesis  1
-logs/3_  3: verify fix                                0
-

Each row has the exact command and full stderr behind it. Auditing is reading three tagged records, not scrolling a transcript. (simulated; to be staged on a real duct bug)

+

Adopted inside tools

+

Tools wrap their own runs so users get records without asking.

+
    +
  • BABS github.com/PennLINC/babs
    Wraps every BIDS App job on SLURM and copies the record next to each subject's results. Why: per-subject peak memory and wall time to size the next batch; the app's stdout and stderr, with its warnings and other internals, for inspecting any job later.
    • +
  • ReproNim/containers github.com/ReproNim/containers
    REPRONIM_USE_DUCT=1 wraps each Singularity container run, e.g. under datalad containers-run. Why: a resource record for every containerized app run, saved with its results.
    • +
  • ReproStim github.com/ReproNim/reprostim
    The video-capture app wraps each ffmpeg recording (on in the default config); the record sits next to the video. Why: resource use for long, unattended capture sessions.
    • +
-

S5Second act: the record as provenance

-

DataLad. datalad run "duct <cmd>" commits command, logs, and the resulting diff together:

-
[DATALAD RUNCMD] excerpt from a real non-nebari datalad run "duct …" (TBD; or one sentence, no excerpt)
-

HPC. fMRIPrep minimal on 26 datasets: wall 3.7–8.2 h, peak RSS median 12.5 GiB, tail to 39.4 GiB; the 40 G grant fit 25 of 26. The outlier (64 BOLD runs) timed out at 24 h, OOM'd at 40 G, finished at 80 G in 43 h, duct peak 52.9 GB.

-

Caveat: sacct MaxRSS sits 5–8 GB above duct's peak on the same jobs (page cache under cgroup v2).

+

Comparison to Related Tools

+ + + + + + + + + +
teeGNU timeductworkflow managers
Snakemake, Nextflow		experiment trackers
MLflow, W&B
wrap any command, no code changes✓✓✓✗ rewrite as a workflow✗ add logging calls to your code
works anywhere, no scheduler✓◐ GNU only◐ Linux, macOS✓✓
stdout/stderr kept✓ merged✗✓✓ per task◐
exit code, invocation, environment✗◐ exit code✓✓◐
peak memory, CPU, whole tree✗◐ one process✓ sampled✓◐
time series, not just a peak✗✗✓◐◐
queryable across runs✗ grep✗✓◐ trace file★
+

✓ yes · ◐ partial · ✗ no · ★ richer than duct. Same one-word change: tee keeps the words, time one number, duct both and the rest.

+

When the others are better: many steps with declared inputs and outputs (a workflow manager knows the graph, reruns only what changed); comparing runs by parameters and metrics (a tracker keeps them with the artifacts, side by side).

-

S8Overhead, measured: duct wrapped around duct

+

Overhead

@@ -248,8 +305,8 @@

S8Overhead, measured: duct wrapped around duct

pids monitoredduct CPU, % of one coreduct RSSrecord bytes / report
11.6 ± 0.237 MB0.6 KB

Outer duct watching an inner duct that watches N idle processes, 5 min per phase, 1 s sampling. CPU is the python process as the outer duct saw it (phase mean; ± is the ps rounding bound), flat to 16 pids, about 2 % at 64 and 256: consistent with a slow rise, not a measured slope. Whole run by time -v, including duct's own ps calls: 3.3 % of one core; duct's sampler misses those few-ms children itself. Laptop i7-1280P, hybrid cores. Records: datalad run in ~/datasets/duct-in-duct.

-
-

S9Honesty box

+
+

Limitations

  • Output is written to disk: do not wrap commands that print secrets.
  • Programs that check isatty() take their non-interactive path (README has workarounds).
    • @@ -258,57 +315,12 @@

    S9Honesty box

  • Resource peaks paid off on SLURM; for agentic work the value has been the logs and con-duct ls. Wrap the runs you cannot afford to repeat.
-
-

S10Try it

- 
pip install con-duct      # or conda-forge
-duct <your command>
-con-duct ls
-

Live demo at the poster: con-duct ls over the real records above.

-
QR
-