Skip to content

Add a twin loop that lifts simulator observations back onto the model - #54

Open
haphantran wants to merge 2 commits into
mainfrom
feature/twin-loop
Open

haphantran wants to merge 2 commits into
mainfrom
feature/twin-loop

Conversation

@haphantran

@haphantran haphantran commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

What this adds

Generation fidelity (benchmarks/fidelity) compares the model with the generated files. The twin loop continues from there: it compares the model with what the generated scene does when it runs, and brings the result back in the model's own terms.

  1. Stamped generation. The Smart Warehouse Omniverse templates now write the ID of the source model element on every layout prim (spatialDsl:elementId) and every warehouse_layout.json record (id).
  2. Observation log. observe.py records the world pose of every stamped prim as a simulator-neutral JSON Lines log. Its header carries a fingerprint of the generated files the run used (the layout JSON, the root scene layer, and its sublayers). The Isaac Sim bridge writes the log with --observe-out. run_standin_sim.py writes the same log with no simulator: the MAS brain drives the robots kinematically and --disturb displaces an element at a chosen time.
  3. Lifting and checking. benchmarks/twin-loop.ts lifts each sample onto a copy of the model (design pose plus the residual between observed and authored scene pose) and evaluates the design-time spatial rules, unchanged, on the lifted model.

The report lists:

  • realization offsets: undriven elements that are not at their authored pose in any sample;
  • runtime drifts: undriven elements that moved away from where they started, with the onset time and the displacement at onset and at the end;
  • rule verdicts that differ between the model and the run, compared sample by sample, with the first and last sample and the number of samples in which they differ;
  • a reconciliation proposal: the model changes that accepting the final observed state would make, and the rule issues that model would gain or lose.

Each finding names the model element, the prim, the generated file, and the template that produced it. A log whose samples miss an authored element is rejected rather than read as "nothing moved", and a log recorded against a different generation of the model is refused with the names of the generated files that differ.

Try it

cd benchmarks && npm run twin-loop

It runs four stand-in scenarios in about 6 seconds and exits non-zero if the findings differ from what was injected:

Scenario Injected Reported
clean nothing no findings; both robots move
runtime-drift Storage Rack #2 pushed 1.2 m into the aisle at t = 20 s one drift from t = 20 s, traced to its prim, warehouse_layout.usda, and the layout template; the rack aisle rule fails at runtime only; accepting the layout would introduce that issue
transient-drift the same push at t = 20 s, pushed back at t = 40 s one drift, -1200 mm at onset and 0 at the end; the aisle rule fails from 20 s to 39.5 s; nothing to reconcile
realization-offset Loading Dock 150 mm above its authored pose from the start one realization offset

benchmarks/README.md has the details and the command for analysing a log recorded in Isaac Sim.

Verified

  • npm run twin-loop: all four scenarios report exactly what was injected.
  • Checked by hand: a log missing one element in one sample is rejected; a log with a different generated layer is refused and names that layer; a log read from inside results/twin-loop/ is analysed (it is read before the folder is cleaned); a design violation that disappears at t = 20 s is reported for t = 20 s to 60 s.
  • npm run fidelity: 105,927 values compared, 0 mismatches, byte-identical regeneration, USD validation passes with the stamped templates.
  • python3 -m unittest test_nav test_brain test_runtime_modes test_layout test_observe: 25 tests pass (the pxr snapshot test runs when usd-core is installed).
  • Frontend: 55 suites, 428 tests pass. Backend and frontend typechecks pass. The new TypeScript passes a strict typecheck.
  • Backend tests were not run locally (no Postgres on the machine used). No backend or shared code changes; CI covers them.

Not verified, and known limits

  • --observe-out has not been run inside Isaac Sim yet. The recording code path in run_warehouse_sim.py is untested there.
  • The stand-in has no physics. It checks the loop and the tracing, not physical behaviour: deviations exist only where a scenario injects them.
  • The generated physics layer gives racks, docks, and conveyors static colliders, so no simulator can show them moving until some elements get dynamic bodies.
  • Lifting covers pose only (position and yaw).

Deployment note

Merging to main deploys to production. The only change that ships in the app is the bundled Smart Warehouse Omniverse example project, whose generated output gains the two ID fields. Everything else is under benchmarks/, examples/, and docs/.

Generation fidelity compares the model with the generated files. The twin
loop compares the model with what the generated scene does when it runs.

- The Smart Warehouse Omniverse templates stamp every layout prim and layout
  record with the ID of its model element (spatialDsl:elementId, id).
- observe.py records the world pose of every stamped prim as a
  simulator-neutral observation log. The Isaac Sim bridge writes it with
  --observe-out; run_standin_sim.py writes the same log with no simulator.
- benchmarks/twin-loop.ts lifts each sample onto a copy of the model (design
  pose plus the residual between observed and authored scene pose) and
  evaluates the design-time spatial rules on it. It reports realization
  offsets, runtime drifts, rule verdicts that differ between model and run,
  and a reconciliation proposal, each traced to the model element, prim,
  generated file, and template.
Copilot AI balanced review requested due to automatic review settings November 13, 2026 03:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues can delete observation inputs and produce incorrect drift, rule-verdict, and reconciliation results through incomplete scene fingerprints and mishandled pose observations.

7 open issues
What changed in this PR

Adds a twin loop that extends generation fidelity checks to runtime behavior, mapping simulator observations back onto the Smart Warehouse model and reusing its spatial rules.

Changes:

  • Stamps generated layout records and USD prims with model element IDs.
  • Adds observation logging, Isaac Sim recording support, and a simulator-free stand-in with injectable disturbances.
  • Adds pose lifting, rule comparisons, reconciliation reports, and three self-checking scenarios.
File Description
frontend/​src/​examples/​data/​smart-warehouse-omniverse-project.json Adds element ID stamps to bundled app templates.
examples/​omniverse-assets/​warehouse_sim/​test_observe.py Tests logging, stand-in disturbances, and USD snapshots.
examples/​omniverse-assets/​warehouse_sim/​run_warehouse_sim.py Adds runtime observation recording options.
examples/​omniverse-assets/​warehouse_sim/​run_standin_sim.py Adds kinematic simulation and injected disturbances.
examples/​omniverse-assets/​warehouse_sim/​README.md Documents recording and observation tests.
examples/​omniverse-assets/​warehouse_sim/​observe.py Defines the log format, recorder, and pose snapshots.
examples/​codegen-projects/​smart-warehouse-omniverse-project.json Adds element ID stamps to standalone generation templates.
docs/​user-guide/​omniverse-code-generation.md Documents generated element IDs and runtime tracing.
benchmarks/​twin-loop.ts Generates bundles, runs scenarios, and analyzes external logs.
benchmarks/​README.md Documents twin-loop setup, usage, and outputs.
benchmarks/​package.json Registers the twin-loop command.
benchmarks/​lib/​twin.ts Implements pose lifting, deviation analysis, and reconciliation reports.

Review effort: Balanced

Comment thread benchmarks/lib/twin.ts Outdated
Comment on lines +267 to +269
const traced = model.elements.filter(element => (
element.presentation?.position3D && authored.has(element.id) && first.poses.has(element.id)
));
Comment thread benchmarks/lib/twin.ts Outdated
};
traced.filter(element => !controlled.has(element.id)).forEach(element => {
const start = first.poses.get(element.id)!;
if (exceeds(residual(authored.get(element.id)!, start), tolerances)) {
Comment thread benchmarks/lib/twin.ts Outdated
Comment on lines +296 to +297
const end = last.poses.get(element.id);
if (onset && end) deviations.push(finding('runtime-drift', element, onset.t, start, end));
Comment thread benchmarks/lib/twin.ts Outdated
Comment on lines +342 to +343
...[...designIssues.entries()].filter(([key]) => !seen.has(key))
.map(([, issue]) => ruleFinding('design-violation-not-observed', issue)),
Comment thread benchmarks/twin-loop.ts
templateId: templates[index].id,
templateName: templates[index].name,
}));
fs.rmSync(outDir, { recursive: true, force: true });
Comment thread benchmarks/twin-loop.ts Outdated
const authored = readObservationLog(authoredPath);

const analyse = (name: string, title: string, observations: ObservationLog): TwinReport => {
if (observations.header.layoutSha256 !== layoutSha256) {
pose["x"] += disturbance["dx"]
pose["y"] += disturbance["dy"]
pose["z"] += disturbance["dz"]
pose["yaw"] += disturbance["dyaw"]
- Reject observation logs whose samples miss authored elements, instead of
  reading the gap as no movement.
- Report a realization offset only when no sample reaches the authored pose.
- Report drifts with the displacement at onset and at the end, so a drift
  that comes back is not shown as zero.
- Compare rule verdicts with the model sample by sample, so a design
  violation that disappears for part of a run is reported.
- Fingerprint the layout JSON, the root scene layer, and its sublayers,
  instead of the layout JSON alone, and name the files that differ.
- Read a log given with --observations before cleaning the results folder.
- Apply stand-in disturbances of robots to the pose the brain sees.
- Add a transient drift scenario.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants