Repository navigation
Add a twin loop that lifts simulator observations back onto the model - #54
Open
haphantran wants to merge 2 commits into
Open
haphantran wants to merge 2 commits into
haphantran wants to merge 2 commits into
Conversation
Generation fidelity compares the model with the generated files. The twin loop compares the model with what the generated scene does when it runs. - The Smart Warehouse Omniverse templates stamp every layout prim and layout record with the ID of its model element (spatialDsl:elementId, id). - observe.py records the world pose of every stamped prim as a simulator-neutral observation log. The Isaac Sim bridge writes it with --observe-out; run_standin_sim.py writes the same log with no simulator. - benchmarks/twin-loop.ts lifts each sample onto a copy of the model (design pose plus the residual between observed and authored scene pose) and evaluates the design-time spatial rules on it. It reports realization offsets, runtime drifts, rule verdicts that differ between model and run, and a reconciliation proposal, each traced to the model element, prim, generated file, and template.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can delete observation inputs and produce incorrect drift, rule-verdict, and reconciliation results through incomplete scene fingerprints and mishandled pose observations.
7 open issues
Reject samples missing expected authored model elements · New Check all samples before classifying a realization offset · New Report the drift onset pose, not only the final pose · New Aggregate design-rule recovery across samples · New Cleanup can delete the observations input before reading it · New Layout hash omits scene layers, allowing stale runs to pass validation · New Apply robot disturbance deltas to the reported pose · New
What changed in this PR
Adds a twin loop that extends generation fidelity checks to runtime behavior, mapping simulator observations back onto the Smart Warehouse model and reusing its spatial rules.
Changes:
- Stamps generated layout records and USD prims with model element IDs.
- Adds observation logging, Isaac Sim recording support, and a simulator-free stand-in with injectable disturbances.
- Adds pose lifting, rule comparisons, reconciliation reports, and three self-checking scenarios.
| File | Description |
|---|---|
frontend/src/examples/data/smart-warehouse-omniverse-project.json |
Adds element ID stamps to bundled app templates. |
examples/omniverse-assets/warehouse_sim/test_observe.py |
Tests logging, stand-in disturbances, and USD snapshots. |
examples/omniverse-assets/warehouse_sim/run_warehouse_sim.py |
Adds runtime observation recording options. |
examples/omniverse-assets/warehouse_sim/run_standin_sim.py |
Adds kinematic simulation and injected disturbances. |
examples/omniverse-assets/warehouse_sim/README.md |
Documents recording and observation tests. |
examples/omniverse-assets/warehouse_sim/observe.py |
Defines the log format, recorder, and pose snapshots. |
examples/codegen-projects/smart-warehouse-omniverse-project.json |
Adds element ID stamps to standalone generation templates. |
docs/user-guide/omniverse-code-generation.md |
Documents generated element IDs and runtime tracing. |
benchmarks/twin-loop.ts |
Generates bundles, runs scenarios, and analyzes external logs. |
benchmarks/README.md |
Documents twin-loop setup, usage, and outputs. |
benchmarks/package.json |
Registers the twin-loop command. |
benchmarks/lib/twin.ts |
Implements pose lifting, deviation analysis, and reconciliation reports. |
Review effort: Balanced
Comment on lines
+267
to
+269
| const traced = model.elements.filter(element => ( | ||
| element.presentation?.position3D && authored.has(element.id) && first.poses.has(element.id) | ||
| )); |
| }; | ||
| traced.filter(element => !controlled.has(element.id)).forEach(element => { | ||
| const start = first.poses.get(element.id)!; | ||
| if (exceeds(residual(authored.get(element.id)!, start), tolerances)) { |
Comment on lines
+296
to
+297
| const end = last.poses.get(element.id); | ||
| if (onset && end) deviations.push(finding('runtime-drift', element, onset.t, start, end)); |
Comment on lines
+342
to
+343
| ...[...designIssues.entries()].filter(([key]) => !seen.has(key)) | ||
| .map(([, issue]) => ruleFinding('design-violation-not-observed', issue)), |
| templateId: templates[index].id, | ||
| templateName: templates[index].name, | ||
| })); | ||
| fs.rmSync(outDir, { recursive: true, force: true }); |
| const authored = readObservationLog(authoredPath); | ||
|
|
||
| const analyse = (name: string, title: string, observations: ObservationLog): TwinReport => { | ||
| if (observations.header.layoutSha256 !== layoutSha256) { |
| pose["x"] += disturbance["dx"] | ||
| pose["y"] += disturbance["dy"] | ||
| pose["z"] += disturbance["dz"] | ||
| pose["yaw"] += disturbance["dyaw"] |
- Reject observation logs whose samples miss authored elements, instead of reading the gap as no movement. - Report a realization offset only when no sample reaches the authored pose. - Report drifts with the displacement at onset and at the end, so a drift that comes back is not shown as zero. - Compare rule verdicts with the model sample by sample, so a design violation that disappears for part of a run is reported. - Fingerprint the layout JSON, the root scene layer, and its sublayers, instead of the layout JSON alone, and name the files that differ. - Read a log given with --observations before cleaning the results folder. - Apply stand-in disturbances of robots to the pose the brain sees. - Add a transient drift scenario.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What this adds
Generation fidelity (
benchmarks/fidelity) compares the model with the generated files. The twin loop continues from there: it compares the model with what the generated scene does when it runs, and brings the result back in the model's own terms.spatialDsl:elementId) and everywarehouse_layout.jsonrecord (id).observe.pyrecords the world pose of every stamped prim as a simulator-neutral JSON Lines log. Its header carries a fingerprint of the generated files the run used (the layout JSON, the root scene layer, and its sublayers). The Isaac Sim bridge writes the log with--observe-out.run_standin_sim.pywrites the same log with no simulator: the MAS brain drives the robots kinematically and--disturbdisplaces an element at a chosen time.benchmarks/twin-loop.tslifts each sample onto a copy of the model (design pose plus the residual between observed and authored scene pose) and evaluates the design-time spatial rules, unchanged, on the lifted model.The report lists:
Each finding names the model element, the prim, the generated file, and the template that produced it. A log whose samples miss an authored element is rejected rather than read as "nothing moved", and a log recorded against a different generation of the model is refused with the names of the generated files that differ.
Try it
It runs four stand-in scenarios in about 6 seconds and exits non-zero if the findings differ from what was injected:
warehouse_layout.usda, and the layout template; the rack aisle rule fails at runtime only; accepting the layout would introduce that issuebenchmarks/README.mdhas the details and the command for analysing a log recorded in Isaac Sim.Verified
npm run twin-loop: all four scenarios report exactly what was injected.results/twin-loop/is analysed (it is read before the folder is cleaned); a design violation that disappears at t = 20 s is reported for t = 20 s to 60 s.npm run fidelity: 105,927 values compared, 0 mismatches, byte-identical regeneration, USD validation passes with the stamped templates.python3 -m unittest test_nav test_brain test_runtime_modes test_layout test_observe: 25 tests pass (the pxr snapshot test runs when usd-core is installed).Not verified, and known limits
--observe-outhas not been run inside Isaac Sim yet. The recording code path inrun_warehouse_sim.pyis untested there.Deployment note
Merging to
maindeploys to production. The only change that ships in the app is the bundled Smart Warehouse Omniverse example project, whose generated output gains the two ID fields. Everything else is underbenchmarks/,examples/, anddocs/.