Add task-first GenSim action and orchestration pipeline - #531

Open

skywhite1024 wants to merge 90 commits into

mainfrom

ljd/action_engine_refactor

Commits on Aug 24, 2026

Align the doc, test, and client with the newest .env setup (image segmentation client)
MuziWong
committed
11f5f0c
Add image generation client + update .env + update doc + test files
MuziWong
committed
4c68adf
Reformat the code, add scene edit basic framework, notice that the docs and tests may not be updated
MuziWong
committed
88de161
Basic design of scene graph
MuziWong
committed
139850b
i
MuziWong
committed
ecaacda
Fixed the prompt of scne understanding: delete the location in description, and add check code
MuziWong
committed
51e1c64
modify the prompt of scene understanding, to avoid a very long description in objects' names
MuziWong
committed
64ca9f1
finish the llm understand scene edit
MuziWong
committed
4236f9d
finish the scene edit plan -> updated scene graph
MuziWong
committed
e52db87
move client try catch outside the understand_scene
MuziWong
committed
95f4759
finish part of the scene edit: from user prompt to simreadyed asset, but no real-world scale...
MuziWong
committed
a72f101
finish basic simready real world scale + semantic-based rotation
MuziWong
committed
a2398eb
Moved the table support infos into simready.
MuziWong
committed
ccab0f3
Finished scene edit
MuziWong
committed
9b27be2
fix some bug before pr
MuziWong
committed
78e01a3
fix a real-world size bug, using z-up mesh
MuziWong
committed
3cd9147
update image-to-scene scene understanding: let vlm give orientation state for assets who need to be calibrated
MuziWong
committed
69431c2
fix big-font problem in asset_masks_with_ids; add scene graph based clibration in image-conditioned scene engine pipeline (but only calibrated the upright bottle-like assets currently)
MuziWong
committed
083d614
run black
MuziWong
committed
c5e9087
run black to tests/
MuziWong
committed
4779f01
replace hard-code bottle-z-up with VLM auto-analyze
MuziWong
committed
4a792f0
make assets group layout optimizer more rubust
MuziWong
committed
d3f4365
delete the hard-coded spatial check in assets' names and descriptionsa
MuziWong
committed
1d5e367
replace jiange's pca heuristic method with VLM operate rotation tools follow the assets' orientation_states
MuziWong
committed
9eb4ca1
ignore the fixed 2d aabb overlap when doing the on-table optimization
MuziWong
committed
ccb4a94
finished the table as root layout optimization (but still needs to be tested)
MuziWong
committed
031b5f8
finish a simple on-relationship optimizer
MuziWong
committed
ae4095e
add the heuristic attaches in collision optimization
MuziWong
committed
73d3c7f
replace the assets gravity settler with gravity settler
MuziWong
committed
a03cda1
use gravity settler in parent surface layout
MuziWong
committed
6f8e26e
Updated docs
MuziWong
committed
a84be90
fix(scene-engine): validate edit categories and preserve batched planar relations
MuziWong
committed
654165c
Outside-table-2d-region checker now does not check those who did not participate in the optimization
MuziWong
committed
e92e978
fix(scene-engine): delete the function export
MuziWong
committed
ad23063
finished scene-graph based on realtion addtion in image as input pipeline(not finished, but can be ran)
MuziWong
committed
301d7eb

Commits on Aug 25, 2026

Delete the debug code for coarse layout from server output
MuziWong
committed
86ef235
support lying object VLM-based simready
MuziWong
committed
0865b90
add: 1. articulated or not 2. segmented RGBA image for each asset
MuziWong
committed
3f8cba3

Commits on Aug 27, 2026

Commits on Aug 28, 2026

Commits on Aug 31, 2026

add articulation affordance | add turn knob

matafela
authored and
skywhite1024
committed
245d716
update

matafela
authored and
skywhite1024
committed
3ec392c
update

matafela
authored and
skywhite1024
committed
647b406
update press button

matafela
authored and
skywhite1024
committed
9d3c9cc
fix test

matafela
authored and
skywhite1024
committed
8c5d6e7
add pull push articulation part

matafela
authored and
skywhite1024
committed
deb94c4
update

matafela
authored and
skywhite1024
committed
91d7137
fix module import

matafela
authored and
skywhite1024
committed
c0962b6
rename

matafela
authored and
skywhite1024
committed
6926307
Add unified policy evaluation for RL checkpoints (#510)

acrlw
authored and
skywhite1024
committed
1b51dd0
Revert "Add unified policy evaluation for RL checkpoints (#510)"

acrlw
authored and
skywhite1024
committed
a4636fa
stash

matafela
authored and
skywhite1024
committed
2953d0b
fix

matafela
authored and
skywhite1024
committed
ccfb7a7
add pour action

matafela
authored and
skywhite1024
committed
34178ca
fix docs

matafela
authored and
skywhite1024
committed
bd3c0de
feat(scene-engine): add versioned authoring API
skywhite1024
committed
adfe065
feat(atomic-actions): preserve GenSim execution contracts
skywhite1024
committed
838b167
feat(gen-sim): migrate action engine onto axis baseline
skywhite1024
committed
168ed64
feat(gen-sim): migrate task orchestration onto scene baseline
skywhite1024
committed
9f28ef0
feat(action-engine): add body-grasp planning and verified cleanup for E2 axis alignment
skywhite1024
committed
33f570d
feat(action-engine): add body-grasp planning and verified cleanup for E2 axis alignment
skywhite1024
committed
e8a724a
feat(gen-sim): add failure continuation and task video archival
skywhite1024
committed
063cd5e
feat(action-engine): implement single-arm E3 pour workflow
skywhite1024
committed
0219d5a
feat(action-engine): add contract-driven payload-aware coordinated manipulation
skywhite1024
committed
ebdb66a
feat(gen-sim): add configurable PGI and Robotiq gripper profiles
skywhite1024
committed
a371e85
fix(gen-sim): preserve source videos and reduce recording frequency
skywhite1024
committed
62ab860
fix(action-engine): align E2 upright orientation to the local z-axis
skywhite1024
committed
f611c0f
feat(gen-sim): add configurable planner modes and runtime diagnostics
skywhite1024
committed
fa26c34
feat(action-engine): extend E4 handover with hold and placement outcomes
skywhite1024
committed
8caf7f7
fix(action-engine): harden E2 and E5 post-action cleanup safety
skywhite1024
committed
ad283d5
refactor(action-engine): use stable entity IDs for object semantics
skywhite1024
committed
d8c5d2c
refactor(action-engine): use stable entity IDs for object semantics
skywhite1024
committed
7def96b
feat(gen-sim): add IK solver profiles and E5 grasp diagnostics
skywhite1024
committed
7304a92
fix(gen-sim): harden coordinated pickment safety
skywhite1024
committed
5f5dbce
temp
skywhite1024
committed
1921e9e
refactor(action-engine): implement E2 upright placement with pickup and transport
skywhite1024
committed
8899abc
Merge remote-tracking branch 'origin/muzi/update_image_input' into ljd/action_engine_refactor_v21
skywhite1024
committed
e4468a8
feat(gen-sim): support articulated objects in scene generation and execution
skywhite1024
committed
e8c86b3

Commits on Sep 1, 2026

Commits on Sep 3, 2026