Skip to content

test(vln): add paired navigation benchmarks - #11

Open
yufoo1 wants to merge 6 commits into
BUAA-CI-LAB:mainfrom
yufoo1:benchmark/vln-navigation
Open

yufoo1 wants to merge 6 commits into
BUAA-CI-LAB:mainfrom
yufoo1:benchmark/vln-navigation

Conversation

@yufoo1

@yufoo1 yufoo1 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add Habitat navigation benchmarks for ActiveVLN and StreamVLN using the versioned model-service HTTP API. Preserve StreamVLN primitive-frame history and independent tensor-batch episodes; record executed actions, model/transport timing, SR/SPL evidence and resumable per-episode results. Validate manifests, checkpoint action grammar, batch identity and failure cleanup.

Pin third_party/embodiinfer to d8cc7105e52ae08302019c620cf656a1c69be2b0, the merge commit of EmbodiInfer PR 12 already on CILAB main. Inference code and device-specific first-128 summaries remain in EmbodiInfer; this change contains no Go2 deployment or asynchronous physical-navigation changes.

Type of change

  • New capability (simulator benchmark and HTTP compatibility)
  • Documentation

Verification

CUDA_VISIBLE_DEVICES='' PYTHONPATH=src NO_PROXY=localhost,127.0.0.1 no_proxy=localhost,127.0.0.1 python -m pytest -q tests/test_activevln_navigation_benchmark.py tests/test_streamvln_navigation_benchmark.py

58 passed after the pointer update; the merge commit has the same source tree as the previously validated Infer revision. Ruff check/format passed for all 13 changed Python files; diff whitespace and Markdown links passed. No GPU, Habitat, real-weight or physical-robot workloads were started for this integration; published measurements refer to their preserved frozen source revisions.

Process boundaries

  • Model services communicate through the versioned HTTP contract; no EmbodiInfer runtime imports enter src/embodirun.
  • Robot/simulator adapters and policy bindings remain separate.
  • No new mandatory dependencies are introduced.

Checklist

  • Conventional Commit subject and configured human DCO sign-off.
  • Benchmark documentation records protocol and measurement boundaries.
  • No weights, datasets, recordings, credentials, private host addresses or personal data are added.
  • Contribution follows the repository's Apache-2.0 license.

Measure ActiveVLN and StreamVLN navigation through the versioned
HTTP model-service contract with fixed manifests, primitive history,
per-episode evidence and explicit tensor-batch validation.

Pin the published model-only EmbodiInfer optimization revision.
Keep physical-robot deployment and asynchronous navigation changes
on their existing branches.

Signed-off-by: yufoo1 <yufoo1.cs@gmail.com>
Signed-off-by: yufoo1 <yufoo1.cs@gmail.com>
@yufoo1 yufoo1 mentioned this pull request Oct 28, 2026
9 tasks done
Signed-off-by: yufoo1 <yufoo1.cs@gmail.com>
Signed-off-by: yufoo1 <yufoo1.cs@gmail.com>
Signed-off-by: yufoo1 <yufoo1.cs@gmail.com>
Signed-off-by: yufoo1 <yufoo1.cs@gmail.com>
@yufoo1
yufoo1 marked this pull request as ready for review October 28, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants