Hi, thanks for releasing X-NavDP!
Appendix B describes training with 56 scene processes, each running 8 environments, giving 448 parallel environments. Each process keeps its assigned scene and periodically switches embodiments. Paper
However, the README example launches 8 processes, and the public implementation uses:
torch.cuda.set_device(local_rank)
scene_index = rank % num_scenes
See GPU assignment and scene assignment.
My understanding is that, on one node with 8 GPUs, this command trains on only 8 fixed scenes. Increasing NPROC_PER_NODE to 16 or 56 would select GPU indices beyond the available devices, rather than automatically assigning multiple scene processes to each GPU.
Could you clarify:
- What hardware and process topology were used for the full 56-scene experiment: number of nodes, GPUs per node, total DDP ranks, and scene processes per GPU?
- If multiple scene processes shared a GPU, could you provide the corresponding GPU-assignment implementation and launch command?
- What is the recommended way to reproduce full 56-scene training on 8 GPUs? Could you also share the GPU model/memory capacity used and, if available, peak memory usage?
If my reading of the implementation is incorrect, please point me to the relevant scheduling logic. An exact launch command/configuration for the Table 1 experiment would be very helpful.
Thank you!
Hi, thanks for releasing X-NavDP!
Appendix B describes training with 56 scene processes, each running 8 environments, giving 448 parallel environments. Each process keeps its assigned scene and periodically switches embodiments. Paper
However, the README example launches 8 processes, and the public implementation uses:
torch.cuda.set_device(local_rank)
scene_index = rank % num_scenes
See GPU assignment and scene assignment.
My understanding is that, on one node with 8 GPUs, this command trains on only 8 fixed scenes. Increasing NPROC_PER_NODE to 16 or 56 would select GPU indices beyond the available devices, rather than automatically assigning multiple scene processes to each GPU.
Could you clarify:
If my reading of the implementation is incorrect, please point me to the relevant scheduling logic. An exact launch command/configuration for the Table 1 experiment would be very helpful.
Thank you!