Skip to content

VR-first UI: floating windows, left-hand card, keyboard, and dictation - #286

Closed
kburzota wants to merge 3 commits into
AgentSystemLabs:mainfrom
kburzota:feat/vr-ui
Closed

kburzota wants to merge 3 commits into
AgentSystemLabs:mainfrom
kburzota:feat/vr-ui

Conversation

@kburzota

@kburzota kburzota commented Oct 3, 2026 •

Copy link
Copy Markdown

Summary

Stacked on #285 (merge that first; this PR’s unique commits sit on top of the WebXR base).

Makes every office window usable inside the headset without a physical keyboard:

  • Open windows are mirrored onto a floating panel you click with the controller rays (iframe windows still leave VR)
  • A palette-style action card on the left controller (Find, Menu, boards, Keyboard, Dictate, Exit VR, …)
  • Full virtual keyboard when a text field is focused; terminal “Type…” opens it instead of exiting VR
  • Hold A to dictate (browser speech recognition, or Whisper via POST /api/transcribe when OPENAI_API_KEY is set)
  • Point at a wall card and pull the trigger to open it in front of you; right grip takes an issue note

Test plan

  • npm run typecheck
  • npm test (613 pass)
  • npm run build
  • Headless canvas preview of keyboard + left-hand card (scripts/xr-ui-preview.png)
  • On a Quest: Enter VR, open Settings from the left-hand card, click around, type with the keyboard
  • Hold A to dictate into a prompt / terminal (with and without browser speech recognition)
  • Trigger on an issues / PR sticky note opens a floating window; right grip takes an issue note
  • Open a terminal panel, use Type…, confirm VR stays on

kburzota and others added 3 commits October 2, 2026 19:20
Enter VR with head-tracked locomotion and controller mappings, and keep the frame loop alive through XR sessions via setAnimationLoop.

Co-authored-by: Cursor <cursoragent@cursor.com>
…R mode.

Co-authored-by: Cursor <cursoragent@cursor.com>
Mirror HTML windows onto a floating panel with controller clicks, add a
left-hand action card, a full virtual keyboard, and hold-A dictation
(Web Speech, or Whisper via /api/transcribe). Wall cards open in front
of you; the right grip takes an issue note.

Co-authored-by: Cursor <cursoragent@cursor.com>
@kburzota

kburzota commented Oct 3, 2026

Copy link
Copy Markdown
Author

Opened against the wrong repo; moved to my fork.

@kburzota kburzota closed this Oct 3, 2026
@schnepg1

schnepg1 commented Oct 3, 2026

Copy link
Copy Markdown

very nice, I had the idea already, too!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants