tutorial-video-simulation

Scripts and tools to generate tutorial videos following a preset procedure. These tools depend on being able to interact with the window using Chrome Devtools on Linux.

Scripts and tools record a set procedure running in its own isolated Xvfb display, never your real desktop session (using the real desktop can cause window targeting issues).

NB, sometimes Chrome will run in both the current and isolated display simultanously.

By presenting apps within Chrome browser windows scripts move the view around move the mouse to set locations using the Chrome DevTools (CDT). Using burn in captions text can be put into the video to highlight offscreen actions like keystrokes and scolling. Clicks are acheived programatically by moving the mouse to the location, burning in a 'click' keystroke and activating the behaviour a click would initiate. This also accounts for poor aim on the part of CDT. While particularly useful for tutorials using browser based GUIs. Chrome can be quite robust at rentering different kinds of app window in chrome windows allowing us to show and point at locations in various kinds of window allowing us to produce windows standing in for other apps like terminal (using ttyd) file navigator (using file://) and text editor. Running these in chrome allows locations to be queryable DOM element, which can be pointed at etc like in a chrome based GUI. We can also apply set dressing (chrome app mode etc) to make these browser windows accessing or emulating other apps look more like the 'real' thing to not confuse viewers.

A useful benefit of generating tutorials this way is that it allows you to regenerate a tutorial as software layout, appearence and behaviour change, as the CDT uses primairly text rendered in the windows it should be able to locate the relevant items (if they are there still) and replicate the same behaviour of the tutorial. The expected procedure for using these scripts in as follows, clone this repo and install dependencies. Install the software needed for the demo, then run the Record.sh for the demo. This means you should be able to install your new version of the software for the tutorial and rerun the record.sh to get an updated video. The idea is that a Record.sh should use the version of the software on path, allowing these video recordings to be simply run as software updates. However, you can have the recording make a new version from an alternative source instead of using that on path (see checkerboard-calibration demo and BUILD_NEW_STRANDBRAID and STRAND_BRAID_TARGET_DIR for an example of this, where a new strand-braid build can be implemented).

note on Chrome app mode: App mode has one side effect worth knowing about: since Chrome no longer draws its own window chrome, openbox decides this window needs a title bar after all and adds one of its own (a normal Chrome window doesn't get one, since openbox recognizes it as already decorated). That extra title bar isn't visible to Chrome's own DOM/CDP measurements, so lib/session.sh's mouse-pointing math reads the window manager's own _NET_FRAME_EXTENTS property (via xprop) to correct for it — a 0,0,0,0 extent (the normal-window case) is a no-op, so this doesn't affect the BUI window's already-correct pointing.

Prerequisites (Linux only)

Six packages are hard requirements:

• ffmpeg — screen capture and caption burn-in
• xdotool — window placement, simulated typing, and keypresses
• Xvfb + openbox — a disposable virtual display and window manager
• ttyd — bridges the terminal's real PTY into a browser window instead of a native terminal emulator (see below for why)
• xprop (part of the x11-utils package) — reads a window's _NET_FRAME_EXTENTS so mouse-pointing can correct for a window-manager- added title bar on the terminal window (see below for why it has one at all)
• python3 is used here and there, mainly in burn_captions.py, needing no extra packages.

Everything else falls back to whatever's already installed instead of requiring anything new. The browser prefers google-chrome/chromium, launched with its own isolated profile, and only falls back to firefox if neither is found — on stock Ubuntu, firefox is a snap package whose sandbox blocks that isolated profile, a known limitation rather than a bug to fix.

Everything else used in the repo should be installed when the software being demonstrated is installed.

Layout

lib/
  session.sh          # shared bash helpers: isolated virtual display, tiled
                       # terminal+browser windows, simulated typing/keys,
                       # screen capture, and a timestamped caption log
  burn_captions.py     # overlays lib/session.sh's caption log onto the
                       # captured video (no dependencies, run with python3)
Strand-Braid-Demos/
  strand-cam-intro/
    record.sh          # tutorial for launching Strand Camera
                        # from the command line, watch the live view, Ctrl+C, relaunch with
                        # `--camera-name` explicit.
  braid-intro/
    record.sh          # tutorial for launching Braid from the
                        # command line: `braid-run config.TOML`, wait for
                        # all cameras to synchronize, scroll up to the QR
                        # code, open the GUI, cycle through every camera,
                        # Ctrl+C, relaunch, then (new, not in the original)
                        # close the GUI window and reopen it via the
                        # terminal's printed URL. 
  checkerboard-calibration/
    record.sh          # regenerates an "intrinsic checkerboard
                        # calibration" tutorial video: launch strand-cam,
                        # enable Checkerboard Calibration, watch detections
                        # accumulate against a real recorded checkerboard
                        # video played back through the `video-file` backend
                        # (no real camera hardware needed), click "Perform
                        # and Save Calibration". Then navigate to the produced calibration.

Running instructions

1. Get the code: git clone this repo. If a demo drives software that isn't already installed, also grab a checkout of that software (see the demo's own notes, e.g. Strand-Braid-Demos.md).
2. Install the hard requirements:

   sudo apt-get update
   sudo apt-get install -y ffmpeg xdotool xvfb openbox ttyd x11-utils

3. Install/build the demoed software, or point the demo at a build of it — see the demo's own notes for the specific override env vars (e.g. checkerboard-calibration's BUILD_NEW_STRANDBRAID/ STRAND_BRAID_TARGET_DIR).
4. Run it:

   cd Strand-Braid-Demos/<demo-directory>
   ./record.sh

   Each demo supports its own env vars (e.g. forcing a hardware/simulation backend) — see that demo's notes for what's available.
5. Check the output: out/<demo-name>.mp4 next to record.sh (plus out/raw.mp4 and out/events.jsonl). Compare it against any existing reference video, tweak sleep durations/captions and rerun if needed. This is also the point of this whole setup: when a demoed app's CLI or GUI drifts and a video goes stale, rerunning record.sh regenerates it against current behavior, with no manual re-recording needed.

License

Extracted from strand-braid and dual-licensed under the same terms, at your option:

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

See COPYRIGHT for details.

Adding another tutorial

Create a new subdirectory with its own record.sh that sources lib/session.sh (however many ../ reach it from wherever the new subdirectory lives — e.g. ../../lib/session.sh for a tutorial nested one level under Strand-Braid-Demos/, matching that group's own scripts) and sets SCRIPT_NAME before doing so (used to namespace that script's temp/work directory). See Strand-Braid-Demos/strand-cam-intro/record.sh for the pattern: start_display → start_capture → open windows → type_in/ send_keys/log_event for the actions being demonstrated → stop_capture → burn_captions.py.