Touchless Computer Controller

Control a Windows desktop with hand gestures, air-drawn shortcuts, and a depth-aware table keyboard. Touchless Computer Controller combines a laptop webcam, MediaPipe, OpenCV, PyAutoGUI, and an optional Intel RealSense D435i into a local, real-time interaction system.

Demo

demo-web.mp4

If the embedded player is unavailable, open the repository copy. The full 1:50 demonstration uses a web-optimized H.264 encode; the original 4K recording is intentionally excluded.

Highlights

• Relative cursor movement that supports lifting and repositioning the hand
• Reliable click, double-click, drag, right-click, and modal scrolling gestures
• Continuous volume control with an on-screen volume indicator
• Window switching and Windows shortcuts through deliberate gestures
• Red air-drawn shortcut traces that disappear automatically
• Green D and e recognition for Win+D and Win+E
• Emergency pause and resume using thumbs-down and thumbs-up
• Optional two-hand Portuguese table keyboard powered by RealSense depth
• Always-on-top camera preview that survives changing windows and Show Desktop
• Local frame processing with no video upload or recording by the application

Gesture reference

Gesture	Action
Point with one index finger	Move the cursor
Touch thumb to a straight index finger	Click; keep holding to drag
Raise index and middle, then touch thumb to index	Double-click
Point while showing the other hand as an open palm	Right-click
Hold a V sign for 0.65 seconds, then move it vertically	Enter and use scroll mode
Hold a fist while scrolling	Exit scroll mode
Raise index, middle, and ring	Temporarily clutch/reposition the pointer
Rotate an open palm left or right	Lower or raise volume
Swipe horizontally with a fist	Switch windows
Raise index and pinky and draw	Run an air-drawn shortcut
Hold the left-facing pistol pose, then draw D or e	Win+D or Win+E
Hold thumbs-down / thumbs-up	Pause / resume
Point one index finger downward	Open the RealSense table keyboard

How it works

Laptop camera -> hand landmarks -> gesture state machine -> desktop actions
                                         |
                                         +-> drawing and volume overlays

Downward-point gesture -> release laptop camera -> RealSense RGB + depth
                                                -> table contact detection
                                                -> Portuguese virtual keyboard

Normal control uses the laptop webcam. MediaPipe landmarks are classified by a stateful gesture controller with activation holds, cooldowns, hysteresis, and latched modes to reduce accidental actions. When table-keyboard mode is requested, the laptop camera is released before the RealSense pipeline opens, preventing camera contention.

Requirements

• Windows 10 or 11
• Python 3.10 or newer (64-bit Python 3.11 or 3.12 recommended)
• A laptop webcam
• Optional: Intel RealSense D435i for the table keyboard
• Even lighting and enough space for the complete hand to remain visible

The gesture controller can run without RealSense. Windows is the primary supported platform because volume integration and window shortcuts use Windows facilities.

Installation

git clone https://github.com/Hugo132645/touchless-controller.git
cd touchless-controller
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -e . --no-deps

requirements.txt installs the complete Windows configuration, including RealSense and native volume support. Contributors can instead install the package and optional dependency groups directly:

pip install -e ".[windows,realsense]"

If the RealSense keyboard is not needed, use pip install -e ".[windows]".

Quick start

Start with dry-run mode. It displays recognition results without sending mouse or keyboard input:

touchless-controller --dry-run

When the gestures look correct, enable desktop control:

touchless-controller

Press Space in the preview to pause and Q to quit. PyAutoGUI's corner fail-safe is also enabled. Keep a physical mouse available while learning or calibrating the gestures.

Using the gestures

Pointer and mouse buttons

Point with the index finger while curling the other fingers. Cursor movement is relative, so leaving the frame and returning does not pull the pointer back toward an edge. Raise index, middle, and ring to clutch: reposition the hand, then return to index-only pointing to establish a new movement anchor.

Touch the thumb to the middle joint of a straight index finger and release before 0.65 seconds for a click. Keep the contact longer to hold the left button while the pointer continues moving, enabling drag-and-drop. Raise the middle finger during the thumb contact for a native double-click. To right-click, continue pointing with one hand and show a fully open palm with the other.

Modal scrolling

Hold a clear V sign for 0.65 seconds to enter scroll mode. Move the two raised fingers up or down for direct, high-sensitivity scrolling. Briefly leave the V pose to reposition; forming it again creates a fresh anchor without a jump. Detection errors do not exit the mode or leak into other actions. Hold a fist for 0.35 seconds to close scroll mode.

Volume and windows

Show a fully open palm. A front-facing palm is neutral; rotate it about 90 degrees in one direction to lower the volume and the opposite direction to raise it. Return to front-facing to stop. A temporary system-style overlay displays the current level.

Make a closed fist and perform a deliberate horizontal swipe to switch windows.

Air drawing and Windows commands

Raise only index and pinky to draw a bright-red shortcut trace. Lower either finger to finish it; the trace remains above other windows for three seconds. Commands are configured under air_draw.shortcuts in the YAML configuration.

For Windows commands, point index and middle to the left while the thumb points up and ring and pinky remain curled. Hold for one second, then draw with the green trace:

• Capital D triggers Win+D and shows the desktop.
• Lowercase e triggers Win+E and opens File Explorer.
• A small or unrecognized drawing presses the Windows key by itself.

Hold a fist briefly to submit the drawing.

Emergency pause

Curl the four fingers and hold thumbs-down for one second to pause all actions. Hold thumbs-up for one second to resume. These gestures are held and latched deliberately to avoid accidental toggles.

RealSense table keyboard

Before opening the keyboard, focus the text field that should receive input. Point one index finger completely downward to switch from the laptop webcam to the D435i. The preview displays a Portuguese keyboard over the downward-facing table view.

Keep hands and objects off the white table during the initial surface calibration. Aligned depth data estimates fingertip contact; either index finger can type. Keys use true key-down/key-up events, so one finger can hold Shift or AltGr while the other selects a letter or symbol. Keycaps change visibly to show the active modifier layer. Select the red EXIT key to close RealSense and return to laptop-camera gestures.

The included configuration currently enables the automatic safe-area layout for the temporary diagonal camera setup. Once the overhead mount is fixed, set keyboard.automatic_layout to false to restore two-corner keyboard calibration.

Configuration

Copy the documented defaults and override values for the current camera and room:

Copy-Item config.example.yaml config.yaml
touchless-controller --config config.yaml

Useful controls include:

Setting	Purpose
camera.index	Laptop webcam selected by OpenCV
control.cursor_gain	Relative pointer movement multiplier
control.smoothing	Cursor responsiveness versus steadiness
control.drag_hold_seconds	Click-to-drag transition time
control.scroll_motion_gain	Scroll distance produced by vertical V movement
control.scroll_invert	Reverses the scroll direction
control.volume_invert	Reverses left/right volume rotation
keyboard.automatic_layout	Chooses automatic or two-corner keyboard placement

See config.example.yaml for every available option and its current tuned value.

Project structure

src/touchless_controller/
├── app.py         # Camera lifecycle and gesture/keyboard mode switching
├── tracking.py    # MediaPipe hand landmark detection
├── controller.py  # Gesture state machine and arbitration
├── keyboard.py    # RealSense table keyboard and contact handling
├── actions.py     # Mouse, keyboard, window, and volume operations
├── overlay.py     # Always-on-top drawing and volume overlays
├── camera.py      # OpenCV and RealSense capture backends
├── geometry.py    # Geometry and air-drawing recognition
├── config.py      # Defaults and YAML configuration loading
└── cli.py         # Command-line entry point

Development

pip install -e ".[dev,windows,realsense]"
pytest
ruff check .

Geometry and keyboard behavior have hardware-independent unit tests. Camera and desktop-integration changes should additionally be verified in --dry-run before a live test.

Privacy and safety

Camera frames are processed locally in memory and are not recorded or uploaded by the application. The MediaPipe hand-landmark model is downloaded on first use and cached in .models/; subsequent tracking is local.

Camera-driven input can still misclassify a pose. Use dry-run first, pause before entering passwords or performing destructive operations, and keep the physical PyAutoGUI corner fail-safe accessible.

Troubleshooting

• Camera does not open: close other camera applications or change camera.index.
• Cursor direction is reversed: adjust camera.mirror for the laptop webcam.
• Scrolling is reversed: set control.scroll_invert: true.
• Gestures trigger too easily: increase the corresponding hold, cooldown, or movement threshold in config.yaml.
• RealSense cannot start: install the realsense extra and confirm the D435i is visible in Windows Device Manager.
• MediaPipe installation fails: use a supported 64-bit Python release, preferably Python 3.11 or 3.12.

License

Released under the MIT License.