Add POET backend model, weights download and skeleton - #121
Draft
C-Achard wants to merge 23 commits into
Draft
Conversation
C-Achard
added
enhancement
Contributor
There was a problem hiding this comment.
Pull request overview
Adds POET pose inference alongside DLCLive, including model-weight downloads, backend selection, and skeleton visualization.
Changes:
- Adds a pluggable pose processor and POET implementation.
- Adds POET weight management and backend-selection UI.
- Adds backend, processor, and dialog tests.
Reviewed changes
Copilot reviewed 14 out of 17 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
.gitignore |
Ignores downloaded temporary weights. |
dlclivegui/config.py |
Adds inference profiling flag. |
dlclivegui/gui/main_window.py |
Integrates backend selection and POET lifecycle. |
dlclivegui/gui/misc/eliding_label.py |
Adds a file-path comment. |
dlclivegui/gui/misc/weights_dialog.py |
Adds the POET download dialog. |
dlclivegui/services/inference/base.py |
Defines backend interfaces and POET enum. |
dlclivegui/services/inference/models/__init__.py |
Initializes the models package. |
dlclivegui/services/inference/models/poet/__init__.py |
Initializes the POET package. |
dlclivegui/services/inference/models/poet/poet_processor.py |
Implements POET inference and pose metadata. |
dlclivegui/services/inference/models/poet/skeleton.py |
Defines the POET skeleton. |
dlclivegui/services/inference/models/poet/weights.py |
Downloads and caches POET weights. |
dlclivegui/services/inference/processor.py |
Adds generic threaded pose processing. |
tests/custom_processors/test_processor_rec_context.py |
Updates renamed processor helper tests. |
tests/gui/main_window/test_backends.py |
Tests backend selection and configuration. |
tests/gui/ui_blocks/test_weights_dialog.py |
Tests weight-dialog behavior. |
tests/services/inference/models/test_poet.py |
Tests POET metadata and downloading. |
tests/services/inference/test_processor.py |
Tests generic pose processing. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+96
to
+103
| try: | ||
| from poet_live import POET, PostProcess | ||
| from poet_live.models.backbone import Backbone, Joiner | ||
| from poet_live.models.position_encoding import ( | ||
| PositionEmbeddingSine, | ||
| ) | ||
| from poet_live.models.transformer import Transformer | ||
| except ImportError as exc: |
Contributor
Author
There was a problem hiding this comment.
Pending decision re: POET updated code hosting
Comment on lines
+159
to
+164
| @torch.no_grad() | ||
| def get_pose( | ||
| self, | ||
| frame: np.ndarray, | ||
| frame_time: float | None = None, | ||
| ) -> np.ndarray | None: |
C-Achard
force-pushed
the
cy/poet-port-existing-demo
branch
from
August 28, 2026 13:11
5b9e10b to
0c52126
Compare
Introduces a new `ENABLE_MODELS_PROFILING` config toggle and adds an abstract `PoseBackend` base interface in inference services. This establishes a shared contract (`init_inference`, `get_pose`, `make_pose_packet`, `close`) for backend implementations and prepares the codebase for consistent backend behavior and optional profiling instrumentation.
Introduce a new `PoseProcessor` service for background pose estimation with a pluggable backend. It runs inference in a dedicated worker thread, uses a bounded queue to handle frame backpressure, emits Qt signals for results/lifecycle/errors, and tracks runtime stats (latency, FPS, dropped frames, and optional profiling timings). The worker also supports graceful stop/reset/shutdown behavior and backend cleanup.
Introduce a new POET inference module under `services/inference/models/poet`, including a `POETBackend` for model initialization and pose decoding, COCO-17 keypoint/skeleton definitions, and a Qt worker to download/cache default POET weights. This lays the groundwork for running POET-based multi-person pose estimation in the GUI.
Refactors inference plumbing to better support POET and improve runtime safety. This adds a string-based `PoseBackends` enum with `POET`, renames the model profiling config flag, tightens `PoseProcessor` typing/Qt imports, and makes worker shutdown/reset and backend-init failure cleanup explicit. It also lazy-loads POET dependencies with a clearer runtime error, decouples POET skeleton metadata from display-layer types, and fixes weight download finalization to atomically replace the configured destination while logging partial-file cleanup failures.
Introduce `create_pose_processor()` in the inference processor module to return the correct processor implementation for each backend (`DLCLiveProcessor` for `dlc`, `PoseProcessor` for `poet`) with explicit backend typing and unsupported-backend validation. Also update POET result construction to use the shared `POET_SKELETON_ID` constant instead of an inline string, keeping skeleton identifiers consistent.
Refactors inference handling to use a backend-agnostic pose processor interface and adds a Models > Pose backend menu to switch between DLCLive and POET. Updates model browsing/validation, signal wiring, control enablement, stats text, and inference lifecycle logic so backend-specific behavior is applied consistently (including recording hooks and shutdown). Also adds POET configuration support and persists the selected backend in settings.
Introduces a new non-modal POET weights dialog that lets users browse existing .pt/.pth files or download the default checkpoint with progress and error handling. Main window integration adds a "Manage POET weights" action, applies selected POET weights to the model path, and persists/restores both backend choice and backend-specific model paths via QSettings so switching between DLC and POET keeps the correct path. The POET default weights directory was also changed to `Path.cwd() / "TEMP_WEIGHTS/poet"`.
Introduce a new `test_processor.py` suite for inference processing. The tests cover processor configuration, pose signal emission with packets, worker startup on first frame, initialization failure handling, queue-drop accounting, reset behavior (including stop-worker failure), and backend factory creation paths for DLC/POET plus unknown-backend rejection.
Introduce GUI tests for backend handling in the main window, focused on the POET integration. The new coverage verifies switching UI/state updates, lazy POET processor creation, blocking backend changes during active inference, per-backend model path restoration, restoring POET from saved settings, and successful POET configuration/processor setup.
Introduce a new test module covering `PoetWeightsDialog` behavior: default state without an initial path, handling a valid initial path, emitting `weights_selected` on use, updating state after browse selection, and restoring button states during download cleanup.
Introduces a new `test_poet.py` suite covering POET inference model behavior and weight download handling. The tests validate checkpoint path/extension errors, ensure pose packets include correct POET metadata (including empty-detection cases), and verify `WeightsDownloadWorker` behavior for reusing existing files, writing downloaded content, emitting progress/finished signals, and cleaning up partial files on download failure.
Refactor the POET weights dialog into a download-only flow: remove manual file selection/use actions, show a fixed destination path with an eliding copy-friendly label, and auto-emit the downloaded path before closing on success. The dialog now provides clearer status/progress messaging and blocks closing while a download is active. Also ignore `/TEMP_WEIGHTS` in git.
Update the POET weights dialog flow to react to `weights_downloaded` instead of manual selection. Downloaded weights are now persisted as the last path, the backend is switched to POET, the model path field is populated automatically, and a status message is shown. The bounding-box coordinate controls were also repositioned in the form to sit with the bbox settings block for clearer UI grouping.
Reworks `test_weights_dialog.py` to match the updated POET download dialog behavior. The tests now validate default destination/status UI, progress updates, successful download completion and cleanup signaling, failure cleanup control restoration, and reject behavior during active vs idle download states.
Replace hardcoded "DLCLive" text in the main window inference startup UI with `_backend_display_name()`. This makes the running button label and initialization success/failure status messages reflect the active backend consistently.
Update `poet_default_weights_dir()` to use `~/.cache/dlclivegui/poet` instead of a working-directory temp path. This makes the weights location stable across runs and avoids writing model files into the project directory.
Refactors the POET processor to import and validate PyTorch only when POET inference is used, avoiding hard import failures at module load time. It adds explicit runtime errors for missing/incomplete torch installs, moves normalization tensor setup into initialization, replaces the decorator with an inference-mode context, and resets normalization state on shutdown for safer lifecycle handling.
Adds explicit cancellation support for POET weight downloads end-to-end. The weights dialog now requests thread interruption when the window is closed during an active download, shows a cancelling status, and prevents immediate close until the worker finishes. The download worker now checks interruption requests inside the read loop, aborts with a cancellation error, and reliably removes partial `.part` files on failure, while also tightening request/open handling with a timeout and clearer variable naming.
Updated pose processing to stop the loop on queue timeout and compute `queue_wait_time` from enqueue timestamp (`enq`) instead of local get-call timing, improving latency accounting. Expanded POET tests by introducing a reusable `urlopen` stub fixture, validating request/timeout behavior, asserting no `finished` signal on download failure, checking error-free successful downloads, and covering `skeleton_id` metadata in empty-detection pose packets.
Makes pose processor shutdown non-throwing by returning a success flag from `reset()`/`shutdown()`, and wires that into the main window so users get a status message when stop is deferred during model initialization. It also refactors the worker loop to honor stop requests around backend init, cleanly close backend state in a single `finally` path, and avoid emitting init errors when a stop was intentional. In POET inference, it now validates input as 3-channel image data and enforces contiguous frame memory before inference to prevent backend/runtime issues.
Revise processor reset tests to match the new non-throwing failure path: reset now returns `False` when the worker does not stop instead of raising. Also add coverage to ensure runtime statistics remain unchanged when reset fails.
Update POET model loading to use `torch.serialization.safe_globals([argparse.Namespace])` and `weights_only=True` when calling `torch.load`. This reduces unsafe pickle deserialization risk while still allowing legacy checkpoint metadata needed by the model state load.
C-Achard
force-pushed
the
cy/poet-port-existing-demo
branch
from
August 28, 2026 13:35
af341ae to
63e490a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add POET model backend, automated weights download, and skeleton display.