From 85f64b7450ec90eacd2126bcdc2045b16e2b9d83 Mon Sep 17 00:00:00 2001 From: Davide Faconti Date: Sat, 14 Nov 2026 15:59:25 +0100 Subject: [PATCH] docs: bring agent instructions and guides in line with the 1.0.0 SDK An audit of every CLAUDE.md, skill and guide against the code found stale claims, samples that do not compile, and history mixed into current docs. - Root CLAUDE.md: read path for plugin authors and ABI changes, correct changelog heading, five floor-surface kinds, real warning flags, CI-only gates; release mechanics move to the new docs/RELEASING.md. - plotjuggler-plugin skill: five families with a new Widget reference, correct toolbox threading, onDataChanged is advisory, closed encoding list, plugin_host for loader tests, null handling, 1.0 pin guidance. - pj_plugins guides: fix non-compiling samples, document readSeries and nulls, resolve the parser-bridge contradiction, consumer CMake targets, remove PR and pre-v4 history. - Reference docs: drop a nonexistent host API from the dialog reference, add missing setters/handlers and utilities, current marker producers, BACKLOG cut to open items, V4_STORE.md -> docs/object_store.md with the correct service id; the historical toolbox gap analysis goes to docs/archive/. - C++ style docs: no tsl::robin_map (not a dependency), real flags. - Header comments: Log `source`, `timestamp_ns` field names. Co-Authored-By: Claude Opus 5.5 --- .claude/commands/test-and-commit.md | 7 +- .claude/skills/cpp20-api-guidelines/SKILL.md | 7 +- .../cpp20-api-guidelines/agents/openai.yaml | 2 +- .../references/review-checklist.md | 6 +- .claude/skills/plotjuggler-plugin/SKILL.md | 116 +++++++--- .../references/builtin-objects.md | 15 +- .../references/data-source.md | 16 +- .../plotjuggler-plugin/references/dialog.md | 2 +- .../references/message-parser.md | 21 +- .../plotjuggler-plugin/references/toolbox.md | 29 ++- .../plotjuggler-plugin/references/widget.md | 131 +++++++++++ .github/workflows/linux-ci.yml | 2 +- CLAUDE.md | 130 +++++------ README.md | 24 +- docs/BACKLOG.md | 41 +--- docs/RELEASING.md | 111 +++++++++ .../toolbox-porting-gap-analysis.md | 0 docs/builtin_type.md | 74 +++--- docs/cpp_design_recommendations.md | 92 ++------ docs/dialog-sdk-reference.md | 54 +++-- V4_STORE.md => docs/object_store.md | 15 +- docs/plot_markers_architecture.md | 61 ++--- docs/plot_markers_use_cases.md | 16 +- docs/sdk-utilities.md | 10 + experimental/parser_module/README.md | 5 +- pj_base/CLAUDE.md | 48 ++-- .../pj_base/builtin/builtin_object.hpp | 2 +- .../builtin/frame_transforms_fields.hpp | 2 +- .../builtin/image_annotations_fields.hpp | 2 +- .../pj_base/builtin/scene_entities_fields.hpp | 4 +- pj_base/proto/pj/README.md | 10 +- pj_plugins/CLAUDE.md | 19 +- pj_plugins/dialog_protocol/CLAUDE.md | 53 ++--- pj_plugins/docs/ARCHITECTURE.md | 218 ++++++++---------- pj_plugins/docs/REQUIREMENTS.md | 48 ++-- pj_plugins/docs/data-source-guide.md | 104 +++++---- pj_plugins/docs/dialog-plugin-guide.md | 50 ++-- pj_plugins/docs/message-parser-guide.md | 59 +++-- pj_plugins/docs/toolbox-guide.md | 139 +++++++---- pj_plugins/docs/widget-guide.md | 20 +- pj_plugins/widget_protocol/CLAUDE.md | 3 + 41 files changed, 1086 insertions(+), 682 deletions(-) create mode 100644 .claude/skills/plotjuggler-plugin/references/widget.md create mode 100644 docs/RELEASING.md rename docs/{ => archive}/toolbox-porting-gap-analysis.md (100%) rename V4_STORE.md => docs/object_store.md (88%) diff --git a/.claude/commands/test-and-commit.md b/.claude/commands/test-and-commit.md index a516649c..f2f89cb5 100644 --- a/.claude/commands/test-and-commit.md +++ b/.claude/commands/test-and-commit.md @@ -1,7 +1,8 @@ Validate and commit staged/unstaged changes. Run these steps in order, stopping on first failure: -1. `./build.sh --debug` — Debug+ASAN build must succeed +1. `./build.sh --debug` — Debug+ASAN build into `build/debug_asan` must succeed 2. `./test.sh` — all tests must pass -3. `git ls-files -z '*.cpp' | xargs -0 clang-tidy-22 -p build` — no clang-tidy warnings +3. `pre-commit run --all-files` — clang-format and file hygiene must be clean +4. `git ls-files -z '*.cpp' | xargs -0 clang-tidy-22 -p build/debug_asan` — no clang-tidy warnings -If all three pass, create a git commit following the repo's commit conventions (see git log for style). If any step fails, fix the issue and re-run from step 1. +If all pass, create a git commit following the repo's commit conventions (see git log for style). If any step fails, fix the issue and re-run from step 1. diff --git a/.claude/skills/cpp20-api-guidelines/SKILL.md b/.claude/skills/cpp20-api-guidelines/SKILL.md index 5667a441..d2d81ab5 100644 --- a/.claude/skills/cpp20-api-guidelines/SKILL.md +++ b/.claude/skills/cpp20-api-guidelines/SKILL.md @@ -1,6 +1,6 @@ --- name: cpp20-api-guidelines -description: "Generate and review C++20 code in a simplicity-first style. Use when tasks involve API design, code reviews, refactors, or coding standards where correctness and usability matter: const-by-default, stateless-first interfaces, minimal abstraction (avoid SOLID-for-its-own-sake), DRY with restraint, testability by design, explicit ownership/lifetimes, PJ::Expected/PJ::Status error handling, and tsl::robin_map containers." +description: "Generate and review C++20 code in a simplicity-first style. Use when tasks involve API design, code reviews, refactors, or coding standards where correctness and usability matter: const-by-default, stateless-first interfaces, minimal abstraction (avoid SOLID-for-its-own-sake), DRY with restraint, testability by design, explicit ownership/lifetimes, and PJ::Expected/PJ::Status error handling." --- # C++20 API Guidelines @@ -28,6 +28,7 @@ description: "Generate and review C++20 code in a simplicity-first style. Use wh - Use `PJ::Expected` (recoverable error with value), `PJ::Status` (recoverable error, no return value), `PJ::Span` (non-owning view). - Success: `PJ::okStatus()`. Error: `PJ::unexpected(msg)`. Check: `.has_value()`. -- Use `tsl::robin_map` for hash containers in `.cpp` files, `std::unordered_map` in headers. - Use `fmt::format` for string formatting. +- Use `std::unordered_map` / `std::unordered_set` for hash containers. Public + headers include only the standard library (plus nlohmann/json where the module + already exposes it); fmt is private to the SDK's own `.cpp` files. - Include minimal code examples only when they clarify contracts or misuse prevention. diff --git a/.claude/skills/cpp20-api-guidelines/agents/openai.yaml b/.claude/skills/cpp20-api-guidelines/agents/openai.yaml index c306c6c8..c6017464 100644 --- a/.claude/skills/cpp20-api-guidelines/agents/openai.yaml +++ b/.claude/skills/cpp20-api-guidelines/agents/openai.yaml @@ -1,4 +1,4 @@ interface: display_name: "C++20 API Guidelines" short_description: "C++20 API rules for simple, misuse-resistant code." - default_prompt: "Apply C++20 API and design rules from cpp_design_recommendations.md prioritizing simplicity, const-by-default, testability, PJ::Expected error handling, tsl::robin_map containers, and explicit ownership." + default_prompt: "Apply C++20 API and design rules from cpp_design_recommendations.md prioritizing simplicity, const-by-default, testability, PJ::Expected error handling, and explicit ownership." diff --git a/.claude/skills/cpp20-api-guidelines/references/review-checklist.md b/.claude/skills/cpp20-api-guidelines/references/review-checklist.md index b3f0cd02..f8c59bf8 100644 --- a/.claude/skills/cpp20-api-guidelines/references/review-checklist.md +++ b/.claude/skills/cpp20-api-guidelines/references/review-checklist.md @@ -21,9 +21,9 @@ Run this checklist before returning recommendations or code. 14. Are objects fully valid after construction (no two-phase init)? ## Container & formatting usage -15. Are hash containers in `.cpp` files using `tsl::robin_map`/`tsl::robin_set` instead of `std::unordered_*`? -16. Are hash containers in `.hpp` headers using `std::unordered_map`/`std::unordered_set`? -17. Are string operations using `fmt::format` where appropriate? +15. Do public headers include only the standard library (and nlohmann/json where the module already exposes it)? +16. Is fmt kept out of public headers (it is a private dependency)? +17. Is a small collection a `std::vector` scan rather than a hash container? 18. Are time values using `int64_t` nanoseconds or `std::chrono` types instead of raw integers? 19. Are public API headers using `PJ::Expected` / `PJ::Status` for error handling? 20. Is `.has_value()` checked before accessing `Expected` values (not `.ok()`)? diff --git a/.claude/skills/plotjuggler-plugin/SKILL.md b/.claude/skills/plotjuggler-plugin/SKILL.md index 0fa0a3b0..13155a8c 100644 --- a/.claude/skills/plotjuggler-plugin/SKILL.md +++ b/.claude/skills/plotjuggler-plugin/SKILL.md @@ -1,12 +1,13 @@ --- name: plotjuggler-plugin description: >- - Write or modify a PlotJuggler plugin (DataSource, MessageParser, Toolbox, or - Dialog) using the plotjuggler_sdk. Use this + Write or modify a PlotJuggler plugin (DataSource, MessageParser, Toolbox, + Dialog, or Widget) using the plotjuggler_sdk. Use this whenever the task is to implement, extend, debug, or build a PlotJuggler extension — importing a file format, streaming live data, parsing message payloads, mapping a custom type to a canonical object or scalars, adding a - data-processing toolbox, or building a plugin configuration dialog — even if + data-processing toolbox, building a plugin configuration dialog, or adding a + dockable panel that shows a custom Qt widget — even if the user does not say the word "plugin". It gives the fast path, correct CMake, and load-bearing ABI/lifetime rules. Do NOT use it for changing the SDK's own ABI/protocol (that is a maintainer task with different rules — see below). @@ -14,7 +15,7 @@ description: >- # Writing a PlotJuggler plugin -PlotJuggler has four **plugin families**. Subclass a family's small C++20 base +PlotJuggler has five **plugin families**. Subclass a family's small C++20 base class and export it from a shared library (`.so`/`.dll`/`.dylib`). `plotjuggler_sdk` supplies each base class. A one-line macro emits the entry point, version symbol, exception trampolines and symbol folding. @@ -24,6 +25,11 @@ This skill is the author-oriented fast path. The repo's `pj_plugins/docs/` guide are the full reference; this steers you to the right one and front-loads the things that silently break a plugin. +Paths in this skill are relative to the SDK repository. An installed package +(Conan, pixi) carries the headers, the CMake helpers and only two guides, +`share/plotjuggler_sdk/docs/{sdk-utilities,provider-guide}.md`; read the other +guides and the examples in the repository. + ## Are you writing a plugin, or changing the SDK? - **Writing a plugin *against* the SDK** → this skill. The macros handle ABI @@ -32,7 +38,7 @@ things that silently break a plugin. - **Changing the SDK's own ABI/protocol** (adding vtable slots, editing a struct, bumping a `PJ_*_PROTOCOL_VERSION`, touching `pj_base/abi/baseline.abi`) → this is a **maintainer** task. Stop and follow `pj_plugins/docs/ARCHITECTURE.md` §0a and - the "Release Versioning" contract in the root `CLAUDE.md`. Different rules apply. + `docs/RELEASING.md`. Different rules apply. ## Step 1 — How do you get the SDK? @@ -43,13 +49,19 @@ plotjuggler_sdk::plugin_sdk)` — only the acquisition step differs: - **Conan package** (`plotjuggler_sdk`) — the common external route; the official plugins use it. Pin the compatibility range explicitly: `plotjuggler_sdk/[>=X.Y.Z <(X+1).0.0]`, where `X.Y.Z` is the version that - introduced the newest SDK feature you actually use (pre-1.0 the upper bound is - `<1.0.0` — the SDK guarantees no breaks within `0.x`). Tip from the official + introduced the newest SDK feature you actually use. On the `0.x` line that is + `[>=0.Y.Z <1.0.0]`: no `0.x` release breaks an earlier one. `1.0.0` is the + first MAJOR and changes source (its `CHANGELOG.md` section lists what). Until + it is released it exists only as `1.0.0-dev.N`: pin that version exactly, + because a range skips pre-releases and two dev builds promise each other + nothing. Tip from the official repo: keep the version in a single `SDK_VERSION` file your `conanfile.py` reads. Then `find_package(plotjuggler_sdk REQUIRED COMPONENTS plugin_sdk)`. - **pixi / conda package** (`plotjuggler_sdk`) — same `find_package` + target once the package is in your environment. -- **git submodule / vendored source** — `add_subdirectory(plotjuggler_sdk)`; the +- **git submodule / vendored source** — set `PJ_BUILD_TESTS` to `OFF` (it + defaults to `ON` and pulls in GTest, the examples and `experimental/`), then + `add_subdirectory(plotjuggler_sdk)`; the same `plotjuggler_sdk::plugin_sdk` alias exists in-tree, and `pj_configure_plugin` / `pj_embed_file` are available too, so the CMake below is identical. (Editing the SDK repo itself is this world: the raw umbrella target is @@ -65,11 +77,16 @@ plotjuggler_sdk::plugin_sdk)` — only the acquisition step differs: | Decode a **byte payload** on a topic into named fields (JSON, protobuf, ROS, custom) | **MessageParser** | `references/message-parser.md` | | A **tool** that reads existing data, transforms it, and writes new topics | **Toolbox** | `references/toolbox.md` | | A **configuration UI** (for a source/parser/toolbox, or standalone) | **Dialog** | `references/dialog.md` | +| A **dockable panel** showing a Qt widget you draw, fed by series the user wires to it | **Widget** | `references/widget.md` | A single `.so` can host **more than one family** — the supported pattern is an owner + its embedded Dialog (one `PJ_*_PLUGIN(...)` macro per family at file scope). Don't ship *unrelated* families in one DSO: catalog discovery reads one -primary family descriptor per library. +primary family descriptor per library. A Widget library is always its own +library: it is the only family that links Qt, and no other family may. + +Functional parser modules under `experimental/parser_module/` are not a plugin +family and are not installed. Do not build a plugin on them. If your data is object-like (image, point cloud, occupancy grid, transforms, markers…) rather than scalar time series, also read `references/builtin-objects.md` @@ -88,17 +105,23 @@ second argument is the **manifest JSON literal** (see Step 5). | MessageParser | `PJ::MessageParserPluginBase` | ⚠ `pj_plugins/sdk/message_parser_plugin_base.hpp` | `registerSchemaHandler()` in ctor/`bindSchema()` | `PJ_MESSAGE_PARSER_PLUGIN(C, m)` | | Toolbox | `PJ::ToolboxPluginBase` | `pj_base/sdk/toolbox_plugin_base.hpp` | `capabilities()` (+ data ops) | `PJ_TOOLBOX_PLUGIN(C, m)` | | Dialog | `PJ::DialogPluginTyped` | `pj_plugins/sdk/dialog_plugin_typed.hpp` + `.../widget_data.hpp` | `manifest()`, `ui_content()`, `widget_data()`, event handlers | `PJ_DIALOG_PLUGIN(C, m)` | +| Widget | `PJ::WidgetPluginBase` | `pj_plugins/sdk/widget_plugin_base.hpp` | `inputs()`, `createWidget()`, `onUpdate()` | `PJ_WIDGET_PLUGIN(C, m)` | + +> ⚠ **Header-location trap.** The DataSource and Toolbox bases live under +> `pj_base/sdk/`, but `MessageParserPluginBase` lives under **`pj_plugins/sdk/`**, +> and so do the Dialog and Widget SDKs (installed there from the +> `dialog_protocol` and `widget_protocol` modules). Do not substitute a +> `pj_base/sdk/` path for those; use the paths in the table. -> ⚠ **Header-location trap.** Three base classes live under `pj_base/sdk/`, but -> `MessageParserPluginBase` lives under **`pj_plugins/sdk/`**, and the Dialog SDK -> lives under `pj_plugins/sdk/` too (installed there from the `dialog_protocol` -> module). Do not substitute a `pj_base/sdk/` path for either one; use the paths -> in the table. +A class inside a namespace, or a plugin linked statically into the host +(`PJ_STATIC_PLUGINS`, the WebAssembly build), registers with +`PJ_*_PLUGIN_NAMED(ns::Class, Class, manifest)`. The quickest correct start is to copy a working example from this repo and edit it: `examples/sdk_consumer/` (minimal external DataSource with the full CMake), -`pj_plugins/examples/mock_*.cpp` (DataSource, parser, toolbox, source+dialog), and -`pj_plugins/dialog_protocol/examples/mock_dialog.cpp` (standalone dialog). Each +`pj_plugins/examples/mock_*.cpp` (DataSource, parser, toolbox, source+dialog), +`pj_plugins/dialog_protocol/examples/mock_dialog.cpp` (standalone dialog) and +`pj_plugins/widget_protocol/examples/mock_widget.cpp` (widget). Each family reference embeds an API skeleton for when the repo isn't at hand (domain placeholders like `openSocket()` are yours to fill in). @@ -128,7 +151,7 @@ add_library(my_plugin SHARED my_plugin.cpp) target_link_libraries(my_plugin PRIVATE plotjuggler_sdk::plugin_sdk) # same in all worlds pj_configure_plugin(my_plugin - FAMILIES data_source # data_source | message_parser | toolbox | dialog + FAMILIES data_source # data_source | message_parser | toolbox | dialog | widget # (list several: `data_source dialog` for an embedded dialog) MANIFEST_FILE ${CMAKE_CURRENT_SOURCE_DIR}/manifest.json MANIFEST_HEADER generated/my_manifest.hpp # -> #include "my_manifest.hpp" @@ -139,9 +162,11 @@ pj_configure_plugin(my_plugin ``` The `plotjuggler_sdk::plugin_sdk` component supplies the common author surface — -base + parser SDK + dialog SDK + these CMake helpers. Compiled provider support -requires `plotjuggler_sdk::source` as well. You do **not** link a separate -dialog target downstream (`pj_dialog_sdk` is an in-tree name). +base + parser SDK + dialog SDK + widget SDK + these CMake helpers. Compiled +provider support requires `plotjuggler_sdk::source` as well. You do **not** link +a separate dialog or widget target downstream (`pj_dialog_sdk` and +`pj_widget_sdk` are in-tree names). A test that loads the built plugin through +the host loaders adds the `plugin_host` component (see Verify). **`pj_configure_plugin` is required in practice.** It: @@ -157,8 +182,12 @@ dialog target downstream (`pj_dialog_sdk` is an in-tree name). `STB_GNU_UNIQUE` symbol leaks. MessageParser manifests **must** include `"encoding": ["json", …]`. -The host routes payloads by these case-sensitive names. -DataSource may add `"file_extensions": [".csv"]`. +The host routes payloads by these case-sensitive names, and admits only the +registered ones: `PJ::registeredParserEncodings()` in +`pj_plugins/host/parser_claim_catalog.hpp` is the list. A parser claiming any +other name is refused when its claims are admitted. +DataSource may add `"file_extensions": [".csv"]`. A Widget manifest needs +`widgets` and `qt_modules` (see `references/widget.md`). `pj_emit_plugin_manifest` (the pre-0.25 name) still works but is deprecated. ### `min_sdk_required` — the host floor, not the build pin @@ -224,15 +253,21 @@ coalescing, etc.) — read them. stray exception's `what()` into the ABI error, non-fallible ones can only swallow it, and an exception escaping a `noexcept` slot terminates the process. Explicit `Status` returns are the only predictable diagnostics. -2. **Keep the data plane on the host's callback thread.** `writeHost()`, - `toolboxHost()`, and `pushMessage()` are single-threaded — a background I/O - thread must buffer under a mutex and hand data off inside `onPoll()`/`poll()`. - The authoritative source is the thread tag on each slot in the protocol - headers: slots tagged `[thread-safe]` — `reportMessage()`, - `isStopRequested()`, `notifyState()`, `requestStop()`, and the toolbox's - `notifyDataChanged()` — may be called from any thread (e.g. reporting a - connection loss from an I/O callback). Everything else stays on the callback - thread. +2. **Respect each host slot's thread tag.** The tag on the slot in the protocol + header is the authority (`data_source_protocol.h`, `toolbox_protocol.h`, + `plugin_data_api.h`); the list below is the common part, not all of it. + - **DataSource:** `writeHost()` and `pushMessage()` are used only inside the + host's calls (`onStart()`/`onPoll()`/`onStop()`). A background I/O thread + buffers under a mutex (`PJ::sdk::DrainQueue`) and hands data off in + `onPoll()`. + - **Toolbox:** writes through `toolboxHost()` may come from the toolbox's own + worker thread, one thread at a time per host object. Every other toolbox + service (object reads, playback, viewport, plot tabs, data processors) is + main-thread: call it from inside a host call or a dialog callback. + - **Any thread:** `reportMessage()`, `isStopRequested()`, `notifyState()`, + `requestStop()` and the toolbox's `notifyDataChanged()`, for example to + report a connection loss from an I/O callback. + - **Widget:** everything is on the GUI thread. 3. **Timestamps are `int64` nanoseconds since the Unix epoch.** Not seconds, not row indices. Extract or synthesize a real absolute nanosecond value. Use `std::chrono::system_clock` for receive-time stamps — **never** @@ -246,7 +281,8 @@ coalescing, etc.) — read them. Dialogs and toolboxes restore from the layout *before data arrives*. Tolerate older configs by migrating or defaulting missing keys. Do not *fail* `loadConfig()` merely because referenced data is not loaded yet. - Re-resolve in `onDataChanged()`. + Resolve it again when the user acts or when you need the data. A toolbox must + not wait for `onDataChanged()`: it is advisory and a host may never call it. 5. **String lifetimes across the ABI.** SDK overrides return `std::string` — the base class buffers it for the ABI, one buffer per slot, invalidated at the next call of the *same* method on the same instance. So never hold the host's @@ -270,9 +306,13 @@ coalescing, etc.) — read them. required keys (+ `encoding` for a parser). - Load it in PlotJuggler, or preferably test the **real built `.so`** through the host loaders (`DataSourceLibrary::load(path)`, - `MessageParserLibrary::load(path)`, …). Inject the path as a compile definition. - Bind SDK test helpers as host services: `pj_base/sdk/testing/parser_write_recorder.hpp` - and `pj_plugins/testing/toolbox_test_store.hpp`. Assert on the writes. + `MessageParserLibrary::load(path)`, …). They are in the `plugin_host` + component: `find_package(plotjuggler_sdk REQUIRED COMPONENTS plugin_sdk plugin_host)` + and link the test to `plotjuggler_sdk::plugin_host`. Inject the path as a compile definition. + Bind SDK test helpers as host services: `pj_base/sdk/testing/parser_write_recorder.hpp`, + `pj_plugins/testing/toolbox_test_store.hpp`, + `pj_plugins/testing/delegated_ingest_fixture.hpp` and, for a widget, + `pj_plugins/sdk/testing/fake_widget_host.hpp`. Assert on the writes. This exercises the host's exact ABI surface via `dlopen`. - Round-trip `saveConfig()` → `loadConfig()` → run with no dialog. @@ -282,8 +322,10 @@ coalescing, etc.) — read them. |---|---| | What each family may do; capabilities; permission matrix; config contract | `pj_plugins/docs/REQUIREMENTS.md` | | How the C ABI / loaders / host bridge work (mostly maintainer detail) | `pj_plugins/docs/ARCHITECTURE.md` | -| Writing each family, in depth | `pj_plugins/docs/{data-source,message-parser,toolbox,dialog-plugin}-guide.md` | +| Writing each family, in depth | `pj_plugins/docs/{data-source,message-parser,toolbox,dialog-plugin,widget}-guide.md` | | Dialog `WidgetData` setters + event-handler signatures | `docs/dialog-sdk-reference.md` | | Builtin object types + their wire codecs | `docs/builtin_type.md`, `pj_base/include/pj_base/builtin/` | -| Object store: publish/read objects, ownership, lazy fetch | `V4_STORE.md` | +| Marking findings on a time-series plot (`PlotMarkers`) | `docs/plot_markers_format.md`, `docs/plot_markers_architecture.md` | +| Helpers that already exist | `docs/sdk-utilities.md` | +| Object store: publish/read objects, ownership, lazy fetch | `docs/object_store.md` | | C++ style, error handling, `PJ::Expected`, naming | `docs/cpp_design_recommendations.md` | diff --git a/.claude/skills/plotjuggler-plugin/references/builtin-objects.md b/.claude/skills/plotjuggler-plugin/references/builtin-objects.md index 696e4f61..3d23206c 100644 --- a/.claude/skills/plotjuggler-plugin/references/builtin-objects.md +++ b/.claude/skills/plotjuggler-plugin/references/builtin-objects.md @@ -8,7 +8,7 @@ faking it with scalar fields. This is the shim that keeps ROS/protobuf/vendor specifics out of PlotJuggler's viewers and storage. Reference: `docs/builtin_type.md`, headers in `pj_base/include/pj_base/builtin/`, -object store ABI in `V4_STORE.md`. +object store ABI in `docs/object_store.md`. ## Decision: builtin type or raw scalars? @@ -20,6 +20,9 @@ object store ABI in `V4_STORE.md`. - 2D overlays on an image → `ImageAnnotations` - 3D frame graph → `FrameTransforms`; camera calibration → `CameraInfo` - occupancy/cost/ESDF → `OccupancyGrid` (+`OccupancyGridUpdate`) or `VoxelGrid` + - 2D grid with several per-cell channels (elevation, layered costmap) → `GridMap` + - triangle mesh with per-vertex / per-face channels → `MeshMap` + (+`MeshMapUpdate`); both need `min_sdk_required` ≥ 1.0.0, a hard floor - marker-style 3D primitives → `SceneEntities`; pose arrays → `PosesInFrame` - mesh asset → `Mesh3D`; inter-frame video → `VideoFrame` - URDF/SDF/MJCF text → `RobotDescription`; log line → `Log` @@ -30,7 +33,7 @@ object store ABI in `V4_STORE.md`. custom payload as a builtin. `BuiltinObjectType::kNone` means "scalar/unknown classification", not "storage forbidden". -The full list (18 concrete types) and their exact fields are in +The full list (20 concrete types) and their exact fields are in `docs/builtin_type.md` and one header each under `pj_base/include/pj_base/builtin/` (the `builtin_object.hpp` enum is the authoritative roster — trust it over prose that may lag behind newer additions). @@ -92,7 +95,8 @@ into the returned object so large payloads stay zero-copy. See - **`pushOwned(topic, ts, bytes)`** — the host copies immediately; you may free your buffer as soon as the call returns. -- **`pushLazy(topic, ts, fetch)`** — the host keeps your closure and calls it only +- **`pushLazy(topic, ts, fetch)`** (DataSource and MessageParser views only; a + Toolbox has `pushOwnedObject` alone) — the host keeps your closure and calls it only when a consumer reads that entry. Capture heavy state **by value** (e.g. a `shared_ptr`), so it stays alive as long as the entry does. Use this for large payloads you would rather read from a file on demand than hold in RAM. @@ -114,8 +118,9 @@ exactly once. Do not keep a raw pointer into anything the closure captured. - `ImageAnnotations.image_topic` explicitly names the base image topic; camera calibration correlates by **name convention** (`/camera_info` ↔ `/image_raw`), not by a reference. -- The annotations codec carries the top-level `timestamp` (only when non-zero) and - `image_topic` (only when non-empty); see `docs/image_annotations_format.md`. +- The annotations codec carries `ImageAnnotations::timestamp_ns` (wire field + `timestamp`, only when non-zero) and `image_topic` (only when non-empty); see + `docs/image_annotations_format.md`. ## Codec round-trips are not bit-exact diff --git a/.claude/skills/plotjuggler-plugin/references/data-source.md b/.claude/skills/plotjuggler-plugin/references/data-source.md index 6e27610e..7d1a6bc1 100644 --- a/.claude/skills/plotjuggler-plugin/references/data-source.md +++ b/.claude/skills/plotjuggler-plugin/references/data-source.md @@ -86,7 +86,7 @@ class MyUdpSource : public PJ::StreamSourceBase { return PJ::okStatus(); } - PJ::Status onPoll() override { // host thread — the only place you touch the host + PJ::Status onPoll() override { // host thread — hand the buffered data to the host here auto batch = pending_.drain(); // swaps the whole queue out under a brief lock auto topic = writeHost().ensureTopic("udp/data"); if (!topic) return PJ::unexpected(topic.error()); @@ -131,8 +131,8 @@ For **delegated** ingest (let a MessageParser decode the bytes): declare header). In `onPoll()`: ```cpp -PJ::sdk::ParserBindingRequest req{.topic_name = topic, .parser_encoding = encoding_, - .parser_config_json = parser_config_}; +PJ::ParserBindingRequest req{.topic_name = topic, .parser_encoding = encoding_, + .parser_config_json = parser_config_}; auto r = ingest_.push(runtimeHost(), /*cache_key=*/topic, req, ts, std::move(bytes)); if (!r) return PJ::unexpected(r.error()); // a failed push IS an error // r == kBindingUnavailable: no parser for that encoding yet — skipped, retried next poll @@ -184,10 +184,12 @@ extension (`pluginExtension(PJ_TOPIC_SUBSCRIPTION_EXTENSION_V1)` returning a liv the hand-off point. - **`FileSourceBase` calls `progressFinish()` for you.** Do not call it yourself from `importData()`; a manual `DataSourcePluginBase` must call it itself. -- **Notify state transitions.** With the raw `DataSourcePluginBase`, call - `runtimeHost().notifyState(state)` on every transition you make, and never - `resume()` from a terminal (`stopped`/`failed`) state — the host makes a new - instance instead. +- **State reports are advisory.** `runtimeHost().notifyState(state)` and + `currentState()` may be ignored by a host: report them, do not depend on + them. An instance is started at most once; the host makes a new instance to + restart, so never `resume()` from a terminal (`stopped`/`failed`) state. +- **An empty cell is a null, not a zero.** Write `PJ::kNull` (or a + `PJ::TypedNull` to create the column on its first row) as the field value. - **Timestamps: ns since epoch** (see SKILL.md rule 3). A row counter is not a timestamp. diff --git a/.claude/skills/plotjuggler-plugin/references/dialog.md b/.claude/skills/plotjuggler-plugin/references/dialog.md index 4d86eb45..254fd046 100644 --- a/.claude/skills/plotjuggler-plugin/references/dialog.md +++ b/.claude/skills/plotjuggler-plugin/references/dialog.md @@ -171,7 +171,7 @@ the plugin still links no Qt. ## Helpers you do not rewrite (connection dialogs) Every streaming source's dialog has the same three sub-problems; the SDK ships the -answers (all `PJ::sdk`, header in parentheses). See SKILL.md Step 4 for the full table. +answers (all `PJ::sdk`, header in parentheses). The full table is `docs/sdk-utilities.md`. - **Parser-encoding combo** (`pj_plugins/sdk/streaming_dialog.hpp`, `pj_plugins/sdk/encoding_utils.hpp`). The host tells the owner which encodings diff --git a/.claude/skills/plotjuggler-plugin/references/message-parser.md b/.claude/skills/plotjuggler-plugin/references/message-parser.md index 04f53402..fd2b61b4 100644 --- a/.claude/skills/plotjuggler-plugin/references/message-parser.md +++ b/.claude/skills/plotjuggler-plugin/references/message-parser.md @@ -60,7 +60,7 @@ class MyParser : public PJ::MessageParserPluginBase { return { .object_type = PJ::sdk::BuiltinObjectType::kNone, // declare what you emit .parse_scalars = - [this](PJ::Timestamp ts, PJ::Span payload) + [](PJ::Timestamp /*transport_ts*/, PJ::Span payload) -> PJ::Expected { std::string_view text(reinterpret_cast(payload.data()), payload.size()); auto value = PJ::parseNumber(text); // locale-independent @@ -97,8 +97,8 @@ Key pieces: ## Route claims and functional extensions -Recompile a handler-registering parser against SDK 0.22 and the base class -automatically exposes both host-facing tables: +A parser that registers `SchemaHandler`s gets both host-facing tables from the +base class, with no code of its own: - `pj.parser_route_claims.v1` reports exact claims from the handler table. It never reports a wildcard. The host synthesizes one universal wildcard scalar @@ -125,6 +125,7 @@ the base class derives these extensions. PJ::Status loadConfig(std::string_view json) override { auto cfg = nlohmann::json::parse(json, nullptr, false); + if (!cfg.is_object()) cfg = nlohmann::json::object(); // empty or invalid config: defaults limit_ = PJ::sdk::arrayLimitFromJson(cfg); // never throws; defaults 500 / clamp use_embedded_ts_ = cfg.value("use_embedded_timestamp", false); return PJ::okStatus(); @@ -134,7 +135,7 @@ the base class derives these extensions. PJ::sdk::arrayLimitToJson(cfg, limit_); // writes canonical + legacy keys return cfg.dump(); } - // in the flatten loop: if (n > limit_.max_size) { if (!limit_.clamp()) skip field; else n = limit_.max_size; } + // in the flatten loop (max_size 0 = unlimited): if (limit_.max_size != 0 && n > limit_.max_size) { if (!limit_.clamp()) skip field; else n = limit_.max_size; } PJ::sdk::ArrayLimit limit_; ``` - `parse()` — legacy direct route; only touch it when porting old code. Plugins @@ -142,9 +143,15 @@ the base class derives these extensions. ## Traps specific to MessageParser -- **`encoding` in the manifest is mandatory and case-sensitive.** The host routes - payloads to your parser by matching these names. `"json"` ≠ `"JSON"`. If it is - missing or mismatched, your parser is never invoked and the plugin looks dead. +- **`encoding` in the manifest is mandatory, case-sensitive and from a closed + list.** The host routes payloads to your parser by matching these names. + `"json"` ≠ `"JSON"`, and a name outside `PJ::registeredParserEncodings()` + (`pj_plugins/host/parser_claim_catalog.hpp`) is refused. If it is missing, + mismatched or unregistered, your parser is never invoked and the plugin looks dead. +- **A parser that returns builtin objects is tied to the SDK MAJOR it was built + with.** The host reads the returned struct directly, so a change to a builtin + struct (1.0.0 changed `Log`, `DepthImage` and the `timestamp_ns` members) + needs a rebuild against the host's MAJOR. - **Never name a topic.** The write path is pre-scoped to the bound topic; your records' field names become `/` automatically. - **Do not require `bindSchema()`.** Schema-less encodings never receive it — diff --git a/.claude/skills/plotjuggler-plugin/references/toolbox.md b/.claude/skills/plotjuggler-plugin/references/toolbox.md index 643d0fd0..05291c23 100644 --- a/.claude/skills/plotjuggler-plugin/references/toolbox.md +++ b/.claude/skills/plotjuggler-plugin/references/toolbox.md @@ -25,7 +25,8 @@ class MyToolbox : public PJ::ToolboxPluginBase { } void onDataChanged() override { - // host tells you the dataset changed; re-read if you cache anything + // Advisory: a host may never call it. Re-read the catalog when the user + // acts, and treat this only as an early hint to drop a cache. } // Do work when the user acts (e.g. from the dialog). Take the timestamps from @@ -90,7 +91,15 @@ Gate on `DataProcessorsHostView::hasTypedRequests()`; the flow (createV2, submit `parseEvaluationReport`) is in `pj_plugins/docs/toolbox-guide.md` → "Typed requests and on-demand evaluation", the contract in the `pj.data_processors.v1` comment of `plugin_data_api.h`. -## Reading a series (Arrow) +## Reading a series + +`toolboxHost().readSeries(field)` returns a `PJ::sdk::MaterializedSeriesView` +that owns the Arrow buffers: `rowCount()`, `timestamps()` (int64 ns) and the +typed `valuesAs*()` pointers. A sample the source did not provide is a **null**: +the `valuesAs*()` slot of a null is unspecified (hosts zero it), so when +`nullCount() != 0` skip the rows where `isNull(row)` before computing anything. + +The raw Arrow path underneath it, for when you need the buffers themselves: ```cpp #include @@ -115,8 +124,11 @@ than row-by-row `appendRecord`. Call it **once per logical operation**, not per row. Forgetting it entirely means your new series never appear in the UI; calling it per record floods the host. -- **Host methods only from the host thread** (SKILL.md rule 2). Background work - must marshal back. +- **Threads** (SKILL.md rule 2). Writes through `toolboxHost()` may come from + your own worker thread, one thread at a time per host object. + `objectReadHost()` and the optional services (playback, viewport, plot tabs, + data processors, MCP endpoint) are main-thread: call them from inside a host + call or a dialog callback. - **Arrow RAII, no dangling pointers** (SKILL.md rule 6). Never keep a raw `ArrowSchema*`/`ArrowArray*` past its holder's scope. - **A `catalogSnapshot()` is a point-in-time view.** After you write and call @@ -127,9 +139,14 @@ than row-by-row `appendRecord`. ## Emitting objects (images, clouds, …) -`registerObjectTopic(source, name, metadata_json)` then +`registerObjectTopic(source, name, PJ::sdk::BuiltinObjectType::kImage)` then `pushOwnedObject(topic, ts, serialized_bytes)` — serialize with the matching -builtin codec first. See `references/builtin-objects.md` and `V4_STORE.md`. +builtin codec first. Use that typed overload: it writes the +`builtin_object_type` metadata viewers discover the type from, which the raw +`metadata_json` overload leaves to you. A toolbox has no lazy push. To attach +an object topic to a dataset that already exists (plot markers on loaded data), +use `registerObjectTopicOnDataset()`, with `setObjectTopicRetention()` to bound +it. See `references/builtin-objects.md` and `docs/object_store.md`. ## Embedding a dialog diff --git a/.claude/skills/plotjuggler-plugin/references/widget.md b/.claude/skills/plotjuggler-plugin/references/widget.md new file mode 100644 index 00000000..bc683b11 --- /dev/null +++ b/.claude/skills/plotjuggler-plugin/references/widget.md @@ -0,0 +1,131 @@ +# Widget plugin + +A Widget plugin contributes dockable panels. Each panel shows a Qt widget the +plugin creates; the user wires series to the widget's named inputs in the host. +The host owns the data and the layout, the plugin owns the pixels. A widget only +consumes: it creates no series and no datasets. + +Full reference: `pj_plugins/docs/widget-guide.md`. Working example without Qt: +`pj_plugins/widget_protocol/examples/mock_widget.cpp`. + +## Header + skeleton + +```cpp +#include +#include +#include + +#include "my_manifest.hpp" // kMyManifest, generated by pj_configure_plugin + +class Battery : public PJ::WidgetPluginBase { + public: + std::vector inputs() const override { + return {PJ::WidgetInput("voltage", "Voltage", "V")}; // name, label, unit + } + void* createWidget(void* parent) override { + label_ = new QLabel(static_cast(parent)); + return label_.data(); + } + void onUpdate(PJ::Span values, double /*time*/) override { + for (const PJ_widget_value_t& v : values) { // one entry per wire: dispatch on v.input + if (v.input == 0 && label_) { + label_->setText(v.kind == PJ_WIDGET_VALUE_NUMBER ? QString::number(v.number, 'f', 1) : QString("-")); + } + } + } + + private: + QPointer label_; // the widget may already be gone when the plugin is destroyed +}; + +PJ_WIDGET_PLUGIN(Battery, kMyManifest) +``` + +One library can offer several widget types: give the class a constructor taking +`std::string_view type` and throw `std::invalid_argument` for an unknown one. + +## Build and manifest + +```cmake +find_package(Qt6 REQUIRED COMPONENTS Widgets) +add_library(my_widgets SHARED battery.cpp) +target_link_libraries(my_widgets PRIVATE plotjuggler_sdk::plugin_sdk Qt6::Widgets) +pj_configure_plugin(my_widgets FAMILIES widget + MANIFEST_FILE ${CMAKE_CURRENT_SOURCE_DIR}/manifest.json + MANIFEST_HEADER generated/my_manifest.hpp MANIFEST_VAR kMyManifest) +``` + +```json +{ + "id": "my-widgets", "name": "My Widgets", "version": "1.0.0", + "min_sdk_required": "0.39.0", + "qt_modules": ["Core", "Gui", "Widgets"], + "widgets": [ + { "type": "battery", "name": "Battery", "icon": "...", + "settings": [ { "key": "min", "type": "number", "default": 0, "label": "Minimum" } ] } + ] +} +``` + +- `widgets` is required and non-empty. `type` is permanent (layouts store + `plugin_id/type`) and must not contain `/`; `name` and `icon` may change. +- `qt_modules` lists the Qt modules the library links, taken only from + `pj_plugins/widget_qt_modules.h` (installed as + `share/plotjuggler_sdk/widget_qt_modules.json`). +- `min_sdk_required` is at least `0.39.0`, the release that introduced the family. +- `settings` is a declarative form the host draws and stores. The plugin has no + settings UI: it receives the values through `loadConfig(json)`. + +## Inputs, values, history + +- **`inputs()`** returns `PJ::WidgetInput{name, label, unit, kind, multiple}`. + `name` is the identity the host matches wires by: non-empty, unique, stable. + `kind` is `kNumber`, `kText` or `kObject` (bytes to decode with the `pj_base` + codecs). `multiple = true` only where every wired series is treated alike (a + table of values). The list may depend on settings; call + `widgetHost().notifyInputsChanged()` when it changes. +- **`onUpdate(values, time)`** carries one value per *wire*: an unwired input has + no entry, a `multiple` input has one per series (`slot`). `time` is in seconds + on the host's display axis. The host also calls it with no changed sample + (after `createWidget`, an accepted `loadConfig`, a wiring change), so treat + every call as "re-read what you need", then repaint. +- **History is requested, not pushed:** + `widgetHost().queryHistory(input, slot, from, to, max_samples, callback)` calls + back once per sample and returns the count the range held before the host's + cap or decimation. +- **Configuration:** `loadConfig(json)` receives every key declared in + `settings` (the host fills in the manifest defaults) with whatever + `saveConfig()` last returned laid over it. Do not reuse a setting key for + internal state. +- **Playback:** the optional `pj.playback.v1` service in the registry passed to + `bind` reads the play state and seeks. Treat it as absent until the lookup succeeds. + +## Traps specific to Widget + +- **Own library, own Qt.** Widget is the only family that links Qt. Never put + another family in the same library, and never link Qt from another family. +- **Same Qt major.minor as the host.** `PJ_WIDGET_PLUGIN` records the Qt version + it was compiled with from `QT_VERSION_*`, so the translation unit that expands + it must include a Qt header. The host refuses a mismatch. No Qt private + headers and no QRhi: they tie the plugin to one exact Qt. +- **Destruction is synchronous.** The host calls `setParent(nullptr)`, then + destroys the instance, whose destructor deletes the widget if it is still + alive. Hold the widget in a `QPointer` and never call `deleteLater` on it. +- **Main thread only.** Do not start a thread that touches the widget or the host. +- **Values are borrowed.** Every pointer in a `PJ_widget_value_t` (`text`, + `bytes`, `type_name`, `label`) is valid only during the call: copy what you keep. +- **Never read data in `paintEvent`.** Keep your own copy from `onUpdate` or + `queryHistory`; the host may refuse a history query from a paint handler. +- **Dispatch on `v.input` and `v.slot`**, never on the position in `values`. +- **Report an `inputs()` failure by throwing.** The host then keeps the inputs + and wires it had, instead of reading the failure as "no inputs". +- **Linux: build on Ubuntu 22.04**, since a library built with a newer libstdc++ + fails to load in the official host. **Windows: MSVC**, and release and debug + runtimes do not mix. + +## Testing + +`pj_plugins/sdk/testing/fake_widget_host.hpp` (`PJ::sdk::testing::FakeWidgetHost`) +is an in-memory `pj.widget_host.v1` host: feed samples per wire, drive +`on_update` at a tracker time, and assert on history queries, caps and +decimation without a running host. diff --git a/.github/workflows/linux-ci.yml b/.github/workflows/linux-ci.yml index cbf1f071..6183b1ca 100644 --- a/.github/workflows/linux-ci.yml +++ b/.github/workflows/linux-ci.yml @@ -234,7 +234,7 @@ jobs: ctest --test-dir build -L wasi --output-on-failure - name: ABI drift gate - # Mechanically enforces the Release Versioning policy (CLAUDE.md): a non-MAJOR + # Mechanically enforces the versioning policy (docs/RELEASING.md): a non-MAJOR # change must not break ABI. Diffs the mock_data_source_plugin canary DSO against # the checked-in pj_base/abi/baseline.abi via libabigail and FAILS on an # ABI-INCOMPATIBLE change (bit 8). Backward-compatible additions (bit 4) pass with diff --git a/CLAUDE.md b/CLAUDE.md index 6cd3a9fd..dc57db49 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,11 +5,17 @@ the SDK as a Conan package or source checkout; SDK changes belong in this repo. ## Read path -Before implementing a helper, consult [Existing SDK utilities](docs/sdk-utilities.md) -and read the matching header. If no entry matches, search the public headers and -`cmake/` before adding an implementation. Then read the relevant module -instructions and task-specific guide. Source providers must also follow -[the provider contract](docs/provider-guide.md). +- **Writing a plugin against the SDK:** the + [plotjuggler-plugin skill](.claude/skills/plotjuggler-plugin/SKILL.md). +- **Changing the SDK:** before implementing a helper, consult + [Existing SDK utilities](docs/sdk-utilities.md) and read the matching header. + If no entry matches, search the public headers and `cmake/` before adding an + implementation. Then read the module instructions and the task's guide from + the table below. Source providers also follow + [the provider contract](docs/provider-guide.md). +- **Changing an ABI struct, vtable, service id, builtin type or wire schema:** + read the [ABI evolution rules](pj_plugins/docs/ARCHITECTURE.md#0a-abi-stability-and-evolution-rules-v5) + and [Versioning and releasing](docs/RELEASING.md) first. The metadata query language and `PJ::common::RollingTransferRate` live in **pj-official-plugins `common/query/` and `common/transfer_rate/`**, respectively. @@ -28,21 +34,23 @@ ownership or storage formats change. Read only the relevant reference sections. | Provider descriptors, origins, artifacts, jobs, limits, completion and Stop | [docs/provider-guide.md](docs/provider-guide.md) | | Plugin authoring, host loaders, discovery and parser routing | [pj_plugins/CLAUDE.md](pj_plugins/CLAUDE.md) | | Canonical object types and codecs | [docs/builtin_type.md](docs/builtin_type.md) | -| ObjectStore services and ownership | [V4_STORE.md](V4_STORE.md) | +| ObjectStore services and ownership | [docs/object_store.md](docs/object_store.md) | | Dialog setters/events | [docs/dialog-sdk-reference.md](docs/dialog-sdk-reference.md) | | Widget plugins (panels with a plugin-created Qt widget) | [pj_plugins/docs/widget-guide.md](pj_plugins/docs/widget-guide.md), [pj_plugins/widget_protocol/CLAUDE.md](pj_plugins/widget_protocol/CLAUDE.md) | | Image annotation / plot marker wire formats | [image_annotations_format.md](docs/image_annotations_format.md), [plot_markers_format.md](docs/plot_markers_format.md) | | C++ style and error handling | [docs/cpp_design_recommendations.md](docs/cpp_design_recommendations.md) | -| Release planning and consumer migration gates | [docs/BACKLOG.md](docs/BACKLOG.md) | +| Versions, host-contract floors, changelog, releases | [docs/RELEASING.md](docs/RELEASING.md) | +| Open SDK items and consumer migration gates | [docs/BACKLOG.md](docs/BACKLOG.md) | -`pj_source` (`plotjuggler_sdk::source`, `PJ::sdk::source`) and `pj_plugins` -depend on `pj_base`; both use nlohmann/json. Provider headers live under -`pj_base/sdk/source/`. +`pj_source` (target `plotjuggler_sdk::source`, namespace `PJ::sdk::source`) is +built from `pj_base/`, with headers under `pj_base/include/pj_base/sdk/source/`. +It and `pj_plugins` depend on `pj_base`; both use nlohmann/json. DataSource/MessageParser/Toolbox C protocols live in `pj_base`; the Dialog protocol lives in `pj_plugins/dialog_protocol/` and the Widget protocol in -`pj_plugins/widget_protocol/`. A Widget plugin links Qt; the SDK never does. The datastore and duplicate -plugin-resolution catalog belong to the application, not this SDK. +`pj_plugins/widget_protocol/`. A Widget plugin links Qt; the SDK never does. +The datastore and duplicate plugin-resolution catalog belong to the +application, not this SDK. The installed `cmake/` helpers (`pj_configure_plugin`, `pj_embed_file`, `pj_harden_plugin_exports`, `pj_add_sdk_test_fixture`) are public API and follow @@ -56,17 +64,25 @@ holds functional parser modules: see ## Build and test ```bash -./build.sh # RelWithDebInfo -./build.sh --debug # Debug + ASAN -./test.sh # all discovered build directories +./build.sh # RelWithDebInfo into build/ +./build.sh --debug # Debug + ASAN into build/debug_asan +./test.sh # every build directory that exists; takes no arguments +ctest --test-dir build/debug_asan -R --output-on-failure # a subset ./test_sdk_install.sh # installed-package consumer +pre-commit run --all-files ``` Dependencies come from `conanfile.py`. Before committing, run `./build.sh --debug && ./test.sh` and pre-commit (clang-format 22.1.0). Packaging changes also require `./test_sdk_install.sh`. -## Release Versioning +CI builds and tests on Linux, Windows (MSVC) and macOS, so module code stays +portable: platform code goes behind `#ifdef` or CMake. Linux CI also runs two +gates `build.sh` does not enable: the ABI drift check (see +[Versioning and releasing](docs/RELEASING.md#abi-and-api-changes)) and the WASI +parser-module conformance (`-DPJ_WASI_SDK_ROOT=`, then `ctest -L wasi`). + +## Versioning Propose a release only according to plugin impact: @@ -77,63 +93,20 @@ Propose a release only according to plugin impact: | Consumer-visible bug fix | PATCH | | Docs, comments, tests or implementation changes invisible to consumers | No bump | -Tail-append optional slots and gate them by `struct_size`; keep minimum vtable -sizes frozen. `abidiff` must show additions only for a MINOR. - -A member change in a builtin object struct (`pj_base/builtin/`) is a MAJOR while -PlotJuggler reads parser objects through the direct C++ bridge; `abidiff` does -not see it. See [ABI evolution rules](pj_plugins/docs/ARCHITECTURE.md#0a-abi-stability-and-evolution-rules-v5). - -The SDK version is **build identity only** — it names what a plugin compiled -against, never what host it runs on. A release that only adds client-side -helpers (static-library code compiled into the plugin) does NOT change runtime -host compatibility and never mints a `pj_base/feature_floors.json` entry; only -host-contract extensions (host-vtable tail slots, versioned service ids, wire -contracts) do, and those MUST be classified there in the same PR (CI-enforced -by `tools/feature_floors/check_feature_floors.py`). The version stays one -monotonic identity; `feature_floors.json` carries the per-release compatibility -semantics; `PJ_ABI_VERSION` remains the orthogonal hard-break counter. -The plugin compatibility contract is `min_sdk_required` (the functioning -floor, the only admission gate) plus the informational -`suggested_sdk_version` (full-feature floor) and `abi_major` for hard -breaks; `min_plotjuggler_version` is deprecated — the decoder keeps reading -it for back-compat, and the host-side gate removal happens in PJ4. -Changelog convention: every release section carries one `Host contract:` line — -either `unchanged (no floor impact)` or `extended: (floor X.Y.Z)`; -the floor checker requires the `extended` form whenever a surface newer than -the last released version exists. -`pj_base/abi/baseline.abi` is refreshed only for an intentional MAJOR break. -See [ABI evolution rules](pj_plugins/docs/ARCHITECTURE.md#0a-abi-stability-and-evolution-rules-v5). - -Consumers pin `plotjuggler_sdk/[>=X.Y.Z <(X+1).0.0]`, explicitly: the lower -bound is the newest feature actually used, the upper bound the next MAJOR. -Pre-1.0 has the same no-breaks guarantee: `[>=0.Y.Z <1.0.0]`. - -`VERSION` is the only hand-maintained version; Conan, CMake and conda derive -theirs from it. Release tags must agree. Tagging or pushing a release requires -explicit user authorization. - -### Development pre-releases - -In-progress SDK work that PJ4 needs before its contract is complete ships as -`X.Y.Z-dev.N` (`VERSION` and the tag `vX.Y.Z-dev.N` agree), not as a public -release. The real `X.Y.Z` is cut once, when the feature's contract is complete. - -- **Publishes:** the tag runs `release.yml` like any release (same matrix, JFrog - upload) and creates a GitHub release marked pre-release; `conda-release.yml` - skips it. PJ4 pins it exactly (`plotjuggler_sdk/0.37.0-dev.1`): Conan version - ranges skip pre-releases unless told to resolve them. -- **No ABI promise between two dev builds.** The MINOR/MAJOR rules above bind - only the final `X.Y.Z`. -- **Names the final version:** `@since`, `feature_floors.json` and the changelog - `Host contract:` floor say `X.Y.Z`, never `X.Y.Z-dev.N`. -- **Ordering is SemVer:** `0.37.0-dev.1 < 0.37.0-dev.2 < 0.37.0`, so a host built - from `0.37.0-dev.1` rejects a plugin whose `min_sdk_required` is `0.37.0`; a - plugin made for a dev host declares `0.37.0-dev.N`. `PJ_SDK_VERSION_*` and the - CMake package version carry `X.Y.Z`; `PJ::sdkVersion()` keeps the full string, - so gate admission with `PJ::SemVer`, not the macros. -- Changelog entries go under `## [Unreleased]`, renamed to `## [X.Y.Z]` at the - final release. +- A member change in a builtin object struct is a MAJOR even though `abidiff` + does not see it: PlotJuggler reads parser objects through the direct C++ bridge. +- A new host-contract surface (host-vtable tail slot, service id, wire flag, + dialog host capability, builtin object type) is classified in + `pj_base/feature_floors.json` in the same PR; the `feature_floors_check` test + fails otherwise. +- Changelog entries go under `## [X.Y.Z] — Unreleased`, with one + `Host contract:` line per release. +- `VERSION` is the only version to edit. In-progress work that PlotJuggler needs + early ships as a `X.Y.Z-dev.N` pre-release. +- Tagging or pushing a release requires explicit user authorization. + +The full rules (floors, plugin compatibility fields, consumer pins, +pre-releases, cutting a release) are in [docs/RELEASING.md](docs/RELEASING.md). ## Coding conventions @@ -141,14 +114,19 @@ release. The real `X.Y.Z` is cut once, when the feature's contract is complete. - `CamelCase` types, `camelBack` functions, `lower_case` variables, `lower_case_` members, `kCamelCase` constants. - Match module ownership: vocabulary in `PJ`, SDK helpers in `PJ::sdk`, - provider helpers in `PJ::sdk::source`, experimental parser modules in `pj`. + provider helpers in `PJ::sdk::source`; the experimental parser-module + authoring kit uses `pj`. - Fallible operations use `PJ::Expected` / `PJ::Status`; invariants use - `PJ_ASSERT`. Warnings: `-Wall -Wextra -Werror`. + `PJ_ASSERT`. +- Warnings are strict (`-Wall -Wextra -Wshadow -Wold-style-cast -Wconversion + -Wpedantic` and more; `/W4` on MSVC) and errors by default + (`PJ_WARNINGS_AS_ERRORS`). The root `CMakeLists.txt` holds the list. ## Documentation requests - "Read all documentation": include every tracked `.md`, including `.claude/`. - "Update documentation": fix outdated claims and add guidance whose absence - caused a bug. + caused a bug. Docs describe the current state; finished plans and historical + analyses go to `docs/archive/`, which is not a reference. - "Check documentation": verify the affected contracts against the code; report discrepancies when edits are outside the authorized task. diff --git a/README.md b/README.md index f17531d5..ce2e9dff 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,10 @@ [![Linux CI](https://github.com/PlotJuggler/plotjuggler_sdk/actions/workflows/linux-ci.yml/badge.svg)](https://github.com/PlotJuggler/plotjuggler_sdk/actions/workflows/linux-ci.yml) [![Windows CI](https://github.com/PlotJuggler/plotjuggler_sdk/actions/workflows/windows-ci.yml/badge.svg)](https://github.com/PlotJuggler/plotjuggler_sdk/actions/workflows/windows-ci.yml) -C++20 foundation libraries for [PlotJuggler](https://github.com/facontidavide/PlotJuggler). +C++20 plugin SDK and host-loading libraries for [PlotJuggler](https://github.com/PlotJuggler/PlotJuggler). +Writing a plugin? Start with the [plugin-authoring guide](.claude/skills/plotjuggler-plugin/SKILL.md) +and the family guides in [`pj_plugins/docs/`](pj_plugins/docs/). Start with [Existing SDK utilities](docs/sdk-utilities.md) before writing helpers. Source providers also follow [the provider contract](docs/provider-guide.md). Both guides ship under `share/plotjuggler_sdk/docs/` in the installed package. @@ -15,7 +17,7 @@ Both guides ship under `share/plotjuggler_sdk/docs/` in the installed package. |--------|-------------|--------------|---------| | **pj_base** | Vocabulary types: `Timestamp`, `DatasetId`, `TopicId`, type trees, `Expected`, `Span` | None | Apache-2.0 | | **pj_source** | Provider descriptors, origins, artifact cache, jobs, limits, presentation, completion and Stop; link `plotjuggler_sdk::source` | pj_base, nlohmann/json | Apache-2.0 | -| **pj_plugins** | C-ABI plugin protocol (DataSource, MessageParser, Dialog, Toolbox, Widget families), C++ SDK base classes, plugin discovery, host-side loaders, and config helpers | pj_base, nlohmann/json | Apache-2.0 | +| **pj_plugins** | C-ABI plugin protocol (DataSource, MessageParser, Dialog, Toolbox, Widget families), C++ SDK base classes, plugin discovery and host-side loaders | pj_base, nlohmann/json | Apache-2.0 | | **cmake/** | Plugin-authoring CMake helpers shipped with `plugin_sdk`: `pj_configure_plugin()` (symbol isolation, export allowlist + gate, manifest validation/sidecar/embed), `pj_embed_file()`, `pj_harden_plugin_exports()`, `pj_add_sdk_test_fixture()` — see `examples/sdk_consumer` | — | Apache-2.0 | ## Getting Started @@ -40,7 +42,8 @@ cd plotjuggler_sdk ./build.sh # RelWithDebInfo (build/) ./build.sh --debug # Debug + ASAN (build/debug_asan) ./test.sh # runs tests in all discovered build dirs -git ls-files -z '*.cpp' | xargs -0 clang-tidy-22 -p build # clang-tidy +pre-commit run --all-files # clang-format and file hygiene +git ls-files -z '*.cpp' | xargs -0 clang-tidy-22 -p build/debug_asan # clang-tidy (not a CI gate) ``` ## Project Layout @@ -49,10 +52,16 @@ git ls-files -z '*.cpp' | xargs -0 clang-tidy-22 -p build # clang-tidy pj_base/ Vocabulary, builtin codecs and sdk/source/ provider support pj_plugins/ C-ABI plugin protocol, SDK, host loaders cmake/ Installed CMake helpers for plugin authors (PjPlugin.cmake, PjSdkTestFixtures.cmake) -examples/sdk_consumer/ find_package() consumer exercising the helpers (built by the release + conda checks) -docs/ Project-wide design guides +examples/sdk_consumer/ find_package() consumer exercising the helpers (built by ./test_sdk_install.sh and the conda recipe test) +experimental/ Built and tested here, never installed (functional parser modules) +tools/feature_floors/ Host-contract floor table checks +packaging/ conan-center-index recipe staging +docs/ Project-wide design guides; RELEASING.md for versions and releases ``` +Conan/CMake components: `base`, `source`, `plugin_sdk` (what a plugin links) and +`plugin_host` (the loaders a host or a plugin test links). + ## License PlotJuggler SDK is **Apache-2.0** ([LICENSE-APACHE](LICENSE-APACHE)); each @@ -60,6 +69,5 @@ source file carries an authoritative `SPDX-License-Identifier` header. You may build **proprietary plugins and applications** on the SDK without restriction; Apache-2.0 also grants an explicit patent license. -The columnar storage engine (formerly the MPL-2.0 `pj_datastore` module) has -moved to the PlotJuggler application repo; plugins load through a stable C ABI -and never link it. See [LICENSE](LICENSE) for the full mapping. +The columnar storage engine belongs to the PlotJuggler application; plugins +reach it through the stable C ABI and never link it. See [LICENSE](LICENSE). diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index d3aa52c2..c56a35e3 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -1,43 +1,24 @@ -# SDK backlog +# Open SDK items and consumer migration gates -Track consumer migrations and deliberate deferrals here. For shipped helpers, +Track open deferrals and consumer migrations here. For shipped helpers, start with [Existing SDK utilities](sdk-utilities.md), then follow -[the provider contract](provider-guide.md). Delivery in this branch does not -imply that consumer repositories have adopted it. +[the provider contract](provider-guide.md). Shipping a helper in the SDK does +not imply that consumer repositories have adopted it. -## 0.31 delivery status +## Open items -The 2026-09-05 promotion plan is implemented below, with the 2026-09-06 -maintainer decisions on query/rate ownership. New consumers use `plotjuggler_sdk::source` and `PJ::sdk::source`. - -| Original item | Status and reuse destination | -|---|---| -| 1. Environment ceilings | Shipped: `envLimit` / `minNonzero` in `source/limits.hpp`; unit conversion and saturation remain caller policy. | -| 2. Duration ceilings | Already shipped: `JobControl::armWatchdog`; consumers replace their deadline loops. | -| 3. Source presentation | Shipped: `SourcePresentation`, `recordSourcePresentation`, `sourcePresentationSettingsGroup` in `source/presentation.hpp`. | -| 4. Capture/completion tracker | Shipped: `IngestOutcomeLedger` in `source/outcome_ledger.hpp`; attachment ordering, synchronization and dataset-survival decisions remain provider responsibilities. | -| 5. Source-record envelope | Shipped: `parseSourceRecordEnvelope` in `source/record_envelope.hpp`, with conformance cases; flat mcap_cloud v1 still requires a versioned restore adapter. | -| 6. Host Stop bridge | Shipped: `StopPoller` in `source/stop_bridge.hpp`; wire provider cancellation to host `requestStop` before joining producers. | -| 7. Time and range math | Shipped: `pj_base/time_format.hpp`, `time_math.hpp`, `slider_window.hpp`; replace consumer copies. | -| 8. Delegated-ingest fixture | Shipped: `PJ::sdk::testing::DelegatedIngestFixture` in `pj_plugins/testing/delegated_ingest_fixture.hpp`. | -| 9. Metadata query language | Relocated to pj-official-plugins `common/query/` (`pj_common_query`, `PJ::query`); reuse its tokenizer/parser/completion/filter, not an SDK copy. | -| 10. Stable table-row identity | Deferred: requires dialog-protocol and host work; do not promote label-to-identity collision workarounds. | -| 11. Rolling transfer rate | Relocated to pj-official-plugins `common/transfer_rate/` (`pj_transfer_rate`, `PJ::common::RollingTransferRate`). | -| 12. Packaged mock plugins | Shipped: four fixture sources under `share/plotjuggler_sdk/test_fixtures/`, built with `pj_add_sdk_test_fixture(name out_target)`. PJ4 can remove its second SDK FetchContent during its pin bump. | - -`source/*.hpp` above means `pj_base/sdk/source/*.hpp`. `time_format` belongs -to pj_base; fixtures belong to pj_plugins/testing. `pj_cloud` remains reserved -for future connection/reconnect code. +- **Stable table-row identity.** Deferred: requires dialog-protocol and host + work; do not promote label-to-identity collision workarounds. ## Consumer migration gates -The following mcap-cloud constraints come from the 2026-09-05 sharing audit of -its SDK 0.20 implementation and toolbox_mosaico. Recheck them against the -consumer branch when adopting 0.31; SDK delivery does not resolve them. +The following mcap-cloud constraints (from a sharing audit of its implementation +and toolbox_mosaico) apply when it adopts the SDK source helpers. Recheck them +against the consumer branch; SDK delivery does not resolve them. ### Preserve behavior while deleting provider copies -- Not every wait is cancellable yet: reconnect-Hello uses +- Not every wait is cancellable: reconnect-Hello uses `sendAndWait(wake_on_cancel=false)` — deleting the deadline mechanism without fixing this adds up to 10 s after watchdog expiry; add a cancellation-during-reconnect-Hello test. diff --git a/docs/RELEASING.md b/docs/RELEASING.md new file mode 100644 index 00000000..26974ab6 --- /dev/null +++ b/docs/RELEASING.md @@ -0,0 +1,111 @@ +# Versioning and releasing the SDK + +What a change must record so that its release is correct, and how a release or +a development pre-release is cut. The bump table is in the root +[CLAUDE.md](../CLAUDE.md#versioning); the ABI rules are in +[ABI evolution rules](../pj_plugins/docs/ARCHITECTURE.md#0a-abi-stability-and-evolution-rules-v5). + +## ABI and API changes + +- Tail-append optional slots and gate them by `struct_size`; keep the minimum + vtable sizes frozen. For a MINOR, `abidiff` must show additions only. +- A member change in a builtin object struct (`pj_base/include/pj_base/builtin/`) + is a MAJOR while PlotJuggler reads parser objects through the direct C++ + bridge. `abidiff` does not see it. +- `pj_base/abi/baseline.abi` is refreshed (target `abi_update_baseline`) only for + an intentional MAJOR break. It stays at the last MAJOR, so the ABI check + reports every addition since. The check runs in Linux CI; to run it locally, + configure with `-DPJ_ENABLE_ABI_CHECK=ON` (needs libabigail's `abidw` and + `abidiff`) and build the `abi_check` target. + +## Host-contract floors + +The SDK version is **build identity only**: it names what a plugin compiled +against, never what host it runs on. `pj_base` and `pj_source` are static +libraries compiled into the plugin, so a release that only adds client-side +helpers does not change runtime host compatibility and never adds a floor. + +A host-contract surface does. The checker inventories five kinds from the +protocol headers (`tools/feature_floors/extract_host_surfaces.py`): + +- host-vtable tail slots; +- versioned host-served service ids (`pj..v`); +- `PJ_*_FLAG*` wire flags; +- dialog host capabilities (`PJ_dialog_host_capability_t` members); +- builtin object types (`PJ_builtin_object_type_t` members), since an older + host cannot decode a type it does not know. + +A PR that adds one classifies it in `pj_base/feature_floors.json` and refreshes +the snapshot with `python3 tools/feature_floors/extract_host_surfaces.py --write`. +The ctest `feature_floors_check` (`tools/feature_floors/check_feature_floors.py`) +fails on an unclassified surface. It is registered only when CMake finds +Python 3. + +The plugin compatibility contract: + +- `min_sdk_required` is the functioning floor and the only admission gate. +- `suggested_sdk_version` is the full-feature floor, informational. A plugin + that declares it also names a `floor_test`. +- `abi_major` marks hard breaks. `PJ_ABI_VERSION` is the orthogonal hard-break + counter. +- `min_plotjuggler_version` is deprecated. The SDK still decodes it, and + PlotJuggler shows it but never gates on it. + +`feature_floors.json` and the plugin-side checker `feature_floor_check.py` are +installed in `share/plotjuggler_sdk/`, so plugin repositories check their own +manifests against the same table. + +## Changelog + +Entries go under `## [X.Y.Z] — Unreleased`. The floor checker finds that section +by the word "Unreleased", so keep it in the heading until the release, then +drop ` — Unreleased`. + +Every release section carries one `Host contract:` line: either +`unchanged (no floor impact)` or `extended: (floor X.Y.Z)`. The +floor checker requires the `extended` form whenever a surface newer than the +last released version exists. + +## Consumer pins + +Consumers pin `plotjuggler_sdk/[>=X.Y.Z <(X+1).0.0]` explicitly: the lower +bound is the newest feature actually used, the upper bound the next MAJOR. On +the `0.x` line that is `[>=0.Y.Z <1.0.0]`; no `0.x` release breaks an earlier one. + +## Version sources + +`VERSION` is the only version to edit for a release. CMake, Conan and conda +derive theirs from it, and the release tag must agree. The conan-center-index +recipe staged under `packaging/` lists its versions by hand and is not part of +the build. + +## Development pre-releases + +In-progress SDK work that PlotJuggler needs before its contract is complete +ships as `X.Y.Z-dev.N` (`VERSION` and the tag `vX.Y.Z-dev.N` agree), not as a +public release. The real `X.Y.Z` is cut once, when the contract is complete. + +- **Publishes:** the tag runs `release.yml` like any release (same build matrix, + JFrog upload) and creates a GitHub release marked pre-release; + `conda-release.yml` skips it. Consumers pin it exactly + (`plotjuggler_sdk/X.Y.Z-dev.N`): Conan version ranges skip pre-releases unless + told to resolve them. +- **No ABI promise between two dev builds.** The MINOR/MAJOR rules bind only the + final `X.Y.Z`. +- **Names the final version:** `@since`, `feature_floors.json` and the changelog + `Host contract:` floor say `X.Y.Z`, never `X.Y.Z-dev.N`. +- **Ordering is SemVer:** `X.Y.Z-dev.1 < X.Y.Z-dev.2 < X.Y.Z`, so a host built + from `X.Y.Z-dev.1` rejects a plugin whose `min_sdk_required` is `X.Y.Z`; a + plugin made for a dev host declares `X.Y.Z-dev.N`. `PJ_SDK_VERSION_*` and the + CMake package version carry `X.Y.Z`; `PJ::sdkVersion()` keeps the full string, + so gate admission with `PJ::SemVer`, not the macros. + +## Cutting a release + +Tagging or pushing a release requires explicit user authorization. + +1. `VERSION` holds the release version; the changelog section is renamed from + `## [X.Y.Z] — Unreleased` to `## [X.Y.Z]` and carries its `Host contract:` line. +2. Push the tag `vX.Y.Z` (or `vX.Y.Z-dev.N`). `release.yml` builds every + platform, uploads to the JFrog Conan remote and creates the GitHub release; + `conda-release.yml` publishes the conda package for a final release. diff --git a/docs/toolbox-porting-gap-analysis.md b/docs/archive/toolbox-porting-gap-analysis.md similarity index 100% rename from docs/toolbox-porting-gap-analysis.md rename to docs/archive/toolbox-porting-gap-analysis.md diff --git a/docs/builtin_type.md b/docs/builtin_type.md index 04b26d96..897aeb09 100644 --- a/docs/builtin_type.md +++ b/docs/builtin_type.md @@ -84,8 +84,8 @@ span points at the bytes; the anchor keeps the underlying allocation alive while consumers use it. **Keep small objects owned.** `ImageAnnotations` and `FrameTransforms` own -their vectors, strings, and scalar fields directly. Future marker types should -follow the same pattern unless they grow payload-sized byte arrays. These values +their vectors, strings, and scalar fields directly. Small value types follow the +same pattern unless they grow payload-sized byte arrays. These values are small enough that the zero-copy anchor pattern is unnecessary. **Serialize only at an explicit boundary.** Large byte-backed types remain @@ -116,8 +116,10 @@ Builtin objects fall into two serialization families: | Owned values | `ImageAnnotations`, `FrameTransforms`, `SceneEntities`, `RobotDescription`, `CameraInfo`, `Log`, `PosesInFrame`, `PlotMarkers` | SDK structs own their vectors/strings/scalars directly. | Canonical codecs serialize the owned value to the protobuf-wire payload described by the `.proto` contract, using shared private wire primitives. `RobotDescription.text` remains raw URDF/SDF/MJCF source inside its small canonical envelope. | Canonical `.proto` files live under `pj_base/proto/pj` and act as the wire -format contract. One file per top-level message, each named after its message -(`Image.proto`, `SceneEntities.proto`, `FrameTransforms.proto`, …). Shared +format contract. One file per builtin type, named after it (`Image.proto`, +`SceneEntities.proto`, `FrameTransforms.proto`, …); a file also holds the nested +messages its type needs (`PointCloud.proto` has `PointField`, `SceneEntities.proto` +the primitives and `SceneEntity`). Shared geometry primitives are grouped in `Geometry.proto`: `Point2`, `Point3`, `Vector2`, `Vector3`, `Quaternion`, and `Pose`. See [`pj_base/proto/pj/README.md`](../pj_base/proto/pj/README.md) for the family @@ -149,29 +151,34 @@ encoder and from a later SDK both read correctly: the host decide that a topic produces images, point clouds, depth images, image annotations, frame transforms, or no builtin object. -| Type | Concrete type | Purpose | -|------|---------------|---------| -| `kNone` | none | Scalar-only schema or unknown object. | -| `kImage` | `PJ::sdk::Image` | Raw or compressed image data. | -| `kPointCloud` | `PJ::sdk::PointCloud` | Packed 3D point records. | -| `kDepthImage` | `PJ::sdk::DepthImage` | Depth pixels plus camera intrinsics. | -| `kImageAnnotations` | `PJ::sdk::ImageAnnotations` | Pixel-space overlay primitives. | -| `kFrameTransforms` | `PJ::sdk::FrameTransforms` | Named 3D frame relationships. | -| `kOccupancyGrid` | `PJ::sdk::OccupancyGrid` | 2D metric occupancy grid (maps, costmaps) in world coordinates. | -| `kCompressedPointCloud` | `PJ::sdk::CompressedPointCloud` | Point cloud delivered in a format-specific compressed binary (e.g. Draco). | -| `kMesh3D` | `PJ::sdk::Mesh3D` | 3D mesh asset in its native binary format (GLTF/GLB/STL/PLY/OBJ/USD/DAE). | -| `kVideoFrame` | `PJ::sdk::VideoFrame` | One frame of an inter-frame-coded video stream (h264/h265/vp9/av1). | -| `kSceneEntities` | `PJ::sdk::SceneEntities` | Procedural 3D scene primitives (arrows, cubes, lines, text, …). | -| `kRobotDescription` | `PJ::sdk::RobotDescription` | Raw URDF/SDF/MJCF text + format hint. | -| `kCameraInfo` | `PJ::sdk::CameraInfo` | Pinhole camera calibration (intrinsics K, distortion D, rectification R, projection P). | -| `kOccupancyGridUpdate` | `PJ::sdk::OccupancyGridUpdate` | Incremental sub-rectangle patch for a previously-published `OccupancyGrid`. | -| `kLog` | `PJ::sdk::Log` | Textual log message (severity level + text + source). | -| `kPosesInFrame` | `PJ::sdk::PosesInFrame` | Array of poses in one frame (PoseArray / particle clouds); styling is viewer-side. | -| `kVoxelGrid` | `PJ::sdk::VoxelGrid` | Dense 3D voxel grid (occupancy/cost/ESDF/semantic); the volumetric sibling of `OccupancyGrid`. | -| `kGridMap` | `PJ::sdk::GridMap` | 2D grid of per-cell channels (elevation maps, layered costmaps) in world coordinates. | -| `kMeshMap` | `PJ::sdk::MeshMap` | Triangle mesh whose vertices and faces carry named data channels, one array per channel (navigation, cost, semantic meshes). | -| `kMeshMapUpdate` | `PJ::sdk::MeshMapUpdate` | Channel patches (dense or sparse) for a previously-published `MeshMap`. | -| `kPlotMarkers` | `PJ::sdk::PlotMarkers` | Findings on a time-series plot (regions, events, bands, and labels). | +| Type | Tag | Concrete type | Purpose | +|------|-----|---------------|---------| +| `kNone` | 0 | none | Scalar-only schema or unknown object. | +| `kImage` | 1 | `PJ::sdk::Image` | Raw or compressed image data. | +| `kPointCloud` | 3 | `PJ::sdk::PointCloud` | Packed 3D point records. | +| `kDepthImage` | 4 | `PJ::sdk::DepthImage` | Depth pixels plus camera intrinsics. | +| `kImageAnnotations` | 5 | `PJ::sdk::ImageAnnotations` | Pixel-space overlay primitives. | +| `kFrameTransforms` | 6 | `PJ::sdk::FrameTransforms` | Named 3D frame relationships. | +| `kOccupancyGrid` | 7 | `PJ::sdk::OccupancyGrid` | 2D metric occupancy grid (maps, costmaps) in world coordinates. | +| `kCompressedPointCloud` | 8 | `PJ::sdk::CompressedPointCloud` | Point cloud delivered in a format-specific compressed binary (e.g. Draco). | +| `kMesh3D` | 9 | `PJ::sdk::Mesh3D` | 3D mesh asset in its native binary format (GLTF/GLB/STL/PLY/OBJ/USD/DAE). | +| `kVideoFrame` | 10 | `PJ::sdk::VideoFrame` | One frame of an inter-frame-coded video stream (h264/h265/vp9/av1). | +| `kSceneEntities` | 11 | `PJ::sdk::SceneEntities` | Procedural 3D scene primitives (arrows, cubes, lines, text, …). | +| `kRobotDescription` | 13 | `PJ::sdk::RobotDescription` | Raw URDF/SDF/MJCF text + format hint. | +| `kCameraInfo` | 14 | `PJ::sdk::CameraInfo` | Pinhole camera calibration (intrinsics K, distortion D, rectification R, projection P). | +| `kOccupancyGridUpdate` | 15 | `PJ::sdk::OccupancyGridUpdate` | Incremental sub-rectangle patch for a previously-published `OccupancyGrid`. | +| `kLog` | 16 | `PJ::sdk::Log` | Textual log message (severity level + text + source). | +| `kPosesInFrame` | 17 | `PJ::sdk::PosesInFrame` | Array of poses in one frame (PoseArray / particle clouds); styling is viewer-side. | +| `kVoxelGrid` | 18 | `PJ::sdk::VoxelGrid` | Dense 3D voxel grid (occupancy/cost/ESDF/semantic); the volumetric sibling of `OccupancyGrid`. | +| `kGridMap` | 20 | `PJ::sdk::GridMap` | 2D grid of per-cell channels (elevation maps, layered costmaps) in world coordinates. | +| `kMeshMap` | 21 | `PJ::sdk::MeshMap` | Triangle mesh whose vertices and faces carry named data channels, one array per channel (navigation, cost, semantic meshes). | +| `kMeshMapUpdate` | 22 | `PJ::sdk::MeshMapUpdate` | Channel patches (dense or sparse) for a previously-published `MeshMap`. | +| `kPlotMarkers` | 19 | `PJ::sdk::PlotMarkers` | Findings on a time-series plot (regions, events, bands, and labels). | + +The tag is the `PJ_builtin_object_type_t` value (`builtin_object_abi.h`). Tags `2` and `12` +are reserved and never reused. `kMeshMap` and `kMeshMapUpdate` are a hard floor: a plugin that +emits either must declare `min_sdk_required` of at least 1.0.0 (see +`pj_base/feature_floors.json`). ### Object-topic renderer metadata @@ -263,7 +270,9 @@ pixel size, 2 bytes for `16UC1` and 4 for `32FC1`), and the field table reports ``` Helpers in `pj_base/builtin/depth_image_utils.hpp` derive common matrices such as -rectification rotation and projection matrix when a consumer wants them. +rectification rotation and projection matrix when a consumer wants them, and +back-project pixels: `PinholeIntrinsics::fromK` inverts `K` once per image +(rejecting a non-finite or singular `K`) and `unprojectPixel` is its one-pixel form. ## PointCloud @@ -522,6 +531,7 @@ axes glyphs, or model (mesh asset) references. | `id` | `std::string` | Republishing with the same `(topic, id)` replaces the previous entity. | | `lifetime_ns` | `int64_t` | `0` means persist until replaced; otherwise expire `lifetime_ns` after `timestamp_ns`. | | `frame_locked` | `bool` | When true, track `frame_id` as it moves; when false, stamp into the fixed frame at publish time. | +| `metadata` | `std::vector` | Arbitrary user-attached key/value pairs; keys should be unique within an entity. | | `arrows` / `cubes` / `spheres` / `cylinders` / `lines` / `triangles` / `texts` / `axes` / `models` | `std::vector<…Primitive>` | Heterogeneous primitive lists. `models` references a mesh asset by `url` or inline `data`. | The `SceneEntities` batch also carries `deletions` (`std::vector`): @@ -530,7 +540,11 @@ stateful stream (e.g. ROS Marker `DELETE` / `DELETEALL`). A deletion is either `kMatchingId` (remove the entity with the given `id`) or `kAll` (clear the topic). Each primitive carries its own `Pose`, geometry-specific size or shape -fields, and color (or per-vertex colors, where applicable). See +fields, and color (or per-vertex colors, where applicable). A `ModelPrimitive` +has `pose`, `scale`, `color` plus `override_color` (tint only when true), and +either a `url` or inline `data` with its MIME `media_type`. The line topology +(`LinePrimitive.Type`, C++ `LineType`) uses Foxglove's numbering (`LINE_STRIP` = 0, +`LINE_LOOP` = 1, `LINE_LIST` = 2). See `pj_base/include/pj_base/builtin/scene_entities.hpp` for the per-primitive fields and `pj_base/proto/pj/SceneEntities.proto` for the wire contract. @@ -616,6 +630,8 @@ plus an optional `category`. | `category` | `std::string` | Source-defined grouping (a tag, a subsystem); may be empty. | The source-location fields (`file`, `function`, `line`) are intentionally omitted. +On the wire `source` is field 4 and fields 5 and 6 are reserved (they are `file` and +`line` in `foxglove.Log`), so a Foxglove-encoded log decodes with those two skipped. `pj_base/builtin/log_codec.hpp` serializes and deserializes this type using the canonical `PJ.Log` protobuf wire format. diff --git a/docs/cpp_design_recommendations.md b/docs/cpp_design_recommendations.md index b3ba3d11..784c3765 100644 --- a/docs/cpp_design_recommendations.md +++ b/docs/cpp_design_recommendations.md @@ -248,27 +248,15 @@ Avoid template metaprogramming when `concepts` + `constexpr if` suffices. ## 13. Library Dependencies -The project uses a small set of third-party libraries alongside the C++20 standard library. - -### String Formatting - -Use `fmt` for all string formatting and concatenation: - -| Task | Use | -|---|---| -| Formatting / concatenation | `fmt::format("prefix: {}", value)` | -| Joining containers | `fmt::join(container, ", ")` inside `fmt::format` | +The SDK's public headers depend only on the C++20 standard library; +nlohmann/json is a public dependency of `pj_source` and `pj_plugins`. +fast_float and fmt are private build details of the SDK's own `.cpp` files and +do not reach plugins, so public headers never include them. ### Hash Containers -**In `.cpp` files:** use `tsl::robin_map` / `tsl::robin_set` for performance-critical hash containers. - -**In `.hpp` headers:** use `std::unordered_map` / `std::unordered_set` to avoid leaking dependencies. - -| Need | Use (in `.cpp`) | Use (in `.hpp`) | -|---|---|---| -| Hash map (default) | `tsl::robin_map` | `std::unordered_map` | -| Hash set (default) | `tsl::robin_set` | `std::unordered_set` | +Use `std::unordered_map` / `std::unordered_set`; the SDK carries no +third-party hash container. ### Error Handling @@ -384,7 +372,7 @@ Practical tips for robust API design: - **Return by value for cheap types** -- Avoid returning `const T&` to internal data when `T` is cheap to copy (e.g., `string_view`, `int`). Return by value instead -- this decouples callers from internal representation and preserves optimization freedom. -- **Avoid unnecessarily strong guarantees** -- Weaker guarantees (e.g., no iterator stability) can enable significantly better performance. Open-addressing hash maps like `tsl::robin_map` outperform `std::unordered_map` precisely because they trade iterator stability for cache locality. +- **Avoid unnecessarily strong guarantees** -- Weaker guarantees (e.g., no iterator stability) can enable significantly better performance, as open-addressing hash maps show by trading iterator stability for cache locality. --- @@ -423,17 +411,17 @@ Pick the simplest container that fits the access pattern. ### Associative (key lookup) -| Need | Container (`.cpp`) | Container (`.hpp`) | -|---|---|---| -| Ordered, unique keys | `std::map` | `std::map` | -| Unordered (faster), unique keys | `tsl::robin_map` | `std::unordered_map` | +| Need | Container | +|---|---| +| Ordered, unique keys | `std::map` | +| Unordered (faster), unique keys | `std::unordered_map` | ### Sets -| Need | Container (`.cpp`) | Container (`.hpp`) | -|---|---|---| -| Ordered | `std::set` | `std::set` | -| Unordered (fastest) | `tsl::robin_set` | `std::unordered_set` | +| Need | Container | +|---|---| +| Ordered | `std::set` | +| Unordered (fastest) | `std::unordered_set` | ### Adaptors @@ -449,47 +437,12 @@ Pick the simplest container that fits the access pattern. ## Compiler Flags -### GCC / Clang - -``` --std=c++20 --Wall -Wextra --Wcast-qual --Wconversion --Wdouble-promotion --Wformat=2 --Wmissing-declarations --Wnon-virtual-dtor --Wold-style-cast --Woverloaded-virtual --Wpointer-arith --Wshadow --Wunused-result --Wvla --Werror # in CI; optional locally -``` - -Clang additional: -``` --Wmost --Wthread-safety # thread-safety static analysis -``` - -Debug/test builds: -``` --fsanitize=address,undefined -``` - -### MSVC - -``` -/std:c++20 -/W4 -/WX # in CI; treat warnings as errors -/permissive- -/DNOMINMAX -/DWIN32_LEAN_AND_MEAN -``` +The root `CMakeLists.txt` owns the warning set; read it there rather than here. +It is strict on every compiler (GCC/Clang: `-Wall -Wextra -Wshadow +-Wold-style-cast -Wconversion -Wpedantic` and more; MSVC: `/W4 /permissive-`), +and warnings are errors unless `PJ_WARNINGS_AS_ERRORS` is turned off. +`./build.sh --debug` adds AddressSanitizer (`PJ_ENABLE_SANITIZERS`); +`PJ_ENABLE_TSAN` selects ThreadSanitizer instead. --- @@ -517,7 +470,6 @@ When reviewing code, classify issues by severity: - `bool` parameter where `enum class` would be clearer - Function with > 3 parameters that could use a struct - Missing `constexpr` on a function that could be constexpr -- Using `std::unordered_map` in `.cpp` files where `tsl::robin_map` would be better ### Suggestion (optional improvement) - Opportunity to use ranges for cleaner pipeline @@ -532,8 +484,6 @@ When reviewing code, classify issues by severity: - [Qt API Design Principles](https://wiki.qt.io/API_Design_Principles) - [C++ Core Guidelines](https://isocpp.github.io/CppCoreGuidelines/) - Scott Meyers, *Effective C++* Item 18; [How Non-Member Functions Improve Encapsulation](https://www.aristeia.com/Papers/CUJ_Feb_2000.pdf) -- [fmt library](https://fmt.dev/) -- type-safe formatting -- [tsl::robin_map](https://github.com/Tessil/robin-map) -- fast open-addressing hash map - [Strong Types for Strong Interfaces](https://www.fluentcpp.com/2016/12/08/strong-types-for-strong-interfaces/) -- Fluent C++ - [When SOLID Breaks: Choose CLARITY](https://krossovochkin.com/posts/2025_05_05_when_solid_breaks_choose_clarity/) - [CppCon 2024: The Most Important C++ Design Guideline is Testability](https://www.classcentral.com/course/youtube-the-most-important-c-design-guideline-is-testability-jody-hagins-cppcon-2024-419955) -- Jody Hagins diff --git a/docs/dialog-sdk-reference.md b/docs/dialog-sdk-reference.md index 550b5828..466985d9 100644 --- a/docs/dialog-sdk-reference.md +++ b/docs/dialog-sdk-reference.md @@ -8,7 +8,7 @@ For the full tutorial, see [dialog-plugin-guide.md](../pj_plugins/docs/dialog-pl ## Runtime Host Information (since 0.21.0) -`DialogPluginBase::hostInfo()` returns an owned `std::optional` +`DialogPluginBase::hostInfo()` returns a `const std::optional&` with `sdk_version`, `plotjuggler_version`, and a capability mask. An empty optional means a pre-0.21 or non-conforming embedding host and is the signal to use a compatible fallback. Test capabilities with @@ -34,6 +34,8 @@ last-writer-wins. ## WidgetData Setters +`WidgetData::toJson()` and `WidgetEventBuilder` replace invalid UTF-8 in strings with U+FFFD, so one bad label cannot drop a whole update. + ### QLineEdit | Method | Description | @@ -184,6 +186,7 @@ Selected result requires at least one non-empty path. | `setListPlaceholder(name, text)` | Centered empty-state hint over the list while it has no items; hidden once items appear. | | `setListItemsDisabled(name, vector)` | Grey out (disable + make unselectable) the items whose text is in the set, keeping them visible; matched by item text. An empty/absent set re-enables every item. | | `setListSelectionMode(name, bool multi)` | Switch a list between single (`false`, default) and multi (`true`, extended) selection. | +| `setListItemColors(name, vector colors)` | Foreground color per item, parallel to the items vector (CSS color string, empty = default palette color). | ### QTableWidget @@ -195,6 +198,10 @@ Selected result requires at least one non-empty path. | `setTableSortIndicator(name, column, ascending)` | Draw the header arrow **without** enabling Qt's sorting — for tables that sort themselves via `onHeaderClicked` (Qt paints an arrow only for its own sorting). Cosmetic; never reorders rows. | | `setSelectedRows(name, vector)` | Set selected row indices | | `setDisabledRows(name, vector)` | Grey out rows (non-selectable) | +| `setVisibleRows(name, vector)` | Live filtering: hide every row whose index is not in the set (empty hides all). To show all rows again pass the full index list. | +| `clearVisibleRows(name)` | Unset a previous `setVisibleRows`; the host then makes **no** visibility change, so it does not re-show hidden rows by itself. | +| `setRowColor(name, row, color_hex)` | Tint one row's background (`"#rrggbb"`; empty clears the override) | +| `setCellTooltip(name, row, col, tooltip)` | Set the tooltip of one cell | | `setTableRadioColumn(name, column, checked_row)` | Render `column` as an exclusive radio group; `checked_row` is selected (-1 = none). Fires `onTableRadioSelected`. | | `appendTableRows(name, seq, rows)` | Delta: append rows without resending `rows` — string or `TableItem` rows (typed rows keep their sort keys via a sparse `append_values` map). See guide → "Table deltas". | | `updateTableCells(name, seq, vector)` | Delta: rewrite individual cells (`{row, col, TableItem}`, plugin row space). Replaces the whole cell — a keyless item clears the sort key. | @@ -276,6 +283,8 @@ table delta protocol's fresh sequence gate and all-or-nothing application rules. | `setChartMarkers(name, vector)` | Overlay markers (events/regions/value-bands) on top of the series | | `clearChart(name)` | Remove chart series and markers | | `setChartZoomEnabled(name, bool)` | Enable chart zoom/pan events | +| `setChartAutoZoom(name, bool)` | Zoom to extents on every series update (`true`) or keep the user's zoom (`false`) | +| `setChartPlaceholder(name, text)` | Overlay hint shown while the chart has no series; hidden once data arrives | ### QFrame Embedded Scene View (since 0.38.0) @@ -356,6 +365,23 @@ Editable multi-marker strip: any number of resizable Region spans + single-point `TimelineMark{int id; bool region; int start; int end;}` — `region=false` is a point Event (`end` ignored). +### RangeSlider (custom widget) + +Two-handle integer range slider. + +| Method | Description | +|--------|-------------| +| `setRangeSliderBounds(name, min, max)` | Integer `[min, max]` bounds. Resets the handle values, so set the values afterwards | +| `setRangeSliderValues(name, lower, upper)` | Handle positions, in slider units | +| `setRangeSliderTimeSpan(name, min_ns, max_ns)` | Map the slider span onto an absolute time window for the handle and duration labels (`0,0` disables) | +| `setRangeSliderMarkers(name, vector)` | Boundary segments (`start`, `end`, `label`) drawn on the slider; empty clears | + +### DateRangePicker (custom widget) + +| Method | Description | +|--------|-------------| +| `setDateRangePlaceholder(name, earliest_iso, latest_iso)` | Soft placeholder hints for the empty from/to fields (ISO-8601 dates; empty clears that side). Not a range constraint | + ### Generic (any widget) | Method | Description | @@ -371,6 +397,9 @@ Editable multi-marker strip: any number of resizable Region spans + single-point |--------|-------------| | `requestAccept()` | Close dialog with OK (one-shot) | | `requestSubDialog(ui_xml)` | Open nested modal sub-dialog | +| `requestSubPanel(ui_xml)` | Open one live, non-blocking sub-panel: its widget events reach the normal handlers and the plugin keeps pushing `WidgetData` to it. The host sends a synthetic `onClicked("subPanelClosed")` when the user dismisses it. Re-requesting while one is open is ignored | +| `closeSubPanel()` | Close the sub-panel opened by `requestSubPanel`; no-op if none is open | +| `requestClose(reason)` | Panel hosts only (ignored by dialogs): close the panel hosting the plugin; `reason` is a plugin-defined string forwarded to the host | --- @@ -392,12 +421,17 @@ Override these in your `DialogPluginTyped` subclass. Return `true` when state ch | `onSelectionChanged(name, items)` | QListWidget, QTableWidget | Vector of selected item texts (table: column-0 text) | | `onItemDoubleClicked(name, index)` | QListWidget, QTableWidget | Row index of double-clicked item | | `onItemDeleteRequested(name, index)` | QListWidget (deletable) | Row whose trash button was clicked (see `setListItemsDeletable`) | +| `onItemContextAction(name, index, action_id)` | QListWidget | Row and id of the context-menu action that fired; the action set is the `pj_context_actions` `.ui` property | +| `onHeaderClicked(name, section)` | QTableWidget | Column whose header was clicked, for tables that sort themselves | | `onTableRadioSelected(name, row)` | QTableWidget radio column | Row whose radio was clicked (see `setTableRadioColumn`) | | `onCodeChanged(name, code)` | QPlainTextEdit code editor | Edited code | | `onCodeChangedWithCursor(name, code, cursor)` | QPlainTextEdit code editor | Edited code + caret offset (`cursor < 0` when no opt-in / not reported); defaults to `onCodeChanged` | | `onItemsDropped(name, items)` | Any widget with `setDropTarget` | Dropped item labels | +| `onDroppedItems(name, items)` | Any widget with `setDropTarget` | `vector`; override instead of `onItemsDropped` to tell same-named series of different datasets apart (`data_source_id` 0 = unknown). Defaults to `onItemsDropped` | | `onChartViewChanged(name, x_min, x_max, y_min, y_max)` | QFrame chart container | Visible chart range | | `onMarkerTimelineChanged(name, marks)` | MarkerTimeline | Full `std::vector` set after a drag/resize/delete | +| `onRangeChanged(name, lower, upper)` | RangeSlider | New handle positions | +| `onDateRangeChanged(name, from_iso, to_iso)` | DateRangePicker | ISO-8601 range bounds (empty = unbounded on that side) | | `onTabChanged(name, index)` | QTabWidget | New tab index | | `onStackedPageChanged(name, index, page_object_name)` | QStackedWidget | New page index and stable Qt `objectName`. Since 0.21.0. | | `onTreeSelectionChanged(name, ids)` | QTreeWidget | Complete logical selected-ID set, including filtered-out selections | @@ -471,23 +505,11 @@ Add a placeholder widget named `pj_parser_slot` in your `.ui`: ``` -### Host Configuration - -Configure the host dialog runtime with a parser dialog provider: - -```cpp -HostDialogRuntimeConfig config; -config.parser_dialog_provider = [&](const std::string& encoding) -> const PJ_dialog_vtable_t* { - return registry.queryParserDialog(encoding); -}; -config.initial_parser_config = saved_parser_config; // Optional - -auto result = runHostDialog(dialog_handle, config); -``` +### Host Behavior -### Behavior +The injection is host behaviour, not SDK code: -1. When the user selects an encoding in `comboBoxProtocol`, the host dialog runtime looks up the parser's dialog vtable +1. When the user selects an encoding in `comboBoxProtocol`, the host looks up the parser's dialog vtable 2. If found, the parser's UI is loaded and injected into `pj_parser_slot` 3. The parser dialog's events and `widget_data()` are handled independently 4. On accept, both configs are returned to the host: the source config and the parser config diff --git a/V4_STORE.md b/docs/object_store.md similarity index 88% rename from V4_STORE.md rename to docs/object_store.md index d266e9c5..e64e419b 100644 --- a/V4_STORE.md +++ b/docs/object_store.md @@ -1,7 +1,7 @@ # ObjectStore Plugin ABI -`pj_datastore::ObjectStore` is the timestamped opaque-payload store that -complements `DataEngine`. `DataEngine` stores typed scalar series; `ObjectStore` +`pj_datastore::ObjectStore` (a type of the PlotJuggler application, not of this +SDK) is the timestamped opaque-payload store that complements `DataEngine`. `DataEngine` stores typed scalar series; `ObjectStore` stores raw bytes for object topics such as structured messages, images, point clouds, annotations, and other domain-specific payloads. @@ -16,7 +16,7 @@ entry point. | `pj.source_object_write.v1` | `PJ_object_write_host_t` | `SourceObjectWriteHostView` | DataSource plugins that create object topics | | `pj.parser_object_write.v1` | `PJ_parser_object_write_host_t` | `ParserObjectWriteHostView` | MessageParser plugins bound to an object topic | | `pj.toolbox_object_read.v1` | `PJ_object_read_host_t` | `ToolboxObjectReadHostView` | Toolbox plugins that read object topics | -| `pj.toolbox.v1` | `PJ_toolbox_host_t` | `ToolboxHostView` | Toolbox plugins that create or annotate object topics | +| `pj.toolbox_write.v1` | `PJ_toolbox_host_t` | `ToolboxHostView` | Toolbox plugins that create or annotate object topics | The service traits live in `pj_base/include/pj_base/sdk/service_traits.hpp`. The raw ABI structs live in `pj_base/include/pj_base/plugin_data_api.h`. @@ -61,7 +61,7 @@ The host-side implementation is `DatastoreParserObjectWriteHost`. ## Toolbox Object Writes -Use `ToolboxHostView` from `pj.toolbox.v1` to publish objects from a toolbox: +Use `ToolboxHostView` from `pj.toolbox_write.v1` to publish objects from a toolbox: - `registerObjectTopic(source, name, metadata_json)` registers under a source created by the toolbox. @@ -74,8 +74,11 @@ Use `ToolboxHostView` from `pj.toolbox.v1` to publish objects from a toolbox: use `1` when republishing a whole marker set. These optional tail slots return an error on older hosts. Reuse them for -[plot markers](docs/plot_markers_architecture.md); no marker-specific store or -per-marker mutation API is needed. +[plot markers](plot_markers_architecture.md); no marker-specific store or +per-marker mutation API is needed. The other path that writes `PlotMarkers` +objects is a `kind="markers"` data processor (`pj.data_processors.v1`, +`DataProcessorsHostView::createMarkers`): the host runs the script and publishes +the set itself. ## Toolbox Object Reads diff --git a/docs/plot_markers_architecture.md b/docs/plot_markers_architecture.md index 2d9db2e4..2d154c33 100644 --- a/docs/plot_markers_architecture.md +++ b/docs/plot_markers_architecture.md @@ -18,7 +18,7 @@ The **producer owns the set and republishes the whole blob** on each change per-id deletes, payload IDs and read-modify-write races. ``` - Producer (toolbox plugin / future Lua / in-process host) + Producer (plugin / host data processor) │ build PlotMarkers set → serialize (PlotMarkers codec) │ register object topic "__markers__/" on the dataset │ push the whole serialized set (republish) @@ -61,17 +61,17 @@ Design notes: - **`ColorRGBA`** is already shared by including `image_annotations.hpp`; do not create another color type. - `BuiltinObjectType::kPlotMarkers`, `PJ_builtin_object_type_t`'s matching tag - and `plot_markers_codec` already ship. Future compatible type additions are - append-only MINOR changes. Refresh `pj_base/abi/baseline.abi` only for an - intentional MAJOR break; see [release versioning](../CLAUDE.md#release-versioning). + and `plot_markers_codec` ship. New builtin types are append-only MINOR changes, and + each new `PJ_builtin_object_type_t` enumerator must be classified in + `pj_base/feature_floors.json` (a plugin emitting it needs that floor). Refresh + `pj_base/abi/baseline.abi` only for an intentional MAJOR break; see + [Versioning and releasing](RELEASING.md#abi-and-api-changes). ## 3. Why ObjectStore + republish (not a dedicated store, not DataEngine) -Markers were originally planned in a dedicated mutable store keyed on "delete one -specific marker". With the in-process `pj_scripting` Lua engine becoming the primary -producer, the model flipped to **producer-owns-the-set + republish**, which removes the -need for store-side per-marker mutation — and that makes the existing **ObjectStore** -the right home (markers become just another builtin object). +The model is **producer-owns-the-set + republish**, which removes the need for +store-side per-marker mutation — and that makes the existing **ObjectStore** the right +home (markers are just another builtin object). - **`DataEngine` (columnar) — no.** Strictly numeric columns (`PrimitiveType`), append-only immutable chunks. A marker is a structured record over a span; it fits @@ -103,25 +103,36 @@ push the whole serialized PlotMarkers set (republish, last-writer win - **Producers creating their own dataset** use `registerObjectTopic(source, …)`. **Producers annotating an existing dataset** (a compiled toolbox over loaded data) use `registerObjectTopicOnDataset(DatasetId, …)` — a tail-appended, **idempotent** - toolbox-host slot that re-resolves the topic on each republish. In-process producers - (host / future Lua) can equally call `ObjectStore` directly. + toolbox-host slot that re-resolves the topic on each republish. - **Per-series read is direct** (one object topic per series). Cross-series aggregation ("all `warning+` markers in the dataset") enumerates the dataset's marker topics (a known limitation, acceptable at marker scale). ## 5. Producers -All produce the **same** `PlotMarkers` set and publish it the same way; they differ -only in where the markers come from: -- **In-process host (today).** The Help → Add Demo Markers action builds a set and - publishes it via `ObjectStore` directly — the simplest exercise of the path. -- **Toolbox plugin (today).** Analysis over already-loaded series → republish its set - via `registerObjectTopicOnDataset` + `pushOwnedObject`. -- **Lua scripting / AI agent (future, `pj_scripting`).** The primary producer: an - in-process script regenerates and republishes its set; the agent most naturally emits - Lua. No new ABI — it rides the same path. -- **Ingestion parser (future).** Markers already present in a recording decode into - `PlotMarkers` — the same role an image parser plays for `ImageAnnotations`. +Every producer publishes the **same** `PlotMarkers` set; they differ in where the markers +come from: +- **Plugins (object-topic writes).** A toolbox or data source builds its set and republishes + it via `registerObjectTopicOnDataset` / `registerObjectTopic` + `pushOwnedObject`, as in §4. +- **Derived markers (`pj.data_processors.v1`).** A plugin, script or agent submits a + `kind="markers"` data processor (`DataProcessorsHostView::createMarkers` in + `pj_base/sdk/plugin_data_api.hpp`): the host runs the script, owns the lifecycle and + publishes the serialized set under `markerObjectTopicName(key)`. With + `PJ_DATA_PROCESSOR_FLAG_EPHEMERAL` the node is a live preview, never persisted. + +### Marker metadata and preview topics + +A marker set is recognized and routed by its object-topic **metadata**, never by name +(`pj_base/sdk/object_topic_metadata.hpp`): `kDerivedMetadataKey` = `kDerivedMarkersValue` +marks a host-published set, `kMarkerScopeMetadataKey` (`kMarkerScopeSeries` / +`kMarkerScopeDataset` / `kMarkerScopeAll`) says which plots draw it, +`kMarkerTargetMetadataKey` names the series a series-scoped set annotates, and +`kMarkerOwnerMetadataKey` names the generator that publishes it. + +`plot_markers.hpp` also defines the helpers producer and overlay share: `markerSeriesKey` +(per-series marker topic from a curve's topic and field names), `kPreviewMarkerTopic` +(marker-topic prefix of an ephemeral preview set) and `isPreviewMarkerTopic` (true for the +bare or object-topic form). ## 6. Rendering (PJ4 host) @@ -134,15 +145,11 @@ only in where the markers come from: marker draws on every plot of its dataset. - **Repaint** on `SessionManager::markersChanged` (a store-agnostic signal a producer fires after republishing) and on time/zoom changes. -- A **markers panel** in `pj_app` (filterable list, click-row → seek) is optional/future. - This is the **`pj_plotting` Qwt path, not the scene2D image path** — markers annotate time-series plots, not image frames. -## 7. Open / deferred +## 7. Limits -- **Headless CLI.** PJ4 is GUI-only today (`pj_app/src/main.cpp` always opens a - window). The data model is headless-ready (Qt-free `pj_runtime`/store), but a true - no-GUI runner is a separate effort; v1 exposes JSON export as a host action. - **Cross-series aggregate query** enumerates topics (per-series read is direct). - **Concurrent writers** to the same topic: the republish model assumes a single owner per marker topic; two live writers to one topic would clobber (last-writer wins). diff --git a/docs/plot_markers_use_cases.md b/docs/plot_markers_use_cases.md index f826cd22..696aba17 100644 --- a/docs/plot_markers_use_cases.md +++ b/docs/plot_markers_use_cases.md @@ -13,10 +13,10 @@ ## 1. Motivation -Plugins — and a future **AI agent** — need a way to put **graphical markers** on -plots (a shaded time region, a point at an event, a value band) and to **ask which -markers exist** on a given series. Use the existing `PlotMarkers` type and codec -through [ObjectStore services](../V4_STORE.md). +Plugins and **AI agents** (through the host's `kind="markers"` data processors) +put **graphical markers** on plots (a shaded time region, a point at an event, a +value band) and **ask which markers exist** on a given series. Use the existing `PlotMarkers` type and codec +through [ObjectStore services](object_store.md). **A marker is a structured finding.** It carries a pass/fail status, severity and category alongside its time anchor. Reports and consumer-side filtering @@ -29,8 +29,8 @@ A marker has three consumers of one data model: Producers edit an in-memory set and republish the whole `PlotMarkers` value; there is no per-marker add/delete API. Toolbox plugins and the host can read -sets through the generic object pipeline. Future agents and ingestion parsers -can use the same type. +sets through the generic object pipeline. Agents and ingestion parsers +use the same type. ## 2. Vocabulary @@ -111,9 +111,7 @@ No payload field sets its scope. `severity ≥ warning` is done on the deserialized set. - **UC-6 — JSON report.** A run exports a report — `overall_status`, plus every - finding with its timestamps and severity. The data model supports this with no GUI; - a true headless CLI entry point is a separate, deferred effort (PJ4 is GUI-only - today), so v1 exposes export as a host action. + finding with its timestamps and severity. The data model supports this with no GUI. - **UC-7 — GUI inspection.** A user sees markers colored by severity, hovers for a tooltip with the semantic fields, and uses a **markers panel** to filter (by diff --git a/docs/sdk-utilities.md b/docs/sdk-utilities.md index 7f6e5922..a6fced59 100644 --- a/docs/sdk-utilities.md +++ b/docs/sdk-utilities.md @@ -36,6 +36,11 @@ from `pj_plugins/dialog_protocol/include/`; Widget headers from | `PJ::sdk::testing::DelegatedIngestFixture` | Delegated-ingest ownership, completion, Stop and discard tests; use plugin_sdk | `pj_plugins/testing/delegated_ingest_fixture.hpp` | | `PJ::sdk::testing::FakeWidgetHost` | In-memory `pj.widget_host.v1` host for Widget plugin tests: per-wire samples at nanosecond store times, inclusive display-time history, caps, decimation, undeliverable samples, `on_update` values at a tracker | `pj_plugins/sdk/testing/fake_widget_host.hpp` | | `PJ::sdk::testing::FakeMcpEndpointHost` | Scriptable `pj.mcp_endpoint.v1` host (held state, token rotation, listener failure, call counts) for plugin tests | `pj_base/sdk/testing/fake_mcp_endpoint_host.hpp` | +| `PJ::testing::ToolboxTestStore` | In-memory toolbox host + runtime host for toolbox unit tests: hand-populated topics/fields served through the Arrow read path, captures `append_record` writes, counts `create_data_source` / `notify_data_changed` | `pj_plugins/testing/toolbox_test_store.hpp` | +| `PJ::sdk::testing::ParserWriteRecorder` | Capture every parser `append_record` / `append_bound_record` as `RecordedRow`s for parser unit tests | `pj_base/sdk/testing/parser_write_recorder.hpp` | +| `PJ::sdk::testing::ParserRuntimeRecorder` | In-memory parser diagnostic sink; strings are copied, so records outlive the parse call | `pj_base/sdk/testing/parser_runtime_recorder.hpp` | +| `startWithGateProbe` | `ProviderJob::start()` with a hook that runs while the worker is parked, to test gate-release races | `pj_base/sdk/testing/provider_job_probe.hpp` | +| `cleanupWithProbe` | `RequestArtifactCache::cleanup()` with a hook between directory scan and eviction, to test concurrent cleanup | `pj_base/sdk/testing/request_cache_probe.hpp` | | `pj_add_sdk_test_fixture(name out_target)` | Build packaged mock plugins; no second SDK source download | `cmake/PjSdkTestFixtures.cmake` | ## Plugin helpers @@ -63,6 +68,10 @@ CMake helpers become available through `find_package(plotjuggler_sdk)`. | `PJ::sdk::ObjectTopicMetadataBuilder` | Build canonical renderer metadata and supplemental media hints | `pj_base/sdk/object_topic_metadata.hpp` | | `PJ::sdk::ObjectBytes` | Own object-read bytes through move-only RAII | `pj_base/sdk/object_bytes.hpp` | | `PJ::sdk::ArrowSchemaHolder, ArrowArrayHolder, ArrowStreamHolder` | Own Arrow out-params; successful writes transfer ownership, failed writes retain it | `pj_base/sdk/arrow.hpp` | +| `PJ::sdk::TimestampPolicy, kCanonicalPolicy, detectTimestampColumn, timestampUnitFromJson, timestampUnitToJson` | Pick the time-axis column of a tabular source (native-type pass, then name pass) and persist its `timestamp_column` / `timestamp_unit` config | `pj_plugins/sdk/timestamp_policy.hpp` | +| `PJ::sdk::ObjectIngestPolicyResolver` | Per-topic > source > type > default policy for deferred object fetch (memory vs re-read cost); host side | `pj_plugins/sdk/object_ingest_policy.hpp` | +| `PJ::sdk::DatasetQualifierSplit, splitDatasetQualifier, qualifiedSeriesName` | Split or compose `dataset_source:topic/field` against the known dataset names; not a persistent identity | `pj_base/sdk/dataset_qualified_name.hpp` | +| `PJ::sdk::MaterializedSeriesView` | Typed view over a series read through `ToolboxHostView::readSeries`: timestamps, values, and `nullCount()` / `isNull()` for the validity bitmap (the `valuesAs*()` pointers do not read it) | `pj_base/sdk/plugin_data_api.hpp` | | `PJ::sdk::parseEvaluationReport` | Parse a data-processor poll report (COMPLETED or FAILED) and derive each bundle's stamp | `pj_plugins/sdk/evaluation_report.hpp` | | `PJ::sdk::storeCatalogSnapshotV2` | Host side: write a catalog snapshot v2 under the caller-size rule, releasing it on refusal | `pj_base/sdk/plugin_data_api.hpp` | | `PJ::sdk::readDataProcessorRequest, readEvaluationBudget` | Host side: decode a typed data-processor request or budget (read-prefix rule, structural rejections) | `pj_base/sdk/data_processors_host.hpp` | @@ -72,6 +81,7 @@ CMake helpers become available through `find_package(plotjuggler_sdk)`. | `PJ::WidgetLibrary, WidgetHandle, WidgetHostBridge` | Host side of the Widget family: load, instantiate per type, serve `pj.widget_host.v1` from callbacks | `pj_plugins/host/widget_library.hpp; widget_handle.hpp; widget_host_bridge.hpp` | | `PJ_WIDGET_QT_MODULES` | Canonical Qt module list a widget may declare in `qt_modules` | `pj_plugins/widget_qt_modules.h` | | `pj_embed_file, pj_configure_plugin` | Embed UI/manifest assets and configure plugin exports/build rules | `cmake/PjPlugin.cmake` | +| `pj_harden_plugin_exports` | Restrict a plugin DSO's exports to the ABI entry points (ELF version script with a post-build gate; Mach-O export list). `pj_configure_plugin` applies it unless `NO_EXPORT_HARDENING` | `cmake/PjPlugin.cmake` | Outside this SDK, reuse pj-official-plugins common/: diff --git a/experimental/parser_module/README.md b/experimental/parser_module/README.md index e2e58985..48e0ff43 100644 --- a/experimental/parser_module/README.md +++ b/experimental/parser_module/README.md @@ -115,13 +115,14 @@ ownership, folder scanning, and rescan policy remain application concerns. Module authors use the standalone C++17 headers under `include/pj_base/parser_module/` through the zero-linkage -`plotjuggler_sdk::parser_module` target. The readers compile ROS 2 concatenated +`plotjuggler_sdk::parser_module` target, an in-tree ALIAS only (nothing here is +installed or packaged). The readers compile ROS 2 concatenated `.msg` bundles or protobuf `FileDescriptorSet` field paths at bind time; the CDR locator supports fixed arrays, bounded and unbounded sequences, bounded strings, and string arrays/sequences, while cyclic or over-depth schemas fail at bind. The typed `ObjectWriter` emits Image, PointCloud, DepthImage, OccupancyGrid, CompressedPointCloud, Mesh3D, VideoFrame, OccupancyGridUpdate, -and VoxelGrid descriptors, including each type's eligible single-splice form. +VoxelGrid and GridMap descriptors, including each type's eligible single-splice form. Module parse callbacks receive the per-message `pj::Timestamp` alongside the payload. `BindingInfo` views expire when `bind()` returns; `owningCopy()` is the fallible retention path. Bulk storage uses nothrow allocation, and protobuf diff --git a/pj_base/CLAUDE.md b/pj_base/CLAUDE.md index cc6e0d54..52c0d363 100644 --- a/pj_base/CLAUDE.md +++ b/pj_base/CLAUDE.md @@ -3,41 +3,52 @@ Before adding a helper, use [Existing SDK utilities](../docs/sdk-utilities.md). Provider work also follows [the provider contract](../docs/provider-guide.md). -`pj_base` owns vocabulary, builtin schemas/codecs, DataSource/MessageParser/Toolbox -C protocols and the DataSource/Toolbox C++ bases. The Dialog protocol and -MessageParser/Dialog bases live in `pj_plugins`. - -`pj_source` headers live here under `sdk/source/`. Link -`plotjuggler_sdk::source` to use them. +`pj_base` owns the vocabulary, the builtin object schemas and codecs, the +DataSource/MessageParser/Toolbox C protocols and the DataSource/Toolbox C++ +bases. The `pj_source` provider library (`plotjuggler_sdk::source`) is built +from this directory too. The Dialog protocol and the MessageParser/Dialog/Widget +bases live in `pj_plugins`. `pj_base` has zero public library dependencies. fast_float and fmt are private build details. It must not depend on Qt, `pj_datastore` or `pj_plugins`. ## Layout -- `include/pj_base/` — vocabulary primitives: `types.hpp`, `time.hpp` (absolute time spine: `Timepoint`/`Duration` + `fromRaw`/`toRaw`), `type_tree.hpp`, `dataset.hpp`, `expected.hpp`, `span.hpp`, `number_parse.hpp`, `assert.hpp`, `diagnostic_sink.hpp`, `buffer_anchor.hpp`. +- `include/pj_base/` — vocabulary primitives: `types.hpp`, `time.hpp` (absolute + time spine: `Timepoint`/`Duration` + `fromRaw`/`toRaw`), `time_math.hpp`, + `time_format.hpp`, `slider_window.hpp`, `type_tree.hpp`, `dataset.hpp`, + `expected.hpp`, `span.hpp`, `number_parse.hpp`, `assert.hpp`, + `diagnostic_sink.hpp`, `buffer_anchor.hpp`. +- `include/pj_base/*_protocol.h`, `plugin_data_api.h`, `builtin_object_abi.h`, + `plugin_abi_export.hpp` — the stable C-ABI surface (plugin families, host + services, descriptor import, parser route claims and functional extension). - `include/pj_base/builtin/` — one struct header (`*.hpp`) and one wire codec - (`*_codec.hpp`) per builtin type. Numeric tags are stable; values 2 and 12 are - permanently reserved. Also contains the tagged, type-erased `BuiltinObject` - holder and type-erased codec dispatcher, and the compile-time field tables + (`*_codec.hpp`) per builtin type, the tagged type-erased `BuiltinObject` + holder and codec dispatcher, and the compile-time field tables (`field_table.hpp`, one `_fields.hpp` per tabled struct). -- `include/pj_base/sdk/` — C++ SDK over the ABI: DataSource + Toolbox `*_plugin_base.hpp`, `service_registry.hpp`/`service_traits.hpp`, host views, Arrow RAII holders, `testing/`. -- `include/pj_base/*_protocol.h`, `plugin_data_api.h`, `builtin_object_abi.h`, `plugin_abi_export.hpp` — the stable C-ABI surface for DataSource/MessageParser/Toolbox (the Dialog protocol header lives in `pj_plugins/dialog_protocol/`). +- `include/pj_base/sdk/` — C++ SDK over the ABI: DataSource + Toolbox + `*_plugin_base.hpp`, `service_registry.hpp`/`service_traits.hpp`, host views, + Arrow RAII holders, `semver.hpp`, `testing/`. `sdk/version.hpp` is generated + from `cmake/version.hpp.in`. +- `include/pj_base/sdk/source/` — `pj_source` provider helpers; see + `../docs/provider-guide.md`. - `proto/pj/` — canonical `.proto` wire contracts for the builtin types (see its README). -- `include/pj_base/sdk/source/` — `pj_source` provider helpers; see `../docs/provider-guide.md`. -- `include/pj_base/time_format.hpp`, `slider_window.hpp` — UTC/duration formatting, checked ISO parsing and slider ranges. -- `src/`, `tests/` — codec/parse impls and gtests. +- `feature_floors.json` — the release that introduced each host-contract + surface; see [Versioning and releasing](../docs/RELEASING.md#host-contract-floors). - `abi/baseline.abi` — golden libabigail dump; the ABI-stability regression baseline. +- `src/`, `tests/` — implementations and gtests. ## Gotchas - ABI numbering is **frozen**. `BuiltinObjectType` (builtin_object.hpp) and `PJ_builtin_object_type_t` (builtin_object_abi.h) share stable numeric values. - Never renumber. Types 2 and 12 are permanently reserved. Append only. + Never renumber. Types 2 and 12 are permanently reserved. Append only, and + classify each new type in `feature_floors.json`. - Every vtable slot is `PJ_NOEXCEPT`. A throw across the ABI boundary calls `std::terminate`. See the header block in `plugin_data_api.h`. - `BuiltinObject` stores its `BuiltinObjectType` tag next to the opaque value. It deliberately avoids `std::variant` for forward compatibility and uses no RTTI. Recover via `obj.get()` / `sdk::typeOf`. See `builtin/builtin_object.hpp`. -- Follow [Release Versioning](../CLAUDE.md#release-versioning) for API/ABI changes; a MINOR does not refresh `abi/baseline.abi`. +- A builtin struct member change is a MAJOR that `abidiff` does not see; follow + [Versioning](../CLAUDE.md#versioning) for any API/ABI change. ## Read deeper | For | Read | @@ -45,7 +56,8 @@ build details. It must not depend on Qt, `pj_datastore` or `pj_plugins`. | Existing numeric/time/source helpers and testing support | [SDK utilities](../docs/sdk-utilities.md) | | Building a source provider | [Provider contract](../docs/provider-guide.md) | | Builtin type design, serialization families, type-erasure rules | `../docs/builtin_type.md` | -| ImageAnnotations canonical wire format | `../docs/image_annotations_format.md` | +| ImageAnnotations / PlotMarkers wire formats | `../docs/image_annotations_format.md`, `../docs/plot_markers_format.md` | +| ObjectStore services | `../docs/object_store.md` | | C++ style / error-handling (`Expected`, `PJ_ASSERT`) | `../docs/cpp_design_recommendations.md` | | Per-family ABI driving order + thread tags | `include/pj_base/data_source_protocol.h`, `message_parser_protocol.h`, `toolbox_protocol.h` | | Writing a DataSource plugin | `include/pj_base/sdk/data_source_patterns.hpp` (start here) | diff --git a/pj_base/include/pj_base/builtin/builtin_object.hpp b/pj_base/include/pj_base/builtin/builtin_object.hpp index e233597b..5825cbc8 100644 --- a/pj_base/include/pj_base/builtin/builtin_object.hpp +++ b/pj_base/include/pj_base/builtin/builtin_object.hpp @@ -72,7 +72,7 @@ enum class BuiltinObjectType : uint16_t { kRobotDescription = 13, ///< sdk::RobotDescription — raw URDF/SDF/MJCF text + format hint. kCameraInfo = 14, ///< sdk::CameraInfo — pinhole camera calibration (K/D/R/P). kOccupancyGridUpdate = 15, ///< sdk::OccupancyGridUpdate — incremental sub-rectangle patch for an OccupancyGrid. - kLog = 16, ///< sdk::Log — textual log message (level + text + name). + kLog = 16, ///< sdk::Log — textual log message (level + text + source). kPosesInFrame = 17, ///< sdk::PosesInFrame — array of poses in one reference frame. kVoxelGrid = 18, ///< sdk::VoxelGrid — dense 3D voxel grid (occupancy/cost/ESDF/semantic). kPlotMarkers = 19, ///< sdk::PlotMarkers — findings on a time-series plot (regions, events, bands, labels). diff --git a/pj_base/include/pj_base/builtin/frame_transforms_fields.hpp b/pj_base/include/pj_base/builtin/frame_transforms_fields.hpp index f9dbf48e..d849d0e2 100644 --- a/pj_base/include/pj_base/builtin/frame_transforms_fields.hpp +++ b/pj_base/include/pj_base/builtin/frame_transforms_fields.hpp @@ -58,7 +58,7 @@ struct FieldTable { static constexpr FieldTableView view{"Pose", Span(fields)}; }; -/// Fields of `FrameTransform`: `timestamp` (int64), `parent_frame_id`, +/// Fields of `FrameTransform`: `timestamp_ns` (int64), `parent_frame_id`, /// `child_frame_id` (strings), `translation` (Vector3), `rotation` (Quaternion). template <> struct FieldTable { diff --git a/pj_base/include/pj_base/builtin/image_annotations_fields.hpp b/pj_base/include/pj_base/builtin/image_annotations_fields.hpp index cddbf7bf..9b75b056 100644 --- a/pj_base/include/pj_base/builtin/image_annotations_fields.hpp +++ b/pj_base/include/pj_base/builtin/image_annotations_fields.hpp @@ -73,7 +73,7 @@ struct FieldTable { static constexpr FieldTableView view{"TextAnnotation", Span(fields)}; }; -/// Fields of `ImageAnnotations`: `timestamp` (int64), `image_topic` +/// Fields of `ImageAnnotations`: `timestamp_ns` (int64), `image_topic` /// (string), `points`, `circles`, `texts` (lists of the primitive structs above). template <> struct FieldTable { diff --git a/pj_base/include/pj_base/builtin/scene_entities_fields.hpp b/pj_base/include/pj_base/builtin/scene_entities_fields.hpp index f875ee7e..10d9766b 100644 --- a/pj_base/include/pj_base/builtin/scene_entities_fields.hpp +++ b/pj_base/include/pj_base/builtin/scene_entities_fields.hpp @@ -160,7 +160,7 @@ struct FieldTable { static constexpr FieldTableView view{"ModelPrimitive", Span(fields)}; }; -/// Fields of `SceneEntity`: `timestamp` (int64), `frame_id`, `id`, +/// Fields of `SceneEntity`: `timestamp_ns` (int64), `frame_id`, `id`, /// `lifetime_ns` (int64), `frame_locked`, `metadata` (list of /// `KeyValuePair`), and the nine primitive lists in `SceneEntity`'s /// declaration order (`arrows`, `cubes`, `spheres`, `cylinders`, `lines`, @@ -187,7 +187,7 @@ struct FieldTable { static constexpr FieldTableView view{"SceneEntity", Span(fields)}; }; -/// Fields of `SceneEntityDeletion`: `type` (enum), `timestamp` (int64), `id`. +/// Fields of `SceneEntityDeletion`: `type` (enum), `timestamp_ns` (int64), `id`. template <> struct FieldTable { static constexpr std::array fields{ diff --git a/pj_base/proto/pj/README.md b/pj_base/proto/pj/README.md index 0ad05839..a8db153d 100644 --- a/pj_base/proto/pj/README.md +++ b/pj_base/proto/pj/README.md @@ -27,6 +27,11 @@ See `docs/builtin_type.md` for the design rationale. - **`CameraInfo.proto`** — pinhole camera calibration (intrinsics, distortion, rectification, projection) so consumers can draw frustums, back-project depth, and rectify; correlated to an image topic by name. - `CameraInfo` +### Log messages + +- **`Log.proto`** — a single textual log message (severity level + text + source + optional category) for a log/console panel; mirrors the core of Foxglove's `Log` (file/line omitted). + - `Log` + ### Byte-backed raster builtins - **`Image.proto`** — single 2D image, raw (`rgb8`, `mono16`, …) or compressed (`jpeg`, `png`, `qoi`) unified under a single `encoding` string. @@ -37,8 +42,6 @@ See `docs/builtin_type.md` for the design rationale. - `OccupancyGrid` - **`OccupancyGridUpdate.proto`** — incremental sub-rectangle patch for a previously-published `OccupancyGrid`; placed by the consumer against the base grid (no own origin/resolution). - `OccupancyGridUpdate` -- **`Log.proto`** — a single textual log message (severity level + text + source + optional category) for a log/console panel; mirrors the core of Foxglove's `Log` (file/line omitted). - - `Log` - **`VideoFrame.proto`** — one frame of an inter-frame-coded video stream (`h264`, `h265`, `vp9`, `av1`) when per-frame `Image` messages would be wasteful. Field layout is wire-identical to `foxglove.CompressedVideo` (timestamp=1, frame_id=2, data=3, format=4), so one decoder parses both. - `VideoFrame` @@ -52,8 +55,9 @@ See `docs/builtin_type.md` for the design rationale. ### 3D scene - **`SceneEntities.proto`** — the workhorse for marker-style 3D visualization; batches procedural primitives sharing a frame and timestamp. - - Primitives: `ArrowPrimitive`, `CubePrimitive`, `SpherePrimitive`, `CylinderPrimitive`, `LinePrimitive`, `TrianglePrimitive`, `TextPrimitive`, `AxesPrimitive` + - Primitives: `ArrowPrimitive`, `CubePrimitive`, `SpherePrimitive`, `CylinderPrimitive`, `LinePrimitive`, `TrianglePrimitive`, `TextPrimitive`, `AxesPrimitive`, `ModelPrimitive` - Entity: `SceneEntity` + - Removal command: `SceneEntityDeletion` - Batch: `SceneEntities` - **`Mesh3D.proto`** — 3D mesh asset delivered in its native binary format (GLTF/GLB/STL/PLY/OBJ/USD/DAE) for URDF-style or scene-mesh visualization. - `Mesh3D` diff --git a/pj_plugins/CLAUDE.md b/pj_plugins/CLAUDE.md index f8ff3f09..265be200 100644 --- a/pj_plugins/CLAUDE.md +++ b/pj_plugins/CLAUDE.md @@ -3,13 +3,14 @@ Before adding a helper, use [Existing SDK utilities](../docs/sdk-utilities.md). Provider work also follows [the provider contract](../docs/provider-guide.md). -`pj_plugins` owns host loaders, discovery, parser routing and the MessageParser -Dialog and Widget authoring APIs. The `plotjuggler_sdk::plugin_sdk` umbrella combines -those headers with `pj_base`. Compiled providers also link `source`. +`pj_plugins` owns host loaders, discovery, parser routing and the MessageParser, +Dialog and Widget authoring APIs. The `plotjuggler_sdk::plugin_sdk` umbrella +combines those headers with `pj_base`; the loaders are the separate +`plotjuggler_sdk::plugin_host` component. Compiled providers also link `source`. Host libraries depend on `pj_base`, never Qt. Datastore bridges and duplicate -plugin-resolution policy belong to the application. Follow the root reading -order, then read this file or [pj_base/CLAUDE.md](../pj_base/CLAUDE.md) as relevant. +plugin-resolution policy belong to the application (`pj_runtime`, +`PluginRuntimeCatalog`). ## Layout - `include/pj_plugins/host/` — DataSource/MessageParser/Toolbox loaders and RAII @@ -17,8 +18,6 @@ order, then read this file or [pj_base/CLAUDE.md](../pj_base/CLAUDE.md) as relev `inspectPluginDso`). `ParserClaimCatalog` and `ParserRouteResolver` handle parser claim admission and per-route resolution. Also contains `ServiceRegistryBuilder`. - The DSO catalog that resolves duplicates and composes loaded plugin families - is **host policy**. It lives in the app (`pj_runtime`, `PluginRuntimeCatalog`). - `include/pj_plugins/sdk/` — SDK pieces that live here, not in `pj_base`: `MessageParserPluginBase`, ingest/timestamp/array policies, streaming source and dialog helpers, endpoint composition, parser trampolines, and the @@ -46,8 +45,8 @@ order, then read this file or [pj_base/CLAUDE.md](../pj_base/CLAUDE.md) as relev modules (authoring kit, native loader and runtime) are experimental and live outside the installed SDK, under `../experimental/parser_module/`. - **Handles keep the DSO mapped.** Every handle holds a `shared_ptr` - library token, exposed via `libraryOwner()`. Destroying or hot-reloading the - loader therefore cannot `dlclose` a live plugin. A lazy ObjectStore payload + library token (`DataSourceHandle` exposes it as `libraryOwner()`). Destroying + or hot-reloading the loader therefore cannot `dlclose` a live plugin. A lazy ObjectStore payload anchor can capture the token to stay safe after the handle dies; its `release` function is plugin code. Dialog handles provide non-owning `borrowed()` handles for source/toolbox embedded dialogs. These must not outlive the owning handle. @@ -59,7 +58,7 @@ order, then read this file or [pj_base/CLAUDE.md](../pj_base/CLAUDE.md) as relev | Source-provider lifecycle and delegated-ingest tests | [Provider contract](../docs/provider-guide.md) | | Family roles, capabilities, permission matrix, config contract | `docs/REQUIREMENTS.md` | | ABI rules, three-level design, loaders, RAII, data-host bridge | `docs/ARCHITECTURE.md` | -| Writing each family | `docs/data-source-guide.md`, `docs/message-parser-guide.md`, `docs/toolbox-guide.md`, `docs/dialog-plugin-guide.md` | +| Writing each family | `docs/data-source-guide.md`, `docs/message-parser-guide.md`, `docs/toolbox-guide.md`, `docs/dialog-plugin-guide.md`, `docs/widget-guide.md` | | Host loader + factory pattern | `include/pj_plugins/host/data_source_library.hpp`, `…/data_source_handle.hpp` | | Discovery from embedded manifests | `include/pj_plugins/host/plugin_catalog.hpp` (the duplicate-resolution catalog is host-side in `pj_runtime`) | | Parser claim admission and route selection | `include/pj_plugins/host/parser_claim_catalog.hpp`, `parser_route_resolver.hpp` | diff --git a/pj_plugins/dialog_protocol/CLAUDE.md b/pj_plugins/dialog_protocol/CLAUDE.md index 11ef0a41..de771804 100644 --- a/pj_plugins/dialog_protocol/CLAUDE.md +++ b/pj_plugins/dialog_protocol/CLAUDE.md @@ -5,45 +5,30 @@ This self-contained nested module contains the toolkit-neutral Dialog C ABI `WidgetEvent`). Host support includes `host/DialogLibrary`, `host/DialogHandle`, `WidgetDataView` and `WidgetEventBuilder`. -Plugins link only `pj_dialog_sdk`, with no Qt. The GUI host renders the `.ui` -XML and relays events over the vtable. +Plugins link only `pj_dialog_sdk` (inside `plotjuggler_sdk::plugin_sdk`), with +no Qt. The GUI host renders the `.ui` XML and relays events over the vtable. +Author guide: [`../docs/dialog-plugin-guide.md`](../docs/dialog-plugin-guide.md); +setter and event reference: [`../../docs/dialog-sdk-reference.md`](../../docs/dialog-sdk-reference.md), +whose "Event dispatch priority and validation" section is the only normative +statement of dispatch order and malformed-event handling. Local traps not visible from the headers: - The `QDialogButtonBox` MUST be named `buttonBox` AND set `standardButtons` in the XML, or the dialog renders with no OK/Cancel and no compile error. - `QTextEdit` / model-based `QTableView` are unsupported by the widget binding — - use `QPlainTextEdit` / `QTableWidget`. See `../docs/dialog-plugin-guide.md`. -- `QStackedWidget` pages are addressed by each direct child page's Qt - `objectName`, not its label or position. `stacked_page` wins over a simultaneous - `stacked_index`; empty/unknown names and negative/out-of-range indexes are - invalid host-side warning/no-ops. See `../docs/dialog-plugin-guide.md`. -- `QTreeWidget` items are addressed only by stable plugin-supplied string IDs, - in a flat `id` / `parent_id` full snapshot. Array order defines unsorted - sibling order; check state is column 0 only. Visibility is tri-state: - absent/unchanged, ID array/filter (empty hides all), or JSON null/reset; the - host adds visible ancestors and keeps selection logical across filtering. - Header and ID arrays are strict: any non-string rejects that whole channel as - unchanged, so malformed arrays never become destructive clears. Activation - columns must be integers in `0..INT_MAX`. - `may_have_children` supports a private placeholder followed by a later full - snapshot. Tree deltas are deferred; any future delta reuses the table - sequence/all-or-nothing contract. See `../docs/dialog-plugin-guide.md`. -- Structured file pickers use stable filter IDs because native display text may - be localized or normalized. As with tree/stacked, the writer serializes caller - input. `WidgetDataView` atomically rejects empty/duplicate IDs, missing selected - IDs, or empty patterns. - The normative dispatch priority, compatibility-key validation, and - malformed/mixed-event rules live only in `../../docs/dialog-sdk-reference.md` - → "Event dispatch priority and validation". - `onClicked` precedes the host picker and result. Put file work in - `onFilePickerResult`. See `../docs/dialog-plugin-guide.md`. -- A table must not combine `sortingEnabled=true` in its `.ui` XML with - `onHeaderClicked`. Qt sorts the view while the plugin reorders the model, - overriding the plugin's order. Use sort keys (`setTableRows` with `TableItem`) - or `onHeaderClicked` — one per table, never both. -- Sortable tables must also leave item drag/drop (`dragEnabled`, `InternalMove`) - OFF: Qt reconstructs dropped cells from serialized display roles, which strips - the typed sort key and leaves a column mixing keyed and keyless cells. + use `QPlainTextEdit` / `QTableWidget`. +- `QStackedWidget` pages are addressed by each direct child page's `objectName`, + not its label or position. +- `QTreeWidget` items are addressed only by stable plugin-supplied string IDs in + a flat full snapshot; a malformed header or ID array is rejected whole, never + read as a clear. Rules: the guide's "Tree Widgets" section. +- File pickers use stable filter IDs. `onClicked` runs before the host picker; + put file work in `onFilePickerResult`. +- A table uses either sort keys (`setTableRows` with `TableItem`) or + `onHeaderClicked`, never both, and never with `sortingEnabled=true` in the + `.ui` together with `onHeaderClicked`. Sortable tables leave item drag/drop + (`dragEnabled`, `InternalMove`) off: Qt rebuilds dropped cells from display + text and drops the sort key. - Headers here install into the SAME `pj_plugins/` include tree as the parent module (merged at install); keep names distinct. - `DialogHandle::borrowed()` / `fromBorrowed()` wrap a source/toolbox-owned diff --git a/pj_plugins/docs/ARCHITECTURE.md b/pj_plugins/docs/ARCHITECTURE.md index 251e3bc1..535f15e8 100644 --- a/pj_plugins/docs/ARCHITECTURE.md +++ b/pj_plugins/docs/ARCHITECTURE.md @@ -3,7 +3,7 @@ ## 0a. ABI stability and evolution rules (v5) Seven rules the loader and every plugin author rely on. Breaking any of -these is an ABI break and requires a future `PJ_ABI_VERSION` bump. +these is an ABI break and requires a `PJ_ABI_VERSION` bump. One classification obligation rides alongside these rules: **a new host surface — a host-vtable tail slot, a versioned service/extension id, a @@ -108,19 +108,21 @@ extension, can no longer emit objects. |---|---| | `"pj..v"` | Stable. Frozen for ≥3 releases before deprecation. | | `"pj.experimental./draft-"` | Unstable. No guarantees. | - `sdk/service_traits.hpp` calls `detail::isValidServiceName()` in a - `static_assert` at every trait's `kName`. Requesting a - `pj.experimental.*` service should log a runtime warning through the - `pj.runtime.v1` log channel. + `sdk/service_traits.hpp` (and `sdk/descriptor_import.hpp`) call + `detail::isValidServiceName()` in a `static_assert` at the `kName` of the + traits they define. `ToolboxRuntimeHostService` and `WidgetHostService` name + their ids without that check, so keep them in the grammar by hand. The SDK + does not log a warning for a `pj.experimental.*` service; accepting one is host + policy. 7. **Exception discipline at the ABI boundary.** Every C ABI entry point (SDK trampolines and host-side service trampolines) must catch all exceptions and convert to a `PJ_error_t` out-param (or a safe default for non-fallible calls). C++ exceptions across - `dlopen` boundaries are undefined behaviour in practice. The - `data_source_trampolines.hpp` / `message_parser_trampolines.hpp` / - `toolbox_trampolines.hpp` files centralize this pattern — mirror it - exactly in any new trampoline. + `dlopen` boundaries are undefined behaviour in practice. The trampolines in + `pj_base/src/data_source_trampolines.cpp`, `pj_base/src/toolbox_trampolines.cpp` + and `pj_plugins/include/pj_plugins/sdk/detail/message_parser_trampolines.hpp` + centralize this pattern — mirror it exactly in any new trampoline. ### abidiff drift gate @@ -132,7 +134,7 @@ libabigail) against a checked-in baseline at | Target | Purpose | |---|---| | `abi_check` | Diff the current build's `mock_data_source_plugin` DSO against `baseline.abi`. Fatal on incompatible changes (libabigail bit 8); warning on backward-compatible additions (bit 4). | -| `abi_update_baseline` | Regenerate `pj_base/abi/baseline.abi` via `abidw` only for an intentional MAJOR break; retain it for compatible tail-slot additions. | +| `abi_update_baseline` | Regenerate `pj_base/abi/baseline.abi` via `abidw`. Run it only for an intentional MAJOR break; a compatible tail-slot addition leaves the baseline untouched. | Adding `PJ_BUILD_TESTS=ON` also registers `abi_check_test` with CTest so `./test.sh` picks it up. The plumbing lives in @@ -164,15 +166,17 @@ covers the extension wiring. Stable MessageParser-specific examples are `"pj.parser_functional.v2"` (`PJ_parser_functional_v2_t` in `pj_base/parser_functional_protocol.h`) plus the -exact route classifier `"pj.parser_route_claims.v1"`. A handler-registering -`MessageParserPluginBase` exposes both automatically. Functional v2 may carry +exact route classifier `"pj.parser_route_claims.v1"`. `MessageParserPluginBase` +always answers the route-claims id (a type without a handler is reported as +declined). It answers the functional id only when the bound type has a handler, +or, before binding, when any handler is registered. Functional v2 may carry one eligible object-field splice. The id `"pj.parser_functional.v1"` is retired and never negotiated. Route classification reports exact handler coverage only, and the host synthesizes the universal wildcard scalar claim. Hosts must query after `bind_schema` and must not cache an earlier absence because schema-generic plugins may register their handler while binding. A -parser that still implements only legacy `parse()` does not advertise the -functional extensions merely because it was rebuilt. +parser that still implements only legacy `parse()` does not advertise +functional v2 merely because it was rebuilt. ### Functional parser modules (experimental) @@ -183,17 +187,15 @@ not covered by the compatibility promise; no host loads modules yet. ## 0. C protocol v4 (current under ABI v5) -All four plugin families (DataSource, MessageParser, Toolbox, Dialog) keep -the v4 C protocol layouts under the v5 boot ABI. Key v4 distinguishing -features (a superset of everything the -previously-circulated pre-v4 design included): +All four plugin families (DataSource, MessageParser, Toolbox, Dialog) use the +v4 C protocol layouts under the v5 boot ABI. Its defining features: - **Arrow C Data Interface at the data boundary.** The write-host vtables expose `append_arrow_stream(ArrowArrayStream*)` as the canonical bulk path; per-record `append_record` / `append_bound_record` remain for streaming producers. Toolbox read-side returns host-owned - `ArrowSchema` + `ArrowArray` via `read_series_arrow` (no more - materialised `std::vector` at the boundary). + `ArrowSchema` + `ArrowArray` via `read_series_arrow` (no materialised + `std::vector` at the boundary). - **PJ_NOEXCEPT on every vtable slot.** Exceptions across `extern "C"` are UB; the noexcept specifier is part of the C++17 function type and enforced at compile time. Trampolines catch and translate internally. @@ -202,24 +204,24 @@ previously-circulated pre-v4 design included): `[thread-safe]` comment (DataSource `start`/`stop` state their rule in words). Each family's plugin vtable also carries a normative per-slot threading table (`data_source_protocol.h`, `toolbox_protocol.h`). - Host-side runtime checking is optional - (reserved for a future `"pj.thread_check.v1"` service). + The SDK does no runtime thread checking. - **Embedded-manifest plugin discovery.** Each DSO exports a family-specific protocol vtable with embedded metadata (`manifest_json` - for data sources, parsers, toolboxes, and newly built dialogs; legacy v4.0 - dialogs fall back to `create()` + `get_manifest()` during inspection). + for data sources, parsers, toolboxes, and dialogs exported with the manifest + macro form; a dialog exported without a manifest literal falls back to + `create()` + `get_manifest()` during inspection). Host-side `PJ::scanPluginDsos(dir)` (in `pj_plugins/host/plugin_catalog.hpp`) walks platform plugin libraries, loads each candidate, validates the ABI and protocol vtable, and parses `id`, `name`, `version`, the optional compatibility fields `min_sdk_required` and `min_plotjuggler_version`, family-specific fields, and optional metadata directly from the embedded manifest. The former is a - concrete SemVer SDK-contract floor and the latter retains its distinct - application-release meaning; see `REQUIREMENTS.md` section 2.1 for the full - compatibility vocabulary and manifest rules. Broken or incompatible + concrete SemVer SDK-contract floor; the latter is deprecated (decoded, never a + gate). See `REQUIREMENTS.md` section 2.1 for the full compatibility vocabulary + and manifest rules. Broken or incompatible candidates are reported as diagnostics while discovery continues. -- **No more RTLD_DEEPBIND.** The loader uses `RTLD_NOW | RTLD_LOCAL` - only (DEEPBIND was a documented ASAN/allocator-interposition trap). +- **No `RTLD_DEEPBIND`.** The loader uses `RTLD_NOW | RTLD_LOCAL` + only (DEEPBIND is a documented ASAN/allocator-interposition trap). Plugin-local symbol isolation is the plugin build's job: `pj_configure_plugin` (`cmake/PjPlugin.cmake`, shipped with `plugin_sdk`) applies hidden visibility, `-Wl,-Bsymbolic-functions`, and a version-script export allowlist gated @@ -229,8 +231,7 @@ previously-circulated pre-v4 design included): manifest-format change. Shared handle ownership controls lifetime, while candidate-file provenance is checked directly at each boot-level symbol. -Structural shape inherited from the pre-v4 design work (carries the -service registry, error out-params, and typed borrowed-dialog patterns): +Structural shape (service registry, error out-params, typed borrowed dialog): - **Service registry as the sole binding mechanism.** Plugin vtables expose a single `bind(ctx, registry, err)` slot. The host registers all services @@ -262,10 +263,9 @@ service registry, error out-params, and typed borrowed-dialog patterns): share is addressed rather than guessed — rules are normative in `plugin_data_api.h` (DATASET-QUALIFIED NAMES), shared parser/composer in `pj_base/sdk/dataset_qualified_name.hpp`. The script payload is **binary-safe** - (`PJ_string_view_t {data,size}`), so the native "door" is WASM bytes through - this same data-only surface (a future host-owned WASM/Python backend is purely - additive and survives plugin unload) — deliberately *not* a C++ kernel vtable - that would dangle on unload. The plugin sees a Qt-free + (`PJ_string_view_t {data,size}`), so a host-owned backend can take bytecode + through this same data-only surface and keep the node alive after the plugin + unloads — deliberately *not* a C++ kernel vtable that would dangle on unload. The plugin sees a Qt-free `sdk::DataProcessorsHostView`. `"pj.settings.v1"` (optional) is a QSettings-like key/value store any plugin family can use for persistent state — the plugin sees a Qt-free @@ -318,8 +318,8 @@ service registry, error out-params, and typed borrowed-dialog patterns): are in `toolbox-guide.md` under "Playback, viewport, and owned tabs" and "Dataset-qualified processor inputs". - **Structured errors everywhere.** All fallible ABI calls take a - `PJ_error_t* out_error` out-parameter. The old per-plugin `get_last_error` - slot is gone. + `PJ_error_t* out_error` out-parameter; there is no per-plugin + `get_last_error` slot. - **Shared write contract, typed ABI services.** DataSource, MessageParser, and Toolbox all write through the same datastore backend and follow the same scalar/Arrow ownership rules, but the ABI keeps three distinct service @@ -328,9 +328,8 @@ service registry, error out-params, and typed borrowed-dialog patterns): The service name selects the family-specific type (`"pj.source_write.v1"`, `"pj.parser_write.v1"`, `"pj.toolbox_write.v1"`), so the compiler prevents a parser from calling source/toolbox-only operations. -- **Typed borrowed dialog.** `get_dialog_context()` returning `void*` is - replaced by `get_dialog()` returning a `PJ_borrowed_dialog_t` fat pointer - `{ctx, const PJ_dialog_vtable_t* vtable}`. +- **Typed borrowed dialog.** `get_dialog()` returns a `PJ_borrowed_dialog_t` fat + pointer `{ctx, const PJ_dialog_vtable_t* vtable}`, not a bare `void*`. - **Family-specific plugin vtables after the common prefix.** DataSource, MessageParser, and Toolbox vtables share `protocol_version, struct_size, create, destroy, manifest_json`; subsequent @@ -458,7 +457,7 @@ pj_datastore/ **Dependency direction:** Installed plugin authors consume `plotjuggler_sdk::plugin_sdk`, which combines `pj_base` with the MessageParser, dialog and widget authoring headers from `pj_plugins`. Host -libraries in `pj_plugins` depend on `pj_base`. `pj_datastore` — now a module in +libraries in `pj_plugins` depend on `pj_base`. `pj_datastore` — a module in the PlotJuggler application repo, not part of this SDK — provides the concrete data-host implementations that bridge plugin writes to the columnar storage engine. @@ -510,10 +509,9 @@ All SDK base classes: **Trampoline pattern:** Each base class has a private set of `static` trampoline functions (e.g. `trampoline_start`) that cast the `void* ctx` to the concrete class, call the virtual, and wrap the result for C ABI return. -MessageParser and Dialog implementations live in `sdk/detail/*_trampolines.hpp`; +The MessageParser implementation lives in `sdk/detail/message_parser_trampolines.hpp`; DataSource and Toolbox implementations live in `pj_base/src/*_trampolines.cpp`; -`WidgetPluginBase` defines its trampolines in its header (the Widget SDK is -header-only). +`DialogPluginBase` and `WidgetPluginBase` define theirs in their own headers. ## 5. Host Loaders @@ -549,29 +547,27 @@ process working directory changes, or dyld reports a symlink-resolved filename. ### 5.1 Host-side diagnostic propagation -Host code that loads plugins accepts an -optional `PJ::DiagnosticSink` (`pj_base/include/pj_base/diagnostic_sink.hpp`) -in its constructor. The sink is a `std::function` -the host invokes for every plugin-load lifecycle event — failed `dlopen`, -missing required manifest fields, malformed JSON, successful loads, -hot-reload detection, etc. Each event carries a level +`PJ::DiagnosticSink` (`pj_base/include/pj_base/diagnostic_sink.hpp`) is a +`std::function`. Each event carries a level (`kInfo`/`kWarning`/`kError`), a `source` string, an optional plugin id, a -message, and a timestamp. - -Embedding apps wire one sink into their host loaders and any application-level -extension services so the GUI can show one chronological diagnostic stream. -Pure-C++ host loaders remain toolkit-free; GUI hosts provide any event-loop -adapter needed to marshal diagnostics onto their UI thread. - -A default-constructed sink discards events at zero cost, so loaders that -take no sink behave as before. +message, and a timestamp. A default-constructed sink discards events at zero +cost; `teeSink(a, b)` forwards to two sinks. + +Only `ParserClaimCatalog` and `ParserRouteResolver` take a sink (constructor +argument or `setDiagnosticSink`); they report claim-admission and route-resolution +events through it. The `*Library::load(path)` loaders return `Expected` with the +failure text, and `scanPluginDsos` reports each unusable candidate in +`PluginScanResult::diagnostics` while discovery continues. An embedding host +forwards these into one sink so the GUI can show a single chronological +diagnostic stream. The host classes stay toolkit-free; GUI hosts provide any +event-loop adapter needed to marshal diagnostics onto their UI thread. ### 5.2 Parser claim catalog and route resolver `ParserClaimCatalog` is the host-side parser dispatch catalog. It admits -transactional claim batches from module manifests or synthesized parser-plugin -coverage, validates stable `(provider_id, claim_id)` identities, and attaches -host-supplied provenance and provider generations. The SDK-owned encoding +transactional claim batches (`admitClaims`) or synthesized parser-plugin +coverage (`admitParserPlugin`), validates stable `(provider_id, claim_id)` +identities, and attaches host-supplied provenance and provider generations. The SDK-owned encoding registry and normalization helpers keep matching case-sensitive and canonical. `ParserRouteResolver` independently selects scalar and object providers. It @@ -703,7 +699,7 @@ All three share a common internal `WriteCore` that handles: route canonical media (images, point clouds, annotations) into the session `ObjectStore` rather than the columnar engine. They forward to the same `ObjectStore::registerTopic` / `pushOwned` the DataSource and - Parser object-write hosts use, so the toolbox host now requires an + Parser object-write hosts use, so the toolbox host requires an `ObjectStore&` at construction alongside the `DataEngine&`. These are **tail slots** appended to `PJ_toolbox_host_vtable_t` under ABI v5 (no version bump): existing slot offsets are unchanged, and `struct_size` @@ -766,30 +762,20 @@ out_array, err)`: | `mock_dialog.cpp` (in `dialog_protocol/examples/`, not `pj_plugins/examples/`) | Standalone dialog: QLineEdit, QSpinBox, QCheckBox, config persistence | | `mock_widget.cpp` (in `widget_protocol/examples/`) | Qt-free widget library: two types, inputs, `on_update`, history, config, fake widget pointer | -### Test files (`pj_plugins/tests/`) +### Tests -| Test | Coverage | -|---|---| -| `data_source_library_test.cpp` | Library loading, vtable validation | -| `file_source_integration_test.cpp` | FileSourceBase end-to-end | -| `delegated_ingest_integration_test.cpp` | _Disabled (pending v3-port: uses removed `bindWriteHost`/`bindRuntimeHost`/`get_last_error`; CMake target commented out). Coverage currently provided by `data_source_library_test.cpp` + `message_parser_library_test.cpp`._ | -| `source_dialog_integration_test.cpp` | DataSource dialog | -| `message_parser_library_test.cpp` | Parser library loading | -| `toolbox_plugin_test.cpp` | Toolbox loading, host binding, read+write flow | -| `dialog_handle_test.cpp` | Owned and borrowed handle lifecycle | -| `dialog_library_test.cpp` | Dialog library loading | -| `dialog_plugin_typed_test.cpp` | Typed event dispatch | -| `widget_data_test.cpp` | WidgetData builder | -| `widget_data_view_test.cpp` | WidgetDataView JSON parsing | -| `widget_event_builder_test.cpp` | Event JSON generation | -| `widget_event_test.cpp` | Event parsing | -| `plugin_lifecycle_test.cpp` | Plugin create/destroy lifecycle | - -## Builtin-object pipeline (PR #86) - -The v4 DataSource runtime host adds a tail slot `push_message` -(offset 88 in `PJ_data_source_runtime_host_vtable_t`) that takes a -deferred byte-fetch callable instead of bytes: +Host-side and plugin-side tests live in `pj_plugins/tests/`, +`pj_plugins/dialog_protocol/tests/` and `pj_plugins/widget_protocol/tests/`; +`pj_base/tests/` holds the ABI layout sentinels. The +`delegated_ingest_integration_test` target is disabled in +`pj_plugins/CMakeLists.txt` (it still uses the removed `bindWriteHost` / +`bindRuntimeHost` / `get_last_error` API); `data_source_library_test` and +`message_parser_library_test` cover delegated ingest instead. + +## Builtin-object pipeline + +The DataSource runtime host's `push_message` tail slot takes a deferred +byte-fetch callable instead of bytes: ```c typedef struct PJ_message_data_fetcher_t { @@ -820,10 +806,11 @@ that cascades `topic > source > type > default`: ObjectStore entry only. Parsers participate through `classifySchema`, `parseScalars`, and -`parseObject`, backed by the per-schema `SchemaHandler` table. Those C++ -methods and their `ScalarRecord` / `ObjectRecord` / `BuiltinObject` values stay -inside the plugin DSO. The base exposes `pj.parser_functional.v2`, which -`MessageParserHandle` negotiates: +`parseObject`, backed by the per-schema `SchemaHandler` table. The base exposes +`pj.parser_functional.v2`, which `MessageParserHandle` negotiates, so those C++ +methods and their `ScalarRecord` / `ObjectRecord` / `BuiltinObject` values need +not cross the DSO boundary. A host may still call them directly through the +deprecated C++ bridge (section 0a); new host code uses v2: - `parse_scalars` calls one caller-owned sink exactly once. Field/name/string views are borrowed only for that callback; the host copies anything it @@ -850,22 +837,21 @@ This deliberately pays one canonical serialization at the DSO boundary and a host-side decode. It avoids the more fragile alternative of sharing STL object layouts, allocators, destructors, or release callbacks for each concrete C++ type. Large input buffers need not be copied before parsing when the host has a -`PayloadView` anchor. Benchmark the serialize/decode cost for image and point -cloud workloads before removing the deprecated fallback; a future optimization -must preserve the same C ownership boundary rather than reintroduce C++ object -sharing. +`PayloadView` anchor. The serialize/decode cost is measurable for image and +point-cloud workloads; an optimization must preserve the same C ownership +boundary rather than reintroduce C++ object sharing. `serializeBuiltinObject` / `deserializeBuiltinObject` dispatch every stable -builtin type, including the newly canonicalized `RobotDescription`. A -zero-length canonical payload is accepted as the valid proto3 default message +builtin type. A zero-length canonical payload is accepted as the valid proto3 default message when the separate type tag is known. Concrete builtins and codecs live under `pj_base/builtin/`; see `docs/builtin_type.md` for the catalog. Parsers built before SDK 0.22 expose no usable functional extension: pre-0.21 builds expose none, and 0.21 builds expose only the retired v1. New plugins and -new host code use v2. PlotJuggler 4 still reads objects through the direct C++ -bridge for every parser; section 0a states what that freezes and when the -freeze ends. +new host code use v2. The direct C++ bridge (a host casting the parser context +to `MessageParserPluginBase*`) is deprecated but supported through 1.x, and +PlotJuggler 4 still reads objects through it for every parser; section 0a states +what it freezes. ## Per-topic pause (demand-driven subscription) @@ -876,8 +862,7 @@ whole-source pause. Two independent additions, both `struct_size`/ `PJ_HAS_TAIL_SLOT`-gated so an old plugin or an old host degrades cleanly: - **Plugin → host advertise.** A second tail slot on the runtime host, - `notify_available_topics(ctx, topics, count, out_error)` (offset 96, - growing `sizeof(PJ_data_source_runtime_host_vtable_t)` 96 → 104), carrying + `notify_available_topics(ctx, topics, count, out_error)`, carrying `PJ_available_topic_t{topic_name, parser_encoding, type_name, schema}` — the same parser-identifying fields as `PJ_parser_binding_request_t` minus `parser_config_json` (not yet known pre-subscription), so the host can @@ -896,14 +881,14 @@ whole-source pause. Two independent additions, both `struct_size`/ call matters). `DataSourceHandle::setActiveTopics` is a no-op when the plugin does not expose the extension. -See `docs/data-source-guide.md` → "Per-topic pause (demand-driven +See [data-source-guide.md](data-source-guide.md) → "Per-topic pause (demand-driven subscription)" for the plugin-author walkthrough. ## Source-cache attachment -SDK 0.28 appends `attach_source_record(ctx, descriptor_json, out_error)` to -the runtime-host vtable at offset 104, growing its size 104 → 112. Providers -call it on the stream thread at download start. The host copies and stores +The runtime-host tail slot `attach_source_record(ctx, descriptor_json, out_error)` +(SDK 0.28.0) lets providers declare their request descriptor. They call it on +the stream thread at download start. The host copies and stores the descriptor bytes verbatim, keys its cache on its own digest, and scopes the record by the provider id from the binding. Matching is byte-exact; the host bounds and parses the descriptor and rejects unknown fields. @@ -919,10 +904,10 @@ are detected with `PJ_HAS_TAIL_SLOT` and reported as an error (no caching). ## Ingest completion and rollback -SDK 0.30 appends `complete_ingest(ctx, completion, out_error)` to the -runtime-host vtable at offset 112 (size 112 → 120). A finite source reports -its terminal outcome — `FAILED` / `CANCELLED` / `COMPLETED` — on the stream -thread once every producer and `push_message` has quiesced. +The runtime-host tail slot `complete_ingest(ctx, completion, out_error)` +(SDK 0.30.0) is how a finite source reports its terminal outcome — `FAILED` / +`CANCELLED` / `COMPLETED` — on the stream thread once every producer and +`push_message` has quiesced. `COMPLETED` attests the ENTIRE declared request: the completion carries the full requested-topic set from the request snapshot (never assembled from the pulls @@ -938,8 +923,8 @@ the host's coverage checks. Cancellation goes through the host's thread-safe stop path first; completion is what the stream thread reports afterwards. -Contexts that never call the slot ingest exactly as before and are simply -never cacheable — cacheability is negotiated from slot presence, not a +A context that never calls the slot ingests normally and is never +cacheable — cacheability is negotiated from slot presence, not a manifest flag. `pj_base/sdk/ingest_completion.hpp` provides the shared fail-closed validator (`copyIngestCompletion`) hosts use to copy the borrowed @@ -950,9 +935,8 @@ and empty. It does not override host failures or authorize an empty requested set or an entirely empty capture. Reuse `IngestOutcomeLedger` and the [provider guide](../../docs/provider-guide.md) to compute the terminal. -The same release also formalizes `discard_parser_ingest` (toolbox runtime -host, offset 40, size 40 → 48): the rollback twin of `release_parser_ingest` -— abort a provisional ingest without committing, dropping staged source -records and vetoing publication. Previously a private plugin-side -compatibility extension; `PJ_TOOLBOX_HAS_DISCARD_PARSER_INGEST` lets plugins -drop their local struct-size probes. +`discard_parser_ingest` (toolbox runtime host tail slot, 0.30.0) is the +rollback twin of `release_parser_ingest`: it aborts a provisional ingest without +committing, dropping staged source records and vetoing publication. +`PJ_TOOLBOX_HAS_DISCARD_PARSER_INGEST` is defined by the public header that +declares the slot. diff --git a/pj_plugins/docs/REQUIREMENTS.md b/pj_plugins/docs/REQUIREMENTS.md index 855f73cb..b438d96d 100644 --- a/pj_plugins/docs/REQUIREMENTS.md +++ b/pj_plugins/docs/REQUIREMENTS.md @@ -50,14 +50,23 @@ Five plugin families exist: ### 2.1 Compatibility vocabulary -Plugin compatibility uses three separate values. They are not aliases and a +Plugin compatibility uses separate values. They are not aliases and a manifest must not substitute one for another: | Value | Meaning | Enforcement | |---|---|---| -| `min_sdk_required` | Minimum PlotJuggler SDK contract whose features the plugin actually requires | Hard host-side load gate before plugin instantiation | -| `min_plotjuggler_version` | Minimum PlotJuggler application release expected by the plugin | Existing application-level advisory policy | -| ABI/dialog protocol version | Binary C ABI and dialog C/wire contract compatibility | Existing strict vtable, `struct_size`, and protocol validation | +| `min_sdk_required` | Minimum PlotJuggler SDK contract whose features the plugin actually requires (the functioning floor) | Hard host-side load gate before plugin instantiation | +| `suggested_sdk_version` | Full-feature floor: the SDK release at which every host surface the plugin uses is available. Declared together with a `floor_test` whenever it is above `min_sdk_required` | Informational; never an admission criterion | +| `min_plotjuggler_version` | Deprecated application-release expectation | Decoded for back-compat and never gates | +| ABI/dialog protocol version | Binary C ABI and dialog C/wire contract compatibility | Strict vtable, `struct_size`, and protocol validation | + +`tools/feature_floors/feature_floor_check.py`, installed with +`feature_floors.json` in `share/plotjuggler_sdk/`, validates a manifest's floors +for plugin repositories: `min_sdk_required` must cover every non-negotiated host +surface the plugin's sources use, and when a used surface was introduced after +it, the manifest declares `suggested_sdk_version` (the newest introducing +release) plus the `floor_test` gtest that proves the plugin works against a +floor-level host. `min_sdk_required` is not automatically the SDK version used to compile the plugin. A plugin compiled with a newer SDK declares the oldest SDK contract @@ -115,13 +124,14 @@ Independent decoder, typically driven by a DataSource via the host. instance — the host creates it and bridges config to parser instances via JSON. - Lifecycle: create → bind → parse* → destroy. -- A parser that registers `SchemaHandler` entries exposes - `pj.parser_functional.v2` and - `pj.parser_route_claims.v1` after schema binding. Functional scalar and object - results cross only through synchronous caller-owned C sinks; v2 may represent - one eligible object bulk field as a payload-relative splice. The host must - never cast the opaque context to - `MessageParserPluginBase` for such a parser. +- `MessageParserPluginBase` always answers `pj.parser_route_claims.v1` (a type + without a handler is declined) and answers `pj.parser_functional.v2` only when + the bound type has a handler (before binding, when any handler is registered). + Functional scalar and object results cross only through synchronous + caller-owned C sinks; v2 may represent one eligible object bulk field as a + payload-relative splice. New host code reads results through v2 rather than + casting the opaque context to `MessageParserPluginBase`; the deprecated bridge + below remains supported. - Route classification reports exact handler-table claims only. The host owns each encoding's universal wildcard scalar claim and resolves scalar and object routes independently. @@ -225,11 +235,14 @@ is bound to one topic. Only Toolbox requires read access: - `catalogSnapshot()` — enumerate available data sources, topics, and fields. -- `readSeriesArrow(field)` — read the full time series for a field as a - host-owned `ArrowSchema` + `ArrowArray` pair (timestamp column + - value column). Plugins wrap the out-params in - `PJ::sdk::ArrowSchemaHolder` / `ArrowArrayHolder` for scope-bound - release. +- `readSeries(field)` — read the full time series for a field as a + `MaterializedSeriesView` (`rowCount()`, `timestamps()`, typed `valuesAs*()` + pointers, `nullCount()` / `isNull(row)`). The value slot of a null sample is + unspecified, so a caller skips rows where `isNull(row)` when `nullCount() != 0`. +- `readSeriesArrow(field)` — the same data as a host-owned `ArrowSchema` + + `ArrowArray` pair (timestamp column + value column), which `readSeries` wraps. + Plugins wrap the out-params in `PJ::sdk::ArrowSchemaHolder` / + `ArrowArrayHolder` for scope-bound release. Materialization/decompression is acceptable when reading actual sample data. @@ -293,13 +306,14 @@ threads must synchronize and call host methods only where those tables allow. | `kCapabilityDelegatedIngest` | Plugin pushes raw bytes for host-side parsing | | `kCapabilitySupportsPause` | pause()/resume() are implemented | | `kCapabilityHasDialog` | Plugin provides a configuration dialog | -| `kCapabilityPerTopicPause` | Host may pause/resume individual topics on a live connection (advertise via `notifyAvailableTopics` + `pj.topic_subscription.v1`) — see `docs/ARCHITECTURE.md` → "Per-topic pause" and `docs/data-source-guide.md` | +| `kCapabilityPerTopicPause` | Host may pause/resume individual topics on a live connection (advertise via `notifyAvailableTopics` + `pj.topic_subscription.v1`) — see [ARCHITECTURE.md](ARCHITECTURE.md#per-topic-pause-demand-driven-subscription) and [data-source-guide.md](data-source-guide.md) | ### Toolbox Capabilities | Flag | Purpose | |---|---| | `kToolboxCapabilityHasDialog` | Plugin provides a persistent UI panel | +| `kToolboxCapabilityNonModalDialog` | The dialog is shown as a non-modal panel rather than a modal pop-up | Read, write, create-source, and destructive-update capabilities are intrinsic to all Toolbox plugins. diff --git a/pj_plugins/docs/data-source-guide.md b/pj_plugins/docs/data-source-guide.md index db931a70..174f8c79 100644 --- a/pj_plugins/docs/data-source-guide.md +++ b/pj_plugins/docs/data-source-guide.md @@ -20,8 +20,8 @@ A DataSource plugin is a shared library (`.so` / `.dylib` / `.dll`) that acquires data — from files, network streams, hardware, etc. — and feeds it -into PlotJuggler. Plugins link only against `pj_base` (no Qt, no host -internals) and communicate through a stable C ABI. +into PlotJuggler. Plugins link only the SDK (`plotjuggler_sdk::plugin_sdk`: no +Qt, no host internals) and communicate through a stable C ABI. ## Choosing a Base Class @@ -42,7 +42,8 @@ only when the supplied state machines do not fit. 1. Pick a base class from the table above. 2. Implement the required virtuals (see Common Patterns below). 3. Export with `PJ_DATA_SOURCE_PLUGIN(YourClass, R"({"id":"...","name":"...","version":"..."})")` -4. Build as a shared library linking `pj_base` +4. Build as a shared library linking `plotjuggler_sdk::plugin_sdk` and call + `pj_configure_plugin(...)` on it (see Build below) For a namespaced class in a static build, provide the getter-symbol token separately: `PJ_DATA_SOURCE_PLUGIN_NAMED(my::Source, MySource, kManifest)`. @@ -58,8 +59,8 @@ prevent runtime failures and confusing host behaviour. **MUST** - Return `PJ::okStatus()` on success and `PJ::unexpected("reason")` on failure from every fallible method. -- Call `[stream-thread]` host methods (`writeHost()`, and `runtimeHost()` - except `reportMessage`/`requestStop`/`isStopRequested`/`notifyState`) only +- Call `[stream-thread]` host methods (`writeHost()`, and the `runtimeHost()` + methods tagged `[stream-thread]` in the table under *Runtime host*) only while the host is inside `start()`/`poll()`/`stop()` — i.e. from `onStart`/`onPoll`/`onStop`/`importData`, one thread at a time. - Make `onStop()` idempotent, and release what a failed `onStart()` acquired: @@ -146,16 +147,20 @@ can read it without creating an instance. ### 4. Build ```cmake +find_package(plotjuggler_sdk REQUIRED COMPONENTS plugin_sdk) + add_library(my_source_plugin SHARED my_source.cpp) -target_link_libraries(my_source_plugin PRIVATE pj_base) +target_link_libraries(my_source_plugin PRIVATE plotjuggler_sdk::plugin_sdk) +pj_configure_plugin(my_source_plugin FAMILIES data_source) ``` -No other dependencies are needed for headless sources. If your source includes -a configuration dialog (see Dialog Integration below), also link `pj_dialog_sdk`: - -```cmake -target_link_libraries(my_source_plugin PRIVATE pj_base pj_dialog_sdk) -``` +`pj_configure_plugin` applies symbol isolation and the export allowlist and +validates `manifest.json` (see `examples/sdk_consumer/CMakeLists.txt`; add +`MANIFEST_HEADER` to embed it in the source). No other dependencies are needed, +including for a configuration dialog (see Dialog Integration below), whose SDK +`plugin_sdk` also provides. Source providers also link `plotjuggler_sdk::source`. +`pj_base`, `pj_dialog_sdk` and `pj_source` are the in-tree names of the targets +behind these. ## Common Patterns @@ -241,7 +246,11 @@ class CsvFileLoader : public PJ::FileSourceBase { auto row_values = splitString(line, ','); for(size_t index = 0; index < row_values.size(); index++) { // if we fail to parse the value string, use PJ::kNull - row_fields[index].value = parseToDouble(row_values[index]).value_or(PJ::kNull); + if (auto parsed = parseToDouble(row_values[index])) { + row_fields[index].value = *parsed; + } else { + row_fields[index].value = PJ::kNull; + } } // Push data. Timestamps are nanoseconds since Unix epoch. // Here we use row number for simplicity — real plugins should @@ -405,7 +414,7 @@ class UdpReceiver : public PJ::StreamSourceBase { int fd_ = -1; std::string config_; std::string encoding_ = "json"; - PJ::Expected binding_ = + PJ::Expected binding_ = PJ::unexpected("unset"); std::atomic running_{false}; @@ -467,21 +476,25 @@ engine. ### Runtime host — control plane Access via `runtimeHost()`. Use this for lifecycle coordination and diagnostics. +The thread tag of each method is normative in `data_source_protocol.h`. -| Method | Purpose | -|---|---| -| `reportMessage(level, text)` | Send info/warning/error to the host UI log. | -| `progressStart(label, total, cancellable)` | Begin a progress bar. | -| `progressUpdate(step)` | Advance progress. Returns false if cancelled. | -| `progressFinish()` | End the progress sequence. | -| `notifyState(state)` | Tell the host your state changed. Advisory: a host may ignore it. | -| `requestStop(terminal_state, reason)` | Ask the host to stop you (self-terminate). | -| `isStopRequested()` | Check if the host wants you to stop. | -| `ensureParserBinding(request)` | Bind a parser for delegated ingest (see below). | -| `pushMessage(handle, timestamp, fetch_message_data)` | Push a message through a parser binding via a deferred fetcher callable; the host invokes it per the active ObjectIngestPolicy (eager/lazy). | -| `notifyAvailableTopics(topics)` | Advertise the full set of topics you *can* stream but have not subscribed, so the host lists and a-priori classifies them with no data flowing. See *Per-topic pause* below. | -| `attachSourceRecord(descriptor_json)` | On the stream thread, declare the request descriptor for the host's source cache. The host stores the bytes verbatim and derives its cache key. Last attach before this context's first `pushMessage` wins; byte-identical repeats before ingest are idempotent. Errors after ingest begins or on hosts that predate the slot never affect ingest. See [Source-cache attachment](ARCHITECTURE.md#source-cache-attachment). | -| `setDatasetMetadata(metadata_json)` | Attach one descriptive JSON object about the loaded artifact (embedded manifests, recording timestamps, file facts) for the host to display generically. Observations, never request identity: it does not affect caching, restore, or trust. Each successful call replaces the whole document; `"{}"` clears it. Old hosts (no slot) return an error and the load proceeds without metadata. | +| Method | Thread | Purpose | +|---|---|---| +| `reportMessage(level, text)` | `[thread-safe]` | Send info/warning/error to the host UI log. | +| `progressStart(label, total, cancellable)` | `[stream-thread]` | Begin a progress bar. | +| `progressUpdate(step)` | `[stream-thread]` | Advance progress. Returns false if cancelled. | +| `progressFinish()` | `[stream-thread]` | End the progress sequence. | +| `notifyState(state)` | `[thread-safe]` | Tell the host your state changed. Advisory: a host may ignore it. | +| `requestStop(terminal_state, reason)` | `[thread-safe]` | Ask the host to stop you (self-terminate). | +| `isStopRequested()` | `[thread-safe]` | Check if the host wants you to stop. | +| `showMessageBox(type, title, message, buttons)` | `[thread-safe]` | Modal message box (`showInfo`/`showWarning`/`showError`/`askContinue`/`askYesNo` wrap it). Blocks until the user answers; the host marshals to its UI thread. A host without modal dialogs answers `kOk`. | +| `listAvailableEncodings()` | `[main-thread]` | JSON array of the parser encodings the host has loaded; empty when none. | +| `ensureParserBinding(request)` | `[stream-thread]` | Bind a parser for delegated ingest (see below). | +| `pushMessage(handle, timestamp, fetch_message_data)` | `[stream-thread]` | Push a message through a parser binding via a deferred fetcher callable; the host invokes it per the active ObjectIngestPolicy (eager/lazy). | +| `notifyAvailableTopics(topics)` | `[stream-thread]` | Advertise the full set of topics you *can* stream but have not subscribed, so the host lists and a-priori classifies them with no data flowing. See *Per-topic pause* below. | +| `attachSourceRecord(descriptor_json)` | `[stream-thread]` | Declare the request descriptor for the host's source cache. The host stores the bytes verbatim and derives its cache key. Last attach before this context's first `pushMessage` wins; byte-identical repeats before ingest are idempotent. Errors after ingest begins or on hosts that predate the slot never affect ingest. See [Source-cache attachment](ARCHITECTURE.md#source-cache-attachment). | +| `completeIngest(outcome, requested_topics, flags)` | `[stream-thread]` | Report a finite ingest's terminal outcome for caching, after every producer and `pushMessage` has quiesced. See [Ingest completion and rollback](ARCHITECTURE.md#ingest-completion-and-rollback). | +| `setDatasetMetadata(metadata_json)` | `[stream-thread]` | Attach one descriptive JSON object about the loaded artifact (embedded manifests, recording timestamps, file facts) for the host to display generically. Observations, never request identity: it does not affect caching, restore, or trust. Each successful call replaces the whole document; `"{}"` clears it. Old hosts (no slot) return an error and the load proceeds without metadata. | ## Optional Features @@ -654,8 +667,8 @@ The host acts on these signals only: `reason` as the failure. - **`runtimeHost().isStopRequested()`** tells you the host (user stop, cancel) wants you to stop. Check it during long operations. -- **`completeIngest()`** reports a finite ingest's terminal outcome for - caching (see *Ingest completion and rollback* in ARCHITECTURE.md). +- **`runtimeHost().completeIngest()`** reports a finite ingest's terminal outcome + for caching (see *Ingest completion and rollback* in ARCHITECTURE.md). `notifyState()`, `currentState()` and the `PJ_data_source_state_t` states (`idle`, `starting`, `running`, `paused`, `stopped`, `failed`, ...) are @@ -690,6 +703,10 @@ it without instantiating the plugin. | `description` | string | no | Short description of the plugin. | | `file_extensions` | string[] | no | File extensions this source handles, e.g. `[".csv", ".tsv"]`. Plugins declaring `kCapabilityFiniteImport` SHOULD include this so the host can build file-dialog filters. | +`min_sdk_required`, `suggested_sdk_version` (with a `floor_test`) and the +deprecated `min_plotjuggler_version` are optional compatibility keys common to +every family; see [REQUIREMENTS.md §2.1](REQUIREMENTS.md#21-compatibility-vocabulary). + Example: ```json { @@ -762,8 +779,8 @@ host synthesise a monotonic timestamp per row. For an Arrow **IPC** byte buffer (file or Flight wire format), use nanoarrow's `ArrowIpcArrayStreamReaderInit` to obtain an `ArrowArrayStream*`. -Pass it to `appendArrowStream()`. v4 no longer exposes a separate IPC-bytes -write slot. +Pass it to `appendArrowStream()`. The write host has no separate IPC-bytes +slot. The raw-pointer overload (`appendArrowStream(topic, ArrowArrayStream*, ...)`) remains an ABI escape hatch. The rvalue-reference form is the documented default. @@ -785,8 +802,9 @@ The normative per-slot table is on `PJ_data_source_vtable_t` in - `[stream-thread]` host methods (`appendRecord()`, `ensureParserBinding()`, `pushMessage()`, progress, ...) only while a `start()`/`poll()`/`stop()` call is in progress, one thread at a time. `[thread-safe]` ones - (`reportMessage()`, `requestStop()`, `isStopRequested()`, `notifyState()`) - from any thread. + (`reportMessage()`, `requestStop()`, `isStopRequested()`, `notifyState()`, + `showMessageBox()`) from any thread; `listAvailableEncodings()` is + `[main-thread]`. If your plugin needs internal threading (e.g. a network receive thread), you must synchronize access to shared state yourself and hand data to `onPoll()`. @@ -873,7 +891,8 @@ with no JSON serialization needed at runtime. │ (business logic) │ │ getDialog() → borrowDialog(...) │ │ │ -│ PJ_DATA_SOURCE_PLUGIN(MySource) │ → exports DataSource vtable +│ PJ_DATA_SOURCE_PLUGIN(MySource, │ → exports DataSource vtable +│ kManifest) │ │ PJ_DIALOG_PLUGIN(MyDialog, │ → exports Dialog vtable │ kManifestJson) │ └──────────────────────────────────────┘ @@ -985,11 +1004,11 @@ See `pj_plugins/docs/dialog-plugin-guide.md` for the dialog protocol itself. DataSource-owned dialog pattern: a combined `.so` with two vtables, shared state via member ownership, and dialog read-only accessors. -## Builtin-object pipeline (PR #86) — `pushMessage` + FetchMessageData +## `pushMessage` and FetchMessageData For sources that fan out raw bytes to a `MessageParser` (MCAP, topic -bridges, future ROS-bag streamers), the runtime host exposes a v2 ingest -slot that takes a deferred callable instead of bytes. The plugin builds +bridges, ROS-bag streamers), the runtime host's `pushMessage` takes a deferred +callable instead of bytes. The plugin builds a closure that knows how to materialise the payload, hands it to the host, and stays policy-agnostic: @@ -1033,10 +1052,11 @@ the ingest thread (`kEager`) or from consumer threads (lazy pulls). Capture file readers / decompressed chunks by `shared_ptr` so the source survives every pending pull. -Reference implementation: `pj_ported_plugins/data_load_mcap` — closure -captures the open `mcap::McapReader` and the message offset, reads the -bytes on demand. See `PLUGIN_DEVELOPMENT.md` in that repo for the -catalog-side companion (`parser_ros`) and a top-down walkthrough. +Reference implementation: `data_load_mcap` in the official plugins repository +(pj-official-plugins) — its closure captures the open `mcap::McapReader` and +the message offset, and reads the bytes on demand. `PLUGIN_DEVELOPMENT.md` in +that repository covers the parser-side companion (`parser_ros`) and gives a +top-down walkthrough. ## Common Mistakes diff --git a/pj_plugins/docs/dialog-plugin-guide.md b/pj_plugins/docs/dialog-plugin-guide.md index 7376dd3c..4a4ddaf0 100644 --- a/pj_plugins/docs/dialog-plugin-guide.md +++ b/pj_plugins/docs/dialog-plugin-guide.md @@ -99,7 +99,8 @@ renders the widgets, and relays events to the plugin over the C vtable. 5. Override the typed event handlers you need (`onTextChanged`, `onIndexChanged`, `onToggled`, etc.) — return `true` when state changes. 6. Export with `PJ_DIALOG_PLUGIN(YourClass, kManifestJson)`. -7. Build as a shared library linking `pj_dialog_sdk`. +7. Build as a shared library linking `plotjuggler_sdk::plugin_sdk` and call + `pj_configure_plugin(... FAMILIES dialog)` on it. For a namespaced class in a static build, provide the getter-symbol token separately: `PJ_DIALOG_PLUGIN_NAMED(my::Dialog, MyDialog, kManifestJson)`. @@ -198,13 +199,14 @@ stored strings and the complete capability mask. Plugin code should normally observe one delivery before first use; the replacement rule makes re-binding deterministic for embedding hosts and tests. -The generated SDK version header and all four feature-test macros are deferred -to their separate 0.21 package: `PJ_SDK_HAS_DIALOG_HOST_INFO`, -`PJ_SDK_HAS_DIALOG_STACKED_WIDGET`, `PJ_SDK_HAS_DIALOG_TREE`, and -`PJ_SDK_HAS_STRUCTURED_FILE_PICKER`. The companion PlotJuggler 4 work is also -deferred: host-info delivery, QStackedWidget and QTreeWidget bindings, and the -shared picker controller with lease/staging support. This SDK branch defines -the toolkit-neutral contracts only. +Two checks answer two different questions. The generated header +`pj_base/sdk/version.hpp` offers `PJ_SDK_VERSION_AT_LEAST(major, minor, patch)` +for the SDK the plugin is compiled against (see +[REQUIREMENTS.md §2.1](REQUIREMENTS.md#21-compatibility-vocabulary)). Whether the +host offers a feature is a runtime question: use `hostInfo()` / `hostHas(capability)` +above. There are no `PJ_SDK_HAS_*` feature-test macros. This SDK defines the +toolkit-neutral contracts only; the host supplies host-info delivery, the +`QStackedWidget` and `QTreeWidget` bindings, and the file-picker controller. ## Step by Step @@ -421,10 +423,16 @@ catalog discovery must instantiate those dialogs to call `manifest()`. ### 6. Build ```cmake +find_package(plotjuggler_sdk REQUIRED COMPONENTS plugin_sdk) + add_library(my_dialog_plugin SHARED my_dialog.cpp) -target_link_libraries(my_dialog_plugin PRIVATE pj_dialog_sdk) +target_link_libraries(my_dialog_plugin PRIVATE plotjuggler_sdk::plugin_sdk) +pj_configure_plugin(my_dialog_plugin FAMILIES dialog) ``` +`pj_configure_plugin` applies symbol isolation and the export allowlist and +validates `manifest.json` (see `examples/sdk_consumer/CMakeLists.txt`). +`pj_dialog_sdk` is the in-tree name of the dialog target behind `plugin_sdk`. No Qt dependency is needed in the plugin — only the host links Qt. ## Manifest Schema @@ -444,7 +452,10 @@ apply in both forms. | `description` | string | no | Short description of the dialog. | The host validates these keys when it inspects the dialog vtable; manifests -missing a required string are rejected. +missing a required string are rejected. `min_sdk_required`, +`suggested_sdk_version` (with a `floor_test`) and the deprecated +`min_plotjuggler_version` are optional compatibility keys common to every +family; see [REQUIREMENTS.md §2.1](REQUIREMENTS.md#21-compatibility-vocabulary). ## The Reactive Loop @@ -637,10 +648,8 @@ and invalid check-state strings reject the whole `tree_items` update. There is no partial tree. `WidgetDataView::treeItems(name, &validation_error)` returns `nullopt` and fills the reason so an embedding host can diagnose the rejection. -V1 deliberately uses complete keyed snapshots; incremental tree deltas are -deferred until profiling justifies them. A future tree-delta design must reuse -the table protocol's fresh sequence-number gate and all-or-nothing application -rules rather than introduce another ordering contract. +Tree updates are always complete keyed snapshots; the protocol has no +incremental tree deltas. > **Note:** `QTextEdit` and `QTableView` (model-based) are not supported by the > widget binding system. Use `QPlainTextEdit` for plain text or code editing, @@ -1169,12 +1178,11 @@ wd.appendTableRows("faultTable", delta_seq_, wd.updateTableCells("faultTable", delta_seq_, {{2, 1, {duration_ms, "1.2 s"}}}); ``` -> **Host support:** this SDK release ships the protocol (setters + the -> `WidgetDataView::tableDelta` decoder); the PlotJuggler host applies deltas -> from its companion release onward. Older hosts ignore the `table_delta` key -> entirely (unknown keys are skipped by design), so emitting deltas against -> one is a harmless no-op — keep a full `setTableRows` fallback if you must -> support them. +> **Host support:** the SDK provides the setters and the +> `WidgetDataView::tableDelta` decoder. A host that does not apply deltas +> ignores the `table_delta` key entirely (unknown keys are skipped by design), +> so emitting deltas against one is a harmless no-op — keep a full +> `setTableRows` fallback if you must support such hosts. ## Choosing a Base Class @@ -1257,7 +1265,7 @@ The source handle owns the dialog object storage and keeps the plugin DSO loaded, so any borrowed dialog handle must be scoped inside the source handle's lifetime. -See `pj_plugins/docs/data-source-guide.md` for the full DataSource-side +See [data-source-guide.md](data-source-guide.md) for the full DataSource-side documentation of this pattern. ## Common Mistakes diff --git a/pj_plugins/docs/message-parser-guide.md b/pj_plugins/docs/message-parser-guide.md index babf3fdb..d67b59ed 100644 --- a/pj_plugins/docs/message-parser-guide.md +++ b/pj_plugins/docs/message-parser-guide.md @@ -52,8 +52,8 @@ the host, which routes them to the appropriate parser based on encoding name. 1. Subclass `PJ::MessageParserPluginBase`. 2. For new parsers, register `SchemaHandler` entries for functional scalar and/or builtin-object output. For a legacy push-only parser, override - `parse()` instead. Optionally override `bindSchema()`, `saveConfig()`, and - `loadConfig()`. + `parse()` instead. Optionally override `bindSchema()` (call the base + implementation first), `saveConfig()`, and `loadConfig()`. 3. Export with `PJ_MESSAGE_PARSER_PLUGIN(YourClass, R"({"id":"...","name":"...","version":"...","encoding":["..."]})")` 4. Build as a shared library linking `plotjuggler_sdk::plugin_sdk` @@ -295,15 +295,17 @@ if (!status) { ### Route claims and functional v2 -Registering at least one `SchemaHandler` automatically exposes two extension -tables; parser authors do not implement them by hand: +`MessageParserPluginBase` answers two extension ids itself; parser authors do +not implement them by hand: -- `pj.parser_route_claims.v1` classifies exact handler-table coverage. It never +- `pj.parser_route_claims.v1` is always answered. It classifies exact + handler-table coverage (a type without a handler is declined) and never reports wildcard claims. The host synthesizes the universal wildcard scalar claim from each manifest encoding. - `pj.parser_functional.v2` delivers scalar records and canonical-wire objects through synchronous caller-owned sinks, including one splice-eligible object - field. + field. It is answered only when the bound type has a handler; before + `bindSchema()`, when any handler is registered. The host resolves scalar and object routes independently. A parser may keep the scalar route for a type it flattens while another provider owns that type's @@ -312,19 +314,27 @@ canonical-object route. ### Schema binding Override `bindSchema()` to receive schema data before parsing begins. The -default implementation is a no-op. +default implementation records the bound type name, which the `SchemaHandler` +dispatch looks up, so an override must call +`MessageParserPluginBase::bindSchema()` first. ```cpp PJ::Status bindSchema(std::string_view type_name, PJ::Span schema) override { + if (auto status = PJ::MessageParserPluginBase::bindSchema(type_name, schema); + !status) { + return status; + } // Store schema for use during parse(). - type_name_ = std::string(type_name); schema_.assign(schema.begin(), schema.end()); // Build lookup tables, compile descriptors, etc. return PJ::okStatus(); } ``` +If the plugin has its own type-name convention, normalize `type_name` before +delegating, and register handlers under that same canonical form. + `type_name` identifies the encoding-specific message type, such as `"sensor_msgs/msg/Imu"` for `ros2msg` or `"my.package.ImuSample"` for Protobuf. `schema` contains encoding-specific bytes, such as ROS `.msg` definition text @@ -467,6 +477,10 @@ it without instantiating the plugin. | `version` | string | yes | Semver version string. | | `encoding` | array of strings | yes | Case-sensitive registered encodings this parser handles, e.g. `["json"]`, `["protobuf"]`, or `["ros1msg", "ros2msg"]`. The host uses this list to match binding requests to parsers; one parser may register more than one encoding. | +`min_sdk_required`, `suggested_sdk_version` (with a `floor_test`) and the +deprecated `min_plotjuggler_version` are optional compatibility keys common to +every family; see [REQUIREMENTS.md §2.1](REQUIREMENTS.md#21-compatibility-vocabulary). + Example: ```json { @@ -611,10 +625,10 @@ to `1.0/0.0`. Tests can assert uniformly without type dispatch code. `message_parser_library_test.cpp` — real-DSO ownership test proving a decoded object survives parser destruction and plugin unload. -## Builtin-object pipeline (PR #86) +## Builtin-object pipeline Beyond the scalar-column output that `parse()` and the -`sdk::ParserWriteHostService` cover, the SDK adds a second, narrow +`sdk::ParserWriteHostService` cover, the SDK offers a second, narrow output channel for canonical builtin objects returned by the parser and visualised by widgets that never learn the source-specific wire format. @@ -633,9 +647,9 @@ handler.parse_object = ``` - `classifySchema` is the *a-priori* declaration — given a type name + - schema bytes, announce which `BuiltinObjectType` (`kImage`, - `kDepthImage`, `kPointCloud`, `kImageAnnotations`, `kFrameTransforms`, - `kNone`) this schema produces. The host consults the answer **before** it + schema bytes, announce which `BuiltinObjectType` this schema produces + (`kNone` for scalar-only; the enum is in `pj_base/builtin/builtin_object.hpp` + and the catalog in [builtin_type.md](../../docs/builtin_type.md)). The host consults the answer **before** it ever sees the payload, so it can pick the right `ObjectIngestPolicy` for the topic. - Classification is side-effect free. `classifySchema` must not emit parser @@ -652,7 +666,7 @@ handler.parse_object = The types above are plugin-author C++ conveniences, not the binary boundary. After `bindSchema`, `MessageParserPluginBase` advertises functional v2 -when at least one handler exists. Its trampolines: +when the bound type has a handler. Its trampolines: 1. invokes `parseScalars` / `parseObject` inside the plugin DSO; 2. delivers scalar fields through one synchronous caller-owned C sink, or @@ -679,15 +693,15 @@ propagate the input buffer without an initial copy. The DSO boundary still performs one canonical serialization and one host decode. This avoids exporting plugin allocators, destructors, RTTI, STL layout or type-erasure internals. Measure this path for image/point-cloud workloads. -Do not bypass it with direct C++ calls. +New host code reads parser output through it rather than through direct C++ +calls; the direct bridge is described below. Parsers built before SDK 0.22 expose no usable functional extension: pre-0.21 builds expose none, and 0.21 builds expose only the retired v1. New host code negotiates v2. -PlotJuggler 4 has not made that move: it reads objects from every parser by -casting the opaque context to `MessageParserPluginBase*` and calling -`parseObject()`. That direct C++ bridge is deprecated but supported through +PlotJuggler 4 reads objects from every parser by casting the opaque context to +`MessageParserPluginBase*` and calling `parseObject()`. That direct C++ bridge is deprecated but supported through 1.x. For a parser author it has one consequence: a parser and the host must agree on the layout of the builtin struct the parser returns, so a parser that emits a type whose struct changed in an SDK release has to be rebuilt against @@ -698,17 +712,16 @@ it when the host moves to v2. Builtin types and their canonical codecs live under `pj_base/builtin/`, one header per type. `serializeBuiltinObject()` / `deserializeBuiltinObject()` -cover every stable builtin tag; `RobotDescription` now has a canonical wire -codec as well. A zero-byte payload is the valid proto3 default message when the +cover every stable builtin tag, `RobotDescription` included. A zero-byte payload is the valid proto3 default message when the separate type tag is known. `sdk::Image` carries an open-ended `std::string encoding` (`"rgb8"`, `"bgr8"`, `"mono8"`, `"jpeg"`, `"png"`, `"compressedDepth"`, …) so raw and compressed images share a single type. New types are appended without changing the `BuiltinObject` type (its tag-plus-opaque-value erasure is forward-compatible by construction). -Reference implementation: `pj_ported_plugins/parser_ros` — -`SchemaHandler` table driven by a static `catalog()`. See -`PLUGIN_DEVELOPMENT.md` in that repo for a top-down walkthrough. +Reference implementation: `parser_ros` in the official plugins repository +(pj-official-plugins) — a `SchemaHandler` table driven by a static `catalog()`. +`PLUGIN_DEVELOPMENT.md` in that repository gives a top-down walkthrough. ## Common Mistakes diff --git a/pj_plugins/docs/toolbox-guide.md b/pj_plugins/docs/toolbox-guide.md index eed21f3c..07ed9095 100644 --- a/pj_plugins/docs/toolbox-guide.md +++ b/pj_plugins/docs/toolbox-guide.md @@ -1,9 +1,10 @@ # Writing a Toolbox Plugin > **Tracks the v5 plugin ABI** (`PJ_ABI_VERSION == 5`). Toolbox plugins -> read time series via the host's `read_series_arrow` slot, which -> returns a caller-owned `ArrowSchema` + `ArrowArray` pair (no more -> materialised `std::vector`). Wrap returns in +> read time series with `ToolboxHostView::readSeries()` (a typed +> `MaterializedSeriesView`) or `readSeriesArrow()`. Both sit on the host's +> `read_series_arrow` slot, which returns a caller-owned `ArrowSchema` + +> `ArrowArray` pair. Wrap raw Arrow out-params in > `PJ::sdk::ArrowSchemaHolder` / `ArrowArrayHolder` for scope-bound > release. See `ARCHITECTURE.md` for the full ABI rules. @@ -28,8 +29,8 @@ Toolbox plugins are long-lived and UI-driven. They may create data sources or transform existing data into new outputs. DataSource is write-only with a streaming lifecycle; MessageParser is headless with request/response calls. -Link only `pj_base`, without Qt or host internals. Plugins communicate through -a stable C ABI. +Link only the SDK (`plotjuggler_sdk::plugin_sdk`), without Qt or host +internals. Plugins communicate through a stable C ABI. Typical Toolbox use cases: FFT analysis, quaternion rotation, Lua scripting editor, custom data transforms. @@ -42,8 +43,8 @@ editor, custom data transforms. 3. Export with `PJ_TOOLBOX_PLUGIN(YourClass, R"({"id":"...","name":"...","version":"..."})")` 4. If you ship an embedded dialog, also declare it as a `DialogPluginTyped` subclass and add `PJ_DIALOG_PLUGIN(YourDialog, kManifestJson)` -5. Build as a shared library linking `pj_base` (+ `pj_dialog_sdk` if - you have a dialog) +5. Build as a shared library linking `plotjuggler_sdk::plugin_sdk` (which + includes the dialog SDK) and call `pj_configure_plugin(...)` on it For namespaced classes in a static build, provide each getter-symbol token separately: `PJ_TOOLBOX_PLUGIN_NAMED(my::Toolbox, MyToolbox, kManifest)` and @@ -63,8 +64,9 @@ the contract is broader than the other families. - Call `runtimeHost().notifyDataChanged()` after any successful write that the user should see in the UI. Coalesce per logical operation, not per record. For example, call once after producing a new series. -- Wrap all `read_series_arrow` returns in `PJ::sdk::ArrowSchemaHolder` / - `ArrowArrayHolder` so the release callbacks fire on scope exit. +- Wrap raw `readSeriesArrow()` out-params in `PJ::sdk::ArrowSchemaHolder` / + `ArrowArrayHolder` so the release callbacks fire on scope exit (`readSeries()` + does it for you). - Persist tool state in `saveConfig()` so a layout reload restores the same view. The host has no ambient persistence. - Call each host method only from the threads its tag allows (see @@ -74,8 +76,9 @@ the contract is broader than the other families. - Throw exceptions across virtual overrides. - Depend on `onDataChanged()` for correctness: it is advisory and a host may never call it. -- Hold an `ArrowSchema*` / `ArrowArray*` past the scope of the holders that - own them — the host may reuse the underlying buffers. +- Hold an `ArrowSchema*` / `ArrowArray*` (or a `MaterializedSeriesView`'s + pointers) past the scope of the holders that own them — the host may reuse the + underlying buffers. - Treat `catalogSnapshot()` as live data. Snapshots are immutable views at acquisition time; reacquire after writes if you need to see your own changes. @@ -155,11 +158,17 @@ PJ_TOOLBOX_PLUGIN(MyToolbox, ### 4. Build ```cmake +find_package(plotjuggler_sdk REQUIRED COMPONENTS plugin_sdk) + add_library(my_toolbox_plugin SHARED my_toolbox.cpp) -target_link_libraries(my_toolbox_plugin PRIVATE pj_base) +target_link_libraries(my_toolbox_plugin PRIVATE plotjuggler_sdk::plugin_sdk) +pj_configure_plugin(my_toolbox_plugin FAMILIES toolbox) ``` -No other dependencies are needed. +If the toolbox embeds a dialog, list `toolbox dialog` in `FAMILIES` +(`examples/sdk_consumer/CMakeLists.txt` shows a plugin with a dialog). No other +dependencies are needed; `pj_base` and `pj_dialog_sdk` are the in-tree names of +the targets behind `plugin_sdk`. ## Lifecycle @@ -203,19 +212,30 @@ data store. | `appendArrowStream(topic, stream, ts_col)` | Hand an `ArrowArrayStream*` (Arrow C Data Interface) to the host for bulk ingest. Same ownership rule as the source write path: success transfers, failure retains. | | `catalogSnapshot()` | Acquire a read-only snapshot of all data sources, topics, and fields. | | `hasCatalogSnapshotV2()`, `catalogSnapshotV2()` (0.38.0) | Like `catalogSnapshot()`, plus every object topic (`objectTopics()`: dataset, builtin type, metadata JSON, and the entry count and raw time range the store retains now). The two halves are not read atomically: a topic created between the reads may appear in one half only. Derived and marker topics are included; tell them apart by metadata (`kDerivedMetadataKey`, the marker keys in `object_topic_metadata.hpp`), never by name. | -| `readSeriesArrow(field, schema*, array*)` | Read one field's full time series into host-owned `ArrowSchema` + `ArrowArray` out-params (two columns: `timestamp` int64 ns, then the typed field value). | +| `readSeries(field)` | Read one field's full time series as a `MaterializedSeriesView`: `rowCount()`, `timestamps()`, typed `valuesAs*()` pointers, `nullCount()` / `isNull(row)`. The normal read path; see *Reading a series*. | +| `readSeriesArrow(field, schema*, array*)` | The same data as host-owned `ArrowSchema` + `ArrowArray` out-params (two columns: `timestamp` int64 ns, then the typed field value); `readSeries` wraps it. | | `registerObjectTopic(source, name, type[, extra_metadata])` | Register a built-in media/object topic under a data source. The typed overload emits the canonical `builtin_object_type` renderer metadata and returns an `ObjectTopicHandle`. | | `registerObjectTopic(source, name, metadata_json)` | Raw-metadata overload for custom or untyped object topics. The store retains the JSON verbatim. | +| `registerObjectTopicOnDataset(dataset, name, type[, extra_metadata])` | Register an object topic on an EXISTING dataset (by `DatasetId`), for annotations such as plot markers on data another source loaded. Idempotent per name: it returns the existing handle. A raw `metadata_json` overload exists. | +| `setObjectTopicRetention(topic, max_entries)` | Keep only the last `max_entries` snapshots of an object topic (0 = unlimited). A producer that republishes a whole set under one topic passes 1. | | `pushOwnedObject(topic, ts, payload)` | Eager-push serialized object bytes into the ObjectStore under an object topic; the host copies the bytes, so the plugin's buffer is free immediately after the call returns. | +Every `toolboxHost()` slot is `[stream-thread]`: it may be called from the +toolbox's own worker thread, one thread at a time per host object (see +[Threading Model](#threading-model)). The object-topic slots are tail slots and +return an error on a host that predates them. + ### Runtime host — control plane Access via `runtimeHost()`. Use this for diagnostics and UI refresh. -| Method | Purpose | -|---|---| -| `reportMessage(level, text)` | Send info/warning/error to the host UI log. | -| `notifyDataChanged()` | Tell the host that data was modified; refresh UI. Idempotent and cheap; coalesce per logical operation, not per record. | +| Method | Thread | Purpose | +|---|---|---| +| `reportMessage(level, text)` | `[thread-safe]` | Send info/warning/error to the host UI log. | +| `notifyDataChanged()` | `[thread-safe]` | Tell the host that data was modified; refresh UI. Idempotent and cheap; coalesce per logical operation, not per record. | +| `createParserIngest(id)` / `createDatasetIngest(id)` | `[thread-safe]` | Create (or fetch) the ingest context of a toolbox-created data source (`id` is the `.id` of `createDataSource`'s handle). Both return a view over the same context: `ensureParserBinding` / `pushMessage` for delegated parsing, progress and stop for direct writes. Drive the returned view from one worker thread at a time. | +| `releaseParserIngest(id)` / `releaseDatasetIngest(id)` | `[thread-safe]` | Flush and destroy the context. Idempotent. Do not use the view afterwards. | +| `discardParserIngest(id)` | `[thread-safe]` | Abort a provisional ingest without committing: staged source records are dropped and any capture is never published. Quiesce every user of the view first and serialize release/discard per dataset id. An older host returns an error; `releaseParserIngest` is not a substitute. | ### Playback, viewport, and owned tabs (SDK 0.28.0) @@ -353,38 +373,70 @@ The contract (lifetime, time flags, report schema, errors) is the (`pj_plugins/sdk/evaluation_report.hpp`); COMPLETED and FAILED share its shape. Read `stopped` and `error`: an empty `bundles` list alone is not "no sample". -### Reading a series via Arrow +### Reading a series -`readSeriesArrow()` is the only read path in v4 — it returns -`ArrowSchema` + `ArrowArray` out-params populated by the host. -Wrap the out-params in the RAII holders from `pj_base/sdk/arrow.hpp` -so they are released automatically at scope exit: +`readSeries(field)` is the normal read path. It returns a `MaterializedSeriesView` +that owns the host's Arrow data and exposes it through typed accessors: ```cpp -#include - void MyToolbox::runFft(PJ::sdk::FieldHandle field) { - PJ::sdk::ArrowSchemaHolder schema; - PJ::sdk::ArrowArrayHolder array; - - auto status = toolboxHost().readSeriesArrow(field, schema.out(), array.out()); - if (!status) { + auto series = toolboxHost().readSeries(field); + if (!series) { runtimeHost().reportMessage(PJ::ToolboxMessageLevel::kError, - "readSeriesArrow failed: " + status.error()); + "readSeries failed: " + series.error()); + return; + } + + const PJ::Span ts = series->timestamps(); // ns since Unix epoch + const double* values = series->valuesAsFloat64(); // nullptr if the field is not float64 + if (values == nullptr) { return; } + const bool has_nulls = series->nullCount() != 0; + for (size_t row = 0; row < series->rowCount(); ++row) { + if (has_nulls && series->isNull(row)) { + continue; // the value slot of a null is unspecified (hosts zero it) + } + consume(ts[row], values[row]); + } +} +// The view releases the Arrow data at scope exit. +``` + +`type()` reports the value column's primitive type; each `valuesAs*()` returns +`nullptr` when the column has another type. A null sample is one the source did +not provide (an empty CSV cell, an absent optional field). Its `valuesAs*()` slot +is unspecified, so skip rows where `isNull(row)` whenever `nullCount() != 0` +instead of reading the slot as a zero. The pointers and span alias the Arrow +buffers and are valid while the view is alive. + +`readSeriesArrow()` returns the same data as raw `ArrowSchema` + `ArrowArray` +out-params, for code that speaks Arrow directly (DuckDB, Polars, pandas via +PyCapsule, ...). Wrap the out-params in the RAII holders from +`pj_base/sdk/arrow.hpp` so they are released at scope exit: + +```cpp +#include + +PJ::sdk::ArrowSchemaHolder schema; +PJ::sdk::ArrowArrayHolder array; - // array.get() now points to a two-column Arrow struct: - // column 0: "timestamp" — int64 nanoseconds since Unix epoch - // column 1: — typed to the field's primitive type - // Walk children[0]->buffers / children[1]->buffers per Arrow spec, - // or hand array.get() directly to analytics code that speaks Arrow - // (DuckDB, Polars, pandas via PyCapsule, …). +auto status = toolboxHost().readSeriesArrow(field, schema.out(), array.out()); +if (!status) { + runtimeHost().reportMessage(PJ::ToolboxMessageLevel::kError, + "readSeriesArrow failed: " + status.error()); + return; } + +// array.get() points to a two-column Arrow struct: +// column 0: "timestamp" — int64 nanoseconds since Unix epoch +// column 1: — typed to the field's primitive type +// Walk children[0]->buffers / children[1]->buffers per the Arrow spec; the +// value column's validity bitmap marks the null samples. // schema and array are released here by their destructors. ``` -**Bulk-write output:** pair `readSeriesArrow` with `appendArrowStream` +**Bulk-write output:** pair `readSeries` / `readSeriesArrow` with `appendArrowStream` to round-trip data through a transform. Use the rvalue-ref overload: ```cpp @@ -397,7 +449,7 @@ auto status = toolboxHost().appendArrowStream( ### Writing object payloads (images, point clouds, annotations) -`readSeriesArrow` / `appendArrowStream` cover *scalar* columns. +`readSeries` / `appendArrowStream` cover *scalar* columns. Use object writes for **canonical media**, such as images, point clouds and annotation overlays. They route to the host `ObjectStore`, not the columnar engine: @@ -456,6 +508,7 @@ PJ::Status loadConfig(std::string_view json) override { | Flag | Value | When to use | |---|---|---| | `kToolboxCapabilityHasDialog` | `1 << 0` | Plugin provides a persistent UI panel | +| `kToolboxCapabilityNonModalDialog` | `1 << 1` | Show the dialog as a non-modal panel rather than a modal pop-up; set it together with `kToolboxCapabilityHasDialog` | ## Manifest Schema @@ -471,6 +524,10 @@ it without instantiating the plugin. | `badge` | string | no | Short label (e.g. `"AI"`, at most 8 characters advised) a host may show next to objects the plugin creates (0.38.0). Absent: host policy (e.g. the plugin name). Read as `PluginDescriptor::badge`. | | `capabilities` | string[] | no | Declarative tags; a host ignores tags it does not know. `"custom_topics_editor"` (`kCustomTopicsEditorCapability`, 0.38.0) marks the editor of the host's Custom topics. Read with `PluginDescriptor::hasCapability`. | +`min_sdk_required`, `suggested_sdk_version` (with a `floor_test`) and the +deprecated `min_plotjuggler_version` are optional compatibility keys common to +every family; see [REQUIREMENTS.md §2.1](REQUIREMENTS.md#21-compatibility-vocabulary). + Example: ```json { @@ -515,7 +572,7 @@ The normative table is on `PJ_toolbox_vtable_t` in `create/release/discardParserIngest` (dataset-ingest) calls are `[thread-safe]`. The ingest view they return is driven one thread at a time. - Every other service (object reads, playback, plot tabs, viewport, data - processors, colormaps, ...) is `[main-thread]` unless its slot says + processors, ...) is `[main-thread]` unless its slot says otherwise: call it from a callback, not from a thread you spawned. ## Testing @@ -550,7 +607,7 @@ TEST(MyToolboxTest, Basic) { The store captures `appendRecord` writes and counts `createDataSource` and `notifyDataChanged` calls. `flatRecords()` gives a flat (timestamp, name, value) view. `writtenRecords()` preserves nested rows of fields. -See `pj_plugins/testing/toolbox_test_store.hpp` for the full API. +See `pj_plugins/include/pj_plugins/testing/toolbox_test_store.hpp` for the full API. ## Examples diff --git a/pj_plugins/docs/widget-guide.md b/pj_plugins/docs/widget-guide.md index 87f12fc5..b5fe4df5 100644 --- a/pj_plugins/docs/widget-guide.md +++ b/pj_plugins/docs/widget-guide.md @@ -145,6 +145,8 @@ has no entry, a `multiple` input has one per wired series (`slot`). Entries are ordered by `input` index, then `slot`, but dispatch on `v.input` (and `v.slot`), never on the position in `values`. `kind == PJ_WIDGET_VALUE_NONE` where the wire has no sample at or before `time`. `time` is on the host's display axis, in seconds. +`flags` is reserved and no bit is defined: a host built against SDK 1.0 or later +writes 0, an older host leaves it unset, so a widget ignores it. `label` is the host's display name of the series behind the wire (for example `topic/field`), set for every kind so a widget that lists its wires, like a table of values, can name each row; it may be empty. Every pointer in a value @@ -229,9 +231,7 @@ Because the SDK stores `settings` as raw JSON, the manifest decoder (`PluginDescriptor::widgets[].settings_json`) checks only that it is an array. The host interprets the entries. -The form is declarative on purpose. A widget that outgrows it is not stuck: a -later tail slot (`create_settings_qwidget`, a settings editor the plugin builds) -or a dynamic `settings_schema` can be added without touching protocol v1. +The form is declarative on purpose: the plugin builds no settings editor. ## Qt rules @@ -272,6 +272,20 @@ or a dynamic `settings_schema` can be added without touching protocol v1. and removal take effect at the next start. - A widget only consumes. It creates no series and no datasets. +## Testing + +`PJ::sdk::testing::FakeWidgetHost` (`pj_plugins/sdk/testing/fake_widget_host.hpp`) +is an in-memory `pj.widget_host.v1` host for unit tests, served through the real +`WidgetHostBridge`, so the widget sees the C ABI. Register `makeHost()` as +`sdk::WidgetHostService`. Add samples per wire with `wire()` and `insert()`; +`values(tracker)` builds the array `on_update` would carry at a display time. +`query_history` follows the protocol (display-time seconds, inclusive range, +ascending order, `max_samples == 0` meaning `default_cap`, `out_total` before +any cap). Samples are stored at integer nanoseconds as a real host stores them, +so put them at epoch nanoseconds to test history reads at epoch-scale display +times. `queries`, `messages` and `inputs_changed_calls` record what the widget +asked for and reported. The fake must outlive the handle made from it. + ## Host side Hosts use `PJ::WidgetLibrary::load(path)` (or `loadStatic(vtable)` for a diff --git a/pj_plugins/widget_protocol/CLAUDE.md b/pj_plugins/widget_protocol/CLAUDE.md index 360358ff..9e0d627a 100644 --- a/pj_plugins/widget_protocol/CLAUDE.md +++ b/pj_plugins/widget_protocol/CLAUDE.md @@ -44,5 +44,8 @@ Local traps not visible from the headers: `plugin_id/type`. - ABI layout pins live in `tests/widget_abi_layout_sentinels_test.cpp` (not in `pj_base`: the family is a `pj_plugins` module). +- Plugin tests use `PJ::sdk::testing::FakeWidgetHost` + (`include/pj_plugins/sdk/testing/fake_widget_host.hpp`), an in-memory + `pj.widget_host.v1`; extend it rather than writing a second fake host. See ../CLAUDE.md for module context.