Export annotations

docs/configuration.rst

@@ -567,6 +567,41 @@ subclass) that unwinds the stack and releases the lock via ``finally`` /
        detected.  Useful when running under a process supervisor (e.g. systemd) that handles the restart externally.
 
 
+Resource Monitoring
+~~~~~~~~~~~~~~~~~~~~
+
+Enabled by default: a background thread samples CPU, memory, disk, network,
+GPU (NVML), and process-level usage and logs each value as a signal, visible
+in Weights Studio like any other loss/metric curve. See
+:doc:`resource_monitoring` for the full metric list, the ``resource_monitoring.yaml``
+schema, and category-level toggles.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 35 15 50
+
+   * - Variable
+     - Default
+     - Description
+   * - ``WEIGHTSLAB_DISABLE_RESOURCE_MONITORING``
+     - ``0``
+     - If set to ``1`` / ``true`` / ``yes`` / ``on``, disables resource
+       monitoring entirely.
+   * - ``WL_RESOURCE_MONITOR_INTERVAL_SECONDS``
+     - ``15``
+     - How often (seconds) the monitor samples and logs a new batch of metrics.
+   * - ``WL_RESOURCE_MONITOR_CATEGORIES``
+     - *(unset — all on)*
+     - Comma-separated category allowlist (``cpu``, ``memory``, ``disk``,
+       ``network``, ``process``, ``gpu``). Anything not listed is disabled.
+   * - ``WL_RESOURCE_MONITOR_DISK_PATH``
+     - OS root
+     - Filesystem path reported by the ``disk`` category's usage metrics.
+   * - ``WL_RESOURCE_MONITOR_CONFIG_PATH``
+     - *(empty)*
+     - Optional directory override for ``resource_monitoring.yaml``.
+
+
 Data and Cache
 ~~~~~~~~~~~~~~
 

docs/export.rst

@@ -0,0 +1,137 @@
+Annotation Export
+==================
+
+WeightsLab can export bounding-box/segmentation annotations to a
+relabeling-tool format, so a dataset (or a slice of one) can be handed off
+for an outsourced relabeling pass. Three ways to trigger it, all backed by
+the same code path:
+
+- **Weights Studio UI** — an "Export" button next to Save/Grid settings, with
+  a format picker (CVAT / Label Studio / V7). Triggers a browser download.
+- **CLI** — ``weightslab export`` connects over gRPC to a running experiment,
+  same as ``weightslab cli``.
+- **Python** — :func:`wl.export_annotations`, called in-process (no gRPC
+  round-trip needed since it already runs alongside the registered dataframe).
+
+Supported formats
+------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 40 40
+
+   * - Format
+     - Output shape
+     - Schema reference
+   * - ``cvat``
+     - A single CVAT XML 1.1 file (one ``<image>`` element per sample, with
+       ``<box>``/``<polygon>`` children).
+     - `CVAT XML format <https://opencv.github.io/cvat/docs/manual/advanced/xml_format/>`_
+   * - ``label_studio``
+     - A single JSON file — a list of "tasks", each with a ``result`` list of
+       ``rectanglelabels``/``polygonlabels`` entries. Coordinates are
+       percentages (0-100) of the image's width/height, per Label Studio's
+       convention.
+     - `Label Studio export format <https://labelstud.io/guide/export.html#JSON>`_
+   * - ``v7``
+     - A zip of one Darwin JSON 2.0 file per image (V7 matches annotations to
+       images by filename on import).
+     - `Darwin JSON reference <https://docs.v7labs.com/reference/darwin-json>`_
+
+Bounding boxes are exported for every format. Segmentation masks are
+converted to polygons via OpenCV contour extraction — this needs the
+optional ``export`` extra:
+
+.. code-block:: bash
+
+   pip install weightslab[export]
+
+Bounding-box-only export needs no extra dependency; if OpenCV isn't
+installed, segmentation samples still export their boxes and a warning is
+logged once, rather than failing the whole export.
+
+Usage
+------
+
+**Python**
+
+.. code-block:: python
+
+   import weightslab as wl
+
+   wl.export_annotations("cvat")                              # everything, under root_log_dir
+   wl.export_annotations("label_studio", "val.json", origin="val_loader")
+   wl.export_annotations("v7", "out/", class_names=["bg", "cat", "dog"])
+   wl.export_annotations("cvat", tags=["ToReview"])           # only samples tagged ToReview
+
+See :doc:`user_functions` for the full :func:`wl.export_annotations` reference.
+
+**CLI**
+
+.. code-block:: bash
+
+   weightslab export --format cvat                      # everything, CVAT XML, into "."
+   weightslab export -f v7 out/ --origin val_loader      # V7/Darwin, val split only
+   weightslab export -f cvat --tag ToReview              # only samples tagged ToReview
+
+Connects over gRPC to a running experiment (``127.0.0.1:50051`` by default),
+same as ``weightslab cli``. See :doc:`user_commands` for every flag.
+
+**Weights Studio UI**
+
+The "Export" button sits next to the Save and Grid settings controls in the
+Details panel. Clicking it opens a small format picker (CVAT / Label
+Studio / V7) with an optional tag selector; the chosen format (and tags, if
+any) trigger an ``ExportAnnotations`` gRPC call and the response downloads
+as a file in your browser.
+
+**In-app chat agent**
+
+Because the chat agent (see :doc:`agent`) has general tool access to the
+live experiment process, you can also just ask for this in plain language --
+e.g. "export the samples tagged ToReview to CVAT format for relabeling" --
+and it calls :func:`wl.export_annotations` with the matching ``tags=``
+argument itself. No special wiring is needed beyond the API existing.
+
+Filtering by tag
+------------------
+
+All three entry points accept a tag filter (``tags=`` in Python, ``--tag`` on
+the CLI, repeatable; the tag picker in the UI) that restricts the export to
+samples carrying **any** of the given tags -- boolean tags set via
+:func:`wl.tag_samples` or categorical values set via
+:func:`wl.set_categorical_tag` both work, since they share the same
+``tag:<name>`` column. Omit it to export every sample. This is the mechanism
+for a "send only what needs another look" relabeling handoff, e.g. tagging
+uncertain samples as ``ToReview`` during data exploration and exporting just
+that subset.
+
+How annotations are resolved
+------------------------------
+
+Every export path collects annotations from the same registered dataframe
+that backs the rest of WeightsLab (`get_dataframe()`), grouping the
+``(sample_id, annotation_id)`` multi-index rows by sample:
+
+- **Boxes** — read from the ``target`` (or ``prediction``, with
+  ``use_predictions=True``) column when it holds coordinate-shaped data
+  (``(x1, y1, x2, y2[, conf][, cls])``), whether that's a single box per
+  sample or several boxes exploded across annotation rows.
+- **Masks -> polygons** — read from the same column when it holds a dense
+  ``(H, W)`` array (pixel value = class id); one polygon per connected
+  region per class id.
+
+Two real gaps in the current data model drive the "best effort" behavior
+below — call these out explicitly if an export looks wrong:
+
+- **No dedicated class-id -> name registry.** Labels are resolved, in order:
+  an explicit ``class_names`` argument; else a ``class_names`` attribute on
+  the dataset object backing the relevant split; else ``"class_<id>"``.
+- **No per-sample stored image path or dimensions.** A real image path is
+  best-effort resolved from a few common dataset attribute names
+  (``image_paths``, ``img_files``, ``images``, ``imgs``, ``files``,
+  ``samples``); dimensions come from that file (via Pillow) or, for
+  segmentation samples, directly from the mask's own shape. When no path
+  resolves, the exported filename is synthetic (``sample_<id>.jpg``) — **no
+  image file is copied or embedded**, so you must ensure the filenames you
+  upload to CVAT/Label Studio/V7 match the ones in the export.

docs/index.rst

@@ -109,6 +109,12 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c
 
       Ask the agent for a branded HTML report: signal health plots, dataset stats, and a written analysis.
 
+   .. grid-item-card:: Annotation Export
+      :link: export
+      :link-type: doc
+
+      Export bounding boxes and segmentation masks to CVAT, Label Studio, or V7 for relabeling.
+
    .. grid-item-card:: gRPC Communication
       :link: grpc/index
       :link-type: doc
@@ -156,6 +162,8 @@ Weightslab is a Python SDK to inspect, monitor, and edit training behavior for c
    agent
    hyperparameters
    logger
+   resource_monitoring
+   export
    checkpointing
    experiment_reports
    .. weights_studio

docs/resource_monitoring.rst

@@ -0,0 +1,174 @@
+Resource Monitoring
+====================
+
+WeightsLab automatically tracks system and process resource usage — CPU,
+memory, disk, network, and GPU — for the whole lifetime of a running
+backend, and logs every value through the same signal pipeline used for
+losses and metrics. The resulting curves appear in Weights Studio exactly
+like any other signal, under graph names prefixed with ``resource/``.
+
+This is enabled by default and requires no setup. It runs independently of
+the training loop — metrics are sampled on a wall-clock interval, not tied
+to training steps, so they keep updating even while training is paused or
+between experiments.
+
+What gets logged
+-----------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 20 45 35
+
+   * - Category
+     - Metrics
+     - Signal names
+   * - ``cpu``
+     - System-wide CPU utilization (%)
+     - ``resource/cpu/utilization_percent``
+   * - ``memory``
+     - System-wide memory utilization (%)
+     - ``resource/memory/system_utilization_percent``
+   * - ``disk``
+     - Disk usage (%, GB) of ``disk_path``; cumulative bytes read/written (MB)
+     - ``resource/disk/utilization_percent``, ``resource/disk/utilization_gb``,
+       ``resource/disk/read_mb``, ``resource/disk/written_mb``
+   * - ``network``
+     - Cumulative bytes sent/received
+     - ``resource/network/bytes_sent``, ``resource/network/bytes_received``
+   * - ``process``
+     - CPU %, thread count, RSS memory (MB/%), and system memory available (MB)
+       for the WeightsLab backend process itself
+     - ``resource/process/cpu_utilization_percent``,
+       ``resource/process/cpu_threads_in_use``,
+       ``resource/process/memory_in_use_mb``,
+       ``resource/process/memory_in_use_percent``,
+       ``resource/process/memory_available_mb``
+   * - ``gpu``
+     - Per-device memory/SM clock (MHz), memory used (bytes/%), temperature (°C)
+     - ``resource/gpu/<index>/memory_clock_mhz``,
+       ``resource/gpu/<index>/sm_clock_mhz``,
+       ``resource/gpu/<index>/memory_allocated_bytes``,
+       ``resource/gpu/<index>/memory_allocated_percent``,
+       ``resource/gpu/<index>/temperature_celsius``
+
+CPU/memory/disk/network/process metrics come from `psutil
+<https://psutil.readthedocs.io/>`_. GPU metrics come from NVML (the
+``pynvml`` import name, shipped by the ``nvidia-ml-py`` package) and are
+per-device — multi-GPU machines get one full set of ``gpu`` signals per
+device index. On a machine with no NVIDIA driver, the ``gpu`` category
+degrades silently to a no-op; every other category is unaffected.
+
+Because sampling is wall-clock driven rather than step-driven, the logged
+"step" for every resource signal is **elapsed seconds since the monitor
+started** — so these curves plot against time, not batch count.
+
+Disabling monitoring
+---------------------
+
+To turn everything off, either:
+
+.. code-block:: bash
+
+   export WEIGHTSLAB_DISABLE_RESOURCE_MONITORING=1
+
+or set ``enabled: false`` in ``resource_monitoring.yaml`` (see below) —
+the YAML value wins if both are set.
+
+Enabling only specific categories
+----------------------------------
+
+Two ways to restrict which categories are sampled:
+
+- **Env var**, comma-separated category list (anything not listed is
+  disabled):
+
+  .. code-block:: bash
+
+     export WL_RESOURCE_MONITOR_CATEGORIES=cpu,memory,gpu
+
+- **YAML file** (``resource_monitoring.yaml``), per-category booleans —
+  lets you leave everything on and disable just one or two:
+
+  .. code-block:: yaml
+
+     resource_monitoring:
+       categories:
+         disk: false
+         network: false
+
+Config file
+------------
+
+Create ``resource_monitoring.yaml`` at your repository root (next to
+``agent_config.yaml``, if you use the agent) to control monitoring without
+touching env vars:
+
+.. code-block:: yaml
+
+   resource_monitoring:
+     enabled: true          # master switch
+     interval_seconds: 15   # how often (seconds) to sample + log a batch of metrics
+     disk_path: "/"         # filesystem path reported by the `disk` category
+     categories:
+       cpu: true
+       memory: true
+       disk: true
+       network: true
+       process: true
+       gpu: true
+
+Config lookup order
+~~~~~~~~~~~~~~~~~~~~
+
+1. ``<WL_RESOURCE_MONITOR_CONFIG_PATH>/.resource_monitoring.yaml`` /
+   ``<WL_RESOURCE_MONITOR_CONFIG_PATH>/resource_monitoring.yaml`` (if
+   ``WL_RESOURCE_MONITOR_CONFIG_PATH`` is set)
+2. Repository-level ``resource_monitoring.yaml``
+3. Package-level ``resource_monitoring.yaml``
+4. Current working directory ``resource_monitoring.yaml``
+
+Any key present in the YAML file overrides the corresponding env var (or
+the built-in default); keys the file omits keep whatever the env var (or
+default) already resolved to.
+
+Environment variables
+-----------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 35 15 50
+
+   * - Variable
+     - Default
+     - Description
+   * - ``WEIGHTSLAB_DISABLE_RESOURCE_MONITORING``
+     - ``0``
+     - If set to ``1`` / ``true`` / ``yes`` / ``on``, disables resource
+       monitoring entirely (no background thread is started).
+   * - ``WL_RESOURCE_MONITOR_INTERVAL_SECONDS``
+     - ``15``
+     - How often (seconds) the monitor samples and logs a new batch of
+       metrics. Clamped to a 1-second floor.
+   * - ``WL_RESOURCE_MONITOR_CATEGORIES``
+     - *(unset — all categories on)*
+     - Comma-separated list of categories to enable
+       (``cpu``, ``memory``, ``disk``, ``network``, ``process``, ``gpu``).
+       When set, any category not listed is disabled.
+   * - ``WL_RESOURCE_MONITOR_DISK_PATH``
+     - OS root (``/`` or ``C:\``)
+     - Filesystem path reported by the ``disk`` category's usage metrics.
+   * - ``WL_RESOURCE_MONITOR_CONFIG_PATH``
+     - *(empty)*
+     - Optional directory override for ``resource_monitoring.yaml``. When
+       set, WeightsLab first checks
+       ``<WL_RESOURCE_MONITOR_CONFIG_PATH>/resource_monitoring.yaml`` before
+       the built-in fallback paths.
+
+Where it runs
+---------------
+
+The monitor is started once, alongside the watchdog, from
+``grpc_serve()`` (``weightslab/trainer/trainer_services.py``) — so it covers
+the whole backend server lifetime, not just active training. It is a single
+daemon thread (``WL-ResourceMonitor``) and stops automatically when the
+process exits.