Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Please check the supported model table in [README.md](README.md) before starting

If you want to add support for a model family that is already listed, please focus on improving the existing implementation instead of opening a duplicate port.

When a loader is registered (or parked), keep the **package catalog** in sync. Installable `ModelPackage` entries must not advertise families that `audiocpp_cli --list-loaders` does not expose. Follow the checklist in [docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md) and run:
When a loader is registered (or parked), keep the **package catalog** in sync. Installable `ModelPackage` entries must not advertise families that `audiocpp_cli --list-loaders` does not expose. If the family takes another model path at load/session time (aligner, VAD, codec, ASR peer, …), override `advertised_companions()` so `--list-loaders --json` exposes those peers. Follow the checklist in [docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md) and run:

```bash
python3 tools/check_loader_catalog_sync.py --self-test
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ Useful CLI features:
- `--help` with `--task` shows task-oriented help
- `--help` with `--model <path>` and optional `--family <family>` shows model-owned request, session, and load options
- `--inspect` prints discovered configs, weights, and capabilities
- `--list-loaders` prints registered model families (`--json` for the machine-readable contract)
- `--list-loaders` prints registered model families (`--json` for the machine-readable contract: tasks, endpoints, and optional runtime `companions`)
- `python tools/model_manager.py list --json` prints installable packages; keep it synced with loaders ([docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md))
- `--batch-text-file <txt>` runs one offline request per non-empty line
- `--batch-text-dir <dir>` runs one offline request per `.txt`, `.md`, or `.json` file, normalizing each file as one paragraph
Expand Down Expand Up @@ -369,7 +369,7 @@ The tool supports three main commands:
- `info <package> --json` prints machine-readable package details
- `install` downloads or converts one package into a models root

The CLI also exposes the runtime loader catalog with `audiocpp_cli --list-loaders --json`, including task and endpoint metadata added by [PR #74](https://github.com/0xShug0/audio.cpp/pull/74).
The CLI also exposes the runtime loader catalog with `audiocpp_cli --list-loaders --json` (`schema_version` 1). That export includes task and endpoint metadata ([PR #74](https://github.com/0xShug0/audio.cpp/pull/74)) and optional per-loader `companions` for runtime peer models (forced aligner, VAD, codec, and similar path options). Integrators should read companions from this contract rather than hard-coding family graphs. See [docs/usage.md](docs/usage.md#discovering-loaders-and-companions) and [docs/maintainers/loader_and_catalog.md](docs/maintainers/loader_and_catalog.md).

Recommended top-level install packages:

Expand Down
45 changes: 45 additions & 0 deletions app/cli/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -584,6 +584,46 @@ int audiocpp_cli_main(int argc, char ** argv) {
for (const auto & endpoint : row.api_endpoints) {
endpoints.push_back(engine::io::json::Value::make_string(endpoint));
}
engine::io::json::Value::Array companions;
for (const auto & companion : row.companions) {
engine::io::json::Value::Object companion_object;
companion_object.emplace("id", engine::io::json::Value::make_string(companion.id));
companion_object.emplace(
"config_key",
engine::io::json::Value::make_string(companion.config_key));
companion_object.emplace(
"scope",
engine::io::json::Value::make_string(companion.scope));
if (!companion.target_family.empty()) {
companion_object.emplace(
"target_family",
engine::io::json::Value::make_string(companion.target_family));
}
companion_object.emplace(
"optional",
engine::io::json::Value::make_bool(companion.optional));
if (!companion.required_for.empty()) {
engine::io::json::Value::Array required_for;
for (const auto & key : companion.required_for) {
required_for.push_back(engine::io::json::Value::make_string(key));
}
companion_object.emplace(
"required_for",
engine::io::json::Value::make_array(std::move(required_for)));
}
if (!companion.label.empty()) {
companion_object.emplace(
"label",
engine::io::json::Value::make_string(companion.label));
}
if (!companion.bundled_default.empty()) {
companion_object.emplace(
"bundled_default",
engine::io::json::Value::make_string(companion.bundled_default));
}
companions.push_back(
engine::io::json::Value::make_object(std::move(companion_object)));
}
engine::io::json::Value::Object loader_object;
loader_object.emplace("tasks", engine::io::json::Value::make_object(std::move(tasks_object)));
loader_object.emplace(
Expand All @@ -592,6 +632,11 @@ int audiocpp_cli_main(int argc, char ** argv) {
loader_object.emplace(
"api_endpoints",
engine::io::json::Value::make_array(std::move(endpoints)));
if (!companions.empty()) {
loader_object.emplace(
"companions",
engine::io::json::Value::make_array(std::move(companions)));
}
loaders_object.emplace(
row.family,
engine::io::json::Value::make_object(std::move(loader_object)));
Expand Down
2 changes: 2 additions & 0 deletions docs/asr.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@

This page covers ASR models. Detailed Qwen3 ASR and forced-alignment notes live in [Qwen3 models](models/qwen3.md).

Runtime peer models (forced aligner, VAD, and similar path options) are advertised on `audiocpp_cli --list-loaders --json` as per-loader `companions`. See [Discovering loaders and companions](usage.md#discovering-loaders-and-companions).

Common CLI shape:

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/community_models/outetts.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ audiocpp_cli --task clon --family outetts \
--max-tokens 1024 --out cloned.wav
```

`--task tts` with the same `--voice-ref` and `--reference-text` options also enables speaker conditioning, which is useful for clients that expose one TTS route. Safetensors packages and older OuteTTS GGUFs do not contain the aligner. For those models, pass `outetts.aligner_model_path`; cloning fails clearly instead of using unreliable estimated word boundaries.
`--task tts` with the same `--voice-ref` and `--reference-text` options also enables speaker conditioning, which is useful for clients that expose one TTS route. Safetensors packages and older OuteTTS GGUFs do not contain the aligner. For those models, pass `outetts.aligner_model_path`; cloning fails clearly instead of using unreliable estimated word boundaries. That peer is also advertised under `loaders.outetts.companions` in `audiocpp_cli --list-loaders --json` (see [usage.md](../usage.md#discovering-loaders-and-companions)).

The installer places `DAC.speech.v1.0` and `Qwen3-ForcedAligner-0.6B` beside the OuteTTS directory. It converts the official DAC checkpoint to a safe tensor source. To do that conversion manually:

Expand Down
40 changes: 38 additions & 2 deletions docs/maintainers/loader_and_catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,44 @@ Do **not** leave a live `SnapshotSource` for a commented-out loader.
Optional catalog↔registry family renames for parked stubs go in
`PARKED_FAMILY_ALIASES` in the sync check (collapse to one id when re-enabling).

## Runtime companions (peer models)

Some loaders take **another model path** at load/session time (forced aligner,
VAD, codec, ASR for best-of-N, etc.). Integrators need a machine-readable way to
discover those peers — not by hard-coding family graphs in UIs.

Advertise them from the loader via `advertised_companions()`. They appear under
each loader in `audiocpp_cli --list-loaders --json` as an optional `companions`
array (`schema_version` stays **1**; additive fields only).

| Field | Meaning |
|---|---|
| `id` | Stable integrator id (e.g. `forced_aligner`) |
| `config_key` | Option key that receives the peer path |
| `scope` | `session`, `load`, or `request` |
| `target_family` | Registered loader family to install/select; omit/empty for external dirs |
| `optional` | Whether the primary can run without the peer |
| `required_for` | Capability/feature keys that need this peer |
| `label` | Short UI label |
| `bundled_default` | Optional in-tree default path (e.g. Silero under `assets/`) |

Rules:

1. Prefer a real `target_family` when the peer is an audio.cpp loader.
2. Non-empty `target_family` must be an **active** registry family (or a bundled
loader such as `silero_vad`).
3. Use empty `target_family` + a clear `label` only for external/non-loader peers
(e.g. Vevo2 Whisper directory).
4. Do **not** invent companions for catalog `parent_package_id` install deps —
those stay install-time only.
5. Keep companions in the loader that owns the path option; do not duplicate a
Studio-owned seed graph.

## Checklist: adding a model family

1. Implement `include/engine/models/<family>/` (or `community_models/`) with a
loader that overrides `advertised_capabilities()` so tasks/endpoints are
explicit.
explicit. Override `advertised_companions()` when the family takes peer paths.
2. Register it in `src/framework/runtime/registry.cpp` (include +
`available_loaders` entry). Prefer the factory name
`make_<family>_loader()` so the id matches the advertised family.
Expand All @@ -66,7 +99,8 @@ python3 tools/model_manager.py list --json
```

Confirm the new family appears in `--list-loaders` and that installable packages
for that family set `"family"` to the same string.
for that family set `"family"` to the same string. Confirm any `companions[].target_family`
values resolve to registered loaders.

## Checklist: parking or removing a family

Expand Down Expand Up @@ -98,6 +132,8 @@ builds. It:
- Parses active vs commented `make_*_loader()` calls in `registry.cpp`
- Compares them to installable standalone packages from `model_manager.py`
- Cross-checks the README recommended package table
- Parses `advertised_companions()` initializers in `**/loader.cpp` and ensures
non-empty `target_family` values are active registry families
- Does **not** require a compiled binary

```bash
Expand Down
4 changes: 4 additions & 0 deletions docs/models/qwen3.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,10 @@ the forced aligner model path. Long audio is split inside the model session
before ASR inference; word timestamps are shifted back onto the original audio
timeline.

Integrators can discover the aligner and VAD peers from
`audiocpp_cli --list-loaders --json` under `loaders.qwen3_asr.companions`
(see [usage.md](../usage.md#discovering-loaders-and-companions)).

`audio_chunk_mode=auto` is the default. For transcript-only ASR, Qwen3 ASR uses
fixed chunks. When word timestamps are requested, it uses bundled Silero VAD
internally to choose speech-aware chunks before running ASR and alignment.
Expand Down
2 changes: 1 addition & 1 deletion docs/tts.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ audiocpp_cli --task vc --family chatterbox --model models/chatterbox --backend c

## MioTTS

MioTTS is a 1.7B voice-clone TTS path that uses MioCodec for acoustic decoding. It requires a reference voice.
MioTTS is a 1.7B voice-clone TTS path that uses MioCodec for acoustic decoding. It requires a reference voice. MioCodec and optional best-of-N ASR peers are advertised under `loaders.miotts.companions` in `audiocpp_cli --list-loaders --json` (see [usage.md](usage.md#discovering-loaders-and-companions)).

| Field | Value |
|---|---|
Expand Down
47 changes: 47 additions & 0 deletions docs/usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,53 @@ Omit these unless you need explicit control. If `--seed` is omitted, models that

`--batch-text-dir` reads `.txt` and `.md` files as plain text. For `.json`, use either a JSON string root or an object with a string `input` or `text` field.

## Discovering loaders and companions

List registered families:

```bash
audiocpp_cli --list-loaders
audiocpp_cli --list-loaders --json
```

`--list-loaders --json` is the machine-readable runtime contract (`schema_version` 1). Each loader entry includes:

| Field | Meaning |
|---|---|
| `tasks` | Supported tasks and run modes |
| `instructions_policy` | How instruction / style text is handled |
| `api_endpoints` | Usual HTTP surfaces for those capabilities |
| `companions` | Optional runtime peer models (omitted when empty) |

Use `companions` when a family needs another model path at load or session time (for example Qwen3 ASR + forced aligner, MioTTS + MioCodec). Do not invent a separate UI graph of peer families — read this export.

Example companion object:

```json
{
"id": "forced_aligner",
"config_key": "qwen3_asr.forced_aligner_model_path",
"scope": "session",
"target_family": "qwen3_forced_aligner",
"optional": true,
"required_for": ["return_timestamps"],
"label": "Forced aligner"
}
```

| Companion field | Meaning |
|---|---|
| `id` | Stable integrator id |
| `config_key` | Option key that receives the peer path (`--session-option` / load option) |
| `scope` | `session`, `load`, or `request` |
| `target_family` | Registered loader family to install or select; omitted for external dirs |
| `optional` | Whether the primary can run without the peer |
| `required_for` | Feature keys that need this peer |
| `label` | Short UI label |
| `bundled_default` | Optional in-tree default path (for example Silero under `assets/`) |

Install-time package dependencies (`parent_package_id` in `model_manager`) are separate from runtime companions. Maintainer rules: [maintainers/loader_and_catalog.md](maintainers/loader_and_catalog.md).

## Model Docs

| Need | Doc |
Expand Down
24 changes: 24 additions & 0 deletions include/engine/framework/runtime/model.h
Original file line number Diff line number Diff line change
Expand Up @@ -105,11 +105,32 @@ class ILoadedVoiceModel {
const SessionOptions & options) const = 0;
};

struct LoaderCompanion {
/** Stable id for integrators (e.g. forced_aligner). */
std::string id;
/** Session/load/request option key that receives the peer path. */
std::string config_key;
/** One of: session, load, request. */
std::string scope = "session";
/**
* Registered loader family to install/select, when the peer is an audio.cpp
* family. Empty when the peer is only a bundled/external directory.
*/
std::string target_family;
bool optional = true;
/** Request/session capability keys that need this peer (e.g. return_timestamps). */
std::vector<std::string> required_for;
std::string label;
/** Optional default path for bundled assets (e.g. Silero under assets/). */
std::string bundled_default;
};

struct LoaderAdvertisement {
std::string family;
CapabilitySet capabilities;
std::string instructions_policy;
std::vector<std::string> api_endpoints;
std::vector<LoaderCompanion> companions;
};

/** Map advertised capabilities to the HTTP surfaces they normally use. */
Expand All @@ -127,10 +148,13 @@ class IVoiceModelLoader {
/**
* Path-free loader catalog for ``--list-loaders --json``.
* Override ``advertised_capabilities`` (and policy when non-default) on each loader.
* Override ``advertised_companions`` when the family loads another family or
* bundled peer at runtime via a path option.
*/
virtual CapabilitySet advertised_capabilities() const;
virtual std::string advertised_instructions_policy() const;
virtual std::vector<std::string> advertised_api_endpoints() const;
virtual std::vector<LoaderCompanion> advertised_companions() const;
LoaderAdvertisement advertise() const;
};

Expand Down
24 changes: 24 additions & 0 deletions src/community_models/outetts/loader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,30 @@ class OuteTTSLoader final : public runtime::IVoiceModelLoader {
public:
std::string family() const override { return "outetts"; }

runtime::CapabilitySet advertised_capabilities() const override {
runtime::CapabilitySet out;
out.supported_tasks = {
{runtime::VoiceTaskKind::Tts, {runtime::RunMode::Offline}},
};
out.supports_speaker_reference = true;
return out;
}

std::vector<runtime::LoaderCompanion> advertised_companions() const override {
return {
{
"forced_aligner",
"outetts.aligner_model_path",
"session",
"qwen3_forced_aligner",
true,
{"speaker_reference"},
"Forced aligner (voice cloning)",
"",
},
};
}

bool can_load(const runtime::ModelLoadRequest &request) const override {
try {
const auto package_spec =
Expand Down
5 changes: 5 additions & 0 deletions src/framework/runtime/model.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -148,12 +148,17 @@ std::vector<std::string> IVoiceModelLoader::advertised_api_endpoints() const {
return default_api_endpoints_for_capabilities(advertised_capabilities());
}

std::vector<LoaderCompanion> IVoiceModelLoader::advertised_companions() const {
return {};
}

LoaderAdvertisement IVoiceModelLoader::advertise() const {
LoaderAdvertisement row;
row.family = family();
row.capabilities = advertised_capabilities();
row.instructions_policy = advertised_instructions_policy();
row.api_endpoints = advertised_api_endpoints();
row.companions = advertised_companions();
return row;
}

Expand Down
25 changes: 25 additions & 0 deletions src/models/miotts/loader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,31 @@ class MioTTSLoader final : public runtime::IVoiceModelLoader {
return out;
}

std::vector<runtime::LoaderCompanion> advertised_companions() const override {
return {
{
"codec",
"codec_model",
"session",
"miocodec",
false,
{},
"MioCodec",
"",
},
{
"best_of_n_asr",
"best_of_n_asr_model",
"session",
"qwen3_asr",
true,
{"best_of_n"},
"ASR for best-of-N scoring",
"",
},
};
}

bool can_load(const runtime::ModelLoadRequest & request) const override {
if (request.family_hint.has_value() && *request.family_hint != family()) {
return false;
Expand Down
Loading
Loading