Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 15 additions & 70 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,50 +78,25 @@ vidxp doctor
vidxp mcp-config
```

`vidxp mcp-config` emits the `mcpServers` JSON used by Claude Desktop and
compatible local stdio clients. Codex has separate configuration: use
`codex mcp add vidxp -- vidxp-mcp --repository default` or configure
`[mcp_servers.vidxp]` in `~/.codex/config.toml`. ChatGPT Desktop and ChatGPT web
use ChatGPT's own connector configuration and do not read Codex's config; hosted
ChatGPT connections use a remote MCP endpoint, with VidXP deployed behind HTTPS
and OIDC.

The CLI works without MCP. Add the browser app with:
Add the browser interface with:

```bash
uv tool install --python 3.14 --torch-backend cpu \
"vidxp[local-worker,mcp,frontend]"
vidxp ui
```

`vidxp ui` binds to loopback by default. Use `vidxp ui --share` only when you
intend to expose the unauthenticated browser interface on the local network.
Streamlit prints its Local and Network URLs when it starts. VidXP disables
Streamlit's first-run email prompt and usage-statistics collection.

`vidxp-api --share` exposes a bearer-protected HTTP API/MCP endpoint on a
trusted LAN. Browser upload tools are omitted unless an explicit HTTPS upload
handoff origin is configured, because a LAN listener alone cannot advertise a
safe browser capability page.

If the `vidxp` command is not found, run `uv tool update-shell` once and reopen
the terminal.
See the [installation guide](INSTALLATION_GUIDE.md) for client-specific MCP
configuration, the HTTP API, and remote server setup.

### 2. Desktop app

Download the installer for Windows, Apple Silicon macOS, or Linux from
[GitHub Releases](https://github.com/grayhatdevelopers/vidxp/releases).

On first launch, choose whether to adopt an existing compatible VidXP
installation without downloading another runtime, or create a private runtime
managed by VidXP Desktop. For a managed runtime, choose the search capabilities,
model location, and optional browser interface; Python and uv do not need to be
installed separately.

Desktop opens its control panel when started. Browser launch is a separate,
explicit **Open VidXP** action. After configuration, closing the control panel
keeps VidXP available through **Manage VidXP**, **Open VidXP**, and **Quit
VidXP** in the system tray.
Connect an existing VidXP installation or let the desktop app manage an isolated
runtime for you. See the [desktop guide](docs/desktop.md) for supported setup
options.

### 3. Docker for a server

Expand All @@ -141,21 +116,12 @@ See the [Coolify guide](docs/deployment/coolify.md) for the complete setup.

## What you can do today

- Build a reusable search library from one video or a whole collection.
- Search dialogue by meaning, even when you do not remember the exact words.
- Find visual moments by describing the scene you are looking for.
- Group recurring faces in a video and render a highlighted actor overlay.
- Search one selected video or every video in the active library.
- Open matching timestamps and export downloadable clips and overlays.
- Receive search and query results as annotated evidence boards, then open the
exact frames or clips worth inspecting.
- Retrieve completed clips through native MCP resources, local stdio paths, or
short-lived resumable HTTPS downloads without embedding video bytes in tool JSON.
- Build searchable libraries from individual videos or whole collections.
- Find dialogue by meaning and visual moments by describing the scene.
- Ask grounded questions and inspect the supporting boards, frames, or clips.
- Group recurring faces and render highlighted actor overlays.
- Keep personal, client, or project libraries separate.
- Follow long indexing jobs, cancel them, and keep the last working index if a
later run fails.
- Use the browser app, automate the CLI, connect an MCP agent, or integrate
VidXP into another application.
- Use VidXP through the desktop app, browser, CLI, MCP, or HTTP API.

## A first search

Expand Down Expand Up @@ -188,31 +154,10 @@ application, or use the HTTP API when VidXP runs as a service.

[![VidXP being used with ChatGPT Desktop AI](./docs/images/claude-with-vidxp.jpg)](https://youtu.be/fa4Zx-bSOh4)

MCP clients can add and discover videos, start indexing, search dialogue and
scenes, ask questions about a library, and create clips or actor overlays.
Local agents can connect over stdio; remote agents can connect to a
self-hosted VidXP server.

Search and question results can include directly inspectable frames and clips, so
agents can show the evidence behind an answer without making users translate raw
timestamps. Evidence rendering is best-effort: a result can still be useful when
an individual frame or clip cannot be produced. Agents can request additional
ranked evidence in small batches without rerunning the search.
For broader result sets, an agent can first show bounded evidence-board pages
with a tile-to-evidence map, then fetch exact frames or clips only for the
selected tiles.

Remote agents can hand users a short-lived page for selecting and uploading
multiple videos. Local agents can ingest approved filesystem paths without moving
video bytes through MCP. VidXP normally indexes successful imports automatically;
registration-only ingestion stops at `registered`, and indexing failures can be
retried without uploading the video again.

Agents can call `get_workspace` before acting to inspect registered media,
active-index coverage, model readiness, and the searchable, queryable,
inspectable, or renderable roles available for each video. Invalid capability
or media selections are rejected before a durable job is queued and include an
actionable next step.
MCP lets AI clients add and index videos, search dialogue and scenes, ask
questions about a library, and return inspectable evidence such as boards,
frames, and clips. Clients can connect locally over stdio or to a self-hosted
VidXP server.

### ChatGPT and Codex skills

Expand Down
26 changes: 19 additions & 7 deletions docs/architecture/platform.md
Original file line number Diff line number Diff line change
Expand Up @@ -920,15 +920,22 @@ guidance.
MCP search and query jobs compile their ranked candidates into annotated,
media-separated JPEG evidence boards before the original job succeeds. The board
uses the existing evidence-frame and artifact services, returns native image
resources plus a tile map to stable evidence IDs, and is included in the same
`get_job` response. The default budget is 24 tiles per page and four pages per
job; `next_start_rank` continues without imposing the standalone-artifact limit.
resources plus a concise index to stable evidence IDs. Active observation uses
compact job summaries and bounded waits; after the job becomes terminal,
`get_job_evidence` returns content-only board images, links, timestamps, and
evidence IDs that remain model-visible in hosts that prioritize
`structuredContent` over MCP content blocks. `get_job` is the explicit full
machine record and does not load or duplicate evidence bytes. The default budget
is 24 tiles per page and four pages per job; `next_start_rank` continues without
imposing the standalone-artifact limit.

Callers may additionally request `keyframes` or `keyframes_and_clips`, with a hard
maximum of five standalone items in the initial job. Scene evidence extracts the
authoritative indexed frame number; other intervals and anonymous actor clusters
use a labeled representative frame. `materialize_job_evidence` accepts one to ten
board evidence IDs for later frame or clip drill-down without rerunning retrieval.
board evidence IDs for later frame or clip drill-down without rerunning retrieval;
it likewise returns content-only images and links rather than duplicating its full
delivery model.
`create_evidence_board` remains for custom selections and continuation pages, and
`create_evidence_clip` remains the single-item fallback. Clip rendering requires
repository write scope; boards and keyframes retain read scope.
Expand Down Expand Up @@ -1056,13 +1063,18 @@ Initial curated tools:
- `get_artifact_download`
- `list_jobs`
- `get_job`
- `get_job_evidence`
- `get_job_status`
- `wait_job`
- `retry_job`
- `cancel_job`

Media and job discovery let an agent recover registered assets and durable work
without carrying IDs across sessions. Generic job polling, retry, and cancellation
cover indexing, search, and query without duplicating operation contracts. Video
bytes remain on the HTTP/tus ingestion boundary rather than crossing MCP.
without carrying IDs across sessions. Compact summaries and `wait_job` provide
bounded observation without repeatedly returning completed result payloads;
retry and cancellation cover indexing, search, and query without duplicating
operation contracts. Video bytes remain on the HTTP/tus ingestion boundary
rather than crossing MCP.

Tool results use real output schemas and structured content. Descriptions remain
short and agent-oriented; full API response schemas are not embedded as prose.
Expand Down
11 changes: 6 additions & 5 deletions docs/deployment/coolify.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,17 +175,18 @@ curl --fail-with-body \
--data '{"modalities":["dialogue","scene","actor"],"capability_options":{}}'
```

The `202 Accepted` response contains the durable `job_id` and a `Location`
header. Poll that location until the job succeeds:
The `202 Accepted` response contains the durable `job_id`. Use the bounded wait
endpoint for compact status, passing the returned observation token on the next
request. Fetch the full job once after it becomes terminal:

```bash
curl --fail-with-body \
--header "Authorization: Bearer ${VIDXP_HTTP_STATIC_BEARER_TOKEN}" \
"https://${VIDXP_PUBLIC_API_HOST}/api/v1/jobs/<job-id>"
"https://${VIDXP_PUBLIC_API_HOST}/api/v1/jobs/<job-id>/wait?timeout_seconds=30"
```

The Streamable HTTP MCP `prepare_models` and `get_job` tools expose the same
operation for an authenticated agent client. Check
The Streamable HTTP MCP `prepare_models`, `wait_job`, and `get_job` tools expose
the same operation for an authenticated agent client. Check
`/api/v1/runtime/readiness` afterward; `/ready` covers control-plane
availability and does not claim that every optional model is prepared.

Expand Down
28 changes: 18 additions & 10 deletions skills/vidxp-find-video-evidence/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,25 @@ description: Use VidXP to search indexed videos, answer grounded questions about
1. Resolve the `vidxp` MCP tools, then call `get_workspace`. If the requested
video is not indexed, explain that it must be indexed first.
2. Use `search_moments` to locate moments or `query_video` for a synthesized,
grounded answer. Set `command.media_id` when the user means one video.
grounded answer. Use `command.query` with `search_moments` and
`command.question` with `query_video`. Set `command.media_id` when the user
means one video.
3. Omit `command.evidence_delivery` for the normal path. The completed job
includes an annotated board covering the ranked results.
4. Poll only that job with `get_job`, honoring `poll_after_seconds`. Search and
query may take time; update the user when the stage changes or about once per
minute, never on every poll and never with an invented ETA.
4. Call `wait_job` for bounded waits. Pass its `observation_token` as
`after_observation_token` on the next wait. When terminal, call
`get_job_evidence` once. It returns the concise evidence index and visual
content without the full structured job dump. Use `get_job` only when exact
machine fields not present in that index are actually needed. Search and
query may take time; update the user
when the stage changes or about once per minute, never after every wait and
never with an invented ETA.
5. Inspect and show the returned board before making visual claims. Use its tile
evidence IDs for follow-up:
- `materialize_job_evidence` accepts up to ten selected IDs and returns
standalone keyframes or clips without rerunning retrieval.
model-visible standalone keyframes or clip links without rerunning retrieval.
- `create_evidence_board` is only for a custom selection or the
`next_start_rank` continuation; poll its returned job ID.
`next_start_rank` continuation; wait on its returned job ID the same way.
6. When standalone artifacts are required in the initial job, put exactly this
inside `command`: `"evidence_delivery": {"mode":
"keyframes_and_clips", "max_items": 3}`. Never send
Expand All @@ -38,11 +45,12 @@ description: Use VidXP to search indexed videos, answer grounded questions about

## Output

- Present returned board images, frames, clips, or working resource links—not
timestamps alone. If a host does not render a link, use
`get_artifact_download`.
- The final response must visibly embed a returned board or frame, or include a
working downloadable resource link—not timestamps alone. Use the returned
`local_path` or `download_url`; never write an unlinked label such as “View
evidence board.” Use `get_artifact_download` only if neither is returned.
- Preserve the source job and evidence IDs. Describe scores as retrieval scores,
and distinguish a visible appearance from a dialogue or caption mention.
- Stop polling on success, failure, or cancellation. An empty result means no
- Stop waiting on success, failure, or cancellation. An empty result means no
matching indexed evidence was found, not that the event is absent from the
original video.
5 changes: 3 additions & 2 deletions skills/vidxp-ingest-video/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ description: Use VidXP to upload, import, register, and automatically index vide
for ordinary content retrieval. Add `actor` only when anonymous recurring-face
clusters are wanted; it does not identify people by name.
3. Call `get_runtime_readiness`. If selected models are missing, submit
`prepare_models` for those modalities and poll its job with `get_job`.
`prepare_models`, use `wait_job` with its observation token for subsequent
bounded waits, then fetch `get_job` once when terminal.
4. Use `ingest_local_media` for one to ten paths accessible to VidXP; otherwise
use `create_media_upload` and give the returned link to the user. Keep
`index_after_import` enabled unless registration-only behavior was requested.
Expand All @@ -30,5 +31,5 @@ description: Use VidXP to upload, import, register, and automatically index vide

- Tell the user that model preparation and indexing can take several minutes.
- Update when the stage changes or about once per minute; do not narrate every
poll or invent an ETA.
status check or invent an ETA.
- Treat files independently so one failure does not hide successful siblings.
48 changes: 48 additions & 0 deletions src/vidxp/api_routes/jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,19 @@
CreateActorOverlayCommand,
CreateIndexCommand,
CreateSnippetCommand,
DEFAULT_JOB_WAIT_SECONDS,
Job,
JobPage,
JobResult,
JobSummary,
JobWaitResult,
ListJobsCommand,
MAX_JOB_WAIT_SECONDS,
Principal,
PrepareModelsCommand,
QueryVideoCommand,
SearchCommand,
Sha256,
)
from vidxp.composition import HttpApplicationContext
from vidxp.core.identifiers import JobId
Expand Down Expand Up @@ -287,6 +292,49 @@ def get_job(
return _http_job(service.jobs.get(job_id))


@router.get(
"/{job_id}/status",
response_model=JobSummary,
operation_id="getJobStatus",
summary="Get compact job status",
dependencies=[Depends(read_principal)],
)
def get_job_status(
job_id: JobId,
service: Annotated[HttpApplicationContext, Depends(context)],
) -> JobSummary:
return service.jobs.summary(job_id)


@router.get(
"/{job_id}/wait",
response_model=JobWaitResult,
operation_id="waitForJobChange",
summary="Wait for a job change",
description=(
"Wait up to 30 seconds for a job to change public stage or become "
"terminal. Evidence rendering is treated as one observable phase "
"rather than one change per artifact. Pass the previous observation "
"token on subsequent requests."
),
dependencies=[Depends(read_principal)],
)
def wait_for_job_change(
job_id: JobId,
service: Annotated[HttpApplicationContext, Depends(context)],
after_observation_token: Sha256 | None = None,
timeout_seconds: Annotated[
int,
Query(gt=0, le=MAX_JOB_WAIT_SECONDS),
] = DEFAULT_JOB_WAIT_SECONDS,
) -> JobWaitResult:
return service.jobs.wait_for_change(
job_id,
after=after_observation_token,
timeout_seconds=timeout_seconds,
)


@router.get(
"/{job_id}/result",
response_model=JobResult,
Expand Down
33 changes: 33 additions & 0 deletions src/vidxp/application_models.py
Original file line number Diff line number Diff line change
Expand Up @@ -1303,6 +1303,10 @@ class PrepareModelsResult(ApplicationModel):

JOB_SCHEMA_VERSION = 2
JOB_PROGRESS_SCHEMA_VERSION = 1
JOB_SUMMARY_SCHEMA_VERSION = 1
JOB_WAIT_RESULT_SCHEMA_VERSION = 1
DEFAULT_JOB_WAIT_SECONDS = 30
MAX_JOB_WAIT_SECONDS = 30


class IndexSnapshotReference(ApplicationModel):
Expand Down Expand Up @@ -1505,6 +1509,35 @@ def _validate_terminal_payload(self) -> "Job":
return self


class JobSummary(ApplicationModel):
"""Compact observation of a durable job without its typed result payload."""

schema_version: Literal[JOB_SUMMARY_SCHEMA_VERSION] = JOB_SUMMARY_SCHEMA_VERSION
job_id: JobId
kind: JobKind
state: JobState
queue: JobQueue
progress: JobProgress | None = None
error: ErrorDetail | None = None
recovery_attempts: int = Field(default=0, ge=0)
created_at: AwareDatetime | None = None
updated_at: AwareDatetime | None = None
terminal: bool
result_available: bool
observation_token: Sha256


class JobWaitResult(ApplicationModel):
"""Result of waiting for a meaningful job state or stage change."""

schema_version: Literal[JOB_WAIT_RESULT_SCHEMA_VERSION] = (
JOB_WAIT_RESULT_SCHEMA_VERSION
)
job: JobSummary
changed: bool
timed_out: bool


class ListJobsCommand(ApplicationModel):
page_size: int = Field(default=50, gt=0, le=100)
cursor: str | None = Field(default=None, min_length=1, max_length=512)
Expand Down
Loading