Skip to content

Add clip-level video understanding for actions and changes #83

Description

@tulayha

VidXP's current visual search examines one frame per second. That works for finding visible objects and people, but a single frame cannot reliably describe movement or a change across time—for example entering versus leaving, picking something up versus putting it down, or one action happening before another.

Add an optional clip-level search path while keeping the existing frame search as the lightweight baseline.

Expected behavior:

  • Split video into reusable shots or short windows with real start and end times.
  • Search across multiple frames when a query depends on motion, order, or change.
  • Return results through the existing evidence boards, keyframes, and downloadable clips.
  • Keep clip encoding replaceable so one model is not wired throughout the application.
  • Let users enable clip-level indexing only when they need it.
  • Measure search quality, indexing time, memory, and storage through Add end-to-end retrieval evaluation #76.

A useful first contribution can add the shared clip representation and one measured implementation. PySceneDetect is a candidate for the initial shot boundaries; VideoPrism, PE Video, and X-CLIP are candidates for comparison rather than required dependencies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions