feat(search): graph search query endpoint with aggregations by dschmidt · Pull Request #3211 · opencloud-eu/opencloud

dschmidt · 2026-07-30T22:12:43Z

This contains a huge libre-graph-api-go bump - so please don't be alarmed about the size of the diff

MS-Graph-style POST /graph/v1beta1/search/query with terms, range, metric and nested sub-aggregations, on both the bleve and OpenSearch backends.

Stacked on #3198.

Caveats

Vendored libre-graph-api-go is interim, generated from feat: add search query endpoint with request examples libre-graph-api#34 rebased onto main. Proper path: land that spec PR upstream (feat: add search query endpoint with request examples libre-graph-api#34) and pin a released version.
Aggregation behavior is pinned in the engine parity suite (AGG-01..09: term buckets, numeric and date ranges, open-ended bounds, metrics, malformed-bound errors), running against bleve and a real OpenSearch. That round also caught and fixed an OpenSearch bug: date range bounds were silently dropped, they now emit a date_range aggregation and malformed bounds are rejected on both engines. Nested sub-aggregations were smoke-tested end-to-end on bleve (artist -> album -> sum(duration)).

Note

Even once this PR and #3210 (honest has-preview + thumbnails relationship) are both merged, search hits still won't carry thumbnails — the $expand=thumbnails wiring for /search/query hits builds on both PRs and currently only exists on the consolidation branch (dschmidt/opencloud@feat/graph-search-full-2, commit 0d94199a: searchEntityThumbnailSet in searchquery.go, preview presence via thumbnail.HasPreviewForMimeType + indexed oc.preview/image-facet dimensions). It needs a follow-up PR once both are in.

codacy-production · 2026-07-30T22:15:31Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 446 complexity

Metric Results

Complexity 446

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

dschmidt · 2026-08-16T09:55:55Z

Follow-up: narrow `Fields` when scanning hits for metrics / sub-aggregations

Not blocking, results are correct. It's the cost profile.

needsSubAggScan widens the page to subAggScanSize (math.MaxInt) for any
aggregation with sub-aggregations or a metric, since neither has a native facet
representation in bleve. Together with bleveReq.Fields = []string{"*"}, a
mediatype:image request carrying one Size/sum metric materialises the whole
matching hit set with every stored field, to add up one number. OpenSearch nests
natively and returns no documents.

Easy to reach from a client: a dashboard asking for a few library metrics over
mediatype:image pulls the whole image library into memory per load. Flat term
and range facets are unaffected.

Options by payoff:

Split scan from page. The scan needs only the metric and sub-agg fields; the
rich Fields set only serves the PageSize window. One request cannot carry
two field sets.
Otherwise narrow Fields to the union of what hydration and aggregations
read, instead of "*".
Bound the scan, mark results approximate.
For real library counters (total bytes of all images), maintain a value at
index time instead.

…egations Rebased dschmidt/libre-graph-api feat/graph-search-full (PR #34) onto opencloud-eu main and regenerated via the repo's woodpecker build-go recipe (openapi-generator v7.23.0, --api-name-suffix Api).

Adds the graph /v1beta1/search/query endpoint, the aggregation proto messages, the service-layer aggregation forwarding/merging, and the recursive bleve aggregation implementation (terms, range, metric, sub-aggregations).

Implements terms, range, metric and sub-aggregations for the OpenSearch backend via a dedicated aggs builder, wiring them through the shared search service.

Range aggregations parsed from/to with ParseFloat only, so date bounds on datetime fields like photo.takenDateTime silently degraded to unbounded numeric ranges. Detect date-formatted bounds (RFC3339 or YYYY-MM-DD), switch the facet to bleve date ranges and read DateRanges from the facet result. Malformed bounds in date mode are rejected.

Metric aggregations (sum/min/max/avg) only worked as sub-aggregations under a terms bucket. Compute top-level metrics by folding the matched hits through the existing accumulator and allow them through the graph layer's numeric field validation.

The cross-space merge only carried buckets, dropping metric results (value/metricKind) from the per-space responses. Reduce metrics with their kind's reducer, keyed by field and kind.

…unds

Regenerate the vendored libre-graph-api-go from the feat/search spec (metricDefinition/searchMetric, @libre.graph.subAggregations, aggregationFilterToken) and add aggregation_filters plus the permissionsActionsAllowedValues entity field to the search protos.

Migrate the graph search query to metricDefinition/searchMetric and the @libre.graph prefixes. Add the aggregationFilterToken round-trip: encode terms/range/or tokens on the response, pass aggregationFilters 1:1 to the search service, decode them and force exact case-sensitive matches in both backends. Map the WebDAV-report facets onto search hits: tags, video, motionPhoto, livePhoto, allowedValues (from the space permission set), me.following (favorite), webUrl (private link, shared helper) and thumbnails via opt-in $expand.

Bleve has count facets only: no metric and no nested facet. So far a metric or a sub-aggregation widened the page to every match and loaded all stored fields of each, then folded the hits in Go. The stored document, extracted text included, was decoded for every match: about 1s and 780MB per 100k matches, independent of the nesting depth. Metrics and nested aggregations now go through an aggCollector hooked into bleve's collector walk via the document-match-handler context key. For every match it visits the doc values of the aggregated fields, the columnar storage bleve's own facets read, and folds them into an accumulator tree. The page stays the size the caller asked for and no stored field is loaded for an aggregation: 7x faster and 80x less memory at 100k matches, within 1.4x of a native bleve facet. Flat terms and range aggregations stay bleve facets. A range parent now carries its sub-aggregations too (the hit fold matched range bucket names against raw values and never attached them), and multi-valued fields count each value like a bleve facet does. The parity suite renders nested results and pins them on both engines: terms in terms, metrics per bucket, terms in numeric and date ranges, three levels, a page of one still aggregating every match, and a malformed bound in a nested range.

dschmidt added the Type:Enhancement label Jul 30, 2026

This was referenced Aug 8, 2026

feat(search): geohash aggregation #3272

Draft

feat(search): honest has-preview prop and thumbnails relationship #3210

Draft

feat(search): sort results via sortProperties / order_by #3298

Draft

dschmidt force-pushed the feat/graph-search-query branch from 1b61ef6 to fb1e41b Compare August 18, 2026 16:51

dschmidt force-pushed the feat/graph-search-query branch 4 times, most recently from c54ffe7 to cc3e6f4 Compare August 31, 2026 18:37

dschmidt force-pushed the feat/graph-search-query branch from cc3e6f4 to af951f6 Compare September 1, 2026 05:48

dschmidt force-pushed the feat/graph-search-query branch from 08eddc9 to ceb7df5 Compare September 1, 2026 07:11

dschmidt mentioned this pull request Sep 1, 2026

feat: add search query endpoint with request examples opencloud-eu/libre-graph-api#34

Open

dschmidt changed the base branch from feat/search-tika-facets to main September 1, 2026 08:34

dschmidt force-pushed the feat/graph-search-query branch 3 times, most recently from 2513263 to 71ca404 Compare September 3, 2026 08:33

dschmidt mentioned this pull request Sep 7, 2026

refactor: graph listing opencloud-eu/web#3321

Draft

7 tasks

dschmidt added 5 commits September 11, 2026 14:30

chore(vendor): regenerate libre-graph-api-go with search query + aggr…

d3391f2

…egations Rebased dschmidt/libre-graph-api feat/graph-search-full (PR #34) onto opencloud-eu main and regenerated via the repo's woodpecker build-go recipe (openapi-generator v7.23.0, --api-name-suffix Api).

feat(search): opensearch aggregations

319ae48

Implements terms, range, metric and sub-aggregations for the OpenSearch backend via a dedicated aggs builder, wiring them through the shared search service.

dschmidt added 7 commits September 11, 2026 14:30

fix(search): merge top-level metric aggregations across spaces

90f9415

The cross-space merge only carried buckets, dropping metric results (value/metricKind) from the per-space responses. Reduce metrics with their kind's reducer, keyed by field and kind.

fix(search): date range aggregations on opensearch, validate range bo…

b058c19

…unds

test(search): pin aggregations in the parity suite

00fe538

feat(search): describe hits from shared spaces as remote items

df5a219

dschmidt force-pushed the feat/graph-search-query branch from fef1116 to 7f7ecbf Compare September 11, 2026 14:30

Navigation Menu

Uh oh!

Uh oh!

feat(search): graph search query endpoint with aggregations - #3211

feat(search): graph search query endpoint with aggregations#3211
dschmidt wants to merge 12 commits into
mainfrom
feat/graph-search-query

dschmidt commented Jul 30, 2026 •

edited

Loading

Uh oh!

codacy-production Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

dschmidt commented Aug 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

dschmidt commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This contains a huge libre-graph-api-go bump - so please don't be alarmed about the size of the diff

Uh oh!

codacy-production Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

dschmidt commented Aug 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Follow-up: narrow Fields when scanning hits for metrics / sub-aggregations

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dschmidt commented Jul 30, 2026 •

edited

Loading

codacy-production Bot commented Jul 30, 2026 •

edited

Loading

dschmidt commented Aug 16, 2026 •

edited

Loading

Follow-up: narrow `Fields` when scanning hits for metrics / sub-aggregations