Evaluate semantic vector recall over the POI corpus

[Medformatik]

Problem

Keyword/category parsing cannot recover every indirect, colloquial, or multilingual place query. A vector index over actual POIs might improve recall, but enabling one without measured quality gates could introduce irrelevant results, high build/storage cost, and local-AI latency.

Current state

• Search composes provider, category, and natural-language stages.
• integrations/search-nlp/ has an Ollama-backed semantic category experiment.
• The checked-in qwen3-embedding:0.6b evaluation did not meet its activation gates.
• Overture ingestion can generate embeddings for conflation, but no pgvector-backed named-POI search index is exposed to users.

Proposed solution

Run an evaluation-first spike for vector recall over the real searchable POI corpus. Compare candidate embedding models and storage approaches, then implement a hybrid lexical/category/proximity/vector path only if it meets explicit gates. Treat the originally suggested nomic-embed-text and pgvector as candidates, not predetermined choices.

Acceptance criteria

• A versioned multilingual corpus covers names, aliases, colloquialisms, categories, misspellings, ambiguity, and negative queries.
• Baselines report Recall@k, precision/nDCG, negative activation, geographic relevance, and lexical recovery.
• At least two local models are benchmarked for quality, p50/p95 latency, memory, build time, disk, and cold start.
• pgvector and any alternative storage strategy use the same corpus/hardware profile.
• The schema uses stable source IDs, supports delete/upsert, and records model/source provenance.
• Incremental OSM/Overture changes update the index without a full rebuild.
• Exact-name and strong lexical matches remain authoritative in hybrid ranking.
• Missing/unhealthy AI degrades to current search.
• Activation thresholds and a stop/no-ship decision are checked into an evaluation report.
• If gates pass, the feature ships behind configuration with regression tests.

Alternatives considered

Keyword aliases are cheap but weak on long-tail phrasing. Generative search is slower and less deterministic. The existing experiment indexes taxonomy labels rather than actual POIs and currently fails its gate.

Area

Search, local AI, data manager, PostgreSQL/PostGIS.

References

• integrations/search-nlp/eval/reports/qwen3-embedding-0.6b-ac6da0dfba84.md
• integrations/search-nlp/semantic-category-index.ts
• services/data-manager/src/jobs/overture/embeddings.ts