free text search doesn't come up with results that I know are in data

[akthom]

In the samples table, I noted some samples with the "Place" value of "Axial seamount summit caldera." I was curious to see all samples from that "place" so i did a free text search for "axial seamount summit caldera." No results came up. I tried searching for just "seamount summit caldera" but it seemed to get stuck loading. so, two issues:

1. why aren't results that I know are in the data not coming up in free text search?
2. why does the search hang or crash after a free text search?

[rdhyee]

Root cause found + fix in PR #327; the "hang" half measured and scoped

@akthom great catch — this was a real and quietly serious bug. Both halves traced:

1) Why search couldn't find "Axial seamount summit caldera"

Search scores label/description/place_name over the facets parquet — but the deployed sample_facets_v3 file predates the #311 place-name pipeline fix, so its place_name was NULL for all 6,026,242 samples. The table's Place column reads a different file (samples_map_lite_v3) that DID get the fix. So the UI was showing you places the search index had never heard of.

Fix: PR #327 — rebuilt the facets file from the live prod data (validated: zero changes to any non-place value; place coverage 37.6%, identical to the lite) and pointed the explorer at it. Verified on a preview with your exact search: 284 results, including IGSN:321000001 "Axial Seamount summit caldera". Your second phrasing ("seamount summit caldera") also returns 284 now.

Worth naming the failure shape: two sibling data files consumed by the same page silently diverged in vintage, and nothing noticed. A broader pipeline↔UI coherence review (with a proposed release-manifest mechanism) is underway so this class of bug gets caught at build/boot time instead of by you.

2) Why it "got stuck loading"

Measured on the preview (fast broadband, cold browser): first search ~9s; an immediate follow-up search ~24s (it queues behind the first search's table/facet/point refreshes). The dominant cold cost is that DuckDB-WASM's range-request fallback (#190) downloads the full ~69 MB facets file on first search — on a slower connection that stretches to minutes, and the UI shows only a static "Building search filter…" the whole time. So: not a crash, but an unbounded-feeling wait with no progress signal.

That latency is a known, documented tradeoff of the current interim search (#168) — the real cure is the FTS track (#169–#172: proper BM25 document index), and your report is good evidence for prioritizing it. Two cheaper interim options if the wait keeps biting: a progress/"first search downloads ~70 MB" hint in the search panel, and/or prewarming the facets file after boot.

— rbotyee 🤖 (Claude, Raymond's assistant)

[akthom]

Reopening - i just tried searching for "seamount" and only 2 results came up (and none of them were the results from yesterday. Something is still going on. is the search case sensitive? if it is, it shouldn't be.

[rdhyee]

The missing-results half is live on production (PR #327 merged, verified fresh-context): "axial seamount summit caldera" → 284 results on isamples.org, and place names in general are now searchable. Leaving this issue open for the second half — the slow/"stuck loading" behavior on cold first search (#190 full-file download + no progress feedback; measured details above). — rbotyee 🤖

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: really fix the slow-search half of this issue, not patch it. What I did: built a proper search index (#170) and a new query path (#171), both reviewed and merged; the fix is now on production behind an opt-in flag.

@akthom the "stuck loading" half of this issue now has its real fix live for you to try: add ?fts=v1 to the Explorer URL — https://isamples.org/explorer.html?fts=v1 — and search normally.

What changes: instead of downloading a 69 MB file on your first search, it fetches only a few small pieces of a pre-built index (typically 2–5 MB per query). Measured on production this morning: "pottery Cyprus" → 1,305 results in under 2 seconds, cold — including on the exact searches that felt stuck before.

Two honest notes: (1) results are ranked by relevance now (BM25), so ordering may differ from the old search; (2) if you search only extremely common words ("material sample"), it says so and shows a best-effort list rather than scanning everything — you'll see a note about which words were ignored. Both behaviors are deliberate and documented.

This stays opt-in until #172's formal benchmark (in progress) confirms it should become the default. Your real-world impressions — especially on your home connection where the old one hurt — are exactly the input that decision wants.