Add /search, /xref and /combine to the HTTP API, plus a lightweight client and a docs site - #82
Open
Robbie1977 wants to merge 13 commits into
Open
Add /search, /xref and /combine to the HTTP API, plus a lightweight client and a docs site#82Robbie1977 wants to merge 13 commits into
Robbie1977 wants to merge 13 commits into
Conversation
…earch & /xref endpoints - docs/vfbconnect-http-api-plan.md: full plan, vfb_connect->query_type map, phasing, 80-peak scaling - clients/vfbquery-client: pure requests+pandas client over v3-cached (no navis); tested live (68 DA1 lPN instances, 107 NBLAST hits) - docs/draft_search_xref_endpoints.py: reviewable draft handlers for /search (C1) and /xref (C3) Decisions: lightweight standalone package; extend v3-cached paths.
…t resolve_entity - Client: search() now runs the exact MCP search_terms edismax query (ontology core, wildcards, mm=45%, qf autosuggest, bq boosts, filter/exclude_types); name->id resolution uses it, resolve_entity dropped. Tested live: 'DA1 lPN' -> FBbt_00067363, get_instances(by name) -> 68 rows. - Draft /search endpoint: corrected to ontology core + real search_terms params; note to share one config with MCP/website (no drift). - Plan C1: reframed as 'expose existing search_terms', with why resolve_entity is the wrong tool.
…e wiring /search Checked geppetto-vfb as directed: five live copies of the free-text search config, all against the Solr ontology core, but the website is NOT sending the same query as the MCP search_terms tool. Differences measured live and factors isolated: - q: website ANDs a wildcard OR-group per token (geppetto-client getResultsSOLR); search_terms wildcards the whole phrase and leans on mm=45%. DA1 lPN 51 vs 718 hits. - fq: website hard-excludes facets_annotation:Deprecated; MCP only demotes it. This (not the tokenisation) is what makes a bare hemibrain bodyId resolve to the right neuron. - bq: website floats Class^200/FBbt^150 over VFB^50 plus DataSet^500/pub^100. - website then explodes synonyms into rows and re-ranks with a ~350-line custom sorter, so users never see raw Solr order - serving 'what the website does' means porting that too. - two copies (spotlight, geppetto-client default) still use the pre-migration schema and one points at solr-dev. Adds docs/search-config-comparison.md + a reproducible docs/compare_search_configs.py, records the open decision in the plan (blocks wiring /search), and makes the client's q construction switchable (q_mode='phrase' = MCP parity, default; 'tokens' = website) so no behaviour changes until the canonical config is agreed.
Search ranking currently exists in five drifted copies (the website's searchConfiguration.js, the geppetto-client default, the MCP, and two others - see docs/search-config-comparison.md). This collapses them to one: search_config.py ports the website's query construction, fq, boost chain and ~370-line comparator, and /search serves it, so the website, the MCP and the Python client can all call the same endpoint and get the same order. The port is bug-compatible on purpose, including JS String.replace(str, str) first-occurrence-only semantics and the empty OR-group a double space produces (measured: 3 hits with it, 0 without, so it is not inert). The comparator is not a consistent total order, so identical ordering cannot be guaranteed by construction and has to be measured: docs/search-parity/ runs the real sorter and refineResults under Node against the same Solr docs and diffs the ordering. Currently byte-identical on 78 queries (22 hand-picked discriminating cases plus a 56-query fuzz sample), including 2432-row result sets, empty input, braces, quotes and apostrophes. Re-run it after touching the ranking. /search deliberately bypasses the Neo4j ProcessPool. Its cost profile is the opposite way round - 150 ms-1.2 s of Solr I/O plus 6-300 ms of Python ranking, with no Neo4j leg - so occupying a pool worker to wait on a socket would both cap search throughput and queue it behind slow connectivity queries, which is backwards: search is how people find the IDs they feed to everything else, so it has to stay responsive while the rest of the box is busy. Instead Solr is awaited on the event loop over a shared keep-alive session and only the ranking is offloaded, to threads rather than processes because pickling ~2000 dicts would cost more than the sort. Concurrency is bounded but queued rather than refused. An availability-only check shed half of a burst of 80 distinct queries with 503 even though it drained in under 4 s; waiting up to search_queue_wait fixes that while still shedding genuine overload. Measured at the workshop target of 80 peak users: 80 concurrent distinct queries all succeed at p50 2.4 s / max 3.5 s, 160 concurrent still all succeed at max 3.9 s, and the realistic case - many people on the same handful of exercise queries - is almost entirely cache hits and coalesced fetches (31 ms p50, >1000 req/s). Verified end-to-end through HTTP against the module-level search() on 15 cases covering rows, limit and all four type filters: identical rows and counts. Cache (611 ms -> 1 ms), coalescing (8 identical concurrent requests -> 1 Solr fetch), the 503 shed path with Retry-After, and the four 400 paths all exercised against a running server.
The client carried its own edismax config — its own q construction, fq, qf and bq — which made it a sixth copy of a search configuration that was already drifting in five places, and it ranked by raw Solr score, so it disagreed with the website on almost every query. All of that is now server-side: search() is a call to /search, and there is no Solr configuration left in the client at all. Two semantic corrections fall out of the move. `rows` is a *ranking* parameter on /search (the comparator can only promote what Solr returned), so page size is now `limit` and `rows` defaults to the website's 500 — previously `rows=50` was silently limiting ranking depth to 50 candidates. And _resolve_to_id asked for rows=1, i.e. one *unranked* candidate; it now asks for limit=1, the top of a fully ranked list. Verified against a local /search: 'DA1 lPN' -> FBbt_00067363, bare hemibrain bodyId '1734350908' -> VFB_jrchjtdb. filter_types/exclude_types are now passed through rather than reimplemented, and boost_types/demote_types are newly exposed. Tests cover what the client is now responsible for — the parameter names it sends, that empty filters are omitted, that candidate depth does not collapse to `limit`, and that ids skip the lookup — plus a live test that asserts the class outranks the individuals for 'DA1 lPN', since that ordering is the point of the endpoint and a plain Solr query does not produce it.
The analysis these docs recorded was written from reading five config files.
Validating the port measured several of its claims false, so this rewrites them
rather than appending caveats.
Corrections:
* pf explains nothing, in any config. `pf=true` is a silent no-op — Solr reads
pf as a field list, so `true` is a field name that does not exist — and even a
correct field list never fires against the website's wildcard OR-group q,
because the expansion leaves no contiguous phrase to boost. Both measured;
`compare_search_configs.py --check-pf` reproduces them, so the claim stays
falsifiable instead of becoming folklore.
* The phrase boost that does the work is appended to bq at request time by the
website's client (`label:"<input>"^3000 synonym:"<input>"^1500`), so it appears
in no config file. Reading the configs cannot find it.
* MBON-a2's top hit is an individual and that is correct. The earlier claim that
the website leads with the class was wrong: FBbt_00047966's synonyms are
["larval MBON-a2", "odd", "MBE7b"], so the class has no MBON-a2 match to boost.
The fix is an ontology synonym, not a search change.
* JS `replace(string, string)` replaces only the first occurrence, so
"MBON-a2-b" normalises to "MBON a2-b". The port is bug-compatible.
* There are six live configs, not five. VFBCircuitBrowser has its own
web-worker sorter and no Class boost, so individuals lead there deliberately.
The load-bearing one: raw Solr order is not what a user sees. Every hit goes
through refineResults and a ~370-line comparator first, so any top-hit
comparison at the Solr level — including earlier revisions of both this document
and this script — describes something nobody experiences. The script now ranks
with the real ported sorter and labels the unranked order as candidates.
That is also why C1's "S — expose existing, don't build" estimate was wrong: the
query is one of three stages, and the two that decide ordering lived only in JS.
Open question 0 (canonical search config) is now decided in favour of the
website's, with the consequence recorded: until the website and the MCP call
/search, this is a fourth consumer agreeing with the website rather than a
single source of truth. The exact-label recall bug (`neuron` absent from all
1370 ranked rows, pushed *down* by bq) is inherited deliberately — the port is
faithful and the config has the bug, and changing VFB search behaviour wants
sign-off rather than a quiet fix inside a port.
Test plan section now carries the measured numbers, including the 80-concurrent
result that only running it revealed: the first implementation refused rather
than queued and returned {503: 40, 200: 40}.
The `neuron` case in comparison.md §5 was described with a mechanism that measurement does not support, and two of its factual claims were wrong. What is actually happening: the label and synonym halves of the runtime phrase boost are additive. `label:"neuron"^3000` alone puts FBbt_00005106 at score-rank 0 — correct. Adding `synonym:"neuron"^1500` drops it to 705, because competitors carry the token in their labels AND their synonyms (`dopaminergic neuron` has `DA neuron`) and so collect 4500, while the term whose label *is* the query has only `nerve cell` and collects 3000. The exact match is penalised for lacking a redundant synonym. The boost is not unselective; it is selective on the wrong thing. Corrected, both falsified by checking the schema rather than reading it: - the schema *does* have an exact field, `label_str` (strings, docValues, case-sensitive), contradicting "no label_s string field"; - `label_autosuggest_e` is not the intended exact field — text_general, tokenized identically to `label`, and unpopulated for the failing document. Scope, which the old text left open: 42/43 sampled labels already retrieve their own term and 41/43 rank 0, and a 70-label single-token sweep found no further failures. `neuron` is the only real miss. But `adult` (score-rank 424 of 500) and `cell` (137) are latent, so this is one indexing change from spreading. Adds check_recall.py, which measures what the parity gate structurally cannot: parity feeds both sorters the same Solr response, so it is blind to a term that never reaches either. Four candidate fixes measured; `label_str:"<term>"^6000` appended to `bq` fixes the miss with zero top-10 churn and no extra request. search_config.py is deliberately untouched. Retrieval changes what every consumer of /search sees and the parity gate cannot catch a regression in it, so this is a recommendation awaiting sign-off, not a change.
Fixes from an adversarial review of the whole branch, plus the CI and harness gaps that review exposed. Service (src/vfbquery/ha_api.py) - 500 bodies no longer ship a traceback. All six sites (get_term_info, run_query, _dispatch_to_pool, handle_search, handle_xref -- owner and coalesced-waiter halves each) go through a new _failure_response, which logs the traceback server-side and returns "ExcType: message". NOTE: this changes the error contract of pre-existing endpoints, not only the two this branch adds. The five coalesced-waiter halves previously never logged at all, so a failure a client saw left nothing on the server. - /xref cache key is now injective. accession and db are free text, so "|".join was not: ?accession=A|B and ?accession=A&db=B| both flattened to one key answering two different questions. Now json.dumps. - on_cleanup uses .get() throughout and suppresses CancelledError. aiohttp runs on_cleanup even when on_startup raised part-way, so indexing there replaced the real startup exception with a KeyError. - _cap_result_rows caps every over-cap key in PAYLOAD_LIST_KEYS, not just the first, and derives `count` from the payload's primary list. Search (src/vfbquery/search_config.py) - build_exact_label_boost also emits un-normalised variants. normalise replaces the first -/_/+ with a space, but label_str holds the label verbatim, so the boost could not fire for the whole hyphenated and underscored half of VFB naming (MBON-a2, fan-shaped body). Recall gate confirms this moved nothing else: churn 0, 53 vs 53 Solr requests. Parity harness (docs/search-parity/) - A query returning zero docs now FAILS. [] == [] was being scored as a pass -- the one failure shape a parity harness must never call success. - That guard immediately caught a real hole: fuzz_queries drew its pool with no fq under sort=short_form asc, i.e. the alphabetical head of the unfiltered index, which is the imported BFO_/CHEBI_/CL_/ENVO_ terms FQ_BASE excludes from /search by design. 15 of 56 fuzz queries returned zero docs and had been passing vacuously. The pool now carries FQ_BASE + FQ_NOT_DEPRECATED and is drawn from six seed-determined random windows rather than one head (one head is all FlyBase alleles: no anatomy class, individual, dataset or publication ever got fuzzed). 78/78 now compare non-vacuously. - check_recall.py measures and gates its "no extra Solr requests" claim instead of asserting it in prose. - scripts/check_gates.sh rejects a valueless trailing --seed instead of silently treating it as "no seed". Client (clients/vfbquery-client/) - VfbError is exported. Catching it is the documented way to handle 503 backpressure, and callers cannot write that retry loop against an exception living in a private module. - Test for 503 -> VfbError at the transport layer (and that a 500 still raises requests.HTTPError, so the mapping is not "everything becomes VfbError"). _FakeResponse can now model a failure at all. CI / tests - tests/conftest.py and tests/test_search_xref_endpoints.py were untracked: test_resilience.py and test_shed_coalescing.py both import `run` from conftest, so a fresh clone and the search-gates workflow errored at collection. - The workflow is committed at docs/ci/search-gates.yml, NOT .github/workflows/. GitHub rejects an entire push whose diff touches .github/workflows/ when the credential lacks `workflow` scope, and the one available here does. Parking it one directory over keeps the CI definition in the diff to review; `git mv docs/ci/search-gates.yml .github/workflows/` from a workflow-scoped checkout activates it. Until then scripts/check_gates.sh is the only thing running these gates. Docs - The parity section now distinguishes the live-loaded sorter from the vendored, pinned copy of refineResults (which will not notice an upstream change on its own), records the vacuous-fuzz finding, and attributes --seed to the parity gate rather than the recall gate. - Client README and client.py document the always-on exact-label boost as the one deliberate departure from website order, and the once-per-site behaviour of Python's default warning filter. All five gates green: unit, client, parity (78 queries), recall (+1 recall, +1 rank0, 0 churn), live. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8ZAhRM7gh8arDx9SuSWcT
Answers the questions that need two queries and a comparison — "neurons in the calyx AND the lateral horn", "MB dopaminergic BUT NOT octopaminergic" — which attendees ask roughly two searches into a workshop, and which previously meant downloading two tables and reconciling them by hand. An expression, not a fixed pair of operands: `[calyx AND lh] NOT gaba`, up to 12 operands and 2000 characters, with OR/NOR loosest, XOR/XNOR next and AND/NAND/NOT tightest, left-associative. Brackets are honoured so the caller can say which comparison happens first, because `A OR B NOT C` means two different things and guessing is not acceptable when the answer is a neuron list somebody will act on. Rows are matched on the identity column — the header whose declared type is `selection_id` — rather than on position or label, so results from query types with entirely different tables still compare. `NeuronInputsTo` is the one type without that header and falls back to `id`; duplicate ids within an operand are folded, which matters more than it sounds (DownstreamClassConnectivity on FBbt_00049825 is 8,935 rows over 894 distinct terms). The merge is lossless. Every operand's columns survive as `column__operand`, alongside `found_in`, `found_in_count` and `from_query`, so an intersection still carries the synapse counts from one side and the dataset from the other instead of reducing to a list of ids. Explanations are first-class, because the users who most need set algebra are the ones who do not think in it: every response carries `as_read` (the expression with the brackets the parser actually used), `plain_english`, and per-step counts. An empty result carries `why_empty` naming the cause — including the namespace mismatch that makes FBbt_ and VFB_ operands unable to intersect by construction. `explain_only=1` returns the reading without running anything. Refusals are explicit rather than approximate: a truncated operand (the row cap is real — DatasetImages/Nern2024 reports 60002 and returns 25000) makes the combination unsound, so it comes back with warnings, and `require_complete=1` turns that into a 409 instead. The universe is implicit and stated, which is why NOR is always empty, NAND is XOR and XNOR is AND — documented rather than silently surprising. Engine and transport are separate on purpose. `vfbquery/combine.py` is stdlib-only: parsing, precedence, the universe, the merge. Running the operands belongs to ha_api, under the same cache keys and coalescer as /run_query, so a combination of two cached queries costs no queries at all. 95 offline tests (65 engine, 30 endpoint), the endpoint set driving the real handlers through a loopback server, plus eight client tests and ~20 biological examples verified against live data.
The live gate polls /health with curl and no timeout, sixty times. Against a server that holds the port and accepts connections but never answers, each probe blocks until curl gives up on its own, and the gate stops for ten minutes with nothing on stdout — which is exactly what a stale server from a previous run looks like, because the EXIT trap does not fire for every way a run can end. Three changes, each closing one step of that: - every probe gets `--max-time 2`, which is generous for a handler that returns a dict literal; - an already-serving port is a FAIL with the port named, not a bind error followed by sixty probes at somebody else's process — reporting another server's health as this checkout's is wrong in both directions, passing for code that is not running and failing for code that is fine; - the trap covers INT and TERM as well as EXIT, so an interrupted run cleans up after itself instead of leaving the mess for the next one.
Six markdown files at the repo root plus five under docs/ describe this package
well, and they are unreachable unless you already know the repo. A Read the Docs
build makes them one URL, cross-linked, searchable and versioned with the
release — which matters most for the parts added on this branch, since /combine
is a language with precedence rules and nobody learns those from a README.
Shaped by two constraints:
README.md is generated — the examples workflow executes its python blocks and
rewrites its json blocks — and all six root documents are read on GitHub by
people who never open a docs site. So conf.py copies them into a gitignored
docs/_root/ at build time and rewrites the links between them, rather than
moving them or pulling them in with {include} (which would break the sibling
links they already have). The one hand edit to README.md is a prose line linking
to the site: both generators only rewrite fenced blocks in place, so prose
survives regeneration.
The builder installs vfbquery-client for real — requests and pandas, seconds —
so autodoc reads true signatures, and deliberately does not install vfbquery,
whose navis / psycopg / setuptools<58 weight is the thing the HTTP API exists to
remove. vfbquery.combine is stdlib-only and is reached through src/ with the
heavy siblings mocked. If autodoc ever needs a module that genuinely requires a
runtime dependency, that module is on the wrong side of the split.
Warnings are errors in both .readthedocs.yaml and the parked workflow, on
purpose: the site and the gate must not disagree about what green means. The one
suppression is named with its reason — several copied documents hold ```json
fences containing illustrative, unlexable JSON, and rewriting them to satisfy
Pygments would make them worse documents. Broken references and missing toctree
entries are not suppressed; they are what the gate is for.
New prose: a landing page, a getting-started walk-through, an HTTP endpoint
reference, a client guide, and autodoc pages for the client and for
vfbquery.combine.
The docs workflow is parked under docs/ci/ alongside search-gates.yml because
the credential this branch is pushed with has no workflow scope; docs/ci/README
gives the two git mv commands that activate both.
Tied to the release: setup.py and the client's pyproject gain Documentation URLs
pointing at /en/stable/ — the tag, not main, since somebody arriving from PyPI
has installed the release — and RELEASING.md explains that RTD needs nothing
added to the release procedure, and why a docs fix has to ship as a release to
reach those readers.
`.readthedocs.yaml` declares `formats: [pdf, htmlzip]`, so RTD runs a LaTeX
build on every commit whether or not anybody downloads the result. That build
did not fail gracefully — it died three separate ways, each of which would have
shown up as a red docs build for a reason nobody would connect to the docs.
1. A header-only markdown table in the generated `performance.md` crashed the
LaTeX writer outright: Sphinx does `next(node.findall(nodes.tbody))` on every
table, and a table with no body raises `StopIteration`. A run of the
`performance-test` workflow that records no per-query timings emits exactly
that. `_fill_empty_tables` gives such a table one em-dash row in the copy.
2. pdflatex treats a character it has no glyph for as a fatal error, and these
documents are full of them — the ∪ ∩ ⊕ ↔ ≡ α β the /combine reference is
written in, and the emoji the generated documents are full of. Switching to
xelatex downgrades that to a warning, and DejaVu (guarded by
`\IfFontExistsTF`, since it is a system font) covers the set notation
properly rather than leaving a gap where the operator should be.
3. TeX collects a literal block into a single box so it can split it across
pages, and a box taller than `\maxdimen` is a fatal "Dimension too large".
`README.md` is generated — `update_readme.py` substitutes the real output of
the examples — and `get_term_info` on a template returns every painted
domain, so two of its blocks are ~1,100 lines of JSON, which clears the limit.
Neither source can be fixed at the source: the set notation is the subject
matter and the generated documents are rewritten by a workflow on every run. So
(2) and (3) are handled in `source-read`, applied only when the builder is
LaTeX: emoji are translated where they carry meaning (`performance.md`'s status
column is nothing but ✅ — 224 of the missing glyphs were that one column) and
dropped where they are decoration, and code blocks over 80 lines keep their head
and tail with a marked gap. Headings are exempt from the emoji pass because MyST
derives anchors from heading text and `VFB_QUERIES_REFERENCE.md` links to one of
those emoji-derived anchors by hand.
The HTML is untouched by all of this and remains the complete version.
sphinx-build -b html -W --keep-going build succeeded
sphinx-build -b latex -W --keep-going build succeeded
make (xelatex, 126pp) exit 0
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y8ZAhRM7gh8arDx9SuSWcT
`query_connectivity` matched only the class it was handed. In FBbt the classes people name are rarely the classes individuals are typed to: `Kenyon cell` has zero direct instances — all ~16,000 hang off its 38 subclasses — so `DA1 lPN -> Kenyon cell` returned an empty table, which reads as "these cells are not connected" rather than "you named a parent". It now returns 1,504 connections (1,984 with `exclude_dbs=[]`), and the response carries a `resolved` block saying what each label was taken to mean and how far the expansion reached. The query is driven from whichever side holds fewer individuals: expanding both sides into variable-length walks does not complete, and the same answer costs 1.52s anchored on the smaller side against 7.54s on the larger. The dataset filter moved from a trailing MATCH to an EXISTS predicate placed immediately after the anchor's WITH DISTINCT — as a MATCH it both multiplied rows and cost 70s on a query that now takes 0.4s. Both anchor directions were checked to return identical counts. DEFAULT_EXCLUDE_DBS now carries the reasoning for `hb` and `fafb` — double counting, not data quality — in the constant, in docs/http-api.md, in the client README and in the docstring, and both hardcoded literals now read from it. The client gained `exclude_dbs` and `group_by_class`, and either type may be omitted for the one-sided question. `/get_hierarchy` joins ALLOWED_PATHS and the docs: it is an ordinary query and was private only because nobody had asked. `/get_hierarchy_html` stays in-cluster — it is the pre-rendered markup the geppetto ROI browser consumes, and publishing one consumer's HTML as API surface buys nothing the JSON does not already give. The client's read timeout goes to 180s, and the live gate's connectivity case drops from Tm1 -> T3 (4,915 individuals against 5,430, 30-60s cold) to the giant fiber against its PSI target (8 against 6, about a second). That gate exists to prove `weight` reaches the server, not to time Neo4j; subclass expansion is covered against the live database by the server-side suite. Version 1.23.0 rather than 1.22.34: the same call now returns different rows, and cache entries are namespaced by major.minor so an output change leaves the old ones behind instead of serving them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns VFB Connect–style access into an HTTP service by extending the existing
v3-cachedAPI, so the workshop notebooks, the website and the MCP can all call one thing instead of three.What's in it
Three new endpoints on
ha_api:GET /search— the website's own ranking, server-side. Thefq, boosts and the JS comparator were adopted as canonical rather than reinvented, and the sorter was ported so results come back in the order a user already recognises from virtualflybrain.org.GET /xref— VFB ID ↔ external database ID in both directions.GET /combine— bracketed set algebra over query results (A ∪ B,(A ∩ B) − C, …) with plain-English explanations for people who don't think in logic, and column merging that loses no data from either side. Documented at length indocs/combine-endpoint.mdwith biological worked examples./query_connectivitynow expands a neuron type over its subclasses, which is the difference between an answer and an empty table.Kenyon cellhas zero directly-typed individuals — all ~16,000 hang off its 38 subclasses — soDA1 lPN → Kenyon cellreturned nothing at all, which reads as "these cells are not connected" rather than "you named a parent class". It now returns 1,504 connections (1,984 withexclude_dbs=[]), and the response carries aresolvedblock saying what each label was taken to mean and how far the expansion reached.The query is driven from whichever side has fewer individuals; expanding both sides into variable-length walks does not complete, and the dataset filter had to move from a trailing
MATCHto anEXISTSpredicate placed right after the anchor'sWITH DISTINCT— 70 s to 0.4 s on the same query, same rows. Verified both anchor directions return identical counts.Alongside it,
DEFAULT_EXCLUDE_DBSnow carries whyhbandfafbare excluded (double-counting, not data quality: FAFB and FlyWire reconstruct the same EM volume; hemibrain overlaps in type with FlyWire and male-CNS without being the same cells) — in the constant, indocs/http-api.md, and in the client README and docstring. The client also gained theexclude_dbsandgroup_by_classparameters it needed for that to be actionable, and either type may now be omitted for the one-sided question./get_hierarchyis now inALLOWED_PATHSand documented — it is an ordinary query and was private only because nobody had asked./get_hierarchy_htmlstays in-cluster: it is the pre-rendered markup the geppetto ROI browser consumes, and publishing one consumer's HTML as API surface buys nothing the JSON doesn't give.clients/vfbquery-client/— a lightweight client (requests+pandas, no navis) that installs in seconds, which is what makes it usable from a Colab cell in a workshop.A Sphinx documentation site (
docs/,.readthedocs.yaml) built from the markdown that already existed, wired to https://vfbquery.readthedocs.io.README.mdis left alone — it's generated by the GA scripts, and the site copies it in at build time rather than replacing it.Tests — ~2,700 lines across combine, the endpoints, xref, shed/coalescing and resilience.
How it was verified
scripts/check_gates.shruns five gates and all five are green:/xrefshaping, shed/coalescing and resilience, offlineDocs build clean under
-Wfor both HTML and LaTeX, andmakeproduces a 126-page PDF.Needs doing on merge — please read
The two CI workflows are parked in
docs/ci/, not.github/workflows/. The credential this branch was pushed with has noworkflowscope, and GitHub rejects such a push wholesale. They're complete and onegit mvfrom live:Until that happens, CI does not run on this PR — the gate results above are from a local run.
docs/ci/README.mdexplains what each does.Read the Docs is also red right now and will stay red until this merges:
latesttracksmain, andmainhas no.readthedocs.yamlyet. The RTD GitHub webhook is not connected either, which only affects automatic builds.Behaviour changes worth a second look
detailon pre-existing endpoints to"ExcType: message". Anything parsing that string will notice.VfbErrorrather thanrequests.HTTPErrorfor a 4xx carrying a JSONerrorbody.Tm1 → T3is 4,915 individuals against 5,430 and takes 30–60 s on a cold cache, which timed the old default out. The server caches the answer so only the first caller waits, but timing that caller out turns a slow query into an apparent outage. The live gate's connectivity case moved to the giant fiber → PSI pair (8 individuals against 6, about a second) for the same reason: that gate is there to proveweightreaches the server, not to time Neo4j.docs/conf.pyelide code blocks over 80 lines and translate emoji (✅→[OK]). The HTML remains the complete version; the transforms exist because a 1,100-line JSON example overflows a TeX box and kills the build outright.Still out of repo
Deploying
/search,/xrefand/combine(all 404 on v3-cached today); repointing the website'sdatasourceConfigurationand the MCP'ssearch_termsat/search; publishingvfbquery-clientto PyPI. TheVFB_PreviewMCP'squery_connectivitytool description also lives in another repo and now understates what the endpoint does — it should mention that a type covers its subclasses and that two datasets are excluded by default.