Skip to content

Narrowing a question by industry made it fail: rank how narrow a side is, instead of asking whether it is - #58

Merged
prayaslashkari merged 3 commits into
developmentfrom
fix/narrowness-ranking
Sep 20, 2026
Merged

prayaslashkari merged 3 commits into
developmentfrom
fix/narrowness-ranking

Conversation

@prayaslashkari

Copy link
Copy Markdown
Collaborator

Follow-up to #55, found by doing the next obvious thing with the map #55 had just repaired.

#55 made "what facilities are within 5 km upstream from PFOS samples in York County" answer in 3s. It returns 418 facilities across 261 distinct industries, including 21 dental offices, 12 hospitals and 8 schools, because block A carries no industry filter and "facility" means anything with a NAICS code. The obvious next step is to filter block A to PFAS source categories.

That returned nothing: a 30s timeout in query planning, on both projections. Adding a filter to a working question made it fail.


The cause, one level above #55

sideIsConstrained answered yes for a region, an entity filter or an IRI pin alike. An industry filter therefore made the anchor count as constrained, the reorder from #55 stopped firing, and the query seeded from block A again. Only now block A meant every facility in that industry nationwide.

The comment two lines above the predicate already read "narrowed is not small: a county of wells is narrowed and still enormous". The predicate did not act on its own warning.

Isolated by varying one thing at a time, all at 5 km, York County, detections only:

Block A Result
no filter, no region works, 418 facilities, 3s
1 code, no region 429, query planning
21 codes, region Maine 429 Cartesian Product, then 500 at 728 MB
21 codes, region York 500 at 728 MB
1 code, region York works, 21 facilities, 13s

Block A had to be small, not merely filtered, and only a region made it small.

The fix

sideIsConstrained becomes sideNarrowness, returning a rank, and the body leads with the higher:

3  an IRI pin        the executor chose this slice; re-seeding discards it
2  a region          bounded by geography, and the graph is indexed for it
1  an entity filter  selective in kind, unbounded in extent
0  nothing

Ties keep the anchor-first order, so this only changes questions where both sides are constrained at different levels. Everything where one side was constrained and the other was not keeps the plan it had. That containment is measured, not argued: of the 375 shapes in the query snapshot, 2 move, both variants with a region on one side and a bare filter on the other.

Measured on the live endpoint, block A carrying no region throughout:

Block A Before After
1 code, unbounded 429, query planning 35 facilities, 16s
1 code, 5 km 429, query planning 22 facilities, 8s
8 codes, 5 km 429, query planning 88 facilities, 3s
21 codes, 5 km 429, query planning 101 facilities, 13s
21 codes, unbounded 429 still fails, 430 MB

The filtered map is the useful one: 101 candidates instead of 418, dentists and schools gone, with sewage treatment, dry cleaners, boatyards, airports and petroleum terminals at the top.

One dashboard question changes plan

samples-downstream-waste-indiana scopes samples to Indiana and filters block C to NAICS 5622 with no region, so its anchor is nationwide and its target is one state. Both orders were run before accepting the change: identical answers, 136 samples and 415 facilities either way, reordered no slower (13s and 2s against 16s and 3s).

check-query-joins had asserted that no prebuilt may reorder. That described the old boolean rather than a rule, so it now expects this one to, and records the measurement as the reason.

Guardrails

30 new assertions in check-query-joins (307 to 337) pin the ranking in both directions: a bare filter must not outrank a region, a region must outrank a bare filter, and for bounded queries the aggregate must seed from the side the body leads with. Mutation-tested: reverting the rank to the old boolean fails them.

All seven checks pass, including the 375-shape snapshot from #55, which is what bounds the blast radius to 2 shapes.

A correction to #55's docs, included here

#55 shipped three documents saying industry selections beyond about ten codes fail, and blaming FILTER(?ic = ?sel || EXISTS { ?ic fio:subcodeOf ?sel }). Both halves are wrong, and both came from a measurement taken before this ranking landed.

  • The limit is not code count, it is code count without a bound. 21 codes answers at 5 km (101 facilities, 13s) and fails only unbounded. A long selection is a reason to set a distance, not to shorten the list.
  • The clause is innocent, and the obvious optimisation is worse. fio:subcodeOf is materialised transitively (325110 is a direct subcode of 32, 325 and 3251), so ?ic fio:subcodeOf? ?sel is an exact rewrite. Measured: 13s with the FILTER, query-planning timeout with the path. Recorded as a rejected design.

Review notes

Three commits: the fix and its assertions, the docs, and the correction above. The snapshot file moves because 2 shapes changed plan; the rest of its 1,400 lines are untouched.

Worth knowing: every correction in this PR came from re-running a measurement rather than from reasoning, and reasoning would have gone the wrong way each time.

Adding an industry filter to block A turned a working question into a failing
one. "What facilities are within 5 km upstream from PFOS samples in York
County" answers in 3s with block A empty; add NAICS 221320 to it and both
projections time out in query planning at 30s.

`sideIsConstrained` was a boolean: region, filter or IRI pin, any of them means
constrained. An industry filter set it to true, the reorder stopped firing, and
the query went back to seeding from block A, which now meant every sewage
treatment facility in the country rather than the 118 sample points in one
county. The comment two lines above it already said "narrowed is not small"; the
predicate did not act on its own warning.

It now returns a rank, and the body leads with the higher one:

  3  an IRI pin. The executor chose this slice.
  2  a region. Bounded by geography, and the graph is indexed for it.
  1  an entity filter only. Selective in kind, unbounded in extent.
  0  nothing.

Ties keep the anchor-first order, so this only changes cases where both sides
are constrained at different levels. Everything where one side was constrained
and the other was not keeps the plan it had, which the snapshot confirms: of 375
shapes, 2 move, both variants with a region on one side and a bare filter on the
other.

Measured on the live endpoint, all with block A carrying no region:

  1 code,  unbounded   429 planning timeout  ->  35 facilities, 16s
  1 code,  5 km        429 planning timeout  ->  22 facilities,  8s
  8 codes, 5 km        429 planning timeout  ->  88 facilities,  3s
  8 codes, unbounded   429 planning timeout  -> 192 facilities, 15s

21 codes still fails. That is a different cost, the `|| EXISTS { ?ic
fio:subcodeOf ?sel }` clause evaluated per selected code and duplicated inside
the bounded aggregate, and it is not addressed here.

One shipped dashboard question changes plan. `samples-downstream-waste-indiana`
has samples scoped to Indiana and NAICS 5622 with no region, so its anchor is
nationwide and its target is one state. Both orders were run: identical answers,
136 samples and 415 facilities either way, reordered no slower at 13s and 2s
against 16s and 3s. `check-query-joins` asserted that no prebuilt may reorder,
which described the old predicate rather than a rule; it now expects this one to
and says why.

30 new assertions pin the ranking in both directions, and a mutation reverting
it to the boolean fails them.
QUERY-MATRIX.md Part 11: what the boolean predicate cost, the rank that replaced
it, the measured before and after with block A unscoped, and the two things it
does not fix.

DEBUGGING.md gains a second 2026-09-20 entry, with the one-variable isolation
table showing that block A had to be small rather than merely filtered, and the
note that a long industry selection fails for a different reason.

W38 entry 17. Written to be read next to entry 15, since this is the same class
of bug one level up and was found by doing the obvious next thing with the map
entry 15 fixed.
…r it

Yesterday's entries said industry selections beyond about ten codes fail, and
blamed the `FILTER(?ic = ?sel || EXISTS { ?ic fio:subcodeOf ?sel })` clause.
Both halves were wrong, and both were written from a measurement taken before
the narrowness ranking landed.

21 codes answers fine at 5 km: 101 facilities in 13s, 84 samples in 23s. It
fails only unbounded, at 430 MB. Code count is a cost on the closure, and a
distance bound already caps it, so a long selection is a reason to set a
distance rather than to shorten the list.

    codes   5 km                    unbounded
    1       22 facilities,  8s      35 facilities, 16s
    8       88 facilities,  3s     192 facilities, 15s
    21     101 facilities, 13s     500, out of memory at 430 MB

The clause is cleared as well, by trying the fix rather than reasoning about it.
`fio:subcodeOf` is materialised transitively (325110 is a direct subcode of 32,
325 and 3251), so `?ic fio:subcodeOf? ?sel` is an exact rewrite of the FILTER.
Measured on the 21-code bounded query: 13s with the FILTER, query-planning
timeout with the path. The FILTER is the fast form. Recorded as a rejected
design so nobody optimises it again.
@railway-app

railway-app Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

🚅 Deployed to the explorer-app-pr-58 environment in sawgraph-explorer

Service Status Web Updated
sawgraph-api ✅ Success (View Logs) Web Sep 20, 2026 at 7:04 am UTC
sawgraph-web ✅ Success (View Logs) Web Sep 20, 2026 at 6:40 am UTC

@railway-app
railway-app Bot temporarily deployed to sawgraph-explorer / explorer-app-pr-58 September 20, 2026 06:40 Destroyed
@prayaslashkari
prayaslashkari merged commit 01a6899 into development Sep 20, 2026
3 checks passed
@prayaslashkari
prayaslashkari deleted the fix/narrowness-ranking branch September 20, 2026 07:17

This branch was successfully deployed

No deployments
sawgraph-explorer / explorer-app-pr-58 — e73d84c6 Deployed Sep 20, 2026 by railway-app[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant