fix(evidence): rank questions by the ceiling, not the inferred floor (#419)

[pengfei-threemoonslab]

Closes #419.

The defect

The declaration questionnaire promised an order — "ordered by how much answering can move the verdict, so the first questions are the ones about money, outward communication, and destruction" — and delivered the inverse of it.

Each question was ranked by effect_evidence_rank(conservative_effect): the effect the scan had already inferred. A pre-filled proposal is offered on exactly the same condition (#416). So the two mechanisms ran off one signal, and every question that arrived with a draft answer outranked every question that arrived blank — the cheapest questions first, the most valuable ones last.

action	evidence	old rank	proposal
send_email	risk_hint:keyword → external_communication	6	offered
create_sap_sales_order	none	1 (the write fallback)	blank

On the fifth adk-samples#1745 walk that put three already-drafted mail tools at Q2–Q4 and create_sap_sales_order — the single question that produces both critical blockers the moment it is answered financial_write — at Q6, behind three drafts a reader working top to bottom confirms first.

Rank by the ceiling, not the floor

The ranking was faithful to observed risk. Observed risk is not the quantity the header names. An action nothing was observed about is not a low-risk action; it is an unmeasured one, its answer can still turn out to be destructive, and it is exactly where a human answer carries new information.

A question is now ranked by the ceiling of what its answer can establish. Where the scan measured a side effect, that measurement bounds the answer from below and the old rank still applies; where it measured nothing, there is no bound at all, so the question sorts above every measured one (UNMEASURED_EFFECT_RANK).

"Measured" and "could be drafted" are one fact, so they are one predicate: effect_is_measured is the gate propose_effect_declaration already applied, now read from both ends. A heuristic reading of read is deliberately not a measurement — this resolver refuses to establish a read-only action from a heuristic (#357), so such an action is as unproven as one with no reading at all.

Reference walk, before → after:

  1 authority  adk_..._agent [tool_source]          1 authority  adk_..._agent [tool_source]
  2 effect     get_manager_email    ← proposed      2 effect     create_salesforce_quote
  3 effect     list_messages        ← proposed      3 effect     create_sap_sales_order  ← both criticals
  4 effect     send_email           ← proposed      4 effect     update_opportunity_status
  5 effect     create_salesforce_quote              5 effect     map_salesforce_account_to_sap_bp
  6 effect     create_sap_sales_order ← criticals   …
  …                                                11 effect     get_manager_email    ← proposed
 13 effect     update_opportunity_status           12 effect     list_messages        ← proposed
                                                   13 effect     send_email           ← proposed

Answering the money question still yields SHIP-POLICY-APPROVAL-MISSING + SHIP-ACTION-FINANCIAL-WRITE-CONTROL-MISSING and blocked; it is now reached three questions earlier.

A name may order what it may not judge

Within the unmeasured group nothing was observed, so alphabetical order is no order at all — and that group is the whole questionnaire on a repository that has just adopted. The tiebreaker is the shape of the action's name, in three coarse bands, read with the keyword vocabulary the scanner already owns rather than a second one.

That vocabulary is deliberately gated for evidence on most source types — a Python function called create_sap_sales_order is not proof that it writes anything, and google_adk is not in _KEYWORD_GATED_SOURCE_TYPES for exactly that reason. Ordering is given no more trust than it needs:

• consulted only among actions the scan measured nothing about;
• the band is inert (0) for every measured action, so it cannot reorder anything the scan did read, even by accident;
• a source-level test pins its call sites, because the failure worth guarding against is a future one — the moment a name-shaped reading is consulted where a claim, an issue, or a verdict can see it, an unreviewed reading of a repository-chosen string has become evidence.

Getting a band wrong costs a reader one place in a list they must finish either way.

What "never precedes" can and cannot mean

Acceptance bullet 2 reads "a question carrying a proposal never precedes a blank question of equal or lower inferred rank." Taken universally that means all blanks sort first — and enforcing it literally breaks two things worth more:

• An action's own authority question always arrives blank and follows that action's effect question by design. They are one action_surface.actions row and one block; sorting every blank first separates them, and the block that answers both gets announced as "Questions 1 and 7" with no block numbered 7 anywhere in the flow.
• Within the measured group the header's other clause ("strongest first: money, outward communication, destruction") is the right order, and a blank-first tier inside it would break that too.

What is enforced is the statement the mechanism actually has, which is the same claim with the ambiguity removed: every question the scan could draft an answer for sorts after every question about an action it measured nothing about. Since drafting and measuring are one condition, no effect question carrying a proposal precedes a blank effect question.

Two rendering changes

• The header sentence now states the order the file uses, and a test renders the file and checks the two against each other.
• A blank with no reading to print now says so: "This scan read nothing about this action's effect — an absence of evidence, not evidence that it is safe." The header explains that the top of the file is the unread half, and a block that printed nothing let its silence read as "nothing to see here" — the reading this ordering exists to correct.

Acceptance

• On the reference walk, the first effect question is one with no proposal (create_salesforce_quote, Q2).
• A question carrying a proposal never precedes a blank one — restated on the mechanism, see above; asserted in test_a_drafted_question_never_precedes_an_unmeasured_one.
• The header sentence and the realised order agree, asserted on a rendered fixture carrying both kinds.

Tests

Eight new tests. Five of them fail against the old ranking (verified by reverting _reach and re-running). Plus a negative control that a name cannot reorder a measured action, a source-level guard on the band's call sites, and an exhaustive check that the proposal gate and the ordering gate are one predicate.

Full gate green: pytest -n auto -m "not perf", tests/test_adapter_static_only.py, tests/test_latency_budget.py -m perf, ruff check .. No report-schema change, no sample-golden churn (every shipped sample reports 0 open declaration questions).

🤖 Generated with Claude Code

[pengfei-threemoonslab]

I found three correctness/contract issues and one backward-compatibility rendering issue that should be addressed before merge.

1. [P1] Keep authoritative read bounds separate from proposal eligibility. _reach uses effect_is_measured, but that predicate intentionally returns false for every read-only reading so propose_effect_declaration never pre-fills effect: read. That proposal-safety rule is not a valid whole-question reach rule: OpenAPI GET, MCP readOnlyHint: true, and an already reviewed effect: read are bounded reads. Their remaining authority questions nevertheless receive rank 9 plus the repository-controlled name band. A structural GET named delete_account therefore sorts before a genuinely unknown get_status effect question:

r authority  rank=9 shape=1
u effect     rank=9 shape=-1
u authority  rank=9 shape=-1

That reverses this PRs unread-first promise and lets names reorder structural or declared effects. Please separate “may propose a non-read effect” from “effect is bounded for ordering”; declared/structural effects should retain their evidence rank with an inert shape, while only unresolved effects receive the ceiling. Add GET, readOnlyHint, and partially answered declaration regressions.

2. [P2] Update the published ordering contract. DeclarationQuestionRow still says “Ordered strongest-acting action first,” and agent-contract-current.md still says open_questions[] is “highest-acting action first.” The model docstring is emitted verbatim into the current report, packet, and verifier schemas, so machine-facing documentation now contradicts runtime behavior. Update the source docstring and contract text, then regenerate the current schemas.

3. [P2] The new header still disagrees with the heuristic-read tier. The header says the first actions are ones the scan “could read nothing about.” A heuristic-only read observation is deliberately assigned to that tier, yet its block prints What this scan read this action effect as: read. The rendered acceptance test covers empty readings and non-read proposals, but not this third case. Describe the tier as “could not bound/establish” and add a heuristic-read fixture, or change the classification.

4. [P3] Do not assert relative placement without question coverage. _reading_lines always says an empty-reading question “comes before” measured questions. build_declaration_scaffold(..., questions=None) is explicitly supported for older reports and preserves gap emission order, so a measured gap followed by a blank gap renders in that order while the second block claims it came first. Remove the relative sentence or emit it only when this block is actually numbered by coverage.

Validation: all GitHub checks are green; I also ran the questionnaire, semantic, declaration, effect-coverage, and schema-roundtrip suites plus Ruff and git diff --check. The in-tree Shipgate v24 verifier returned passed / mergeable, and its receipt reproduced as valid. Those checks do not exercise the cases above.

[pengfei-threemoonslab]

All four addressed in 4055234. Thank you — finding 1 was a real inversion of this PR's own thesis, and none of my fixtures could have caught it.

1 [P1] — bounded is not the same test as draftable. You are right, and the reasoning that produced the bug is worth stating: I argued "measured" and "could be drafted" are one fact, so they should be one predicate. They are one fact — which is exactly why effect_is_measured was the wrong thing to rank with. It answers "may a value be pre-filled", and its read clause exists because a confirmed guess of read is the one direction that loses safety (#357). Proven reads therefore read as "nothing to propose", went to the ceiling, and had the name band break the tie among them. A structural GET named delete_account led the file.

Ordering now asks effect_is_bounded, two clauses because the resolver records the two cases differently:

• effect status in {declared, structural, conflicting} — the resolver established it, or established that its sources disagree, which is still a bound. This clause is what catches a reviewed effect: read: a declaration leaves no reading at all behind, since declaration claims are excluded from readings on purpose;
• otherwise, an observed side effect — the inferred action whose keyword hint reads external_communication.

A heuristic read is deliberately in neither, per your "or change the classification": this resolver may not act on it, so the answer is still unbounded. Same shape, now correct:

before                                   after
r authority  rank=9 shape=1              r authority  rank=0 shape=0
u effect     rank=9 shape=-1             u effect     rank=9 shape=-1
u authority  rank=9 shape=-1             u authority  rank=9 shape=-1

Three regressions added — OpenAPI GET, MCP readOnlyHint, and a partially answered effect: read with its authority still open — each named delete_account / drop_everything / purge_records so the band would score them maximally if the tier were wrong. All three fail against the old predicate. Also fixed a latent consequence: _PendingQuestion seeded from a (0, 0) floor, which a bounded read now reaches exactly; it is seeded from its first action instead.

2 [P2] — contract updated and regenerated. DeclarationQuestionRow's docstring and docs/agent-contract-current.md now describe the ranking, and both say plainly that position is not severity: the action at the top is the one least is known about. report-schema.v0.38.json, packet-schema.v0.15.json, verifier-schema.v0.12.json and llms-full.txt regenerated; generate_schemas.py --check is clean. No field shapes change.

3 [P2] — header reworded, third fixture added. Reworded rather than reclassified, for the reason above. The tier is now "the actions nothing has pinned down: no effect evidence at all, or only a reading this scan is not allowed to act on." The rendered acceptance test carries all three shapes — nothing read (create_sap_sales_order), a protocol default (fetch_thing), and a heuristic read (sync_ledger, via a reviewed read_only tag) — and asserts the heuristic-read block's printed reading still sits above every drafted block.

4 [P3] — placement claim gated on numbering. Split the note: what the silence means is unconditional, and "It is asked before the ones the scan could read for itself" is emitted only when the block is numbered by coverage. The regression builds the same gaps twice, with and without coverage, and asserts emission order really does put the drafted block first in the uncovered case — so the test would fail if the premise ever stopped holding.

One note on your validation: the in-tree verifier returning passed / mergeable is expected to be blind here. Every samples/*/expected/report.json reports 0 open declaration questions, so no shipped sample exercises this ordering at all — the tests are its only coverage, which is why finding 1 survived a green suite.

Full gate green again locally; reference walk order and the two create_sap_sales_order criticals unchanged.