Skip to content

feat(dialect-rustfs-admin): claim RustFS's metadata=true listings - #1600

Merged
overtrue merged 4 commits into
mainfrom
p9/1588-metadata-listing-extension
Nov 11, 2026
Merged

overtrue merged 4 commits into
mainfrom
p9/1588-metadata-listing-extension

Conversation

@overtrue

@overtrue overtrue commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1588

What

RustFS's metadata route claims two S3-shaped listings before its admin router and its S3 service:
GET /{bucket}?list-type=2&metadata=true and GET /{bucket}?versions&metadata=true. This PR serves
them in the RustFS dialect as rustfs:ListObjectsV2WithMetadata and
rustfs:ListObjectVersionsWithMetadata. Without these rows a gateway-fronted RustFS answers either
501 or a standard listing that silently drops the metadata the client asked for.

  • Inventory. scripts/rustfs_metadata_listing_routes.py is a narrow reader of
    rustfs/src/app/metadata_route.rs (metadata_operation and check_metadata_access), and
    gen_rustfs_admin_route_inventory.py records its two rows first in extension_routes. Each row
    has a new query_requires field (metadata=equals:true), a new claimed_by field, and the auth
    class SignedOrAnonymousThenBucketListing. I regenerated the JSON from the RustFS tree at the
    pinned commit 443e8afe (a tarball fetched by that SHA). Before the change, regenerating the
    untouched script reproduced the committed JSON byte for byte, so the method is faithful.
  • Generator (xtask/src/rustfs_admin_dialect/extension.rs). An extension row can now carry
    further query rules. Overlap with standard rows and with other extension rows is computed over
    every rule. A Claimer (admin router or metadata route) decides the evidence URL, the prose and
    the floor. FIRST_PRECEDENCE goes from 80 to 78, so the two new rows sit at 78 and 79 and the
    admin router's eight keep 80 to 87.
  • Floor. Both operations use admin::listing_floor = OperationFloor::builtin(name, S3). That
    is exactly the floor of ListObjectsV2 and ListObjectVersions, recorded in ADR-0050, which
    narrows ADR-0024 (f), ADR-0035 and ADR-0021 for these two operations only. A presigned ruling on
    a listing row is refused.
  • Legacy selection (crates/core/src/route/legacy_rustfs.rs). An Extension now takes a list
    of conditions, and the two listings come first, as RustFS asks its metadata route first.
  • Authorisation. s3:ListBucket (V2) and s3:ListBucketVersions (versions) on the bucket.
    There is no body codec: the host's handler builds the listing document, with each entry's
    <UserMetadata>, as an AdminResponse.

SigV2: what it admits, and the planned refusal

Where SigV2 is admitted. A SigV2 request reaches these two operations exactly where it reaches
the standard listings:

  • Presigned: refused without asking on the default floor. On the RustFS profile's floor
    (SigV2 compatibility, presigned on every standard operation) it is asked the listing action for
    the presigner.
  • Header-signed: accepted on every floor. SigV2 header authentication is on by default
    (SigV2Policy::HeaderOnly, docs/security-model.md).

Evidence for this:

  • Gateway: SecurityFloor::enforce_scheme_allowed (crates/sig/src/floor.rs) asks the same
    four questions of this floor and of ListObjectsV2's: a SigV4-only requirement, privilege, the
    SigV2 presigned switch, and the presigned policy.
  • RustFS: s3s at RustFS's pinned rev 5761ddfe verifies a matched custom route's signature,
    SigV2 included, before handing it the request (crates/s3s/src/ops/mod.rs prepare →
    crates/s3s/src/ops/signature.rs check). The metadata route then refuses no signature family
    itself, unlike the admin router.

The consequence. SigV2 signs only its fixed sub-resource list
(crates/sig/src/sig_v2/string_to_sign.rs), and neither metadata nor list-type is on it.
Under SigV2, whoever holds a listing URL, or has captured a SigV2 header-signed listing request,
can obtain every object's user metadata, which the signer did not sign for. They append
metadata=true, the signature still verifies, and the request becomes the metadata listing under
the signer's listing action on the same bucket. Before this PR the same edit could only widen the
listing itself, as an appended prefix can. It applies to:

  • a captured header-signed request, inside the skew window, on every floor;
  • a presigned URL, where SigV2 presigned URLs are admitted (the RustFS profile's floor, not the
    default floor).

Under SigV4 the same edit is refused, because SigV4 signs the whole query.

Pinned in both directions (crates/goldens/src/rustfs_admin_dialect/metadata_listing_tests.rs):

  • SigV2 presigned with metadata=true appended: admitted on the profile floor, refused without
    asking on the default floor.
  • SigV2 header-signed with metadata=true appended: admitted on both floors.
  • SigV2 header-signed with the signed versions key added, or with a forged Authorization:
    refused without asking on both floors.
  • The SigV4 presigned form of the edit: refused without asking.

ADR-0050 states the consequence in these words, and docs/security-model.md gains the note it
owes for a route selected by a parameter outside SigV2's signed set.

Ruling: parity in this PR. This PR keeps parity with RustFS, which serves the same requests
with the same s3s key set, and does not refuse SigV2. Planned change: a follow-up applies the
"SigV2 is refused under every policy" floor rule that #1592 is adding to these two
operations, which removes both forms. It is not implemented here.

Known gateway-versus-RustFS selection differences

From the independent review's ledger, with the trailing-slash rows now traced through the service:

Input (bucket GET) RustFS Gateway Action asked Direction
list-type=2&metadata=%74rue (any encoded true) V2 metadata listing ListObjectsV2 s3:ListBucket both gateway returns less
versions&metadata=%74rue versions metadata listing ListObjectVersions s3:ListBucketVersions both gateway returns less
list-type=%32&metadata=true V2 metadata listing table: ListObjects; legacy: ListObjectsV2 s3:ListBucket all gateway returns less; the two selections already differ here
%6Detadata=true, %76ersions, list%2Dtype=2 metadata listing (keys decoded) refused at the wire layer none fail closed (pinned)
list-type=2&list-type=1&metadata=true V2 metadata listing (first value) refused as a duplicate none fail closed (pinned)
/bkt/?list-type=2&metadata=true metadata listing metadata listing s3:ListBucket same (pinned)
/bkt//?list-type=2&metadata=true metadata listing (all slashes trimmed) object target; 501 in an assembly without GetObject none here differs (pinned)
/bkt%2Fkey?list-type=2&metadata=true metadata listing on a bucket named bkt/key profile: object target, GetObject s3:GetObject on the object differs; the RustFS side looks like its own bug
Website-domain host with metadata=true website route wins no website route n/a out of scope

+, mixed case and double encoding produce no difference: neither side can read true or the key
from them.

What the resolver could express

DialectRoute.selector is a &[Predicate] conjunction, so core's resolver already handled
Method ∧ Target ∧ QueryEquals("list-type","2") ∧ QueryEquals("metadata","true"). The
standard table already uses two-key rows such as intelligent-tiering + id. Two other pieces
were limited to one discriminator, and this PR extends both:

  • the generator (ExtensionRoute.query_discriminator, overlaps_standard, overlaps_extension
    and render each took one key);
  • legacy selection (Extension.when: Cond).

The predicate, and the evidence for it

Source: https://github.com/rustfs/rustfs/blob/443e8afee194fe38b32d62a9289a99c1f703715c/rustfs/src/app/metadata_route.rs.
In my own words: it is a bucket GET (path-style or virtual-host root); the first metadata value
must be exactly the case-sensitive string true; versions (present, any value) is tried before a
first list-type of 2; anything else goes on to the admin router and the S3 service; a signed
caller is resolved and an unsigned one stays anonymous; the action is ListBucketVersions or
ListBucket on the bucket.

So:

  • metadata=false, metadata=, a bare metadata, TRUE, True, 1 and truee resolve as today.
  • metadata=true with list-type=1, with no selector, on an object path, or on another
    sub-resource also resolves as today.
  • versions&list-type=2&metadata=true is the versions listing.

Known divergence. Values are compared as they arrived, as every routing value in the gateway
is. RustFS's selector decodes them, so metadata=%74rue and list-type=%32 are metadata listings
on RustFS and not here. No SigV4 client sends those spellings. Tests pin the current behaviour.

Route-table golden: unchanged

crates/core/tests/golden/route-table.txt renders only the standard table built from the pinned
AWS model (generated_entries() + SHADOWING, see crates/core/tests/golden.rs). Dialect rows are
declared through DialectBuilder::declare and never enter that rendering, and that is also true
of the eight existing extension rows. The golden test passes unchanged (git diff is empty), so
no existing row moved and no row was added there. The rows that do move are in the dialect's own
generated overlay (crates/dialect-rustfs-admin/src/table/overlay.rs, not protected):

Row Precedence Change Why
rustfs:ListObjectVersionsWithMetadata 78 added RustFS asks its metadata route before the admin router, and tries versions first
rustfs:ListObjectsV2WithMetadata 79 added second arm of the same selector
the eight admin-router rows 80 to 87 unchanged FIRST_PRECEDENCE moved to 78 for exactly this

Each new row declares a shadowing decision over every standard GET bucket row it can meet (for
example ListObjectsV2, ListObjectVersions, ListObjects and GetBucketAcl) and over the five
admin-router GET bucket rows. The versions row also shadows the V2 row. All of these are computed
from the built table and pinned by an_extension_row_shadows_exactly_the_rows_of_its_cell_it_can_be_named_with.
The eight existing operation modules change by one line each: requires: &[] in RECORD.

BREAKING

The PR touches no protected file except a new ADR and its index row. It is marked BREAKING because
hosts see different behaviour.

Migration. A host that installs the RustFS dialect and registers the standard listing
operations must also register handlers for rustfs:ListObjectsV2WithMetadata and
rustfs:ListObjectVersionsWithMetadata. fold_every_extension walks both. Each handler answers
the listing document with <UserMetadata> per entry, as RustFS's execute_list_objects_v2m and
execute_list_object_versions_m do.

If a host does not register them: a request carrying metadata=true with list-type=2 or
versions is answered 501 NotImplemented by name. It is no longer served as a standard listing
without the metadata. Requests without metadata=true are unaffected.

A host on the RustFS profile without the dialect installed (ServiceBuilder::rustfs_profile()
turns on legacy selection, and the dialect is a separate crate) now answers both shapes 501
before authentication. Before this PR it served the standard listing. Legacy selection names the
rustfs: operation and the table has no such row. crates/fs/tests/crud/listing.rs
n_a_metadata_listing_without_the_dialect_is_501_on_the_rustfs_profile pins both directions on
the reference backend under the profile:

  • the standard listings are still 200;
  • both metadata shapes are 501 NotImplemented with no listing body;
  • the same request without the profile is the standard listing.

Decision: 501 is kept. The alternative would make legacy selection fall back to the
standard listing when the dialect is absent. That is the silent drop of the extension this PR
exists to remove: the client asked for user metadata and would get a listing without it and no
signal. 501 tells it. Hosts that do not need the extension are unaffected unless their clients
send metadata=true. I checked the in-repo profile users: no test in crates/fs or
crates/difftest sends metadata=true, and the only new reference is the test above.
Migration for such a host: install rustfs_admin_dialect() and register the two handlers, or
accept the 501.

Under the whole RustFS profile, the posture line presigned_allowed_ops now names the two
operations, exactly where it names the standard listings. lambda_tests was updated to say so.

Tests

Negative cases outnumber positive ones.

File Positive Negative
crates/dialect-rustfs-admin/tests/integration/metadata_listing.rs 6 routing + 3 declaration 12
crates/goldens/src/rustfs_admin_dialect/metadata_listing_tests.rs 4 (+4 both ways) 7
crates/fs/tests/crud/listing.rs (profile, no dialect) 0 (+1 both ways) 0
crates/core/src/route/tests/legacy_rustfs.rs 1 1
xtask/src/rustfs_admin_dialect/metadata_listing_tests.rs 3 5
crates/goldens/src/migration_inventory/rustfs_admin_routes/tests.rs 1 5
scripts/test_metadata_listing_routes.py (new CI step) 1 9

The dialect integration tests run under both selections and compare with the dialect absent.
They cover:

  • every near miss: the metadata value, a missing or other listing selector, an object or service
    target, other methods, other sub-resources, other key spellings and repeats;
  • virtual-host and path-style forms;
  • precedence over x-id and the admin router's discriminators.

The floor test compares each floor with the standard listing's on privilege, schemes, SigV4
requirement, bearer claim and admission under all four anonymous × presigned policies. Both
directions are asserted: refused under the default floor, admitted under delegation.

The goldens tests cover the assembled service:

  • signed requests are asked exactly the listing action on the bucket;
  • unsigned requests are refused without asking on the default floor;
  • an unsigned request is asked as an anonymous caller on the RustFS profile floor, and refused
    before the handler when the policy denies it or allows only another action;
  • a presigned request is refused without asking on the default floor and admitted on the profile
    floor;
  • metadata=true appended to a presigned listing URL is refused without asking.

Existing tests adapted, with no assertion weakened:

  • extension index offsets, now shifted by the two leading rows;
  • counts: 8 → 10 records and 352 → 354 generated files;
  • the behind-a-standard-row test needs 13 synthetic rows instead of 11, because precedences now
    start at 78;
  • the two admin-router-only floor tests now iterate the eight admin-router records, which are
    unchanged;
  • the shadow-census test now models every query rule of a row.

Mutation table

Every mutation was applied to the PR head on its own:

# Mutation Went red (local)
M1 metadata=true dropped from the V2 row's selector and overlay record n_a_metadata_value_other_than_true_is_the_standard_listing, n_a_misspelled_key_is_unchanged, n_a_path_named_like_a_key_is_unchanged, n_the_two_selections_never_disagree_on_a_metadata_request, each_metadata_listing_declares_its_listing_action_codec_and_row
M2 listing_floor made privileged (OperationFloor::custom) each_metadata_listing_floor_is_exactly_its_standard_listings_floor
M3 listing_floor widened to builtin_presigned each_metadata_listing_floor_is_exactly_its_standard_listings_floor
M4 legacy selection loses the metadata gate on the V2 row n_a_metadata_listing_near_miss_is_the_ordinary_operation, two_operation_keys_select_the_first_in_legacy_order
M5 legacy selection's versions listing made unreachable a_metadata_listing_selects_the_dialect_operation_ahead_of_every_key
M6 generator overlap between extension rows ignores further rules n_rows_whose_rules_cannot_meet_declare_no_overlap
M7 generator stops pairing a claimer with its auth class n_a_claimer_with_the_other_claimers_auth_class_is_refused
M8 generator admits a presigned ruling on a listing row n_a_presigned_ruling_on_a_listing_row_is_refused
M9 Python reader accepts any metadata gate test_n_a_changed_gate_value_is_refused, test_n_a_case_insensitive_gate_is_refused
M10 inventory reader stops checking further rules n_an_extension_route_with_an_unknown_gate_rule_is_refused, n_an_extension_route_with_an_empty_gate_key_is_refused
M11 V2 listing authorised by s3:ListBucketVersions each_metadata_listing_declares_its_listing_action_codec_and_row
M12 listing_floor made SigV4-only (require_sigv4_under_every_policy) a_sigv2_presigned_listing_is_admitted_exactly_where_a_standard_one_is, metadata_true_appended_to_a_sigv2_presigned_listing_is_the_same_callers_listing
M13 legacy selection names neither metadata listing, as before this PR n_a_metadata_listing_without_the_dialect_is_501_on_the_rustfs_profile (fs), the_metadata_listing_is_ahead_of_every_named_sub_resource
M14 SigV2 signs metadata as a sub-resource metadata_true_appended_to_a_sigv2_presigned_listing_is_admitted_only_where_sigv2_presigned_is, metadata_true_appended_to_a_sigv2_header_signed_listing_is_the_signers_metadata_listing

Not mutated, because they pin behaviour this PR does not implement:

  • n_an_encoded_key_is_refused_before_routing (the wire layer's refusal of an encoded name);
  • the double-slash half of a_trailing_slash_bucket_path_is_the_listing_and_a_double_slash_is_an_object (path addressing).

All fourteen mutations ran locally (cargo test -p <crate> <filter>, and the Python reader test for M9), each applied alone to this head and reverted afterwards. CI runners were budgeted at one run per task, so the throwaway draft PR #1601 was closed and its run cancelled before it reported.

PR checklist

  • The four-command gate passed: fmt, clippy on changed crates, crate tests, and cargo xtask verify --crate rustfs-gateway-dialect-rustfs-admin. The full workspace runs in CI.
  • New or changed public API has rustdoc (admin::listing_floor, ExtensionRouteRecord::requires, the two operations)
  • Negative test cases outnumber positive ones
  • Every new assertion was mutated (table above)
  • No Protected File touched except a new ADR and its index row; BREAKING and a migration path are given above
  • No unsafe introduced
  • No inventory / linkme introduced
  • No #[non_exhaustive] added to a DTO
  • No s3s code copied, and no RustFS code copied; evidence is a URL plus a self-written summary
  • No credentials, signatures, or expected-signature values reachable from logs or error bodies
  • No plan document, analysis report, or working note committed
  • Commits follow Conventional Commits; PR title ≤72 characters
  • A Handoff comment has been appended to the issue

Role Verdicts

  • simplicity-adversary: attacked the new Claimer enum (two variants, both rendered and both tested), the requires field versus a second parallel extension mechanism (rejected: one generator path keeps the eight rows byte-identical except requires: &[]), the one-line admin::listing_floor (kept because the generated module names its floor through admin::, as for every other floor), and the separate Python reader module (one caller, split to keep the inventory script under 800 lines) — no break found
  • security-adversary: crates/sig/src/sig_v2/string_to_sign.rs:93 SigV2 signs neither metadata nor list-type, so metadata=true appended to a SigV2 presigned listing URL (RustFS profile floor) or to a captured SigV2 header-signed listing (every floor, inside the skew window) selects the metadata listing under the signer's listing action and discloses every object's user metadata; kept for RustFS parity, pinned in both directions in crates/goldens/src/rustfs_admin_dialect/metadata_listing_tests.rs, stated in ADR-0050 and docs/security-model.md, with the SigV2 refusal of rustfs:StsFormPost admits SigV2 where RustFS refuses it #1592 as the planned follow-up. Also attacked routing ambiguity (sub-resources, x-id, admin-router keys, versions with list-type=2, virtual-host and path style, encoded keys, trailing and double slashes, repeats, case), the floor against ListObjectsV2's under all four policies, anonymous requests without a policy, the wrong listing action, SigV4 presigned append, and the RustFS profile without the dialect (501): no further break

RustFS's metadata route claims GET /{bucket}?list-type=2&metadata=true and
GET /{bucket}?versions&metadata=true ahead of its admin router and S3
service. Record both in the route inventory (read from metadata_route.rs),
generate them as rustfs:ListObjectsV2WithMetadata and
rustfs:ListObjectVersionsWithMetadata at precedences 78 and 79, each keyed
on its listing selector plus metadata=true, on the standard listing floor
(ADR-0050), and teach the legacy selection the same two rows.

Fixes #1588
Support ADR-0050's SigV2 presigned sentence with the gateway's scheme
check and s3s's custom-route signature path at RustFS's pinned revision,
record that SigV2 does not sign metadata or list-type, so an appended
metadata=true on a SigV2 presigned V2 listing selects the metadata
listing for the same caller and action, and pin both with goldens tests.
The independent review of #1600 found two statements untrue as written.
Pin, both ways and in both SigV2 forms, that an appended metadata=true
on a SigV2 presigned or captured header-signed listing selects the
metadata listing for the signer wherever that form is admitted, and
state that consequence and the planned SigV2 refusal (#1592) in ADR-0050
and docs/security-model.md. Pin the 501 a RustFS-profile host without
the dialect now answers, on the filesystem reference backend. Add the
encoded-key, trailing-slash and sub-resource cases.
@overtrue

Copy link
Copy Markdown
Collaborator Author

The independent review's findings are addressed in bff179e8. CI is green on cac25834, the merge of main into this branch: 34 of 34 checks pass.

  • Finding 1 (SigV2):
    • The SigV4 test's doc now says the guarantee is SigV4-only.
    • The SigV2 property is pinned in both directions for both forms. Presigned: admitted on the profile floor, refused without asking on the default floor. Header-signed: admitted on both floors, refused when a signed key is added or the Authorization is forged.
    • The consequence is stated plainly in ADR-0050 and docs/security-model.md.
    • The SigV2 refusal of rustfs:StsFormPost admits SigV2 where RustFS refuses it #1592 is recorded as the planned follow-up and is not implemented here.
  • Finding 2 (migration):
    • The BREAKING note now covers a RustFS-profile host without the dialect.
    • crates/fs/tests/crud/listing.rs pins that case on the reference backend: 200 for the standard listings, 501 for both metadata shapes.
    • The 501 is kept, and the PR description gives the reasoning.
    • No crates/fs or crates/difftest test sends metadata=true.
  • Findings 3 and 4:
    • New cases cover encoded keys (refused at the wire layer), a trailing slash (still the listing), a double slash (an object), and sub-resources beside the selectors as real requests.
    • The review's difference ledger is in the description, with the slash rows traced.
  • Mutations M13 and M14 were run locally; both went red.

@overtrue
overtrue marked this pull request as ready for review November 11, 2026 09:49
@overtrue
overtrue merged commit baf0fb2 into main Nov 11, 2026
34 checks passed
@overtrue
overtrue deleted the p9/1588-metadata-listing-extension branch November 11, 2026 09:49
overtrue added a commit that referenced this pull request Nov 11, 2026
overtrue added a commit that referenced this pull request Nov 11, 2026
Takes #1600 (ADR-0050, the metadata listings). This branch's ADR becomes
ADR-0051 (file, title, index row and every reference of this branch;
#1600's own ADR-0050 references are untouched).

Extends the SigV2 refusal to rustfs:ListObjectsV2WithMetadata and
rustfs:ListObjectVersionsWithMetadata through admin::listing_floor and
the generator's floor text. SigV2 does not sign `metadata`, so an
appended metadata=true turned a SigV2 listing into a user-metadata
disclosure. SigV4 and unsigned listings stay exactly as on the
standard listings, which keep admitting SigV2. This is a deliberate
difference from RustFS, which serves the SigV2 form (recorded in
ADR-0051).

Tests flipped deliberately:
- metadata_listing_tests: the three tests that pinned SigV2 presigned,
  appended-presigned and appended-header listings as admitted are now
  refusals (403 AccessDenied, no authorizer question) on both floors,
  plus a correctly signed SigV2 metadata listing, also refused;
- dialect metadata_listing: the floor equals the standard listing
  floor except refuses_sigv2, which the standard floors do not;
- #1583's SigV2 census includes the zip rows (670-8), its control keeps
  only the OIDC row, and the dialect census record becomes
  the_zip_download_refuses_sigv2_without_requiring_sigv4.

Also: the sig test-target guard lists sigv2_refused_floor. This fixes
both CI failures on 88147f9 (Static checks; Guard self-test 5,
"fails on the current tree"). The redundant zip assertions are dropped
from extensions.rs to keep it within 800 lines; census.rs keeps them.

Mutations, all run locally, each reverted, all red:
M1 floor ignores the rule; M2 predicate loses the SigV4 implication;
M3 setter also requires SigV4; M4 refusal after the credential parse;
M5 STS floor without the rule; M6 zip floor without the rule (census
included); M7 legacy stand-in admits SigV2; M8 deaf credential
recorder; M9 no-op unknown-key rewrite; M10 and M11 both registry
directions; M12 the rule refuses every family; M13 predicate always
true; M14 metadata listing floor without the rule.
overtrue added a commit that referenced this pull request Nov 11, 2026
main gained ADR-0050 with #1600, so the record of this change
moves to the next free number: its file, title, index row and every
reference to it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Claim RustFS's metadata=true listing extension as dialect operations

1 participant