Repository navigation
feat(dialect-rustfs-admin): claim RustFS's metadata=true listings - #1600
Merged
Merged
Conversation
RustFS's metadata route claims GET /{bucket}?list-type=2&metadata=true and
GET /{bucket}?versions&metadata=true ahead of its admin router and S3
service. Record both in the route inventory (read from metadata_route.rs),
generate them as rustfs:ListObjectsV2WithMetadata and
rustfs:ListObjectVersionsWithMetadata at precedences 78 and 79, each keyed
on its listing selector plus metadata=true, on the standard listing floor
(ADR-0050), and teach the legacy selection the same two rows.
Fixes #1588
This was referenced Nov 11, 2026
Support ADR-0050's SigV2 presigned sentence with the gateway's scheme check and s3s's custom-route signature path at RustFS's pinned revision, record that SigV2 does not sign metadata or list-type, so an appended metadata=true on a SigV2 presigned V2 listing selects the metadata listing for the same caller and action, and pin both with goldens tests.
The independent review of #1600 found two statements untrue as written. Pin, both ways and in both SigV2 forms, that an appended metadata=true on a SigV2 presigned or captured header-signed listing selects the metadata listing for the signer wherever that form is admitted, and state that consequence and the planned SigV2 refusal (#1592) in ADR-0050 and docs/security-model.md. Pin the 501 a RustFS-profile host without the dialect now answers, on the filesystem reference backend. Add the encoded-key, trailing-slash and sub-resource cases.
Collaborator
Author
|
The independent review's findings are addressed in
|
overtrue
marked this pull request as ready for review
November 11, 2026 09:49
13 of 14 tasks
overtrue
added a commit
that referenced
this pull request
Nov 11, 2026
Renumber this branch's ADR to 0051: #1600 took 0050.
overtrue
added a commit
that referenced
this pull request
Nov 11, 2026
Takes #1600 (ADR-0050, the metadata listings). This branch's ADR becomes ADR-0051 (file, title, index row and every reference of this branch; #1600's own ADR-0050 references are untouched). Extends the SigV2 refusal to rustfs:ListObjectsV2WithMetadata and rustfs:ListObjectVersionsWithMetadata through admin::listing_floor and the generator's floor text. SigV2 does not sign `metadata`, so an appended metadata=true turned a SigV2 listing into a user-metadata disclosure. SigV4 and unsigned listings stay exactly as on the standard listings, which keep admitting SigV2. This is a deliberate difference from RustFS, which serves the SigV2 form (recorded in ADR-0051). Tests flipped deliberately: - metadata_listing_tests: the three tests that pinned SigV2 presigned, appended-presigned and appended-header listings as admitted are now refusals (403 AccessDenied, no authorizer question) on both floors, plus a correctly signed SigV2 metadata listing, also refused; - dialect metadata_listing: the floor equals the standard listing floor except refuses_sigv2, which the standard floors do not; - #1583's SigV2 census includes the zip rows (670-8), its control keeps only the OIDC row, and the dialect census record becomes the_zip_download_refuses_sigv2_without_requiring_sigv4. Also: the sig test-target guard lists sigv2_refused_floor. This fixes both CI failures on 88147f9 (Static checks; Guard self-test 5, "fails on the current tree"). The redundant zip assertions are dropped from extensions.rs to keep it within 800 lines; census.rs keeps them. Mutations, all run locally, each reverted, all red: M1 floor ignores the rule; M2 predicate loses the SigV4 implication; M3 setter also requires SigV4; M4 refusal after the credential parse; M5 STS floor without the rule; M6 zip floor without the rule (census included); M7 legacy stand-in admits SigV2; M8 deaf credential recorder; M9 no-op unknown-key rewrite; M10 and M11 both registry directions; M12 the rule refuses every family; M13 predicate always true; M14 metadata listing floor without the rule.
This was referenced Nov 11, 2026
overtrue
added a commit
that referenced
this pull request
Nov 11, 2026
main gained ADR-0050 with #1600, so the record of this change moves to the next free number: its file, title, index row and every reference to it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1588
What
RustFS's metadata route claims two S3-shaped listings before its admin router and its S3 service:
GET /{bucket}?list-type=2&metadata=trueandGET /{bucket}?versions&metadata=true. This PR servesthem in the RustFS dialect as
rustfs:ListObjectsV2WithMetadataandrustfs:ListObjectVersionsWithMetadata. Without these rows a gateway-fronted RustFS answers either501or a standard listing that silently drops the metadata the client asked for.scripts/rustfs_metadata_listing_routes.pyis a narrow reader ofrustfs/src/app/metadata_route.rs(metadata_operationandcheck_metadata_access), andgen_rustfs_admin_route_inventory.pyrecords its two rows first inextension_routes. Each rowhas a new
query_requiresfield (metadata=equals:true), a newclaimed_byfield, and the authclass
SignedOrAnonymousThenBucketListing. I regenerated the JSON from the RustFS tree at thepinned commit
443e8afe(a tarball fetched by that SHA). Before the change, regenerating theuntouched script reproduced the committed JSON byte for byte, so the method is faithful.
xtask/src/rustfs_admin_dialect/extension.rs). An extension row can now carryfurther query rules. Overlap with standard rows and with other extension rows is computed over
every rule. A
Claimer(admin router or metadata route) decides the evidence URL, the prose andthe floor.
FIRST_PRECEDENCEgoes from 80 to 78, so the two new rows sit at 78 and 79 and theadmin router's eight keep 80 to 87.
admin::listing_floor=OperationFloor::builtin(name, S3). Thatis exactly the floor of
ListObjectsV2andListObjectVersions, recorded in ADR-0050, whichnarrows ADR-0024 (f), ADR-0035 and ADR-0021 for these two operations only. A presigned ruling on
a listing row is refused.
crates/core/src/route/legacy_rustfs.rs). AnExtensionnow takes a listof conditions, and the two listings come first, as RustFS asks its metadata route first.
s3:ListBucket(V2) ands3:ListBucketVersions(versions) on the bucket.There is no body codec: the host's handler builds the listing document, with each entry's
<UserMetadata>, as anAdminResponse.SigV2: what it admits, and the planned refusal
Where SigV2 is admitted. A SigV2 request reaches these two operations exactly where it reaches
the standard listings:
(SigV2 compatibility, presigned on every standard operation) it is asked the listing action for
the presigner.
(
SigV2Policy::HeaderOnly,docs/security-model.md).Evidence for this:
SecurityFloor::enforce_scheme_allowed(crates/sig/src/floor.rs) asks the samefour questions of this floor and of
ListObjectsV2's: a SigV4-only requirement, privilege, theSigV2 presigned switch, and the presigned policy.
5761ddfeverifies a matched custom route's signature,SigV2 included, before handing it the request (
crates/s3s/src/ops/mod.rsprepare→crates/s3s/src/ops/signature.rscheck). The metadata route then refuses no signature familyitself, unlike the admin router.
The consequence. SigV2 signs only its fixed sub-resource list
(
crates/sig/src/sig_v2/string_to_sign.rs), and neithermetadatanorlist-typeis on it.Under SigV2, whoever holds a listing URL, or has captured a SigV2 header-signed listing request,
can obtain every object's user metadata, which the signer did not sign for. They append
metadata=true, the signature still verifies, and the request becomes the metadata listing underthe signer's listing action on the same bucket. Before this PR the same edit could only widen the
listing itself, as an appended
prefixcan. It applies to:default floor).
Under SigV4 the same edit is refused, because SigV4 signs the whole query.
Pinned in both directions (
crates/goldens/src/rustfs_admin_dialect/metadata_listing_tests.rs):metadata=trueappended: admitted on the profile floor, refused withoutasking on the default floor.
metadata=trueappended: admitted on both floors.versionskey added, or with a forgedAuthorization:refused without asking on both floors.
ADR-0050 states the consequence in these words, and
docs/security-model.mdgains the note itowes for a route selected by a parameter outside SigV2's signed set.
Ruling: parity in this PR. This PR keeps parity with RustFS, which serves the same requests
with the same s3s key set, and does not refuse SigV2. Planned change: a follow-up applies the
"SigV2 is refused under every policy" floor rule that #1592 is adding to these two
operations, which removes both forms. It is not implemented here.
Known gateway-versus-RustFS selection differences
From the independent review's ledger, with the trailing-slash rows now traced through the service:
GET)list-type=2&metadata=%74rue(any encodedtrue)ListObjectsV2s3:ListBucketbothversions&metadata=%74rueListObjectVersionss3:ListBucketVersionsbothlist-type=%32&metadata=trueListObjects; legacy:ListObjectsV2s3:ListBucketall%6Detadata=true,%76ersions,list%2Dtype=2list-type=2&list-type=1&metadata=true/bkt/?list-type=2&metadata=trues3:ListBucket/bkt//?list-type=2&metadata=true501in an assembly withoutGetObject/bkt%2Fkey?list-type=2&metadata=truebkt/keyGetObjects3:GetObjecton the objectmetadata=true+, mixed case and double encoding produce no difference: neither side can readtrueor the keyfrom them.
What the resolver could express
DialectRoute.selectoris a&[Predicate]conjunction, so core's resolver already handledMethod ∧ Target ∧ QueryEquals("list-type","2") ∧ QueryEquals("metadata","true"). Thestandard table already uses two-key rows such as
intelligent-tiering+id. Two other pieceswere limited to one discriminator, and this PR extends both:
ExtensionRoute.query_discriminator,overlaps_standard,overlaps_extensionand
rendereach took one key);Extension.when: Cond).The predicate, and the evidence for it
Source: https://github.com/rustfs/rustfs/blob/443e8afee194fe38b32d62a9289a99c1f703715c/rustfs/src/app/metadata_route.rs.
In my own words: it is a bucket
GET(path-style or virtual-host root); the firstmetadatavaluemust be exactly the case-sensitive string
true;versions(present, any value) is tried before afirst
list-typeof2; anything else goes on to the admin router and the S3 service; a signedcaller is resolved and an unsigned one stays anonymous; the action is
ListBucketVersionsorListBucketon the bucket.So:
metadata=false,metadata=, a baremetadata,TRUE,True,1andtrueeresolve as today.metadata=truewithlist-type=1, with no selector, on an object path, or on anothersub-resource also resolves as today.
versions&list-type=2&metadata=trueis the versions listing.Known divergence. Values are compared as they arrived, as every routing value in the gateway
is. RustFS's selector decodes them, so
metadata=%74rueandlist-type=%32are metadata listingson RustFS and not here. No SigV4 client sends those spellings. Tests pin the current behaviour.
Route-table golden: unchanged
crates/core/tests/golden/route-table.txtrenders only the standard table built from the pinnedAWS model (
generated_entries()+SHADOWING, seecrates/core/tests/golden.rs). Dialect rows aredeclared through
DialectBuilder::declareand never enter that rendering, and that is also trueof the eight existing extension rows. The golden test passes unchanged (
git diffis empty), sono existing row moved and no row was added there. The rows that do move are in the dialect's own
generated overlay (
crates/dialect-rustfs-admin/src/table/overlay.rs, not protected):rustfs:ListObjectVersionsWithMetadataversionsfirstrustfs:ListObjectsV2WithMetadataFIRST_PRECEDENCEmoved to 78 for exactly thisEach new row declares a shadowing decision over every standard
GETbucket row it can meet (forexample
ListObjectsV2,ListObjectVersions,ListObjectsandGetBucketAcl) and over the fiveadmin-router
GETbucket rows. The versions row also shadows the V2 row. All of these are computedfrom the built table and pinned by
an_extension_row_shadows_exactly_the_rows_of_its_cell_it_can_be_named_with.The eight existing operation modules change by one line each:
requires: &[]inRECORD.BREAKING
The PR touches no protected file except a new ADR and its index row. It is marked BREAKING because
hosts see different behaviour.
Migration. A host that installs the RustFS dialect and registers the standard listing
operations must also register handlers for
rustfs:ListObjectsV2WithMetadataandrustfs:ListObjectVersionsWithMetadata.fold_every_extensionwalks both. Each handler answersthe listing document with
<UserMetadata>per entry, as RustFS'sexecute_list_objects_v2mandexecute_list_object_versions_mdo.If a host does not register them: a request carrying
metadata=truewithlist-type=2orversionsis answered501 NotImplementedby name. It is no longer served as a standard listingwithout the metadata. Requests without
metadata=trueare unaffected.A host on the RustFS profile without the dialect installed (
ServiceBuilder::rustfs_profile()turns on legacy selection, and the dialect is a separate crate) now answers both shapes
501before authentication. Before this PR it served the standard listing. Legacy selection names the
rustfs:operation and the table has no such row.crates/fs/tests/crud/listing.rsn_a_metadata_listing_without_the_dialect_is_501_on_the_rustfs_profilepins both directions onthe reference backend under the profile:
200;501 NotImplementedwith no listing body;Decision:
501is kept. The alternative would make legacy selection fall back to thestandard listing when the dialect is absent. That is the silent drop of the extension this PR
exists to remove: the client asked for user metadata and would get a listing without it and no
signal.
501tells it. Hosts that do not need the extension are unaffected unless their clientssend
metadata=true. I checked the in-repo profile users: no test incrates/fsorcrates/difftestsendsmetadata=true, and the only new reference is the test above.Migration for such a host: install
rustfs_admin_dialect()and register the two handlers, oraccept the
501.Under the whole RustFS profile, the posture line
presigned_allowed_opsnow names the twooperations, exactly where it names the standard listings.
lambda_testswas updated to say so.Tests
Negative cases outnumber positive ones.
crates/dialect-rustfs-admin/tests/integration/metadata_listing.rscrates/goldens/src/rustfs_admin_dialect/metadata_listing_tests.rscrates/fs/tests/crud/listing.rs(profile, no dialect)crates/core/src/route/tests/legacy_rustfs.rsxtask/src/rustfs_admin_dialect/metadata_listing_tests.rscrates/goldens/src/migration_inventory/rustfs_admin_routes/tests.rsscripts/test_metadata_listing_routes.py(new CI step)The dialect integration tests run under both selections and compare with the dialect absent.
They cover:
metadatavalue, a missing or other listing selector, an object or servicetarget, other methods, other sub-resources, other key spellings and repeats;
x-idand the admin router's discriminators.The floor test compares each floor with the standard listing's on privilege, schemes, SigV4
requirement, bearer claim and admission under all four anonymous × presigned policies. Both
directions are asserted: refused under the default floor, admitted under delegation.
The goldens tests cover the assembled service:
before the handler when the policy denies it or allows only another action;
floor;
metadata=trueappended to a presigned listing URL is refused without asking.Existing tests adapted, with no assertion weakened:
start at 78;
unchanged;
Mutation table
Every mutation was applied to the PR head on its own:
metadata=truedropped from the V2 row's selector and overlay recordn_a_metadata_value_other_than_true_is_the_standard_listing,n_a_misspelled_key_is_unchanged,n_a_path_named_like_a_key_is_unchanged,n_the_two_selections_never_disagree_on_a_metadata_request,each_metadata_listing_declares_its_listing_action_codec_and_rowlisting_floormade privileged (OperationFloor::custom)each_metadata_listing_floor_is_exactly_its_standard_listings_floorlisting_floorwidened tobuiltin_presignedeach_metadata_listing_floor_is_exactly_its_standard_listings_floormetadatagate on the V2 rown_a_metadata_listing_near_miss_is_the_ordinary_operation,two_operation_keys_select_the_first_in_legacy_ordera_metadata_listing_selects_the_dialect_operation_ahead_of_every_keyn_rows_whose_rules_cannot_meet_declare_no_overlapn_a_claimer_with_the_other_claimers_auth_class_is_refusedn_a_presigned_ruling_on_a_listing_row_is_refusedmetadatagatetest_n_a_changed_gate_value_is_refused,test_n_a_case_insensitive_gate_is_refusedn_an_extension_route_with_an_unknown_gate_rule_is_refused,n_an_extension_route_with_an_empty_gate_key_is_refuseds3:ListBucketVersionseach_metadata_listing_declares_its_listing_action_codec_and_rowlisting_floormade SigV4-only (require_sigv4_under_every_policy)a_sigv2_presigned_listing_is_admitted_exactly_where_a_standard_one_is,metadata_true_appended_to_a_sigv2_presigned_listing_is_the_same_callers_listingn_a_metadata_listing_without_the_dialect_is_501_on_the_rustfs_profile(fs),the_metadata_listing_is_ahead_of_every_named_sub_resourcemetadataas a sub-resourcemetadata_true_appended_to_a_sigv2_presigned_listing_is_admitted_only_where_sigv2_presigned_is,metadata_true_appended_to_a_sigv2_header_signed_listing_is_the_signers_metadata_listingNot mutated, because they pin behaviour this PR does not implement:
n_an_encoded_key_is_refused_before_routing(the wire layer's refusal of an encoded name);a_trailing_slash_bucket_path_is_the_listing_and_a_double_slash_is_an_object(path addressing).All fourteen mutations ran locally (
cargo test -p <crate> <filter>, and the Python reader test for M9), each applied alone to this head and reverted afterwards. CI runners were budgeted at one run per task, so the throwaway draft PR #1601 was closed and its run cancelled before it reported.PR checklist
cargo xtask verify --crate rustfs-gateway-dialect-rustfs-admin. The full workspace runs in CI.admin::listing_floor,ExtensionRouteRecord::requires, the two operations)BREAKINGand a migration path are given aboveunsafeintroducedinventory/linkmeintroduced#[non_exhaustive]added to a DTORole Verdicts
Claimerenum (two variants, both rendered and both tested), therequiresfield versus a second parallel extension mechanism (rejected: one generator path keeps the eight rows byte-identical exceptrequires: &[]), the one-lineadmin::listing_floor(kept because the generated module names its floor throughadmin::, as for every other floor), and the separate Python reader module (one caller, split to keep the inventory script under 800 lines) — no break foundmetadatanorlist-type, sometadata=trueappended to a SigV2 presigned listing URL (RustFS profile floor) or to a captured SigV2 header-signed listing (every floor, inside the skew window) selects the metadata listing under the signer's listing action and discloses every object's user metadata; kept for RustFS parity, pinned in both directions in crates/goldens/src/rustfs_admin_dialect/metadata_listing_tests.rs, stated in ADR-0050 and docs/security-model.md, with the SigV2 refusal of rustfs:StsFormPost admits SigV2 where RustFS refuses it #1592 as the planned follow-up. Also attacked routing ambiguity (sub-resources,x-id, admin-router keys,versionswithlist-type=2, virtual-host and path style, encoded keys, trailing and double slashes, repeats, case), the floor againstListObjectsV2's under all four policies, anonymous requests without a policy, the wrong listing action, SigV4 presigned append, and the RustFS profile without the dialect (501): no further break