diff --git a/.agents/languages/cpp.md b/.agents/languages/cpp.md index c5a45fa410..c5689969b0 100644 --- a/.agents/languages/cpp.md +++ b/.agents/languages/cpp.md @@ -18,6 +18,18 @@ Load this file when changing `cpp/`, Cython build plumbing, or C++ xlang behavio resource amplification, publish reference or cache state that survives root cleanup, or return success past the required safepoint. Do not add per-field checks, cursor rollback, or tests that pin the first detection point solely to make an error earlier or more precise. +- Every public `Fory` and `ThreadSafeFory` root serialization or deserialization overload freezes + explicit registration before codec work. Route facade and resolver registration through one + authoritative frozen flag for that registry. Do not add another lifecycle flag or a multi-state + machine around the existing read/write-context resolver construction. +- A root freezes explicit registration before the existing read/write-context construction clones + the registered resolver tables. Context construction must not eagerly complete every registered + `TypeInfo`; complete metadata only when a context first uses that type for metadata, + struct-version, or skip behavior. Keep ordinary type lookup free of completion work. +- Keep the registration check out of normal runtime lookup hot paths. +- Direct `Fory` and its resolver are creator-thread-owned. Configure `ThreadSafeFory` before + concurrent use and let that facade own first-root synchronization; do not add a resolver mutex to + support concurrent registration that neither facade permits. - Put private methods last in class definitions, immediately before private fields. - Do not redesign alias-based or low-level public type shapes to add convenience methods unless the user explicitly asks for that API change. - For cross-language feature ports, match protocol behavior but use idiomatic C++ ownership and layering instead of mirroring Java structure literally. diff --git a/.agents/languages/csharp.md b/.agents/languages/csharp.md index d8c856b59a..a3f9e0d4b6 100644 --- a/.agents/languages/csharp.md +++ b/.agents/languages/csharp.md @@ -9,6 +9,15 @@ Load this file when changing `csharp/` or C# xlang behavior. - C# code must build without compiler or analyzer warnings. Treat warnings as blockers in project, test, and generated code. - Fory C# requires .NET SDK `8.0+` and C# `12+`. - Use `dotnet format` to keep C# code style consistent. +- A direct C# `Fory` registry and the `ThreadSafeFory` public registration boundary each own one + authoritative frozen flag. The first root sets the owning flag before codec work and leaves it + set after failure. Explicit registration checks that flag before mutation. `ThreadSafeFory` keeps + its existing registration callbacks only to configure newly created child runtimes; do not + turn that list into another registry lifecycle state. +- Generated or custom serializer construction and `TypeInfo` creation can execute application + code. Complete them before explicit ID or name publication, then recheck the same `Fory` flag + immediately before publishing that registration. Do not publish an intermediate unregistered + `TypeInfo` for a custom binding. - Generated C# gRPC service companions are compiler-owned files that depend on application-provided gRPC packages, not `csharp/src/Fory`. Keep gRPC package references out of the Fory runtime package. - C# generated schema modules are source-file owners. Service companions must use that module's `ThreadSafeFory` and must not introduce namespace-owned aliases or duplicate serializer registration paths. - C# external-type serialization is target-keyed. A local diff --git a/.agents/languages/dart.md b/.agents/languages/dart.md index a4d76cc700..c09f9670ff 100644 --- a/.agents/languages/dart.md +++ b/.agents/languages/dart.md @@ -98,8 +98,8 @@ Load this file when changing `dart/`. - Do not add parallel header-low/header-high slot caches or multi-slot recent caches in TypeMeta hot paths to chase benchmark gaps. Header-cache hits must use the concrete checked cache owner directly; if a hit hint is needed, cache one TypeInfo/TypeMeta object and compare the protocol-defined top 52 header bits on that object, not separate low/high header fields or benchmark-pattern state. - The top 52 TypeDef/TypeMeta header bits are the schema identity. The full low 12 bits belong only to the current frame and must not participate in hit selection. On a hit, decode the current body size from its low eight bits and any extended-size varuint, prove those bytes readable, and skip exactly that body. Do not validate reserved/compress flags, compare cached or local low bits, parse or rehash the body, repeat schema or policy validation, or grow low-bit sentinels, accepted-header fields, parallel header slots, or benchmark-pattern state. The cold miss path owns low-flag validation. - Dart expected-type TypeDef reads should compare only the top 52 bits of the expected `TypeInfo` object's cached local TypeDef header before consulting the parsed-metadata map. A match is a direct local-schema hit: use the current frame's size encoding only for bounds and skip, add the expected type to the per-read shared type table, and do not validate its low flags, publish to `ParsedTypeMetaCache`, record a remote schema version, or parse/hash the body. -- Dart local TypeDef construction is registration-owned: record registrations - and finalize their dependent TypeDefs and struct serializers before the first +- Dart local TypeDef construction is registration-owned: each explicit registration constructs its + dependent TypeDefs and struct serializers before the first root read or write. The first `serialize`, `serializeTo`, `serializeBuiltin`, `serializeBuiltinTo`, `deserialize`, or `deserializeFrom` call permanently freezes that `Fory` instance's resolver; diff --git a/.agents/languages/go.md b/.agents/languages/go.md index 5192fc3e7b..3b68aa70bf 100644 --- a/.agents/languages/go.md +++ b/.agents/languages/go.md @@ -7,6 +7,12 @@ Load this file when changing `go/fory/` or Go xlang behavior. - Run Go commands from within `go/fory/`. - Changes under `go/` must pass formatting and tests. - The Go implementation focuses on fast serializers. +- A Go `Fory` instance has one authoritative registry-frozen flag. The first root serialization or + deserialization sets it before codec work and leaves it set after failure. Explicit registration + checks that flag before mutation. Do not add another registry lifecycle state or alter existing + registration semantics beyond that boundary check. + `threadsafe.Fory` has no registration API or facade registry: configure every pooled child in the + factory passed to `NewWithFactory` before returning it. - Go `ReadContext` intentionally defers codec errors to existing `HasError` or `CheckError` boundaries. After an error, work may continue only while it remains panic- and bounds-safe and cannot cause disproportionate work or allocation, publish state that survives root cleanup, or diff --git a/.agents/languages/java.md b/.agents/languages/java.md index a96d04b7f0..4512f0675f 100644 --- a/.agents/languages/java.md +++ b/.agents/languages/java.md @@ -33,6 +33,14 @@ Load this file when changing anything under `java/` or when Java drives a cross- values; use qualified names only when a real name conflict requires it. - If you run temporary tests with `java -cp`, run `mvn -T16 install -DskipTests` first so local Fory jars are current. - `WriteContext`, `ReadContext`, and `CopyContext` must stay explicit. Do not reintroduce `ThreadLocal` or ambient runtime-context patterns. +- Java scoped meta-share TypeInfo occurrences are root-local, and the current table size is their + protocol visibility boundary. Reset tables of at most 8192 entries by setting the size to zero; + do not null retained slots. Only when the size exceeds 8192 may cleanup replace the backing array, + and it must restore a small eight-slot array instead of retaining or allocating 8192 slots. Keep + the normal root-cleanup path allocation-free, and do not add count-shape specializations. +- Deserialization failures must not copy or retain the active reference table or materialized graph + in the exception. Root reset owns releasing operation-local graph state; keep failure reporting + bounded independently of graph size. - Java root deserialization graph memory budgeting belongs to `ReadContext` and is initialized by `Fory` root APIs. Public config is `maxGraphMemoryBytes` with fixed `128 MiB` default. Positive explicit values override the default; @@ -76,6 +84,41 @@ Load this file when changing anything under `java/` or when Java drives a cross- work, dynamic stream bytes-read accounting, or stale narrower-scope formulas. - Generated serializers must not retain runtime context fields. `Fory` should stay a root-operation facade rather than accumulating serializer or convenience state. - When the serializer class and constructor shape are known at the call site, prefer direct constructor lambdas or direct instantiation over reflective `Serializers.newSerializer(...)`. +- Each natural Java registry or public facade boundary owns one authoritative lifecycle flag. A + concrete resolver owns its local frozen flag; a thread-safe facade uses the shared registry's + frozen flag for the facade-wide boundary. The first root serialization or deserialization sets the + owning fact before codec work and never clears it, including after failure. Every explicit type, + serializer, module, name, ID, or type-checker binding checks that fact before mutation. + Disallow-list changes use the bound checker's resolver listeners. Do not add another lifecycle + state or a parallel registration-commit path. +- Direct and thread-safe facades expose module registration before their first root. Kotlin and + Scala registration extensions target `BaseFory`; do not narrow them to concrete `Fory` or make + builder installation the only thread-safe path. A thread-safe facade's shared registry owns its + registration boundary so a root started by any child closes the facade before registration can + mutate another child. Non-root `execute` and copy operations remain concurrent and do not freeze + registration. Complete facade registration before concurrent serialization, deserialization, + copy, or `execute` calls begin; do not serialize those operations behind a registration lock. +- `ForyModule.install` is registration-only setup. It may install nested modules and construct + child-specific serializers, but must not start a root through the supplied child or a captured + direct or thread-safe facade. Do not propagate the facade lifecycle owner into raw child + registration to support this invalid reentrancy; frozen-facade replay needs the unexposed child + to remain governed by its local resolver until it adopts the shared snapshot. +- Live thread-safe facade registration carries the existing `SharedRegistry` check through + application-controlled serializer preparation and runs it immediately before child publication. + Replay into a new, unexposed thread-local child instead uses that child's local resolver check, + then freezes the child onto the shared snapshot before exposure. Do not make every child resolver + consult the shared frozen flag: that would reject the required frozen-facade setup replay. +- `ThreadSafeFory.execute` exposes one borrowed child only for the callback. Do not retain that + child or register through it; use the facade registration methods so every current and future + child receives the same setup. +- Serializer completion used by lazy, JIT, and generated serializers is an internal resolver-owned + operation. It remains valid after registration freezes and must not be treated as explicit + registration. +- Registration freeze does not disable native runtime type resolution. When class registration is + not required, native roots may discover an unregistered runtime class and materialize its + resolver-owned `TypeInfo`, descriptor, serializer, or JIT cache entry after freeze. This runtime + cache materialization must not create or change an explicit class, serializer, ID, name, or + policy registration. - For GraalVM, use `fory codegen` to generate serializers when building native images. Do not add reflection configuration except for JDK `proxy`. - In Java native mode (`xlang=false`), only `Types.BOOL` through `Types.STRING` share type IDs with xlang mode. Other native-mode type IDs differ. - Choose one serializer ownership location per logical Java type family. Add native/xlang serializer variants only when the wire format or constructor contract truly differs. diff --git a/.agents/languages/javascript.md b/.agents/languages/javascript.md index 534e49481b..32f3978959 100644 --- a/.agents/languages/javascript.md +++ b/.agents/languages/javascript.md @@ -11,6 +11,19 @@ Load this file when changing `javascript/`. - Preserve generated serializer hot paths that bind writer, reader, ref, resolver, and metadata locals in outer closures; do not replace them with per-call context lookups without a measured reason. - Do not add parallel header-low/header-high slot caches in TypeMeta hot paths to chase benchmark gaps. Header-cache hits must use the concrete checked cache owner directly; if a small hit hint is needed, cache TypeMeta objects themselves and compare `TypeMeta.headerHash`, not separate low/high header fields or benchmark-pattern state. - JavaScript TypeMeta header cache hits should compare the 52-bit TypeMeta header hash directly. The hash is precise in JS `Number` and already includes the low header bits as hash input; do not add extra low-bit fields, sentinel state, nullable accepted headers, or parallel slot arrays around it. +- JavaScript `Fory` owns the one authoritative registration-frozen flag. The first root + serialization or deserialization sets it before codec work and leaves it set after failure. + `TypeResolver` owns registration maps, not a second lifecycle flag. +- Codegen hooks can start a root while an explicit registration graph is being generated. Recheck + the `Fory`-owned flag before every later serializer publication in that graph; do not add a + lifecycle flag to `TypeResolver`, stage the graph, or roll registry entries back. +- A failed root releases its operation-local reference and metadata state before the exception + escapes. The next root entry releases state retained by the previous successful operation before + reusing the context. Keep the successful root exit allocation-free and do not copy Java + backing-array retention policies onto native JavaScript arrays. Read-side metadata + occurrence arrays use native replacement reset. The MetaString and TypeMeta writer owner tables each have + their own logical size: reset active owner IDs and that table's logical size without clearing + bounded backing, and replace either backing only after its root has more than 8192 owners. - Runtime value carriers such as decimal or reduced-precision numeric types belong under the core `types/` ownership boundary, with imports, exports, and codegen externals updated together. - Keep `TypeInfo` as schema metadata. Compatibility-sensitive decisions belong on `TypeResolver` or explicit operations, not as retained resolver state on metadata objects. - Normalize optional boolean config values at config construction; do not carry `null` through runtime paths when it means `false`. diff --git a/.agents/languages/kotlin.md b/.agents/languages/kotlin.md index 49b2444389..82dbcc7485 100644 --- a/.agents/languages/kotlin.md +++ b/.agents/languages/kotlin.md @@ -14,6 +14,15 @@ Load this file when changing `kotlin/` or compiler code that generates Kotlin so Fory. Do not auto-install a new serializer for an existing type-registered Kotlin class unless the wire format matches the previous serializer family and old-payload/new-runtime compatibility is tested. +- Kotlin registration extensions target `BaseFory` so direct and thread-safe facades share the same + pre-root registration API, including module installation. Complete thread-safe facade + registration before concurrent serialization, deserialization, copy, or execution begins. +- Explicit type, serializer, enum, and union registration checks the receiving `BaseFory` facade or + natural registry owner's one frozen flag before mutation. Keep generated serializer construction + on the existing direct resolver path; do not add a parallel registration path or lifecycle state. +- Combined generated structural registration attaches the serializer with `setSerializer` after + registering the canonical STRUCT `TypeInfo`; `registerSerializer` would incorrectly reclassify + that wire identity as EXT. Generated unions use `registerUnion`. - When adding Kotlin gRPC service companions, emit Kotlin source only. Reuse the generated schema module's `ThreadSafeFory` and KSP-generated schema serializers, and keep grpc-java/grpc-kotlin dependencies application-owned instead of adding them as hard `fory-kotlin` dependencies. diff --git a/.agents/languages/python.md b/.agents/languages/python.md index aed074dfbf..c5707ba2c6 100644 --- a/.agents/languages/python.md +++ b/.agents/languages/python.md @@ -11,6 +11,26 @@ Load this file when changing `python/`, Cython serialization, or Python xlang be - Python mode is the pure-Python xlang implementation and is mainly for debugging and testing. - Cython mode is the default high-performance implementation. - Cython mode owns the hot runtime path. Do not duplicate core runtime types between Python and Cython, tunnel Python facade methods into hidden Cython internals, or keep dead shims unless the user explicitly needs a compatibility module path. +- A direct Python `TypeResolver` owns one authoritative `_registry_frozen` flag. In Cython mode the + compiled `TypeResolver` is the active owner instead, and the Python resolver delegates explicit + registration checks to that compiled flag; do not mirror the flag between the two resolvers. + Roots set their active owner before codec work and never clear it, including after failure. + `ThreadSafeFory` owns its own `_registry_frozen` flag for the public registration boundary over + pooled children. Its existing callback list configures newly created children; it is not another + lifecycle state. +- Explicit type, serializer, name, and ID registration checks the frozen flag before mutation. + Automatic IDs remain registration-owned and must not turn native runtime discovery into explicit + registration. +- A Python wire name or user ID identifies one `TypeInfo`. Reject explicit or native-discovery + collisions before publishing resolver maps. Lazy TypeDef completion preserves a configured + serializer, does not retain partial state, and restores the prior serializer and TypeDef after + failed completion without adding a lifecycle state. +- Registry freeze prohibits explicit type and serializer registration after the first root; it + does not prohibit native runtime type resolution. Non-strict native writes may discover runtime + classes or callables, and reads may resolve those authorized by the deserialization policy. Both + paths may materialize resolver-owned type information or serializer cache entries without + creating or changing an explicit type, serializer, ID, name, or policy registration. Do not + describe these operations as late registration. - Use explicit Cython fields and methods for fixed hot-path shapes. Avoid `__getattr__`, generic `object` fields, public bridge internals, or `Fory` backreferences where ownership can stay explicit. - Keep Python and Cython context/ref-tracking branch conditions and stack mutations semantically aligned unless a documented intentional difference exists. - Root deserialization graph memory budget state belongs to pure-Python and Cython `ReadContext`. diff --git a/.agents/languages/scala.md b/.agents/languages/scala.md index ecfd3b07e1..aaa38d0c4c 100644 --- a/.agents/languages/scala.md +++ b/.agents/languages/scala.md @@ -9,6 +9,15 @@ Load this file when changing `scala/`. - Scala supports the JVM and GraalVM Native Image, not Android. Do not add Android-specific Scala sources, tests, resources, R8 metadata, compiler plugins, macros, dependencies, or compatibility design. +- Scala registration extensions target `BaseFory` so direct and thread-safe facades share the same + pre-root registration API, including module installation. Complete thread-safe facade + registration before concurrent serialization, deserialization, copy, or execution begins. +- Explicit type, serializer, enum, and union registration checks the receiving `BaseFory` facade or + natural registry owner's one frozen flag before mutation. Keep generated serializer construction + on the existing direct resolver path; do not add a parallel registration path or lifecycle state. +- Combined generated structural registration attaches the serializer with `setSerializer` after + registering the canonical STRUCT `TypeInfo`; `registerSerializer` would incorrectly reclassify + that wire identity as EXT. Generated unions use `registerUnion`. ## Commands diff --git a/.agents/languages/swift.md b/.agents/languages/swift.md index 1df5b89d3d..6e9a1fdfe2 100644 --- a/.agents/languages/swift.md +++ b/.agents/languages/swift.md @@ -37,6 +37,13 @@ Load this file when changing `swift/` or Swift xlang behavior. ignored declaration fields are budget-only and must not enter target access, construction, metadata, or wire code. Omitted large value storage must be declared explicitly and ignored. +- Swift registry lifecycle uses one authoritative frozen flag set by the first root serialization or + deserialization. Do not add another lifecycle state or cache freeze failure separately. Registered + TypeInfo owns lazy TypeMeta completion after freeze; do not add an eager whole-registry metadata + pass. +- Serializer static properties are application code and can start a root during registration. + Recheck the same resolver flag after the last static property access and immediately before + returning from an idempotent registration or publishing `TypeInfo` by ID or name. - Direct `Any` and `AnyObject` root overloads remain disfavored forwarding facades over `DynamicSerializer` and `DynamicSerializer`, including their Data-buffer forms. Arbitrary protocol roots explicitly select `DynamicSerializer`. Do not add an unconstrained diff --git a/AGENTS.md b/AGENTS.md index 10f82994a0..c3cccbe796 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -152,6 +152,21 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th than conflating them with `readData`. - For remote TypeDef/TypeMeta reads, the checked metadata cache is the only owner of remote "already validated" state. Cache hit means the header was previously parsed, body/hash-validated, policy-checked, and published by that cache, so the hot path must skip the body and use cached metadata without extra validation, hashing, limit checks, exact-local checks, allocation, or policy work. The protocol-defined 52-bit TypeDef/TypeMeta header hash is the unique schema identity, so a known expected local header/hash match is a local-schema hit and must not recompare field arrays or metadata bodies. The low 12 header bits belong only to the current frame; on a hit, use its current size and optional extension for bounds and skip, but do not validate its reserved or compression flags. A local hit uses the local TypeInfo/TypeMeta without schema-version counting or cache publish. Cache miss is the only path that parses and validates non-local metadata, including low flags, and enforces limits. If the local header becomes available only after that first parse, compare its 52-bit hash with the validated received hash; equality selects the local owner without a second byte or field comparison. Only a non-local miss publishes remote metadata to the cache. Do not add nullable accepted-header fields, sentinel headers, per-TypeInfo markers, pending metadata state, parallel header-low/header-high slots, or parallel acceptance state for this decision. If a runtime needs a metadata hit hint, cache the concrete checked metadata owner object, such as the TypeInfo, TypeDef, or TypeMeta used by that runtime, and compare its validated header identity directly. - Checked MetaString caches follow the same rule: validate and publish only on cache miss; on cache hit, skip the encoded body and use the cached value without rehashing, comparing body bytes, or repeating validation. The protocol-defined wire hash alone is the MetaString cache identity; the current frame length is used only for bounds checking and advancing the reader, and must not participate in hit selection. Do not add hit-time byte or length comparison or parallel acceptance state for MetaString caches. +- Java scoped meta-share TypeInfo occurrences are root-local, and their current table size is the + protocol visibility boundary. Root cleanup must reset tables of at most 8192 entries by setting + only the size to zero; do not clear retained slots. When the size exceeds 8192, replace the + backing array with an eight-slot array. Do not add per-count or benchmark-shape specializations + to this path. +- A failed JavaScript root releases its operation-local reference and metadata state before the + exception escapes. The next root entry releases state retained by the previous successful root. + Keep successful root exit allocation-free, and do not copy Java backing-array retention policies + onto native JavaScript arrays. Read-side metadata + occurrence arrays use native replacement reset. The MetaString and TypeMeta writer owner tables each have their own + logical size: reset active owner IDs and that table's logical size without clearing bounded + backing, and replace either backing only after its root has more than 8192 owners. +- Root failure exceptions must not copy or retain the operation reference table or materialized + object graph for diagnostics. Root cleanup owns releasing that graph, and failure reporting must + remain bounded independently of graph size. - When a user corrects a non-obvious invariant, encode it in the nearest source comment before continuing, and also update `AGENTS.md`, `.agents/**`, docs, or specs when the rule is reusable beyond one file. Do not rely only on chat history, task notes, commit messages, or benchmark logs for corrections that protect security, protocol behavior, ownership, naming, or hot-path performance. - Reject semantic hacks. Do not bypass broken semantics by deleting cases, simplifying callers, adding coercion hooks, or using workaround fallbacks; fix the underlying bug and prove it with focused tests. - Protect hot paths. Avoid per-call allocations, callback objects, result tuples or records, unnecessary runtime branches, and wrapper-class substitutions in hot codec/runtime paths; prefer conditional imports and allocation-free concrete implementations where they fit the language. @@ -170,8 +185,34 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th maps, generated descriptors, metadata, serializers, or caches. Do not support post-use registration through cache invalidation, descriptor refresh, serializer rebinding, metadata rebuilding, or other late-registration - machinery. Registration-order finalization before the first root operation - remains registration-owned and must not create a runtime invalidation path. + machinery. Keep one authoritative frozen flag for each natural registration + owner or public facade boundary. A thread-safe facade with its own public + registration surface may own that boundary flag, but must not mirror a child + registry's lifecycle. Registration preparation can execute application code + through serializer construction, static serializer metadata, or codegen hooks. + Recheck the same owner flag after the last such callback and immediately before + explicit registry publication. Do not add another lifecycle state, a registration + commit or rollback path, or eager whole-registry preparation solely to implement + freeze. Copy operations and + facade execution callbacks do not freeze registration unless they start a + root serialization or deserialization operation. + Registry freeze does not make native runtime type resolution immutable. When + registration is not required, Java and Python native modes may still discover + an unregistered runtime type and materialize resolver-owned type information, + descriptors, serializers, or JIT code while processing a root. Lazy, JIT, and + generated serializer completion for an existing binding is likewise allowed + after freeze. These runtime cache operations must not create or change an + explicit type, serializer, ID, name, or policy registration. + Each explicit name and ID has one type-information owner. Native discovery + must reject an identity collision instead of replacing that owner. Lazy + metadata or serializer completion preserves the selected explicit serializer + and does not retain partial state; failure leaves the prior valid state. + Java `Fory.register(ForyModule)` and the corresponding `BaseFory` operation remain available on + direct and thread-safe facades before the first root. Kotlin and Scala registration extensions + target `BaseFory` so the same API works with `Fory`, `ThreadLocalFory`, and pooled + `ThreadSafeFory` implementations. Module installation is registration-only setup: it may install + nested modules and child-specific serializers, but must not start a root through the supplied + child or the direct or thread-safe facade installing the module. - Use semantic naming only. Name things after protocol or domain concepts, not history, runtime origin, or workaround style; avoid vague names such as `Internal`, `java_style_*`, `Runtime`, `Session`, `Plan`, `Payload`, or `Binding` when they do not name the real concept. Keep class, method, function, and variable names concise; do not encode the whole scenario or implementation history into one identifier. Never name a class or method with a `Plan` suffix; use the real domain concept instead. For Fory codec/read APIs, do not use generic `payload` naming; name the exact owner and data shape, such as bytes, body, frame, field, string, list, map, compressed bytes, or primitive-array encoding. - Keep one implementation path. Do not keep parallel helpers, serializers, harnesses, wrappers, or registration flows for the same concept; extend the existing owner path instead of inventing another one. - Follow current scope exactly. The latest explicit user instruction overrides earlier plans, and when scope narrows, remove leaked out-of-scope edits immediately. @@ -188,6 +229,7 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th - Do not allow implementation drift from the design document. - Do not compromise design decisions to make implementation easier. - Do not leave workaround code behind. +- Do not introduce unnecessary abstractions or concepts. - All code must have a clean owner model; the wrong owner model or abstraction is unacceptable. - Do not leave ugly or temporary code behind. - Do not leave legacy, dead, useless, or stale code, tests, or docs behind. @@ -198,6 +240,8 @@ This is the entry point for AI guidance in Apache Fory. Read this file first, th - Do not preserve legacy, dead, or useless code, tests, or docs unless the user explicitly requests it. - Ignore internal API compatibility unless the user explicitly requests it. Do not keep shims, wrappers, or transitional paths only to preserve internal call sites. - Performance is the top priority. Do not introduce regressions without explicit justification. +- Do not add object allocation to hot paths. Breaking internal compatibility is acceptable; remove + obsolete code, tests, and docs instead of preserving compatibility-only paths. - "Refactor" means changing structure, ownership, naming, or API shape without changing behavior, wire format, or implementation strategy unless the user explicitly asks for those changes. - Do not make design tradeoffs the user did not request. If a refactor appears to require a behavior, logic, protocol, or performance tradeoff, stop and ask. - Treat existing low-level or optimized code as deliberate by default. During a refactor, preserve the current implementation strategy unless the user explicitly asks to redesign or optimize it. diff --git a/cpp/fory/serialization/context.cc b/cpp/fory/serialization/context.cc index 35c2a8c153..2d1b7fd5c1 100644 --- a/cpp/fory/serialization/context.cc +++ b/cpp/fory/serialization/context.cc @@ -79,6 +79,7 @@ WriteContext::write_type_meta(const std::type_index &type_id) { // This ensures consistent indexing when the same type is written via // either type_index or TypeInfo* path FORY_TRY(type_info, type_resolver_->get_type_info(type_id)); + FORY_RETURN_NOT_OK(type_resolver_->ensure_type_meta(type_info)); write_type_meta(type_info); return Result(); } @@ -122,6 +123,13 @@ void WriteContext::write_type_meta(const TypeInfo *type_info) { buffer_.write_bytes(type_info->type_def.data(), type_info->type_def.size()); } +void WriteContext::ensure_type_meta(const TypeInfo *type_info) { + auto result = type_resolver_->ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(!result.ok())) { + set_error(std::move(result).error()); + } +} + /// write pre-encoded meta string to buffer (avoids re-encoding on each write) static void write_encoded_meta_string(Buffer &buffer, const CachedMetaString &encoded) { @@ -189,6 +197,12 @@ WriteContext::write_enum_type_info(const TypeInfo *type_info) { } else if (type_id == static_cast(TypeId::NAMED_ENUM)) { if (config_->compatible) { // write type meta inline using streaming protocol + if (FORY_PREDICT_FALSE(!type_info->type_meta)) { + ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(has_error())) { + return Unexpected(error()); + } + } write_type_meta(type_info); } else { // write pre-encoded namespace and type_name @@ -281,6 +295,12 @@ WriteContext::write_any_type_info(const TypeInfo *type_info) { case TypeId::COMPATIBLE_STRUCT: case TypeId::NAMED_COMPATIBLE_STRUCT: // write type meta inline using streaming protocol + if (FORY_PREDICT_FALSE(!type_info->type_meta)) { + ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(has_error())) { + return Unexpected(error()); + } + } write_type_meta(type_info); break; case TypeId::NAMED_ENUM: @@ -289,6 +309,12 @@ WriteContext::write_any_type_info(const TypeInfo *type_info) { case TypeId::NAMED_UNION: if (config_->compatible) { // write type meta inline using streaming protocol + if (FORY_PREDICT_FALSE(!type_info->type_meta)) { + ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(has_error())) { + return Unexpected(error()); + } + } write_type_meta(type_info); } else { // write pre-encoded namespace and type_name @@ -352,8 +378,7 @@ WriteContext::write_struct_type_info(const std::type_index &type_id) { return Result(); } -Result -WriteContext::write_struct_type_info(const TypeInfo *type_info) { +void WriteContext::write_struct_type_info(const TypeInfo *type_info) { uint32_t fory_type_id = type_info->type_id; // write type_id @@ -368,11 +393,23 @@ WriteContext::write_struct_type_info(const TypeInfo *type_info) { case TypeId::COMPATIBLE_STRUCT: case TypeId::NAMED_COMPATIBLE_STRUCT: // write type meta inline using streaming protocol + if (FORY_PREDICT_FALSE(!type_info->type_meta)) { + ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(has_error())) { + return; + } + } write_type_meta(type_info); break; case TypeId::NAMED_STRUCT: if (config_->compatible) { // write type meta inline using streaming protocol + if (FORY_PREDICT_FALSE(!type_info->type_meta)) { + ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(has_error())) { + return; + } + } write_type_meta(type_info); } else { // write pre-encoded namespace and type_name @@ -380,8 +417,9 @@ WriteContext::write_struct_type_info(const TypeInfo *type_info) { write_encoded_meta_string(buffer_, *type_info->encoded_namespace); write_encoded_meta_string(buffer_, *type_info->encoded_type_name); } else { - return Unexpected( + set_error( Error::invalid("Encoded meta strings not initialized for struct")); + return; } } break; @@ -389,8 +427,6 @@ WriteContext::write_struct_type_info(const TypeInfo *type_info) { // STRUCT type - just writing type_id is sufficient break; } - - return Result(); } void WriteContext::reset() { @@ -698,6 +734,16 @@ ReadContext::read_type_meta_owner(const TypeInfo *expected_type_info) { ReadTypeInfo{local_type_info, local_type_info}); return local_type_info; } + if (FORY_PREDICT_FALSE( + !local_type_info->type_meta && + is_struct_type(static_cast(local_type_info->type_id)))) { + FORY_RETURN_NOT_OK(type_resolver_->ensure_type_meta(local_type_info)); + if (has_local_meta_hash(local_type_info, meta_hash)) { + reading_type_infos_.push_back( + ReadTypeInfo{local_type_info, local_type_info}); + return local_type_info; + } + } } FORY_TRY(remote_schema_key, check_remote_type_meta_limit(*parsed_meta)); @@ -708,7 +754,8 @@ ReadContext::read_type_meta_owner(const TypeInfo *expected_type_info) { cached->concrete_owner = local_type_info; if (local_type_info) { // Have local type - assign dispatch IDs by comparing schemas. - // Note: Extension types don't have type_meta (only structs do) + // Extension types have no local TypeMeta; only structs can provide local + // field metadata. if (local_type_info->type_meta) { FORY_RETURN_NOT_OK(TypeMeta::assign_local_dispatch_ids( local_type_info->type_meta.get(), parsed_meta->field_infos)); diff --git a/cpp/fory/serialization/context.h b/cpp/fory/serialization/context.h index f6173f8738..27877c0e5b 100644 --- a/cpp/fory/serialization/context.h +++ b/cpp/fory/serialization/context.h @@ -262,7 +262,8 @@ class WriteContext { /// Subsequent occurrences: writes (index << 1) | 1 as reference. Result write_type_meta(const std::type_index &type_id); - /// write TypeMeta inline using TypeInfo pointer (fast path). + /// write TypeMeta inline using a TypeInfo whose metadata is ready (fast + /// path). /// First occurrence: writes (index << 1) | 0 followed by TypeDef bytes. /// Subsequent occurrences: writes (index << 1) | 1 as reference. void write_type_meta(const TypeInfo *type_info); @@ -298,9 +299,9 @@ class WriteContext { /// Fastest path for writing struct type info when TypeInfo is already known. /// Avoids type_index creation and lookup overhead. /// - /// @param type_info Pointer to the TypeInfo (must be valid) - /// @return Success or error - Result write_struct_type_info(const TypeInfo *type_info); + /// @param type_info Pointer to a valid TypeInfo. This method completes its + /// metadata lazily when the configured wire mode needs it. + void write_struct_type_info(const TypeInfo *type_info); /// Fastest path - write struct type_id directly without any lookups. /// Use this when the type_id is already known (e.g., from a cache). @@ -345,6 +346,8 @@ class WriteContext { void reset(); private: + FORY_NOINLINE void ensure_type_meta(const TypeInfo *type_info); + // Error state - accumulated during serialization, checked at the end Error error_; diff --git a/cpp/fory/serialization/fory.h b/cpp/fory/serialization/fory.h index d9087038c1..068742ec0f 100644 --- a/cpp/fory/serialization/fory.h +++ b/cpp/fory/serialization/fory.h @@ -192,9 +192,6 @@ class ForyBuilder { bool compatible_set_ = false; std::shared_ptr type_resolver_; - /// Helper to get or create type resolver and finalize it - std::shared_ptr get_finalized_resolver(); - friend class Fory; friend class ThreadSafeFory; }; @@ -297,9 +294,7 @@ class BaseFory { /// fory.register_struct(1); /// ``` template Result register_struct(uint32_t type_id) { - return register_type([this, type_id]() { - return type_resolver_->template register_by_id(type_id); - }); + return type_resolver_->template register_by_id(type_id); } /// Register a struct type with namespace and type name. @@ -320,9 +315,7 @@ class BaseFory { template Result register_struct(const std::string &ns, const std::string &type_name) { - return register_type([this, &ns, &type_name]() { - return type_resolver_->template register_by_name(ns, type_name); - }); + return type_resolver_->template register_by_name(ns, type_name); } /// Register a struct type with a name. @@ -362,9 +355,7 @@ class BaseFory { /// fory.register_enum(1); /// ``` template Result register_enum(uint32_t type_id) { - return register_type([this, type_id]() { - return type_resolver_->template register_by_id(type_id); - }); + return type_resolver_->template register_by_id(type_id); } /// Register an enum type with namespace and type name. @@ -385,9 +376,7 @@ class BaseFory { template Result register_enum(const std::string &ns, const std::string &type_name) { - return register_type([this, &ns, &type_name]() { - return type_resolver_->template register_by_name(ns, type_name); - }); + return type_resolver_->template register_by_name(ns, type_name); } /// Register an enum type with a name. @@ -417,9 +406,7 @@ class BaseFory { /// @param type_id Unique numeric identifier for this union type. /// @return Success or error if registration fails. template Result register_union(uint32_t type_id) { - return register_type([this, type_id]() { - return type_resolver_->template register_union_by_id(type_id); - }); + return type_resolver_->template register_union_by_id(type_id); } /// Register a union type with namespace and type name. @@ -432,9 +419,7 @@ class BaseFory { template Result register_union(const std::string &ns, const std::string &type_name) { - return register_type([this, &ns, &type_name]() { - return type_resolver_->template register_union_by_name(ns, type_name); - }); + return type_resolver_->template register_union_by_name(ns, type_name); } /// Register a union type with a name. @@ -458,9 +443,7 @@ class BaseFory { /// @return Success or error if registration fails. template Result register_extension_type(uint32_t type_id) { - return register_type([this, type_id]() { - return type_resolver_->template register_ext_type_by_id(type_id); - }); + return type_resolver_->template register_ext_type_by_id(type_id); } /// Register an extension type with namespace and type name. @@ -473,10 +456,7 @@ class BaseFory { template Result register_extension_type(const std::string &ns, const std::string &type_name) { - return register_type([this, &ns, &type_name]() { - return type_resolver_->template register_ext_type_by_name(ns, - type_name); - }); + return type_resolver_->template register_ext_type_by_name(ns, type_name); } /// Register an extension type with a name. @@ -507,22 +487,7 @@ class BaseFory { return std::make_pair(std::move(ns), std::move(type_name)); } - template - Result register_type(RegisterFn &&fn) { - std::lock_guard lock(registration_mutex_); - if (FORY_PREDICT_FALSE(registration_locked_)) { - return Unexpected(Error::invalid( - "Cannot register types after first serialize/deserialize call")); - } - return std::forward(fn)(); - } - protected: - void lock_registration() const { - std::lock_guard lock(registration_mutex_); - registration_locked_ = true; - } - /// Protected constructor - only derived classes can instantiate. explicit BaseFory(const Config &config, std::shared_ptr resolver) @@ -538,8 +503,6 @@ class BaseFory { Config config_; std::shared_ptr type_resolver_; - mutable std::mutex registration_mutex_; - mutable bool registration_locked_{false}; }; // ============================================================================ @@ -571,8 +534,8 @@ class Fory : public BaseFory { /// @return Vector containing serialized bytes, or error. template Result, Error> serialize(const T &obj) { - if (FORY_PREDICT_FALSE(!finalized_)) { - ensure_finalized(); + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); } WriteContextGuard guard(*write_ctx_); Buffer &buffer = write_ctx_->buffer(); @@ -592,30 +555,10 @@ class Fory : public BaseFory { /// @return Number of bytes written, or error. template Result serialize(OutputStream &output_stream, const T &obj) { - if (FORY_PREDICT_FALSE(!finalized_)) { - ensure_finalized(); + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); } - WriteContextGuard guard(*write_ctx_); - output_stream.reset(); - write_ctx_->set_output_stream(&output_stream); - Buffer &buffer = write_ctx_->buffer(); - buffer.bind_output_stream(&output_stream); - auto serialize_result = serialize_impl(obj, buffer); - if (FORY_PREDICT_FALSE(!serialize_result.ok())) { - buffer.clear_output_stream(); - write_ctx_->set_output_stream(nullptr); - return Unexpected(std::move(serialize_result).error()); - } - output_stream.force_flush(); - buffer.clear_output_stream(); - write_ctx_->set_output_stream(nullptr); - if (FORY_PREDICT_FALSE(output_stream.has_error())) { - return Unexpected(output_stream.error()); - } - if (FORY_PREDICT_FALSE(write_ctx_->has_error())) { - return Unexpected(write_ctx_->take_error()); - } - return output_stream.flushed_bytes(); + return serialize_stream(output_stream, obj); } /// Serialize an object to a std::ostream. @@ -626,8 +569,11 @@ class Fory : public BaseFory { /// @return Number of bytes written, or error. template Result serialize(std::ostream &ostream, const T &obj) { + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); + } StdOutputStream output_stream(ostream); - return serialize(output_stream, obj); + return serialize_stream(output_stream, obj); } /// Serialize an object to an existing Buffer (fastest path). @@ -639,16 +585,10 @@ class Fory : public BaseFory { template FORY_ALWAYS_INLINE Result serialize_to(Buffer &buffer, const T &obj) { - if (FORY_PREDICT_FALSE(!finalized_)) { - ensure_finalized(); + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); } - // Swap in the caller's buffer so all writes go there. - buffer.swap(write_ctx_->buffer()); - auto result = serialize_impl(obj, write_ctx_->buffer()); - buffer.swap(write_ctx_->buffer()); - // reset internal state after use without clobbering caller buffer. - write_ctx_->reset(); - return result; + return serialize_buffer(buffer, obj); } /// Serialize an object to an existing byte vector (zero-copy). @@ -664,12 +604,14 @@ class Fory : public BaseFory { template Result serialize_to(std::vector &output, const T &obj) { + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); + } // Wrap the output vector in a Buffer for zero-copy serialization // writer_index starts at output.size() for appending Buffer buffer(output); - // Forward to Buffer version - auto result = serialize_to(buffer, obj); + auto result = serialize_buffer(buffer, obj); // Resize vector to actual written size output.resize(buffer.writer_index()); @@ -684,19 +626,7 @@ class Fory : public BaseFory { /// @return Deserialized object, or error. template Result deserialize(const uint8_t *data, size_t size) { - if (FORY_PREDICT_FALSE(!finalized_)) { - ensure_finalized(); - } - if (data == nullptr) { - return Unexpected(Error::invalid("Data pointer is null")); - } - if (size == 0) { - return Unexpected(Error::invalid("Data size is zero")); - } - - Buffer buffer(const_cast(data), static_cast(size), - false); - return deserialize_buffer(buffer); + return deserialize_bytes(data, size); } /// Deserialize an object from a byte vector. @@ -706,7 +636,7 @@ class Fory : public BaseFory { /// @return Deserialized object, or error. template Result deserialize(const std::vector &data) { - return deserialize(data.data(), data.size()); + return deserialize_bytes(data.data(), data.size()); } /// Deserialize an object from a Buffer, updating the buffer's reader_index. @@ -719,8 +649,8 @@ class Fory : public BaseFory { /// @param buffer Buffer to read from. Its reader_index will be updated. /// @return Deserialized object, or error. template Result deserialize(Buffer &buffer) { - if (FORY_PREDICT_FALSE(!finalized_)) { - ensure_finalized(); + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); } return deserialize_buffer(buffer); } @@ -735,20 +665,10 @@ class Fory : public BaseFory { /// @return Deserialized object, or error. template Result deserialize(InputStream &input_stream) { - struct StreamShrinkGuard { - InputStream *input_stream = nullptr; - ~StreamShrinkGuard() { - if (input_stream != nullptr) { - input_stream->shrink_buffer(); - } - } - }; - StreamShrinkGuard shrink_guard{&input_stream}; - Buffer &buffer = input_stream.get_buffer(); - if (FORY_PREDICT_FALSE(!finalized_)) { - ensure_finalized(); + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); } - return deserialize_buffer(buffer); + return deserialize_stream(input_stream); } /// Deserialize an object from StdInputStream. @@ -757,7 +677,10 @@ class Fory : public BaseFory { /// @param stream Input stream wrapper to read from. /// @return Deserialized object, or error. template Result deserialize(StdInputStream &stream) { - return deserialize(static_cast(stream)); + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); + } + return deserialize_stream(stream); } // ========================================================================== @@ -777,37 +700,31 @@ class Fory : public BaseFory { ReadContext &read_context() { return *read_ctx_; } private: - /// Constructor for ForyBuilder - resolver will be finalized lazily. + /// Constructor for ForyBuilder - operation contexts are initialized lazily. explicit Fory(const Config &config, std::shared_ptr resolver) - : BaseFory(config, std::move(resolver)), finalized_(false), + : BaseFory(config, std::move(resolver)), precomputed_header_(compute_header(config.xlang)) {} - /// Constructor for ThreadSafeFory pool - resolver is already finalized. - struct PreFinalized {}; - explicit Fory(const Config &config, std::shared_ptr resolver, - PreFinalized) - : BaseFory(config, std::move(resolver)), finalized_(false), - precomputed_header_(compute_header(config.xlang)) { - // Pre-finalized, immediately create contexts - ensure_finalized(); + /// Create a runtime whose operation contexts clone an already prepared + /// resolver. + static std::unique_ptr + create_with_contexts(const Config &config, + std::shared_ptr resolver) { + auto fory = std::unique_ptr(new Fory(config, std::move(resolver))); + fory->write_ctx_.emplace(config, fory->type_resolver_->clone()); + fory->read_ctx_.emplace(config, fory->type_resolver_->clone()); + return fory; } - /// Finalize the type resolver on first use. - void ensure_finalized() { - if (!finalized_) { - lock_registration(); - auto final_result = type_resolver_->build_final_type_resolver(); - FORY_CHECK(final_result.ok()) - << "Failed to build finalized TypeResolver: " - << final_result.error().to_string(); - // Replace with finalized resolver - auto finalized_resolver = std::move(final_result).value(); + /// Freeze registration and initialize operation contexts. + void ensure_contexts_initialized() { + if (!write_ctx_.has_value()) { + FORY_CHECK(!read_ctx_.has_value()); + auto context_resolver = type_resolver_->build_context_type_resolver(); // Create contexts with cloned resolvers - write_ctx_.emplace(config_, finalized_resolver->clone()); - read_ctx_.emplace(config_, finalized_resolver->clone()); - // Store finalized resolver - type_resolver_ = std::move(finalized_resolver); - finalized_ = true; + write_ctx_.emplace(config_, context_resolver->clone()); + read_ctx_.emplace(config_, context_resolver->clone()); + type_resolver_ = std::move(context_resolver); } } @@ -837,6 +754,76 @@ class Fory : public BaseFory { ", local xlang=" + std::string(config_.xlang ? "true" : "false")); } + template + Result serialize_stream(OutputStream &output_stream, + const T &obj) { + WriteContextGuard guard(*write_ctx_); + output_stream.reset(); + write_ctx_->set_output_stream(&output_stream); + Buffer &buffer = write_ctx_->buffer(); + buffer.bind_output_stream(&output_stream); + auto serialize_result = serialize_impl(obj, buffer); + if (FORY_PREDICT_FALSE(!serialize_result.ok())) { + buffer.clear_output_stream(); + write_ctx_->set_output_stream(nullptr); + return Unexpected(std::move(serialize_result).error()); + } + output_stream.force_flush(); + buffer.clear_output_stream(); + write_ctx_->set_output_stream(nullptr); + if (FORY_PREDICT_FALSE(output_stream.has_error())) { + return Unexpected(output_stream.error()); + } + if (FORY_PREDICT_FALSE(write_ctx_->has_error())) { + return Unexpected(write_ctx_->take_error()); + } + return output_stream.flushed_bytes(); + } + + template + FORY_ALWAYS_INLINE Result serialize_buffer(Buffer &buffer, + const T &obj) { + // Swap in the caller's buffer so all writes go there. + buffer.swap(write_ctx_->buffer()); + auto result = serialize_impl(obj, write_ctx_->buffer()); + buffer.swap(write_ctx_->buffer()); + // reset internal state after use without clobbering caller buffer. + write_ctx_->reset(); + return result; + } + + template + Result deserialize_bytes(const uint8_t *data, size_t size) { + if (FORY_PREDICT_FALSE(!write_ctx_.has_value())) { + ensure_contexts_initialized(); + } + if (data == nullptr) { + return Unexpected(Error::invalid("Data pointer is null")); + } + if (size == 0) { + return Unexpected(Error::invalid("Data size is zero")); + } + + Buffer buffer(const_cast(data), static_cast(size), + false); + return deserialize_buffer(buffer); + } + + template + Result deserialize_stream(InputStream &input_stream) { + struct StreamShrinkGuard { + InputStream *input_stream = nullptr; + ~StreamShrinkGuard() { + if (input_stream != nullptr) { + input_stream->shrink_buffer(); + } + } + }; + StreamShrinkGuard shrink_guard{&input_stream}; + Buffer &buffer = input_stream.get_buffer(); + return deserialize_buffer(buffer); + } + /// Core serialization implementation. /// TypeMeta is written inline using streaming protocol (no deferred writing). template @@ -914,6 +901,12 @@ class Fory : public BaseFory { if (write_root_type_info_ != nullptr && write_root_type_info_key_ == ctid) { return write_root_type_info_; } + return cache_write_root_type_info(ctid); + } + + template + FORY_NOINLINE Result + cache_write_root_type_info(uint64_t ctid) { FORY_TRY(type_info, write_ctx_->type_resolver().template get_type_info()); write_root_type_info_key_ = ctid; @@ -921,7 +914,6 @@ class Fory : public BaseFory { return type_info; } - bool finalized_; uint8_t precomputed_header_; std::optional write_ctx_; std::optional read_ctx_; @@ -962,24 +954,28 @@ class ThreadSafeFory : public BaseFory { public: template Result, Error> serialize(const T &obj) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->serialize(obj); } template Result serialize(OutputStream &output_stream, const T &obj) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->serialize(output_stream, obj); } template Result serialize(std::ostream &ostream, const T &obj) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->serialize(ostream, obj); } template Result serialize_to(Buffer &buffer, const T &obj) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->serialize_to(buffer, obj); } @@ -987,28 +983,34 @@ class ThreadSafeFory : public BaseFory { template Result serialize_to(std::vector &output, const T &obj) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->serialize_to(output, obj); } template Result deserialize(const uint8_t *data, size_t size) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->template deserialize(data, size); } template Result deserialize(const std::vector &data) { - return deserialize(data.data(), data.size()); + ensure_resolver_initialized(); + auto fory_handle = fory_pool_.acquire(); + return fory_handle->template deserialize(data.data(), data.size()); } template Result deserialize(InputStream &input_stream) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->template deserialize(input_stream); } template Result deserialize(StdInputStream &stream) { + ensure_resolver_initialized(); auto fory_handle = fory_pool_.acquire(); return fory_handle->template deserialize(stream); } @@ -1016,26 +1018,21 @@ class ThreadSafeFory : public BaseFory { private: explicit ThreadSafeFory(const Config &config, std::shared_ptr resolver) - : BaseFory(config, std::move(resolver)), finalized_resolver_(), - finalized_once_flag_(), fory_pool_([this]() { - return std::unique_ptr(new Fory( - config_, get_finalized_resolver(), Fory::PreFinalized{})); + : BaseFory(config, std::move(resolver)), shared_resolver_(), + resolver_once_flag_(), fory_pool_([this]() { + // Every public root prepares the shared resolver before pool + // acquisition. Fory owns the per-runtime context clones. + return Fory::create_with_contexts(config_, shared_resolver_); }) {} - std::shared_ptr get_finalized_resolver() const { - std::call_once(finalized_once_flag_, [this]() { - lock_registration(); - auto final_result = type_resolver_->build_final_type_resolver(); - FORY_CHECK(final_result.ok()) - << "Failed to build finalized TypeResolver: " - << final_result.error().to_string(); - finalized_resolver_ = std::move(final_result).value(); + void ensure_resolver_initialized() const { + std::call_once(resolver_once_flag_, [this]() { + shared_resolver_ = type_resolver_->build_context_type_resolver(); }); - return finalized_resolver_->clone(); } - mutable std::shared_ptr finalized_resolver_; - mutable std::once_flag finalized_once_flag_; + mutable std::shared_ptr shared_resolver_; + mutable std::once_flag resolver_once_flag_; util::Pool fory_pool_; friend class ForyBuilder; @@ -1045,23 +1042,13 @@ class ThreadSafeFory : public BaseFory { // ForyBuilder Implementation // ============================================================================ -inline std::shared_ptr ForyBuilder::get_finalized_resolver() { - if (!type_resolver_) { - type_resolver_ = std::make_shared(); - } - type_resolver_->apply_config(normalized_config()); - auto final_result = type_resolver_->build_final_type_resolver(); - FORY_CHECK(final_result.ok()) << "Failed to build finalized TypeResolver: " - << final_result.error().to_string(); - return std::move(final_result).value(); -} - inline Fory ForyBuilder::build() { if (!type_resolver_) { type_resolver_ = std::make_shared(); } type_resolver_->apply_config(normalized_config()); - // Don't finalize yet - allow type registration, finalize on first use + // Allow type registration until the first root operation initializes its + // contexts. return Fory(config_, type_resolver_); } @@ -1070,7 +1057,7 @@ inline ThreadSafeFory ForyBuilder::build_thread_safe() { type_resolver_ = std::make_shared(); } type_resolver_->apply_config(normalized_config()); - // ThreadSafeFory builds finalized resolver lazily + // ThreadSafeFory freezes and clones its shared resolver on the first root. return ThreadSafeFory(config_, type_resolver_); } diff --git a/cpp/fory/serialization/serialization_test.cc b/cpp/fory/serialization/serialization_test.cc index e7ea4989a9..dc1a054cf9 100644 --- a/cpp/fory/serialization/serialization_test.cc +++ b/cpp/fory/serialization/serialization_test.cc @@ -76,6 +76,16 @@ struct NestedStruct { FORY_STRUCT(NestedStruct, point, label); }; +struct UnregisteredField { + int32_t value = 0; + FORY_STRUCT(UnregisteredField, value); +}; + +struct MissingFieldOwner { + UnregisteredField field; + FORY_STRUCT(MissingFieldOwner, field); +}; + enum class Color { RED, GREEN, BLUE }; enum class SignedScopedStatus : int32_t { NEG = -3, ZERO = 0, LARGE = 42 }; FORY_ENUM(SignedScopedStatus, NEG, ZERO, LARGE); @@ -181,6 +191,45 @@ inline std::vector buffer_bytes(Buffer &buffer) { buffer.data() + buffer.writer_index()); } +class RegistryProbeInputStream final : public InputStream { +public: + explicit RegistryProbeInputStream(Fory &fory) : fory_(fory) {} + + Result fill_buffer(uint32_t) override { + return Unexpected(Error::io_error("No input available")); + } + + Result read_to(uint8_t *, uint32_t) override { + return Unexpected(Error::io_error("No input available")); + } + + Result skip(uint32_t) override { + return Unexpected(Error::io_error("No input available")); + } + + Result unread(uint32_t) override { + return Unexpected(Error::io_error("No input available")); + } + + void shrink_buffer() override {} + + Buffer &get_buffer() override { + auto result = fory_.register_struct<::SimpleStruct>(1); + registration_rejected_ = !result.ok(); + return active_buffer_ == nullptr ? buffer_ : *active_buffer_; + } + + void bind_buffer(Buffer *buffer) override { active_buffer_ = buffer; } + + bool registration_rejected() const { return registration_rejected_; } + +private: + Fory &fory_; + Buffer buffer_; + Buffer *active_buffer_ = nullptr; + bool registration_rejected_ = false; +}; + template void test_roundtrip(const T &original, bool should_equal = true) { auto fory = @@ -1334,6 +1383,60 @@ TEST(SerializationTest, RegistrationByNameFailureDoesNotLeakTypeInfo) { EXPECT_EQ(dotted_type_name.error().code(), ErrorCode::Invalid); } +TEST(SerializationTest, UnusedTypeMetaStaysLazy) { + auto fory = Fory::builder().xlang(true).compatible(true).build(); + ASSERT_TRUE(fory.register_struct("demo", "UsedStruct").ok()); + ASSERT_TRUE(fory.register_struct("demo", "UnusedStruct").ok()); + + auto serialized = fory.serialize(SimpleStruct{1, 2}); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); + + auto used = + fory.write_context().type_resolver().get_type_info(); + auto unused = + fory.write_context().type_resolver().get_type_info(); + ASSERT_TRUE(used.ok()); + ASSERT_TRUE(unused.ok()); + EXPECT_NE(used.value()->type_meta, nullptr); + EXPECT_EQ(unused.value()->type_meta, nullptr); + EXPECT_TRUE(unused.value()->type_def.empty()); +} + +TEST(SerializationTest, UnneededTypeMetaStaysLazy) { + auto fory = Fory::builder() + .xlang(true) + .compatible(false) + .check_struct_version(false) + .build(); + ASSERT_TRUE(fory.register_struct(1).ok()); + + auto serialized = fory.serialize(SimpleStruct{1, 2}); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); + + auto type_info = + fory.write_context().type_resolver().get_type_info(); + ASSERT_TRUE(type_info.ok()); + EXPECT_EQ(type_info.value()->type_meta, nullptr); + EXPECT_TRUE(type_info.value()->type_def.empty()); +} + +TEST(SerializationTest, TypeMetaFailureIsAtomic) { + auto fory = Fory::builder().xlang(true).compatible(true).build(); + ASSERT_TRUE( + fory.register_struct("demo", "MissingOwner").ok()); + + auto serialized = fory.serialize(MissingFieldOwner{}); + ASSERT_FALSE(serialized.ok()); + + auto owner = + fory.write_context().type_resolver().get_type_info(); + ASSERT_TRUE(owner.ok()); + EXPECT_EQ(owner.value()->type_meta, nullptr); + EXPECT_TRUE(owner.value()->type_def.empty()); + EXPECT_FALSE( + fory.register_struct("demo", "MissingField").ok()); +} + static std::vector make_remote_type_meta(const std::string &type_name, const std::string &field) { std::vector fields; @@ -1658,10 +1761,10 @@ TEST(SerializationTest, ExpectedLocalTypeMetaStaysRootLocal) { ASSERT_TRUE( fory.register_extension_type("example", "ExpectedExt").ok()); ASSERT_TRUE(fory.register_union("example", "ExpectedUnion").ok()); - auto finalized = fory.serialize(SimpleStruct{}); - ASSERT_TRUE(finalized.ok()) << finalized.error().to_string(); + auto serialized = fory.serialize(SimpleStruct{}); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); - ReadContext ctx(fory.config(), fory.type_resolver().clone()); + ReadContext ctx(fory.config(), fory.write_context().type_resolver().clone()); auto struct_info = ctx.type_resolver().get_type_info(); auto enum_info = ctx.type_resolver().get_type_info(); auto ext_info = ctx.type_resolver().get_type_info(); @@ -1718,8 +1821,8 @@ TEST(SerializationTest, LocalTypeMetaPrecedesRemoteCache) { auto fory = Fory::builder().xlang(true).compatible(true).build(); ASSERT_TRUE( fory.register_enum("example", "WarmLocal").ok()); - auto finalized = fory.serialize(SignedScopedStatus::ZERO); - ASSERT_TRUE(finalized.ok()) << finalized.error().to_string(); + auto serialized = fory.serialize(SignedScopedStatus::ZERO); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); auto expected = fory.type_resolver().get_type_info(); ASSERT_TRUE(expected.ok()) << expected.error().to_string(); const std::vector &type_def = expected.value()->type_def; @@ -1763,8 +1866,8 @@ TEST(SerializationTest, StaticTypeMetaChecksOwner) { ASSERT_TRUE(fory.register_enum("example", "EnumB").ok()); ASSERT_TRUE(fory.register_union("example", "UnionA").ok()); ASSERT_TRUE(fory.register_union("example", "UnionB").ok()); - auto finalized = fory.serialize(SimpleStruct{}); - ASSERT_TRUE(finalized.ok()) << finalized.error().to_string(); + auto serialized = fory.serialize(SimpleStruct{}); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); ReadContext ctx(fory.config(), fory.type_resolver().clone()); auto struct_b = ctx.type_resolver().get_type_info(); @@ -1773,6 +1876,7 @@ TEST(SerializationTest, StaticTypeMetaChecksOwner) { ASSERT_TRUE(struct_b.ok()); ASSERT_TRUE(enum_b.ok()); ASSERT_TRUE(union_b.ok()); + ASSERT_TRUE(ctx.type_resolver().ensure_type_meta(struct_b.value()).ok()); expect_static_meta_ref_mismatch(ctx, struct_b.value()); expect_static_meta_ref_mismatch(ctx, enum_b.value()); @@ -1801,8 +1905,8 @@ TEST(SerializationTest, CachedTypeMetaChecksOwner) { ASSERT_TRUE( fory.register_enum("example", "CacheEnumA").ok()); ASSERT_TRUE(fory.register_union("example", "CacheUnionA").ok()); - auto finalized = fory.serialize(SimpleStruct{}); - ASSERT_TRUE(finalized.ok()) << finalized.error().to_string(); + auto serialized = fory.serialize(SimpleStruct{}); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); ReadContext ctx(fory.config(), fory.type_resolver().clone()); expect_cached_owner_mismatch( @@ -1827,8 +1931,8 @@ TEST(SerializationTest, StaticCollectionChecksOwner) { fory.register_struct("example", "CollectionA").ok()); ASSERT_TRUE( fory.register_struct("example", "CollectionB").ok()); - auto finalized = fory.serialize(SimpleStruct{}); - ASSERT_TRUE(finalized.ok()) << finalized.error().to_string(); + auto serialized = fory.serialize(SimpleStruct{}); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); auto first = make_remote_type_meta("CollectionB", "first_remote"); auto second = make_remote_type_meta("CollectionB", "second_remote"); @@ -1867,8 +1971,8 @@ TEST(SerializationTest, StaticMapChecksOwner) { .build(); ASSERT_TRUE(fory.register_struct("example", "MapA").ok()); ASSERT_TRUE(fory.register_struct("example", "MapB").ok()); - auto finalized = fory.serialize(SimpleStruct{}); - ASSERT_TRUE(finalized.ok()) << finalized.error().to_string(); + auto serialized = fory.serialize(SimpleStruct{}); + ASSERT_TRUE(serialized.ok()) << serialized.error().to_string(); auto first = make_remote_type_meta("MapB", "first_remote"); auto second = make_remote_type_meta("MapB", "second_remote"); @@ -2059,7 +2163,7 @@ TEST(SerializationTest, IdExtDoesNotUseTypeMetaLimits) { EXPECT_EQ(decoded.value(), IdLimitExt{42}); } -TEST(SerializationTest, LocalTypeMetaFinalizationIgnoresReceiveBodyLimit) { +TEST(SerializationTest, LocalMetaIgnoresReceiveLimit) { auto fory = Fory::builder() .xlang(true) .compatible(true) @@ -2439,6 +2543,85 @@ TEST(SerializationTest, ConfigurationBuilder) { EXPECT_FALSE(compatible_with_version_check.config().check_struct_version); } +// ============================================================================ +// Registration Lifecycle Tests +// ============================================================================ + +TEST(SerializationTest, DirectFailedRootFreezes) { + auto source_resolver = std::make_shared(); + auto fory = Fory::builder() + .xlang(true) + .compatible(false) + .track_ref(false) + .type_resolver(source_resolver) + .build(); + + auto root_result = fory.deserialize(nullptr, 0); + ASSERT_FALSE(root_result.ok()); + + auto facade_registration = fory.register_struct<::SimpleStruct>(1); + ASSERT_FALSE(facade_registration.ok()); + + auto type_info = source_resolver->get_type_info_by_id( + static_cast(TypeId::STRING)); + ASSERT_TRUE(type_info.ok()); + ASSERT_EQ(type_info.value()->harness.any_write_fn, nullptr); + ASSERT_EQ(type_info.value()->harness.any_read_fn, nullptr); + + auto late_registration = register_any_type(*source_resolver); + ASSERT_FALSE(late_registration.ok()); + EXPECT_EQ(type_info.value()->harness.any_write_fn, nullptr); + EXPECT_EQ(type_info.value()->harness.any_read_fn, nullptr); + EXPECT_FALSE( + source_resolver->get_type_info(std::type_index(typeid(std::string))) + .ok()); +} + +TEST(SerializationTest, ThreadSafeRegistrationFreezes) { + auto fory = Fory::builder() + .xlang(true) + .compatible(false) + .track_ref(false) + .build_thread_safe(); + ASSERT_TRUE(fory.register_struct<::ComplexStruct>(1).ok()); + + ::ComplexStruct original{"Alice", 30, {"reading", "coding"}}; + auto bytes_result = fory.serialize(original); + ASSERT_TRUE(bytes_result.ok()) + << "Serialization failed: " << bytes_result.error().to_string(); + + auto late_registration = fory.register_struct<::SimpleStruct>(2); + EXPECT_FALSE(late_registration.ok()); +} + +TEST(SerializationTest, InputStreamFreezesBeforeAccess) { + auto fory = + Fory::builder().xlang(true).compatible(false).track_ref(false).build(); + RegistryProbeInputStream input_stream(fory); + + EXPECT_FALSE(fory.deserialize(input_stream).ok()); + EXPECT_TRUE(input_stream.registration_rejected()); +} + +TEST(SerializationTest, ThreadSafeFailedRootFreezes) { + auto source_resolver = std::make_shared(); + auto fory = Fory::builder() + .xlang(true) + .compatible(false) + .track_ref(false) + .type_resolver(source_resolver) + .build_thread_safe(); + + auto root_result = fory.deserialize(nullptr, 0); + ASSERT_FALSE(root_result.ok()); + + auto facade_registration = fory.register_struct<::SimpleStruct>(1); + ASSERT_FALSE(facade_registration.ok()); + + auto late_registration = register_any_type(*source_resolver); + ASSERT_FALSE(late_registration.ok()); +} + // ============================================================================ // Thread Safety Tests // ============================================================================ @@ -2484,27 +2667,6 @@ TEST(SerializationTest, ThreadSafeForyMultiThread) { EXPECT_EQ(success_count.load(), k_num_threads * k_iterations_per_thread); } -TEST(SerializationTest, ThreadSafeForyRejectsRegistrationAfterFirstSerialize) { - auto fory = Fory::builder() - .xlang(true) - .compatible(false) - .track_ref(false) - .build_thread_safe(); - ASSERT_TRUE(fory.register_struct<::ComplexStruct>(1).ok()); - - ::ComplexStruct original{"Alice", 30, {"reading", "coding"}}; - auto bytes_result = fory.serialize(original); - ASSERT_TRUE(bytes_result.ok()) - << "Serialization failed: " << bytes_result.error().to_string(); - - auto late_registration = fory.register_struct<::SimpleStruct>(2); - EXPECT_FALSE(late_registration.ok()); - ASSERT_FALSE(late_registration.ok()); - EXPECT_EQ(late_registration.error().code(), ErrorCode::Invalid); - EXPECT_NE(late_registration.error().to_string().find("Cannot register types"), - std::string::npos); -} - TEST(SerializationTest, TemporalCarriersAreHashable) { std::unordered_map date_map; date_map[Date(0)] = "epoch"; diff --git a/cpp/fory/serialization/skip.cc b/cpp/fory/serialization/skip.cc index 6d79c34a0c..e3c904f6fc 100644 --- a/cpp/fory/serialization/skip.cc +++ b/cpp/fory/serialization/skip.cc @@ -98,8 +98,11 @@ void skip_fields(ReadContext &ctx, const std::vector &field_infos) { void skip_struct_data(ReadContext &ctx, const TypeInfo &type_info) { if (!type_info.type_meta) { - ctx.set_error(Error::type_error("TypeMeta not found for struct skip")); - return; + auto result = ctx.type_resolver().ensure_type_meta(&type_info); + if (FORY_PREDICT_FALSE(!result.ok())) { + ctx.set_error(std::move(result).error()); + return; + } } if (ctx.check_struct_version()) { (void)ctx.read_int32(ctx.error()); @@ -537,11 +540,17 @@ void skip_struct(ReadContext &ctx, const FieldType &) { } } - if (!type_info || !type_info->type_meta) { - ctx.set_error( - Error::type_error("TypeInfo or TypeMeta not found for struct skip")); + if (!type_info) { + ctx.set_error(Error::type_error("TypeInfo not found for struct skip")); return; } + if (!type_info->type_meta) { + auto result = ctx.type_resolver().ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(!result.ok())) { + ctx.set_error(std::move(result).error()); + return; + } + } skip_fields(ctx, type_info->type_meta->get_field_infos()); } @@ -673,9 +682,11 @@ void skip_unknown(ReadContext &ctx) { case TypeId::NAMED_COMPATIBLE_STRUCT: { // For struct types, we already have the type_info with field_infos if (!type_info->type_meta) { - ctx.set_error( - Error::type_error("TypeMeta not found for UNKNOWN struct skip")); - return; + auto result = ctx.type_resolver().ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(!result.ok())) { + ctx.set_error(std::move(result).error()); + return; + } } skip_fields(ctx, type_info->type_meta->get_field_infos()); return; diff --git a/cpp/fory/serialization/struct_serializer.h b/cpp/fory/serialization/struct_serializer.h index d04707935a..521e8f6a72 100644 --- a/cpp/fory/serialization/struct_serializer.h +++ b/cpp/fory/serialization/struct_serializer.h @@ -4614,10 +4614,7 @@ struct Serializer>> { return; } const TypeInfo *type_info = type_info_res.value(); - auto write_result = ctx.write_struct_type_info(type_info); - if (FORY_PREDICT_FALSE(!write_result.ok())) { - ctx.set_error(std::move(write_result).error()); - } + ctx.write_struct_type_info(type_info); } /// Read and validate type info. @@ -4691,6 +4688,14 @@ struct Serializer>> { write_data_generic(obj, ctx, has_generics); } + static FORY_NOINLINE void ensure_type_meta(WriteContext &ctx, + const TypeInfo *type_info) { + auto result = ctx.type_resolver().ensure_type_meta(type_info); + if (FORY_PREDICT_FALSE(!result.ok())) { + ctx.set_error(std::move(result).error()); + } + } + static void write_data(const T &obj, WriteContext &ctx) { // Only write struct version hash when check_struct_version is enabled, // matching Java's behavior in ObjectSerializer.write(). @@ -4701,10 +4706,11 @@ struct Serializer>> { return; } const TypeInfo *type_info = type_info_res.value(); - if (!type_info->type_meta) { - ctx.set_error( - Error::type_error("Type metadata not initialized for struct")); - return; + if (FORY_PREDICT_FALSE(!type_info->type_meta)) { + ensure_type_meta(ctx, type_info); + if (FORY_PREDICT_FALSE(ctx.has_error())) { + return; + } } int32_t local_version = TypeMeta::compute_struct_version(*type_info->type_meta); @@ -4733,10 +4739,11 @@ struct Serializer>> { return; } const TypeInfo *type_info = type_info_res.value(); - if (!type_info->type_meta) { - ctx.set_error( - Error::type_error("Type metadata not initialized for struct")); - return; + if (FORY_PREDICT_FALSE(!type_info->type_meta)) { + ensure_type_meta(ctx, type_info); + if (FORY_PREDICT_FALSE(ctx.has_error())) { + return; + } } int32_t local_version = TypeMeta::compute_struct_version(*type_info->type_meta); @@ -4952,10 +4959,13 @@ struct Serializer>> { return T{}; } local_type_info = local_type_info_res.value(); - if (!local_type_info->type_meta) { - ctx.set_error(Error::type_error( - "Type metadata not initialized for requested struct")); - return T{}; + if (FORY_PREDICT_FALSE(!local_type_info->type_meta)) { + auto meta_result = + ctx.type_resolver().ensure_type_meta(local_type_info); + if (FORY_PREDICT_FALSE(!meta_result.ok())) { + ctx.set_error(std::move(meta_result).error()); + return T{}; + } } int32_t local_version = TypeMeta::compute_struct_version(*local_type_info->type_meta); @@ -4973,10 +4983,13 @@ struct Serializer>> { return T{}; } local_type_info = local_type_info_res.value(); - if (!local_type_info->type_meta) { - ctx.set_error(Error::type_error( - "Type metadata not initialized for requested struct")); - return T{}; + if (FORY_PREDICT_FALSE(!local_type_info->type_meta)) { + auto meta_result = + ctx.type_resolver().ensure_type_meta(local_type_info); + if (FORY_PREDICT_FALSE(!meta_result.ok())) { + ctx.set_error(std::move(meta_result).error()); + return T{}; + } } } @@ -5043,10 +5056,13 @@ struct Serializer>> { return T{}; } const TypeInfo *local_type_info = local_type_info_res.value(); - if (!local_type_info->type_meta) { - ctx.set_error(Error::type_error( - "Type metadata not initialized for requested struct")); - return T{}; + if (FORY_PREDICT_FALSE(!local_type_info->type_meta)) { + auto meta_result = + ctx.type_resolver().ensure_type_meta(local_type_info); + if (FORY_PREDICT_FALSE(!meta_result.ok())) { + ctx.set_error(std::move(meta_result).error()); + return T{}; + } } int32_t local_version = TypeMeta::compute_struct_version(*local_type_info->type_meta); @@ -5086,8 +5102,15 @@ struct Serializer>> { return T{}; } - // In compatible mode with type info provided, use schema evolution path - if (ctx.is_compatible() && type_info.type_meta) { + // In compatible mode with type info provided, use schema evolution path. + if (ctx.is_compatible()) { + if (FORY_PREDICT_FALSE(!type_info.type_meta)) { + auto meta_result = ctx.type_resolver().ensure_type_meta(&type_info); + if (FORY_PREDICT_FALSE(!meta_result.ok())) { + ctx.set_error(std::move(meta_result).error()); + return T{}; + } + } return read_compatible(ctx, &type_info); } diff --git a/cpp/fory/serialization/type_info.h b/cpp/fory/serialization/type_info.h index 48564019f5..37441f737e 100644 --- a/cpp/fory/serialization/type_info.h +++ b/cpp/fory/serialization/type_info.h @@ -136,14 +136,14 @@ struct TypeInfo { std::string type_name; bool register_by_name = false; bool is_external = false; - std::unique_ptr type_meta; + mutable std::unique_ptr type_meta; std::vector sorted_indices; fory::flat_hash_map name_to_index; - std::vector type_def; + mutable std::vector type_def; Harness harness; - // TypeInfo and its harness are immutable after registration. Cache the last - // read target so repeated root operations avoid walking the declared base - // graph; ThreadSafeFory uses distinct cloned TypeInfo owners per pooled Fory. + // Registration data and the harness are immutable after registration. + // Operation-context clones complete TypeMeta lazily and cache the last read + // target; ThreadSafeFory never shares these mutable caches between workers. mutable const std::type_info *cached_read_target = nullptr; mutable Harness::ReadAsFn cached_read_as_fn = nullptr; // Pre-encoded meta strings for efficient writing (avoids re-encoding on each diff --git a/cpp/fory/serialization/type_resolver.cc b/cpp/fory/serialization/type_resolver.cc index edf8fc60d3..ca90419d50 100644 --- a/cpp/fory/serialization/type_resolver.cc +++ b/cpp/fory/serialization/type_resolver.cc @@ -1762,91 +1762,59 @@ TypeResolver::get_type_info(const std::type_index &type_index) const { return entry->second; } -Result, Error> -TypeResolver::build_final_type_resolver() { - auto final_resolver = std::make_unique(); - - // copy configuration - final_resolver->compatible_ = compatible_; - final_resolver->xlang_ = xlang_; - final_resolver->check_struct_version_ = check_struct_version_; - final_resolver->track_ref_ = track_ref_; - final_resolver->finalized_ = true; - - // Build mapping from old pointers to new pointers for rebuilding lookup maps - fory::flat_hash_map ptr_map; +Result TypeResolver::check_registration() { + if (FORY_PREDICT_FALSE(registry_frozen_)) { + return Unexpected(Error::invalid( + "TypeResolver registry is frozen, cannot register more types")); + } + FORY_CHECK(std::this_thread::get_id() == registration_thread_id_) + << "TypeResolver registration methods must be called from the same " + "thread that created the TypeResolver"; + return Result(); +} - // Deep clone all existing TypeInfo objects - for (const auto &info : type_infos_) { - auto cloned = info->deep_clone(); - TypeInfo *new_ptr = cloned.get(); - ptr_map[info.get()] = new_ptr; - final_resolver->type_infos_.push_back(std::move(cloned)); - } - auto remap_type_info = [&ptr_map](const TypeInfo *old_ptr) { - auto *entry = ptr_map.find(old_ptr); - FORY_CHECK(entry != nullptr); - return entry->second; - }; +std::unique_ptr TypeResolver::build_context_type_resolver() { + registry_frozen_ = true; + return clone(); +} - // Rebuild lookup maps with new pointers - for (const auto &[key, old_ptr] : type_info_by_ctid_) { - final_resolver->type_info_by_ctid_.put(key, remap_type_info(old_ptr)); - } - for (const auto &[key, old_ptr] : type_info_by_id_) { - final_resolver->type_info_by_id_.put(key, remap_type_info(old_ptr)); +Result TypeResolver::ensure_type_meta(const TypeInfo *type_info) { + if (FORY_PREDICT_TRUE(type_info != nullptr && type_info->type_meta)) { + return Result(); } - for (const auto &[key, old_ptr] : user_type_info_by_id_) { - final_resolver->user_type_info_by_id_.put(key, remap_type_info(old_ptr)); + if (FORY_PREDICT_FALSE(type_info == nullptr)) { + return Unexpected(Error::invalid("TypeInfo is null")); } - for (const auto &[key, old_ptr] : type_info_by_name_) { - final_resolver->type_info_by_name_[key] = remap_type_info(old_ptr); - } - for (const auto &[key, old_ptr] : type_info_by_runtime_type_) { - final_resolver->type_info_by_runtime_type_[key] = remap_type_info(old_ptr); - } - - for (const auto &[key, old_ptr] : partial_type_infos_) { - final_resolver->partial_type_infos_.put(key, remap_type_info(old_ptr)); - } - - // Process all partial type infos to build complete type metadata - for (const auto &[rust_type_id, partial_ptr] : - final_resolver->partial_type_infos_) { - // Call the harness's sorted_field_infos function to get complete field info - FORY_TRY(sorted_fields, - partial_ptr->harness.sorted_field_infos_fn(*final_resolver)); - - // Build complete TypeMeta - TypeMeta meta = TypeMeta::from_fields( - partial_ptr->type_id, partial_ptr->namespace_name, - partial_ptr->type_name, partial_ptr->register_by_name, - partial_ptr->user_type_id, std::move(sorted_fields)); - - // Serialize TypeMeta to bytes - FORY_TRY(type_def, meta.to_bytes()); - - // Update the TypeInfo in place - partial_ptr->type_def = std::move(type_def); - - // Parse the serialized TypeMeta back to create unique_ptr - Buffer buffer(partial_ptr->type_def.data(), - static_cast(partial_ptr->type_def.size()), false); - buffer.writer_index(static_cast(partial_ptr->type_def.size())); - // This metadata was just generated from local registration state. Remote - // receive limits are enforced only on remote metadata parse/cache-miss - // paths, so large trusted local schemas do not fail during finalization. - FORY_TRY(parsed_meta, - TypeMeta::from_bytes(buffer, nullptr, - std::numeric_limits::max(), - std::numeric_limits::max())); - partial_ptr->type_meta = std::move(parsed_meta); + if (FORY_PREDICT_FALSE(!registry_frozen_)) { + return Unexpected(Error::invalid( + "Type metadata is available only after registration is frozen")); } - - // Clear partial_type_infos in the final resolver since they're all completed - final_resolver->partial_type_infos_.clear(); - - return final_resolver; + if (FORY_PREDICT_FALSE(type_info->harness.sorted_field_infos_fn == nullptr)) { + return Unexpected( + Error::type_error("Type metadata builder is not available")); + } + + FORY_TRY(sorted_fields, type_info->harness.sorted_field_infos_fn(*this)); + TypeMeta meta = + TypeMeta::from_fields(type_info->type_id, type_info->namespace_name, + type_info->type_name, type_info->register_by_name, + type_info->user_type_id, std::move(sorted_fields)); + FORY_TRY(type_def, meta.to_bytes()); + + Buffer buffer(type_def.data(), static_cast(type_def.size()), false); + buffer.writer_index(static_cast(type_def.size())); + // Local registration metadata is trusted. Remote receive limits apply only + // to remote cache misses, not to this context-local completion path. + FORY_TRY(parsed_meta, + TypeMeta::from_bytes(buffer, nullptr, + std::numeric_limits::max(), + std::numeric_limits::max())); + + // Publish only after every fallible step succeeds. The TypeMeta pointer is + // the completion condition, so a failed attempt leaves no partial state. + type_info->type_def = std::move(type_def); + type_info->type_meta = std::move(parsed_meta); + return Result(); } std::unique_ptr TypeResolver::clone() const { @@ -1857,7 +1825,7 @@ std::unique_ptr TypeResolver::clone() const { cloned->xlang_ = xlang_; cloned->check_struct_version_ = check_struct_version_; cloned->track_ref_ = track_ref_; - cloned->finalized_ = finalized_; + cloned->registry_frozen_ = registry_frozen_; // Build mapping from old pointers to new pointers fory::flat_hash_map ptr_map; @@ -1891,9 +1859,6 @@ std::unique_ptr TypeResolver::clone() const { for (const auto &[key, old_ptr] : type_info_by_runtime_type_) { cloned->type_info_by_runtime_type_[key] = remap_type_info(old_ptr); } - // Note: Don't copy partial_type_infos_ - clone should only be used on - // finalized resolvers - return cloned; } diff --git a/cpp/fory/serialization/type_resolver.h b/cpp/fory/serialization/type_resolver.h index 07344e68a3..753102196a 100644 --- a/cpp/fory/serialization/type_resolver.h +++ b/cpp/fory/serialization/type_resolver.h @@ -1359,22 +1359,12 @@ class TypeResolver { template Result register_any_type(); - /// Builds the final TypeResolver by completing all partial type infos - /// created during registration. - /// - /// This method processes all types that were registered. During registration, - /// types are stored in `partial_type_infos` without their complete - /// type metadata to avoid circular dependencies. This method: - /// - /// 1. Iterates through all partial type infos - /// 2. Calls their `sorted_field_infos` function to get complete field - /// information - /// 3. Builds complete TypeMeta and serializes it to bytes - /// 4. Returns a new TypeResolver with all type infos fully initialized - /// - /// @return A new TypeResolver with all type infos fully initialized and ready - /// for use. - Result, Error> build_final_type_resolver(); + /// Permanently freezes registration and clones the resolver for operation + /// contexts. Type metadata is completed by the context clone when used. + std::unique_ptr build_context_type_resolver(); + + /// Complete one TypeInfo's metadata after registration is frozen. + Result ensure_type_meta(const TypeInfo *type_info); /// Deep clones the TypeResolver for use in a new context. /// @@ -1516,7 +1506,7 @@ class TypeResolver { void register_type_internal_runtime(const std::type_index &type_index, TypeInfo *info); - void check_registration_thread(); + Result check_registration(); void register_builtin_types(); @@ -1526,7 +1516,10 @@ class TypeResolver { bool track_ref_; std::thread::id registration_thread_id_; - bool finalized_; + bool registry_frozen_; + // Registration is creator-thread-only. Shared facades are configured before + // concurrent use and own first-root synchronization; the resolver must not + // duplicate that synchronization around its hot lookup state. // Primary storage - owns all TypeInfo objects std::vector> type_infos_; @@ -1538,7 +1531,6 @@ class TypeResolver { util::U32PtrMap type_info_by_id_{256}; util::U64PtrMap user_type_info_by_id_{256}; fory::flat_hash_map type_info_by_name_; - util::U64PtrMap partial_type_infos_{256}; // For runtime polymorphic lookups (smart pointers) - uses std::type_index fory::flat_hash_map type_info_by_runtime_type_; @@ -1554,7 +1546,7 @@ class TypeResolver { inline TypeResolver::TypeResolver() : compatible_(false), xlang_(false), check_struct_version_(true), track_ref_(true), registration_thread_id_(std::this_thread::get_id()), - finalized_(false) { + registry_frozen_(false) { register_builtin_types(); } @@ -1565,14 +1557,6 @@ inline void TypeResolver::apply_config(const Config &config) { track_ref_ = config.track_ref; } -inline void TypeResolver::check_registration_thread() { - FORY_CHECK(std::this_thread::get_id() == registration_thread_id_) - << "TypeResolver registration methods must be called from the same " - "thread that created the TypeResolver"; - FORY_CHECK(!finalized_) - << "TypeResolver has been finalized, cannot register more types"; -} - template inline void *TypeResolver::harness_struct_read_as(ReadContext &ctx, const TypeInfo *type_info) { @@ -1712,8 +1696,8 @@ template const TypeMeta &TypeResolver::struct_meta() { constexpr uint64_t ctid = type_index(); TypeInfo *info = type_info_by_ctid_.get_or_default(ctid, nullptr); FORY_CHECK(info != nullptr) << "Type not registered"; - FORY_CHECK(info->type_meta) - << "Type metadata not initialized for requested struct"; + auto result = ensure_type_meta(info); + FORY_CHECK(result.ok()) << result.error().to_string(); return *info->type_meta; } @@ -1721,8 +1705,8 @@ template TypeMeta TypeResolver::clone_struct_meta() { constexpr uint64_t ctid = type_index(); TypeInfo *info = type_info_by_ctid_.get_or_default(ctid, nullptr); FORY_CHECK(info != nullptr) << "Type not registered"; - FORY_CHECK(info->type_meta) - << "Type metadata not initialized for requested struct"; + auto result = ensure_type_meta(info); + FORY_CHECK(result.ok()) << result.error().to_string(); return *info->type_meta; } @@ -1752,7 +1736,7 @@ get_type_info_with_resolver(TypeResolver &resolver) { } template Result TypeResolver::register_any_type() { - check_registration_thread(); + FORY_RETURN_IF_ERROR(check_registration()); using ChronoTimestamp = std::chrono::time_point; if constexpr (std::is_same_v || @@ -1790,7 +1774,7 @@ template Result TypeResolver::register_any_type() { template Result TypeResolver::register_by_id(uint32_t type_id) { - check_registration_thread(); + FORY_RETURN_IF_ERROR(check_registration()); if (type_id == kInvalidUserTypeId) { return Unexpected(Error::invalid( "type_id must be in range [0, 0xfffffffe] for register_by_id")); @@ -1814,8 +1798,7 @@ Result TypeResolver::register_by_id(uint32_t type_id) { // Register and get back the stored pointer FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - // Also register for runtime polymorphic lookups and partial type infos - partial_type_infos_.put(ctid, stored_ptr); + // Also register for runtime polymorphic lookups. register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } else if constexpr (std::is_enum_v) { @@ -1830,7 +1813,6 @@ Result TypeResolver::register_by_id(uint32_t type_id) { } FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - partial_type_infos_.put(ctid, stored_ptr); register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } else { @@ -1845,7 +1827,7 @@ template Result TypeResolver::register_by_name(const std::string &ns, const std::string &type_name) { - check_registration_thread(); + FORY_RETURN_IF_ERROR(check_registration()); if (type_name.empty()) { return Unexpected( Error::invalid("type_name must be non-empty for register_by_name")); @@ -1871,7 +1853,6 @@ TypeResolver::register_by_name(const std::string &ns, } FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - partial_type_infos_.put(ctid, stored_ptr); register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } else if constexpr (std::is_enum_v) { @@ -1884,7 +1865,6 @@ TypeResolver::register_by_name(const std::string &ns, } FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - partial_type_infos_.put(ctid, stored_ptr); register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } else { @@ -1897,7 +1877,7 @@ TypeResolver::register_by_name(const std::string &ns, template Result TypeResolver::register_ext_type_by_id(uint32_t type_id) { - check_registration_thread(); + FORY_RETURN_IF_ERROR(check_registration()); if (type_id == kInvalidUserTypeId) { return Unexpected(Error::invalid("type_id must be in range [0, 0xfffffffe] " "for register_ext_type_by_id")); @@ -1912,7 +1892,6 @@ Result TypeResolver::register_ext_type_by_id(uint32_t type_id) { build_ext_type_info(actual_type_id, user_type_id, "", "", false)); FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - partial_type_infos_.put(ctid, stored_ptr); register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } @@ -1921,7 +1900,7 @@ template Result TypeResolver::register_ext_type_by_name(const std::string &ns, const std::string &type_name) { - check_registration_thread(); + FORY_RETURN_IF_ERROR(check_registration()); if (type_name.empty()) { return Unexpected(Error::invalid( "type_name must be non-empty for register_ext_type_by_name")); @@ -1938,14 +1917,13 @@ TypeResolver::register_ext_type_by_name(const std::string &ns, type_name, true)); FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - partial_type_infos_.put(ctid, stored_ptr); register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } template Result TypeResolver::register_union_by_id(uint32_t type_id) { - check_registration_thread(); + FORY_RETURN_IF_ERROR(check_registration()); if (type_id == kInvalidUserTypeId) { return Unexpected(Error::invalid( "type_id must be in range [0, 0xfffffffe] for register_union_by_id")); @@ -1959,7 +1937,6 @@ Result TypeResolver::register_union_by_id(uint32_t type_id) { false)); FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - partial_type_infos_.put(ctid, stored_ptr); register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } @@ -1968,7 +1945,7 @@ template Result TypeResolver::register_union_by_name(const std::string &ns, const std::string &type_name) { - check_registration_thread(); + FORY_RETURN_IF_ERROR(check_registration()); if (type_name.empty()) { return Unexpected(Error::invalid( "type_name must be non-empty for register_union_by_name")); @@ -1985,7 +1962,6 @@ TypeResolver::register_union_by_name(const std::string &ns, ns, type_name, true)); FORY_TRY(stored_ptr, register_type_internal(ctid, std::move(info))); - partial_type_infos_.put(ctid, stored_ptr); register_type_internal_runtime(std::type_index(typeid(T)), stored_ptr); return Result(); } diff --git a/csharp/src/Fory/Fory.cs b/csharp/src/Fory/Fory.cs index cf0684b2e9..0b1bb1c319 100644 --- a/csharp/src/Fory/Fory.cs +++ b/csharp/src/Fory/Fory.cs @@ -30,6 +30,7 @@ public sealed class Fory private readonly TypeResolver _typeResolver; private WriteContext _writeContext; private ReadContext _readContext; + private bool _registryFrozen; internal Fory(Config config) { @@ -67,9 +68,13 @@ public static ForyBuilder Builder() /// Type to register. /// Numeric type identifier used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public Fory Register(uint typeId) { - _typeResolver.Register(typeof(T), typeId); + EnsureRegistrationOpen(); + TypeInfo typeInfo = ResolveRegistrationTypeInfo(); + _typeResolver.Register(typeof(T), typeId, typeInfo); return this; } @@ -79,10 +84,14 @@ public Fory Register(uint typeId) /// Type to register. /// Name used on the wire. A dotted name is split at the last dot. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public Fory Register(string name) { + EnsureRegistrationOpen(); (string namespaceName, string typeName) = TypeResolver.SplitTypeName(name); - _typeResolver.Register(typeof(T), namespaceName, typeName); + TypeInfo typeInfo = ResolveRegistrationTypeInfo(); + _typeResolver.Register(typeof(T), namespaceName, typeName, typeInfo); return this; } @@ -93,9 +102,14 @@ public Fory Register(string name) /// Namespace used on the wire. /// Type name used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public Fory Register(string typeNamespace, string typeName) { - _typeResolver.Register(typeof(T), typeNamespace, typeName); + EnsureRegistrationOpen(); + TypeResolver.ValidateSplitTypeName(typeNamespace, typeName); + TypeInfo typeInfo = ResolveRegistrationTypeInfo(); + _typeResolver.Register(typeof(T), typeNamespace, typeName, typeInfo); return this; } @@ -106,10 +120,13 @@ public Fory Register(string typeNamespace, string typeName) /// Serializer implementation used for . /// Numeric type identifier used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public Fory Register(uint typeId) where TSerializer : Serializer, new() { - TypeInfo typeInfo = _typeResolver.RegisterSerializer(); + EnsureRegistrationOpen(); + TypeInfo typeInfo = CreateCustomTypeInfo(); _typeResolver.Register(typeof(T), typeId, typeInfo); return this; } @@ -121,11 +138,14 @@ public Fory Register(uint typeId) /// Serializer implementation used for . /// Name used on the wire. A dotted name is split at the last dot. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public Fory Register(string name) where TSerializer : Serializer, new() { + EnsureRegistrationOpen(); (string namespaceName, string typeName) = TypeResolver.SplitTypeName(name); - TypeInfo typeInfo = _typeResolver.RegisterSerializer(); + TypeInfo typeInfo = CreateCustomTypeInfo(); _typeResolver.Register(typeof(T), namespaceName, typeName, typeInfo); return this; } @@ -138,11 +158,14 @@ public Fory Register(string name) /// Namespace used on the wire. /// Type name used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public Fory Register(string typeNamespace, string typeName) where TSerializer : Serializer, new() { + EnsureRegistrationOpen(); TypeResolver.ValidateSplitTypeName(typeNamespace, typeName); - TypeInfo typeInfo = _typeResolver.RegisterSerializer(); + TypeInfo typeInfo = CreateCustomTypeInfo(); _typeResolver.Register(typeof(T), typeNamespace, typeName, typeInfo); return this; } @@ -155,13 +178,24 @@ public Fory Register(string typeNamespace, string typeName) /// Serialized bytes. public byte[] Serialize(in T value) { + _registryFrozen = true; ByteWriter writer = _writeContext.Writer; writer.Reset(); + // Serializer lookup is part of the root and may fail before codec entry, so establish the + // root's clean context before invoking generated or application serializer factories. + _writeContext.ResetFor(writer); Serializer serializer = _typeResolver.GetSerializer(); WriteHead(writer); - _writeContext.ResetFor(writer); RefMode refMode = Config.TrackRef ? RefMode.Tracking : RefMode.NullOnly; - serializer.Write(_writeContext, value, refMode, true, false); + try + { + serializer.Write(_writeContext, value, refMode, true, false); + } + catch + { + _writeContext.Reset(); + throw; + } _writeContext.RefWriter.Reset(); return writer.ToArray(); @@ -188,11 +222,13 @@ public void Serialize(IBufferWriter output, in T value) /// Thrown when trailing bytes remain after decoding. public T Deserialize(ReadOnlySpan payload) { + _registryFrozen = true; ByteReader reader = _readContext.Reader; reader.Reset(payload); T value = DeserializeFromReader(reader); if (reader.Remaining != 0) { + _readContext.Reset(); ThrowUnexpectedTrailingBytes(); } @@ -208,11 +244,21 @@ public T Deserialize(ReadOnlySpan payload) /// Thrown when trailing bytes remain after decoding. public T Deserialize(byte[] payload) { + _registryFrozen = true; ByteReader reader = _readContext.Reader; - reader.Reset(payload); + try + { + reader.Reset(payload); + } + catch + { + _readContext.Reset(); + throw; + } T value = DeserializeFromReader(reader); if (reader.Remaining != 0) { + _readContext.Reset(); ThrowUnexpectedTrailingBytes(); } @@ -258,6 +304,7 @@ private static void ThrowInvalidRootHeader(byte bitmap) => [MethodImpl(MethodImplOptions.AggressiveInlining)] internal T DeserializeFromReader(ByteReader reader) { + _registryFrozen = true; ReadContext readContext = _readContext; readContext.ResetFor(reader); readContext._remainingGraphMemoryBytes = Config.MaxGraphMemoryBytes; @@ -288,4 +335,37 @@ internal T DeserializeFromReader(ByteReader reader) } } + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private void EnsureRegistrationOpen() + { + if (_registryFrozen) + { + ThrowRegistryFrozen(); + } + } + + private TypeInfo CreateCustomTypeInfo() + where TSerializer : Serializer, new() + { + TypeInfo typeInfo = TypeInfo.Create(typeof(T), new TSerializer()); + // Serializer construction and TypeInfo creation can execute application code. Recheck + // after both so a reentrant root cannot be followed by resolver publication. + EnsureRegistrationOpen(); + return typeInfo; + } + + private TypeInfo ResolveRegistrationTypeInfo() + { + TypeInfo typeInfo = _typeResolver.ResolveRegistrationTypeInfo(typeof(T)); + // Construction may start a root. Keep the candidate unpublished until the Fory-owned + // check succeeds; Register then publishes the binding and explicit ID or name together. + EnsureRegistrationOpen(); + return typeInfo; + } + + [MethodImpl(MethodImplOptions.NoInlining)] + private static void ThrowRegistryFrozen() => + throw new InvalidOperationException( + "types and serializers must be registered before the first serialization or deserialization operation"); + } diff --git a/csharp/src/Fory/ThreadSafeFory.cs b/csharp/src/Fory/ThreadSafeFory.cs index 9639f9161d..eb8d673aae 100644 --- a/csharp/src/Fory/ThreadSafeFory.cs +++ b/csharp/src/Fory/ThreadSafeFory.cs @@ -16,6 +16,7 @@ // under the License. using System.Buffers; +using System.Runtime.CompilerServices; namespace Apache.Fory; @@ -28,12 +29,13 @@ public sealed class ThreadSafeFory : IDisposable private readonly object _registrationLock = new(); private readonly List> _registrations = []; private readonly ThreadLocal _threadLocalFory; + private int _registryFrozen; private bool _disposed; internal ThreadSafeFory(Config config) { _config = config; - _threadLocalFory = new ThreadLocal(CreatePerThreadFory, trackAllValues: true); + _threadLocalFory = new ThreadLocal(CreatePerThreadFory); } /// @@ -42,11 +44,13 @@ internal ThreadSafeFory(Config config) public Config Config => _config; /// - /// Registers a user type by numeric type identifier for all current and future thread-local runtimes. + /// Registers a user type by numeric type identifier on this thread-safe runtime. /// /// Type to register. /// Numeric type identifier used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public ThreadSafeFory Register(uint typeId) { ApplyRegistration(fory => fory.Register(typeId)); @@ -54,39 +58,47 @@ public ThreadSafeFory Register(uint typeId) } /// - /// Registers a user type by name for all current and future thread-local runtimes. + /// Registers a user type by name on this thread-safe runtime. /// /// Type to register. /// Name used on the wire. A dotted name is split at the last dot. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public ThreadSafeFory Register(string name) { + EnsureRegistrationOpen(); _ = TypeResolver.SplitTypeName(name); ApplyRegistration(fory => fory.Register(name)); return this; } /// - /// Registers a user type by namespace and name for all current and future thread-local runtimes. + /// Registers a user type by namespace and name on this thread-safe runtime. /// /// Type to register. /// Namespace used on the wire. /// Type name used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public ThreadSafeFory Register(string typeNamespace, string typeName) { + EnsureRegistrationOpen(); TypeResolver.ValidateSplitTypeName(typeNamespace, typeName); ApplyRegistration(fory => fory.Register(typeNamespace, typeName)); return this; } /// - /// Registers a user type by numeric type identifier with a custom serializer for all thread-local runtimes. + /// Registers a user type by numeric type identifier with a custom serializer on this thread-safe runtime. /// /// Type to register. /// Serializer implementation used for . /// Numeric type identifier used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public ThreadSafeFory Register(uint typeId) where TSerializer : Serializer, new() { @@ -95,31 +107,37 @@ public ThreadSafeFory Register(uint typeId) } /// - /// Registers a user type by name with a custom serializer for all thread-local runtimes. + /// Registers a user type by name with a custom serializer on this thread-safe runtime. /// /// Type to register. /// Serializer implementation used for . /// Name used on the wire. A dotted name is split at the last dot. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public ThreadSafeFory Register(string name) where TSerializer : Serializer, new() { + EnsureRegistrationOpen(); _ = TypeResolver.SplitTypeName(name); ApplyRegistration(fory => fory.Register(name)); return this; } /// - /// Registers a user type by namespace and name with a custom serializer for all thread-local runtimes. + /// Registers a user type by namespace and name with a custom serializer on this thread-safe runtime. /// /// Type to register. /// Serializer implementation used for . /// Namespace used on the wire. /// Type name used on the wire. /// The same runtime instance. + /// Registration closes permanently when the first serialization or deserialization attempt begins, including an attempt that fails. + /// Registration has closed because a root operation was attempted. public ThreadSafeFory Register(string typeNamespace, string typeName) where TSerializer : Serializer, new() { + EnsureRegistrationOpen(); TypeResolver.ValidateSplitTypeName(typeNamespace, typeName); ApplyRegistration(fory => fory.Register(typeNamespace, typeName)); return this; @@ -133,6 +151,7 @@ public ThreadSafeFory Register(string typeNamespace, string type /// Serialized bytes. public byte[] Serialize(in T value) { + BeginRoot(); return Current.Serialize(in value); } @@ -144,6 +163,7 @@ public byte[] Serialize(in T value) /// Value to serialize. public void Serialize(IBufferWriter output, in T value) { + BeginRoot(); Current.Serialize(output, in value); } @@ -155,6 +175,7 @@ public void Serialize(IBufferWriter output, in T value) /// Deserialized value. public T Deserialize(ReadOnlySpan payload) { + BeginRoot(); return Current.Deserialize(payload); } @@ -209,14 +230,56 @@ private void ApplyRegistration(Action registration) lock (_registrationLock) { ThrowIfDisposed(); + if (_registryFrozen != 0) + { + ThrowRegistryFrozen(); + } + _registrations.Add(registration); - foreach (Fory fory in _threadLocalFory.Values) + } + } + + private void EnsureRegistrationOpen() + { + lock (_registrationLock) + { + ThrowIfDisposed(); + if (_registryFrozen != 0) { - registration(fory); + ThrowRegistryFrozen(); + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private void BeginRoot() + { + // Freeze before Current can create a per-thread runtime so roots and registrations + // linearize against one boundary and every runtime gets the same configuration. + if (Volatile.Read(ref _registryFrozen) == 0) + { + FreezeRegistry(); + } + } + + [MethodImpl(MethodImplOptions.NoInlining)] + private void FreezeRegistry() + { + lock (_registrationLock) + { + ThrowIfDisposed(); + if (_registryFrozen == 0) + { + Volatile.Write(ref _registryFrozen, 1); } } } + [MethodImpl(MethodImplOptions.NoInlining)] + private static void ThrowRegistryFrozen() => + throw new InvalidOperationException( + "types and serializers must be registered before the first serialization or deserialization operation"); + private void ThrowIfDisposed() { if (_disposed) diff --git a/csharp/src/Fory/TypeResolver.cs b/csharp/src/Fory/TypeResolver.cs index 9eb0cc9da1..dc0fe861af 100644 --- a/csharp/src/Fory/TypeResolver.cs +++ b/csharp/src/Fory/TypeResolver.cs @@ -109,7 +109,7 @@ private static class GenericTypeCache private readonly Dictionary<(string NamespaceName, string TypeName), TypeInfo> _byTypeName = []; private readonly UInt64Map _typeInfos = new(); private ulong _versionHash; - private bool _finalized; + private bool _versionHashReady; /// /// Registers a generated enum or union serializer factory for a runtime target type. @@ -188,7 +188,7 @@ private static UInt64Map CreateTypeMap(params (Type Key, Type Value)[] ent public Serializer GetSerializer() { - if (_finalized) + if (_versionHashReady) { ulong version = _versionHash; GenericTypeCacheEntry? cacheEntry = Volatile.Read(ref GenericTypeCache.Entry); @@ -209,7 +209,7 @@ public TypeInfo GetTypeInfo(Type type) public TypeInfo GetTypeInfo() { - if (_finalized) + if (_versionHashReady) { ulong version = _versionHash; GenericTypeCacheEntry? cacheEntry = Volatile.Read(ref GenericTypeCache.Entry); @@ -220,7 +220,7 @@ public TypeInfo GetTypeInfo() } TypeInfo typeInfo = GetTypeInfo(typeof(T)); - EnsureFinalizedVersion(); + EnsureVersionHash(); Volatile.Write( ref GenericTypeCache.Entry, new GenericTypeCacheEntry(_versionHash, typeInfo)); @@ -453,6 +453,25 @@ internal IReadOnlyList TypeMetaFields(TypeInfo typeInfo, bool return typeInfo.TypeMetaFields(trackRef); } + private TypeInfo ResolveTypeInfo( + Type type, + TypeInfo? explicitTypeInfo, + ulong typeKey) + { + TypeInfo typeInfo = explicitTypeInfo ?? CreateBindingCore(type); + if (typeInfo.Type != type) + { + throw new InvalidDataException($"serializer type mismatch for {type}, got {typeInfo.Type}"); + } + + if (_typeInfos.TryGetValue(typeKey, out TypeInfo? previous)) + { + typeInfo = typeInfo.WithRegistrationFrom(previous); + } + + return typeInfo; + } + private TypeInfo GetOrCreateTypeInfo(Type type, TypeInfo? explicitTypeInfo) { ulong typeKey = TypeMapKey.Get(type); @@ -469,41 +488,42 @@ private TypeInfo GetOrCreateTypeInfo(Type type, TypeInfo? explicitTypeInfo) } } - TypeInfo typeInfo = explicitTypeInfo ?? CreateBindingCore(type); - if (typeInfo.Type != type) - { - throw new InvalidDataException($"serializer type mismatch for {type}, got {typeInfo.Type}"); - } - - if (_typeInfos.TryGetValue(typeKey, out TypeInfo? previous)) - { - typeInfo = typeInfo.WithRegistrationFrom(previous); - } - + TypeInfo typeInfo = ResolveTypeInfo(type, explicitTypeInfo, typeKey); _typeInfos.Set(typeKey, typeInfo); - InvalidateFinalizedVersion(); + InvalidateVersionHash(); return typeInfo; } - internal TypeInfo RegisterSerializer() - where TSerializer : Serializer, new() + internal TypeInfo ResolveRegistrationTypeInfo(Type type) { - TypeInfo typeInfo = TypeInfo.Create(typeof(T), new TSerializer()); - RegisterSerializer(typeof(T), typeInfo); - return typeInfo; + return ResolveRegistrationTypeInfo(type, null); } - internal void RegisterSerializer(Type type, TypeInfo typeInfo) + private TypeInfo ResolveRegistrationTypeInfo(Type type, TypeInfo? explicitTypeInfo) { - GetOrCreateTypeInfo(type, typeInfo); + ulong typeKey = TypeMapKey.Get(type); + if (_typeInfos.TryGetValue(typeKey, out TypeInfo? existing)) + { + if (explicitTypeInfo is null || ReferenceEquals(existing, explicitTypeInfo)) + { + return existing; + } + + if (existing.IsRegistered) + { + throw new InvalidDataException($"cannot override serializer for registered type {type}"); + } + } + + return ResolveTypeInfo(type, explicitTypeInfo, typeKey); } internal void Register(Type type, uint id, TypeInfo? explicitTypeInfo = null) { - TypeInfo typeInfo = GetOrCreateTypeInfo(type, explicitTypeInfo).WithTypeIdRegistration(id); + TypeInfo typeInfo = ResolveRegistrationTypeInfo(type, explicitTypeInfo).WithTypeIdRegistration(id); _typeInfos.Set(TypeMapKey.Get(type), typeInfo); _byUserTypeId[id] = typeInfo; - InvalidateFinalizedVersion(); + InvalidateVersionHash(); } internal static (string NamespaceName, string TypeName) SplitTypeName(string name) @@ -543,41 +563,41 @@ internal static void ValidateSplitTypeName(string namespaceName, string typeName internal void Register(Type type, string namespaceName, string typeName, TypeInfo? explicitTypeInfo = null) { ValidateSplitTypeName(namespaceName, typeName); - TypeInfo typeInfo = GetOrCreateTypeInfo(type, explicitTypeInfo); + TypeInfo typeInfo = ResolveRegistrationTypeInfo(type, explicitTypeInfo); MetaString namespaceMeta = MetaStringEncoder.Namespace.Encode(namespaceName, TypeMetaEncodings.NamespaceMetaStringEncodings); MetaString typeNameMeta = MetaStringEncoder.TypeName.Encode(typeName, TypeMetaEncodings.TypeNameMetaStringEncodings); typeInfo = typeInfo.WithTypeNameRegistration(namespaceMeta, typeNameMeta); _typeInfos.Set(TypeMapKey.Get(type), typeInfo); _byTypeName[(namespaceName, typeName)] = typeInfo; - InvalidateFinalizedVersion(); + InvalidateVersionHash(); } /// - /// Returns a finalized semantic resolver version used by generated/static caches. + /// Returns the semantic resolver version used by generated/static caches. /// The version is computed lazily and changes whenever bindings/registrations change. /// /// Resolver version token. public ulong VersionHash() { - EnsureFinalizedVersion(); + EnsureVersionHash(); return _versionHash; } - private void InvalidateFinalizedVersion() + private void InvalidateVersionHash() { - _finalized = false; + _versionHashReady = false; _versionHash = 0; } - private void EnsureFinalizedVersion() + private void EnsureVersionHash() { - if (_finalized) + if (_versionHashReady) { return; } _versionHash = ComputeVersionHash(); - _finalized = true; + _versionHashReady = true; } private ulong ComputeVersionHash() diff --git a/csharp/tests/Fory.Tests/ClassInheritanceTests.cs b/csharp/tests/Fory.Tests/ClassInheritanceTests.cs index 32153a21db..47bd44ff51 100644 --- a/csharp/tests/Fory.Tests/ClassInheritanceTests.cs +++ b/csharp/tests/Fory.Tests/ClassInheritanceTests.cs @@ -180,7 +180,8 @@ public void FlattenedHierarchyRoundTrips(bool compatible, bool trackRef) .Compatible(compatible) .TrackRef(trackRef) .Build() - .Register(6401); + .Register(6401) + .Register(6408); InheritedLeaf value = new() { PublicValue = 13, @@ -205,7 +206,6 @@ public void FlattenedHierarchyRoundTrips(bool compatible, bool trackRef) Assert.Equal(23, decoded.HiddenValue); Assert.Equal(trackRef, ReferenceEquals(decoded, decoded.Self)); - fory.Register(6408); InheritedMiddle middle = new() { PublicValue = 29, diff --git a/csharp/tests/Fory.Tests/ExternalTypeSerializationTests.cs b/csharp/tests/Fory.Tests/ExternalTypeSerializationTests.cs index 3df1b9cb4b..66d8b23cc2 100644 --- a/csharp/tests/Fory.Tests/ExternalTypeSerializationTests.cs +++ b/csharp/tests/Fory.Tests/ExternalTypeSerializationTests.cs @@ -401,6 +401,8 @@ public void CustomSerializerReplacesGenerated() ExternalFields value = new() { Count = 19, Name = "custom" }; ForyRuntime generated = ForyRuntime.Builder().Build(); generated.Register(6106); + Assert.Throws( + () => generated.Register(6107)); byte[] generatedBytes = generated.Serialize(value); ForyRuntime custom = ForyRuntime.Builder().Build(); @@ -411,8 +413,6 @@ public void CustomSerializerReplacesGenerated() Assert.NotEqual(generatedBytes, customBytes); Assert.Equal(value.Count, decoded.Count); Assert.Equal(value.Name, decoded.Name); - Assert.Throws( - () => generated.Register(6107)); } [Fact] diff --git a/csharp/tests/Fory.Tests/ForyRuntimeTests.cs b/csharp/tests/Fory.Tests/ForyRuntimeTests.cs index 36e6ccfbce..d093e061a3 100644 --- a/csharp/tests/Fory.Tests/ForyRuntimeTests.cs +++ b/csharp/tests/Fory.Tests/ForyRuntimeTests.cs @@ -834,18 +834,12 @@ public void ThreadSafeForyPropagatesRegistrationsToThreads() } [Fact] - public void ThreadSafeForyRegistrationAppliesToInitializedThreadLocalInstance() + public void ThreadSafeForyRejectsLateRegister() { using ThreadSafeFory fory = ForyRuntime.Builder().TrackRef(true).BuildThreadSafe(); _ = fory.Serialize(1); - fory.Register(952); - Node source = new() { Value = 7 }; - source.Next = source; - Node decoded = fory.Deserialize(fory.Serialize(source)); - Assert.Equal(7, decoded.Value); - Assert.NotNull(decoded.Next); - Assert.Same(decoded, decoded.Next); + Assert.Throws(() => fory.Register(952)); } [Fact] @@ -3099,7 +3093,7 @@ public void TypeResolverVersionHashIncludesUnregisteredTypeBindings() } [Fact] - public void TypeResolverVersionHashIsStableWithinSameFinalizedResolver() + public void VersionHashIsStable() { TypeResolver resolver = new(); _ = resolver.GetTypeInfo>(); diff --git a/csharp/tests/Fory.Tests/RuntimeEdgeCaseTests.cs b/csharp/tests/Fory.Tests/RuntimeEdgeCaseTests.cs index b817b99036..26f50d0c60 100644 --- a/csharp/tests/Fory.Tests/RuntimeEdgeCaseTests.cs +++ b/csharp/tests/Fory.Tests/RuntimeEdgeCaseTests.cs @@ -15,6 +15,7 @@ // specific language governing permissions and limitations // under the License. +using System.Buffers; using System.Numerics; using Apache.Fory; using ForyRuntime = Apache.Fory.Fory; @@ -59,6 +60,13 @@ public sealed class DecimalEnvelope public sealed class CustomPayloadSerializer : Serializer { + public static Action? ConstructionAction; + + public CustomPayloadSerializer() + { + ConstructionAction?.Invoke(); + } + public override CustomPayload DefaultValue => null!; public override void WriteData(WriteContext context, in CustomPayload value, bool hasGenerics) @@ -77,6 +85,119 @@ public override CustomPayload ReadData(ReadContext context) } } +[ForyStruct] +public sealed class FrozenPayload +{ + public int Value { get; set; } +} + +public sealed class FrozenPayloadSerializer : Serializer +{ + public static int Constructions; + + public FrozenPayloadSerializer() + { + Interlocked.Increment(ref Constructions); + } + + public override FrozenPayload DefaultValue => null!; + + public override void WriteData(WriteContext context, in FrozenPayload value, bool hasGenerics) + { + _ = hasGenerics; + context.Writer.WriteVarInt32(value.Value); + } + + public override FrozenPayload ReadData(ReadContext context) + { + return new FrozenPayload { Value = context.Reader.ReadVarInt32() }; + } +} + +public enum LookupFailureValue +{ + Zero, +} + +public sealed class LookupFailureSerializer : Serializer +{ + public static Action? ConstructionAction; + + public LookupFailureSerializer() + { + ConstructionAction?.Invoke(); + } + + public override LookupFailureValue DefaultValue => LookupFailureValue.Zero; + + public override void WriteData(WriteContext context, in LookupFailureValue value, bool hasGenerics) + { + _ = context; + _ = value; + _ = hasGenerics; + } + + public override LookupFailureValue ReadData(ReadContext context) + { + _ = context; + return LookupFailureValue.Zero; + } +} + +public enum RegistrationValue +{ + Zero, +} + +public sealed class RegistrationValueSerializer : Serializer +{ + public static Action? ConstructionAction; + public static int ConstructionCount; + + public RegistrationValueSerializer() + { + ConstructionCount++; + ConstructionAction?.Invoke(); + } + + public override RegistrationValue DefaultValue => RegistrationValue.Zero; + + public override void WriteData(WriteContext context, in RegistrationValue value, bool hasGenerics) + { + _ = context; + _ = value; + _ = hasGenerics; + } + + public override RegistrationValue ReadData(ReadContext context) + { + _ = context; + return RegistrationValue.Zero; + } +} + +public sealed class FailingWritePayload +{ + public int Value { get; set; } +} + +public sealed class FailingWriteSerializer : Serializer +{ + public override void WriteData(WriteContext context, in FailingWritePayload value, bool hasGenerics) + { + _ = context; + _ = value; + _ = hasGenerics; + throw new InvalidOperationException("write failure"); + } + + public override FailingWritePayload ReadData(ReadContext context) + { + _ = context; + return new(); + } +} + public sealed class RuntimeEdgeCaseTests { [Fact] @@ -739,10 +860,9 @@ public void SplitTypeNameRejectsDots() } [Fact] - public void ThreadSafeDottedSerializerNameRoundTrip() + public void ThreadSafeDottedNameRoundTrip() { using ThreadSafeFory fory = ForyRuntime.Builder().BuildThreadSafe(); - _ = fory.Serialize(1); fory.Register("test.custom_payload"); CustomPayload decoded = fory.Deserialize( @@ -753,14 +873,302 @@ public void ThreadSafeDottedSerializerNameRoundTrip() } [Fact] - public void DeserializeRejectsTrailingBytes() + public void FrozenRegistryRejectsBeforeMutation() { ForyRuntime fory = ForyRuntime.Builder().Build(); - byte[] payload = fory.Serialize(123); + _ = fory.Serialize(1); + FrozenPayloadSerializer.Constructions = 0; + + Action[] registrations = + [ + () => fory.Register(711), + () => fory.Register(string.Empty), + () => fory.Register("test", "bad.name"), + () => fory.Register(712), + () => fory.Register(string.Empty), + () => fory.Register("test", "bad.name"), + ]; + + foreach (Action registration in registrations) + { + Assert.Throws(registration); + } + + Assert.Equal(0, FrozenPayloadSerializer.Constructions); + } + + [Fact] + public void CustomSerializerRechecksFreeze() + { + ForyRuntime fory = ForyRuntime.Builder().Build(); + CustomPayloadSerializer.ConstructionAction = () => _ = fory.Serialize(1); + try + { + Assert.Throws( + () => fory.Register(721)); + } + finally + { + CustomPayloadSerializer.ConstructionAction = null; + } + + TypeInfo typeInfo = ReadContextFor(fory).TypeResolver.GetTypeInfo(typeof(CustomPayload)); + Assert.False(typeInfo.IsRegistered); + Assert.NotEqual(typeof(CustomPayloadSerializer), typeInfo.SerializerType); + } + + [Fact] + public void GeneratedRegistrationRechecksFreeze() + { + TypeResolver.RegisterGenerated(); + foreach (bool registerByName in new[] { false, true }) + { + ForyRuntime fory = ForyRuntime.Builder().Build(); + RegistrationValueSerializer.ConstructionCount = 0; + RegistrationValueSerializer.ConstructionAction = () => _ = fory.Serialize(1); + try + { + Assert.Throws(() => + { + if (registerByName) + { + fory.Register("test.registration_value"); + } + else + { + fory.Register(722); + } + }); + } + finally + { + RegistrationValueSerializer.ConstructionAction = null; + } + + TypeInfo typeInfo = ReadContextFor(fory).TypeResolver.GetTypeInfo(typeof(RegistrationValue)); + Assert.False(typeInfo.IsRegistered); + Assert.Equal(2, RegistrationValueSerializer.ConstructionCount); + } + } + + [Fact] + public void FailedRootFreezesRegistry() + { + ForyRuntime fory = ForyRuntime.Builder().Build(); + ReadContext context = ReadContextFor(fory); + context.AppendReadMetaString(MetaString.Empty('_', '_')); + + Assert.ThrowsAny(() => fory.Deserialize((byte[])null!)); + Assert.Null(context.GetReadMetaString(0)); + Assert.Throws(() => fory.Register(713)); + } + + [Fact] + public void FailedWriteRestoresState() + { + ForyRuntime fory = ForyRuntime.Builder().TrackRef(true).Build(); + fory.Register(718); + FailingWritePayload value = new() { Value = 1 }; + + Assert.Throws(() => fory.Serialize(value)); + Assert.Equal(0u, WriteContextFor(fory).RefWriter.ReserveRefId()); + Assert.Throws(() => fory.Register(719)); + + Assert.Throws(() => fory.Serialize(value)); + Assert.Equal(7, fory.Deserialize(fory.Serialize(7))); + } + + [Fact] + public void FailedLookupRestoresWriteState() + { + TypeResolver.RegisterGenerated(); + ForyRuntime fory = ForyRuntime.Builder().TrackRef(true).Build(); + WriteContext context = WriteContextFor(fory); + _ = context.RefWriter.ReserveRefId(); + Assert.True(context.AssignTypeMetaIndexIfAbsent(typeof(FrozenPayload)).IsNew); + Assert.True(context.AssignMetaStringIndexIfAbsent(MetaString.Empty('_', '_')).IsNew); + bool lookupStarted = false; + LookupFailureSerializer.ConstructionAction = () => + { + lookupStarted = true; + throw new InvalidOperationException("serializer lookup failed"); + }; + try + { + Assert.ThrowsAny(() => fory.Serialize(LookupFailureValue.Zero)); + Assert.True(lookupStarted); + } + finally + { + LookupFailureSerializer.ConstructionAction = null; + } + + Assert.Equal(0u, context.RefWriter.ReserveRefId()); + Assert.True(context.AssignTypeMetaIndexIfAbsent(typeof(FrozenPayload)).IsNew); + Assert.True(context.AssignMetaStringIndexIfAbsent(MetaString.Empty('_', '_')).IsNew); + } + + [Fact] + public void FailedReaderRootFreezesRegistry() + { + ForyRuntime fory = ForyRuntime.Builder().Build(); + + Assert.ThrowsAny( + () => fory.DeserializeFromReader(new ByteReader(Array.Empty()))); + Assert.Throws(() => fory.Register(714)); + } + + [Fact] + public void ThreadSafeFailedRootFreezesRegistry() + { + using ThreadSafeFory fory = ForyRuntime.Builder().BuildThreadSafe(); + + Assert.ThrowsAny(() => fory.Deserialize(Array.Empty())); + Action[] registrations = + [ + () => fory.Register(string.Empty), + () => fory.Register("test", "bad.name"), + () => fory.Register(string.Empty), + () => fory.Register("test", "bad.name"), + ]; + + foreach (Action registration in registrations) + { + Assert.Throws(registration); + } + } + + [Fact] + public void ThreadSafeOutputFreezesRegistry() + { + using ThreadSafeFory fory = ForyRuntime.Builder().BuildThreadSafe(); + ArrayBufferWriter output = new(); + + fory.Serialize(output, 1); + + Assert.Throws(() => fory.Register(720)); + } + + [Fact] + public async Task ThreadSafeRootAndRegistrationRace() + { + using ThreadSafeFory fory = ForyRuntime.Builder().BuildThreadSafe(); + using Barrier start = new(2); + Exception? registrationError = null; + + Task root = Task.Run(() => + { + start.SignalAndWait(); + _ = fory.Serialize(1); + }); + Task registration = Task.Run(() => + { + start.SignalAndWait(); + try + { + fory.Register(716); + } + catch (Exception error) + { + registrationError = error; + } + }); + + await Task.WhenAll(root, registration); + Assert.True(registrationError is null or InvalidOperationException); + Assert.Throws(() => fory.Register(717)); + + if (registrationError is null) + { + FrozenPayload value = new() { Value = 42 }; + Assert.Equal(value.Value, fory.Deserialize(fory.Serialize(value)).Value); + } + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void TrailingBytesResetReadState(bool useSpan) + { + ForyRuntime writer = NewCompatibleTimeFory(); + byte[] payload = writer.Serialize(new TimeEnvelope { Dates = [new DateOnly(2024, 1, 2)] }); + ForyRuntime probe = NewCompatibleTimeFory(); + _ = probe.DeserializeFromReader(new ByteReader(payload)); + Assert.NotNull(ReadContextFor(probe).GetTypeMetaRef(0)); + + ForyRuntime reader = NewCompatibleTimeFory(); byte[] invalidPayload = [.. payload, 0x7F]; - InvalidDataException exception = Assert.Throws(() => fory.Deserialize(invalidPayload)); - Assert.Contains("unexpected trailing bytes", exception.Message, StringComparison.Ordinal); + _ = useSpan + ? Assert.Throws(() => DeserializeSpan(reader, invalidPayload)) + : Assert.Throws( + () => reader.Deserialize(invalidPayload)); + ReadContext context = ReadContextFor(reader); + Assert.Null(context.GetTypeMetaRef(0)); + Assert.Null(context.GetReadMetaString(0)); + } + + [Theory] + [InlineData(false)] + [InlineData(true)] + public void TrailingFailureKeepsTypeMetaCache(bool useSpan) + { + ForyRuntime fory = ForyRuntime.Builder() + .Compatible(false) + .MaxSchemaVersionsPerType(1) + .Build(); + ReadContext context = ReadContextFor(fory); + TypeMeta first = ReadAndStoreTypeMeta(context, RemoteStructTypeMeta(901, "first")); + ulong firstHash = EncodedTypeMetaHash(first); + byte[] invalidPayload = [.. fory.Serialize(123), 0x7F]; + + if (useSpan) + { + Assert.Throws(() => DeserializeIntSpan(fory, invalidPayload)); + } + else + { + Assert.Throws(() => fory.Deserialize(invalidPayload)); + } + + Assert.True(context.TryGetTypeMetaByHash(firstHash, out _)); + Assert.Throws( + () => ReadAndStoreTypeMeta(context, RemoteStructTypeMeta(901, "second"))); + } + + private static ForyRuntime NewCompatibleTimeFory() + { + ForyRuntime fory = ForyRuntime.Builder().Compatible(true).Build(); + fory.Register(701); + return fory; + } + + private static void DeserializeSpan(ForyRuntime fory, byte[] payload) + { + _ = fory.Deserialize(payload.AsSpan()); + } + + private static void DeserializeIntSpan(ForyRuntime fory, byte[] payload) + { + _ = fory.Deserialize(payload.AsSpan()); + } + + private static ReadContext ReadContextFor(ForyRuntime fory) + { + System.Reflection.FieldInfo? field = typeof(ForyRuntime).GetField( + "_readContext", + System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic); + Assert.NotNull(field); + return Assert.IsType(field.GetValue(fory)); + } + + private static WriteContext WriteContextFor(ForyRuntime fory) + { + System.Reflection.FieldInfo? field = typeof(ForyRuntime).GetField( + "_writeContext", + System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic); + Assert.NotNull(field); + return Assert.IsType(field.GetValue(fory)); } [Fact] diff --git a/docs/compiler/generated-code/java.md b/docs/compiler/generated-code/java.md index 43f5326927..25dd187f7b 100644 --- a/docs/compiler/generated-code/java.md +++ b/docs/compiler/generated-code/java.md @@ -92,6 +92,8 @@ public final class Animal extends Union { Each JVM schema generates a `ForyModule`. Imported schema modules are installed through `fory.register(...)`, so shared imports are deduplicated by the Fory instance. +Module installation is registration-only setup and must not start a root serialization or +deserialization operation on the runtime being configured. ```java public final class AddressbookForyModule implements org.apache.fory.ForyModule { diff --git a/docs/object-serialization/core-concepts.md b/docs/object-serialization/core-concepts.md index bdb8e9b877..328bc859af 100644 --- a/docs/object-serialization/core-concepts.md +++ b/docs/object-serialization/core-concepts.md @@ -39,7 +39,9 @@ only values. Use [Row Format](../row-format/index.md) for trusted analytical row A Fory instance owns its mode, schema behavior, reference settings, registered types, custom serializers, and read limits. Configure and register the instance before its first root serialization or deserialization operation, then reuse it. Registration is frozen after the first -root operation so the same instance always resolves a type in the same way. +root attempt, even when that operation fails, so its explicit type and serializer mappings stay +fixed. Native modes may still perform documented lazy runtime-type discovery when registration is +not required. Thread-safety differs by Fory implementation. Some implementations provide a thread-safe wrapper or pool; others use one instance per thread or task. Follow the selected language guide instead of sharing an ordinary @@ -53,8 +55,9 @@ model should read this value_; a field schema describes _what data that model co A statically known field can use its declared type directly. A dynamic field also carries the concrete type needed for interfaces, abstract classes, trait objects, broad object types, -or heterogeneous values. Dynamic typing is more flexible but requires every possible concrete type -to be registered and supported by the selected mode. +or heterogeneous values. Every concrete type must be supported by the selected mode and explicitly +registered unless that language's native-mode guide documents lazy discovery. Xlang mode and +registration-required configurations require explicit registration. In xlang mode, peers must coordinate the same portable type identity and mapping. Native mode may use implementation-specific identities and language-specific types. See diff --git a/docs/object-serialization/cpp/type-registration.md b/docs/object-serialization/cpp/type-registration.md index 62f525ef94..13169fc4bf 100644 --- a/docs/object-serialization/cpp/type-registration.md +++ b/docs/object-serialization/cpp/type-registration.md @@ -29,6 +29,12 @@ Apache Fory™ requires explicit type registration for struct types. This design - **Type Safety**: Detects type mismatches at deserialization time - **Polymorphic Serialization**: Enables serialization of polymorphic objects via smart pointers +## Registration Lifecycle + +Complete all registrations before the first root serialization or deserialization call. The first +root attempt permanently freezes that Fory instance's registry, even when the operation fails. +Create a new Fory instance when a different registry is required. + ## Registering Structs Use `register_struct(type_id)` to register a struct type: diff --git a/docs/object-serialization/csharp/thread-safety.md b/docs/object-serialization/csharp/thread-safety.md index 0681c04219..fa83b6b6c3 100644 --- a/docs/object-serialization/csharp/thread-safety.md +++ b/docs/object-serialization/csharp/thread-safety.md @@ -53,9 +53,9 @@ Parallel.For(0, 64, i => ## Registration Behavior -- `ThreadSafeFory.Register(...)` stores registrations centrally. -- Existing per-thread Fory instances are updated. -- New threads receive all previous registrations automatically. +- Register every type before the first serialization or deserialization attempt. +- Starting the first root permanently freezes registration, including when that root fails. +- Later registration throws `InvalidOperationException` before changing the registry. ## Disposal diff --git a/docs/object-serialization/csharp/type-registration.md b/docs/object-serialization/csharp/type-registration.md index 0c6b629310..7f275c6e47 100644 --- a/docs/object-serialization/csharp/type-registration.md +++ b/docs/object-serialization/csharp/type-registration.md @@ -67,9 +67,20 @@ Name-based custom serializer registration is also supported: fory.Register("com.example.MyType"); ``` +## Registration Lifecycle + +A `Fory` instance accepts registration only before its first root serialization or deserialization +attempt. Starting that operation permanently closes registration, even when the operation fails. +Every later registration throws `InvalidOperationException`. + +To configure additional types, build a new `Fory` instance, complete its registrations, and then +use that new instance for serialization or deserialization. + ## Thread-Safe Registration -`ThreadSafeFory` exposes the same registration APIs. Registrations are propagated to all per-thread Fory instances. +`ThreadSafeFory` exposes the same registration APIs. Register every type before the first +serialization or deserialization attempt. Starting the first root permanently freezes +registration, even when the root fails. A later registration throws `InvalidOperationException`. ```csharp using ThreadSafeFory fory = Fory.Builder().BuildThreadSafe(); @@ -90,7 +101,7 @@ fory.Register(101); Registering a derived class does not make an unannotated base class serializable. - For the split overloads, `typeName` must be non-empty and must not contain dots. -- Register before high-volume serialization workloads to avoid missing type metadata. +- Complete registration before the first root serialization or deserialization attempt. ## Related Topics diff --git a/docs/object-serialization/go/configuration.md b/docs/object-serialization/go/configuration.md index 8f59db06e2..ff1603548d 100644 --- a/docs/object-serialization/go/configuration.md +++ b/docs/object-serialization/go/configuration.md @@ -386,11 +386,16 @@ type Request struct { Payload string } -f := threadsafe.New( - fory.WithXlang(true), - fory.WithMaxDepth(30), -) -f.RegisterStruct(Request{}, 1) +f := threadsafe.NewWithFactory(func() *fory.Fory { + inner := fory.New( + fory.WithXlang(true), + fory.WithMaxDepth(30), + ) + if err := inner.RegisterStruct(Request{}, 1); err != nil { + panic(err) + } + return inner +}) // Process requests concurrently for req := range requests { diff --git a/docs/object-serialization/go/native.md b/docs/object-serialization/go/native.md index cf72cb866c..445f75e10c 100644 --- a/docs/object-serialization/go/native.md +++ b/docs/object-serialization/go/native.md @@ -79,8 +79,13 @@ import ( "github.com/apache/fory/go/fory/threadsafe" ) -f := threadsafe.New(fory.WithXlang(false), fory.WithTrackRef(true)) -_ = f.RegisterStruct(Order{}, 100) +f := threadsafe.NewWithFactory(func() *fory.Fory { + inner := fory.New(fory.WithXlang(false), fory.WithTrackRef(true)) + if err := inner.RegisterStruct(Order{}, 100); err != nil { + panic(err) + } + return inner +}) ``` ## Schema Evolution diff --git a/docs/object-serialization/go/security.md b/docs/object-serialization/go/security.md index 0ad71c5513..3a6af4586e 100644 --- a/docs/object-serialization/go/security.md +++ b/docs/object-serialization/go/security.md @@ -30,7 +30,7 @@ Before deserialization: - Authenticate the sender and protect message integrity at the transport or storage layer. - Enforce request or file size, timeout, and concurrency limits outside Fory. - Register only the application types the endpoint accepts and configure the reader before its - first root operation. + first root operation. The first root attempt freezes registration even if it fails. - Validate the deserialized value against application authorization and domain rules before use. ## Built-in safeguards diff --git a/docs/object-serialization/go/thread-safety.md b/docs/object-serialization/go/thread-safety.md index 89b5367667..8a2bcca3c3 100644 --- a/docs/object-serialization/go/thread-safety.md +++ b/docs/object-serialization/go/thread-safety.md @@ -114,32 +114,44 @@ err = threadsafe.Unmarshal(data, &target) ## Type Registration -Type registration should be done before concurrent use: +Each pooled `Fory` instance owns its registry. Configure every instance in `NewWithFactory` before +returning it to the pool: ```go -f := threadsafe.New() - -// Register types BEFORE concurrent access -f.RegisterStruct(User{}, 1) -f.RegisterStruct(Order{}, 2) +f := threadsafe.NewWithFactory(func() *fory.Fory { + inner := fory.New(fory.WithXlang(true)) + if err := inner.RegisterStruct(User{}, 1); err != nil { + panic(err) + } + if err := inner.RegisterStruct(Order{}, 2); err != nil { + panic(err) + } + return inner +}) -// Now safe to use concurrently go func() { - f.Serialize(&User{ID: 1}) + _, _ = f.Serialize(&User{ID: 1}) }() ``` -### Thread-Safe Registration +The factory is the sole registration path for the thread-safe wrapper. The wrapper has no registry +of its own, and a registration applied to one pooled instance would not configure future instances. +Every factory invocation must return an instance with the same configuration and registrations. -The thread-safe wrapper handles registration safely: +For a directly owned `Fory`, register types on that instance before its first root operation. +Starting serialization or deserialization permanently freezes that instance's registry, including +when the root fails: ```go -// Safe: Registration is synchronized -f := threadsafe.New() -f.RegisterStruct(User{}, 1) // Thread-safe +inner := fory.New(fory.WithXlang(true)) +if err := inner.RegisterStruct(User{}, 1); err != nil { + panic(err) +} + +_, _ = inner.Serialize(&User{ID: 1}) ``` -However, for best performance, register all types at startup before concurrent use. +Later registration on `inner` returns `fory.ErrRegistryFrozen` without changing its registry. ## Zero-Copy Considerations @@ -195,8 +207,13 @@ func BenchmarkNonThreadSafe(b *testing.B) { } func BenchmarkThreadSafe(b *testing.B) { - f := threadsafe.New() - f.RegisterStruct(User{}, 1) + f := threadsafe.NewWithFactory(func() *fory.Fory { + inner := fory.New(fory.WithXlang(true)) + if err := inner.RegisterStruct(User{}, 1); err != nil { + panic(err) + } + return inner + }) user := &User{ID: 1, Name: "Alice"} for i := 0; i < b.N; i++ { @@ -235,12 +252,13 @@ for i := 0; i < numWorkers; i++ { For dynamic goroutine count or simplicity: ```go -// Single shared instance -var f = threadsafe.New() - -func init() { - f.RegisterStruct(User{}, 1) -} +var f = threadsafe.NewWithFactory(func() *fory.Fory { + inner := fory.New(fory.WithXlang(true)) + if err := inner.RegisterStruct(User{}, 1); err != nil { + panic(err) + } + return inner +}) func handleRequest(user *User) []byte { // Safe from any goroutine @@ -252,11 +270,13 @@ func handleRequest(user *User) []byte { ### HTTP Handler Example ```go -var fory = threadsafe.New() - -func init() { - fory.RegisterStruct(Response{}, 1) -} +var serializer = threadsafe.NewWithFactory(func() *fory.Fory { + inner := fory.New(fory.WithXlang(true)) + if err := inner.RegisterStruct(Response{}, 1); err != nil { + panic(err) + } + return inner +}) func handler(w http.ResponseWriter, r *http.Request) { response := &Response{ @@ -265,7 +285,7 @@ func handler(w http.ResponseWriter, r *http.Request) { } // Safe: threadsafe.Fory handles concurrency - data, err := fory.Serialize(response) + data, err := serializer.Serialize(response) if err != nil { http.Error(w, err.Error(), 500) return @@ -319,23 +339,21 @@ f := threadsafe.New() data, _ := f.Serialize(value1) // Already copied ``` -### Registering Types Concurrently +### Registering Only One Pooled Instance ```go -// RISKY: Concurrent registration -go func() { - f.RegisterStruct(TypeA{}, 1) -}() -go func() { - f.Serialize(value) // May not see TypeA -}() +// WRONG: a configured instance cannot be installed into threadsafe.New. +inner := fory.New(fory.WithXlang(true)) +_ = inner.RegisterStructByName(TypeA{}, "example.TypeA") +f := threadsafe.New(fory.WithXlang(true)) ``` -**Fix**: Register all types before concurrent use. +`f` creates different pooled instances, so the registration on `inner` has no effect. Configure the +registration inside `NewWithFactory` so every pooled instance receives it. ## Best Practices -1. **Register types at startup**: Before any concurrent operations +1. **Configure registrations in the factory**: Every pooled instance must receive the same setup 2. **Clone data if keeping references**: With non-thread-safe instance 3. **Use per-worker instances for hot paths**: Eliminates pool contention 4. **Profile before optimizing**: Thread-safe overhead may be negligible diff --git a/docs/object-serialization/go/type-registration.md b/docs/object-serialization/go/type-registration.md index 9367affdd3..9a6005f9db 100644 --- a/docs/object-serialization/go/type-registration.md +++ b/docs/object-serialization/go/type-registration.md @@ -126,9 +126,15 @@ f1.RegisterStruct(User{}, 1) f2.RegisterStruct(User{}, 1) ``` +The thread-safe wrapper creates multiple `Fory` instances. Configure registrations in +`threadsafe.NewWithFactory` so every pooled instance receives the same registry before use; the +wrapper does not expose registration methods. + ## Registration Timing -Register types after creating a Fory instance and before any serialize/deserialize calls: +Register types after creating a Fory instance and before the first serialization or deserialization +attempt. Starting that first root permanently freezes the instance registry, even when the root +fails. Later registration returns `fory.ErrRegistryFrozen` without changing the registry: ```go f := fory.New(fory.WithXlang(true)) @@ -228,7 +234,8 @@ fory.register_by_name::("example.User")?; ## Best Practices -1. **Register early**: Register all types at application startup before any serialization +1. **Register early**: Register all types at application startup before any serialization or + deserialization 2. **Be consistent**: Use the same ID or name across all languages and all instances 3. **Register all types**: Include nested struct types, not just top-level types 4. **Prefer IDs for performance**: Numeric IDs have lower serialization overhead than names diff --git a/docs/object-serialization/java/type-registration.md b/docs/object-serialization/java/type-registration.md index 774b27ca3f..c17d5aa676 100644 --- a/docs/object-serialization/java/type-registration.md +++ b/docs/object-serialization/java/type-registration.md @@ -44,8 +44,16 @@ Automatically assigned IDs depend on registration order, so readers and writers same classes in the same order. With explicit IDs, the order may differ, but each ID must map to the same class on both sides. -Complete class and serializer registration before the first `serialize`, `deserialize`, or `copy` -call. Later registration attempts are rejected. +Complete explicit class and serializer registration before the first `serialize` or `deserialize` +call. Starting either operation permanently freezes registration even if the operation fails. Copy +operations and `ThreadSafeFory#execute` do not freeze registration unless the supplied callback +starts serialization or deserialization. Later explicit registration attempts are rejected. + +For a thread-safe facade, complete registration before concurrent serialization, deserialization, +copy, or `execute` calls begin. + +In native mode with registration disabled, allowed unregistered runtime classes may still be used +after this boundary. That behavior does not reopen or change explicit registration. `registerSerializer(Foo.class, ...)` is sufficient to use `Foo` when class registration is enabled. Use `registerSerializerAndType(Foo.class, ...)` when you also want Fory to assign a numeric type ID. @@ -97,7 +105,7 @@ Fory fory = Fory.builder().withXlang(false) `STRICT` rejects every class outside the allow list. `WARN` rejects disallowed classes and logs a warning for classes outside the allow list. `DISABLE` skips allow-list checking. -Configure disallow rules before the first `serialize`, `deserialize`, or `copy` call. To use +Configure disallow rules before the first `serialize` or `deserialize` call. To use different disallow rules later, create a new Fory instance. ## Limit Max Deserialization Depth @@ -117,7 +125,8 @@ Fory fory = Fory.builder() 1. Keep class registration enabled for untrusted input. 2. Prefer explicit numeric IDs when readers and writers can share a stable ID mapping. 3. Use the same registration order on both sides when IDs are assigned automatically. -4. Configure all classes, serializers, and disallow rules before the first operation. +4. Configure all classes, serializers, and disallow rules before the first root serialization or + deserialization operation. 5. Configure `AllowListChecker` when class registration is disabled. ## Related Topics diff --git a/docs/object-serialization/javascript/type-registration.md b/docs/object-serialization/javascript/type-registration.md index a0c471c075..e2b4a3a109 100644 --- a/docs/object-serialization/javascript/type-registration.md +++ b/docs/object-serialization/javascript/type-registration.md @@ -117,6 +117,10 @@ fory.register(Type.enum("example.status", Status)); Registration is per `Fory` instance. If you create two instances, you need to register schemas in both. +Register every type and custom serializer before the first root serialization or deserialization +attempt. Starting that first root operation permanently closes registration for the instance, even +if the operation fails. Create a new `Fory` instance when you need a different registration set. + ## What `register` Returns `fory.register(schema)` returns a bound serializer pair: diff --git a/docs/object-serialization/kotlin/configuration.md b/docs/object-serialization/kotlin/configuration.md index f58b2d83a9..4abe134dfc 100644 --- a/docs/object-serialization/kotlin/configuration.md +++ b/docs/object-serialization/kotlin/configuration.md @@ -83,6 +83,12 @@ object ForyHolder { } ``` +`ForyModule` registration and Kotlin reified registration extensions target `BaseFory`, so they are +available on both direct and thread-safe facades. Complete registration before the facade's first +root serialization or deserialization, and before concurrent use of a thread-safe facade begins. +Module installation is registration-only setup and must not start root serialization or +deserialization through the runtime or facade being configured. + ### Using Builder Methods ```kotlin diff --git a/docs/object-serialization/kotlin/static-generated-serializers.md b/docs/object-serialization/kotlin/static-generated-serializers.md index fe42f884b1..1f4a815e14 100644 --- a/docs/object-serialization/kotlin/static-generated-serializers.md +++ b/docs/object-serialization/kotlin/static-generated-serializers.md @@ -273,7 +273,9 @@ fory.register("example.User") `ForyKotlin.builder()` installs the Kotlin serializer bootstrap for the Fory instance. The `fory.register(...)` extension registers your xlang schema type -name and resolves the generated serializer from the target class. +name and resolves the generated serializer from the target class. The extension +targets `BaseFory`, so it works with direct and thread-safe facades before their +first root serialization or deserialization. Do not register or reference generated serializer classes in application code. Fory resolves them from the registered target class. diff --git a/docs/object-serialization/python/configuration.md b/docs/object-serialization/python/configuration.md index 04652322e7..798e1ee9b5 100644 --- a/docs/object-serialization/python/configuration.md +++ b/docs/object-serialization/python/configuration.md @@ -82,6 +82,14 @@ class ThreadSafeFory: ## Key Methods ```python +fory = pyfory.ThreadSafeFory(xlang=True) + +# Complete registration before the first root operation. Choose one form: +fory.register(MyClass, type_id=123) +# fory.register(MyClass, name="my.package.MyClass") +# fory.register(MyClass, type_id=123, serializer=MySerializer) +# fory.register(MyClass, name="my.package.MyClass", serializer=MySerializer) + # Serialization (serialize/deserialize are identical to dumps/loads) data: bytes = fory.serialize(obj) obj = fory.deserialize(data) @@ -89,14 +97,6 @@ obj = fory.deserialize(data) # Alternative API (aliases) data: bytes = fory.dumps(obj) obj = fory.loads(data) - -# Type registration by id -fory.register(MyClass, type_id=123) -fory.register(MyClass, type_id=123, serializer=custom_serializer) - -# Type registration by name -fory.register(MyClass, name="my.package.MyClass") -fory.register(MyClass, name="my.package.MyClass", serializer=custom_serializer) ``` ## Xlang And Native Mode Comparison diff --git a/docs/object-serialization/python/index.md b/docs/object-serialization/python/index.md index cf05c7510a..d066d46ab5 100644 --- a/docs/object-serialization/python/index.md +++ b/docs/object-serialization/python/index.md @@ -111,8 +111,10 @@ for t in threads: t.join() - **Instance Pool**: Maintains a pool of `Fory` instances protected by a lock for thread safety - **Shared Configuration**: All registrations must be done upfront and are applied to all instances -- **Same API**: Drop-in replacement for `Fory` class with identical methods -- **Registration Safety**: Prevents registration after first use to ensure consistency +- **Shared Root API**: Provides the same root serialization and deserialization operations as + `Fory` +- **Registration Safety**: Prevents explicit registration after the first root serialization or + deserialization attempt **When to Use:** diff --git a/docs/object-serialization/python/schema-evolution.md b/docs/object-serialization/python/schema-evolution.md index a8927c92c0..8ebf0ba2f2 100644 --- a/docs/object-serialization/python/schema-evolution.md +++ b/docs/object-serialization/python/schema-evolution.md @@ -60,25 +60,27 @@ class SlotMessage: import pyfory from dataclasses import dataclass -# Version 1: Original class +# Version 1: Writer schema @dataclass -class User: +class UserV1: name: str age: pyfory.Int32 -f = pyfory.Fory(xlang=True) -f.register(User, name="User") -data = f.dumps(User("Alice", 30)) +writer = pyfory.Fory(xlang=True) +writer.register(UserV1, name="User") +data = writer.dumps(UserV1("Alice", 30)) -# Version 2: Add new field (backward compatible) +# Version 2: Reader schema with a new field @dataclass -class User: +class UserV2: name: str age: pyfory.Int32 email: str = "unknown@example.com" # New field with default -# Can still deserialize old data -user = f.loads(data) +# Register the reader schema on a separate instance. +reader = pyfory.Fory(xlang=True) +reader.register(UserV2, name="User") +user = reader.loads(data) print(user.email) # "unknown@example.com" ``` diff --git a/docs/object-serialization/python/security.md b/docs/object-serialization/python/security.md index df3c5cba6b..5ece2a3607 100644 --- a/docs/object-serialization/python/security.md +++ b/docs/object-serialization/python/security.md @@ -29,15 +29,17 @@ Before deserialization: - Authenticate the sender and protect message integrity at the transport or storage layer. - Enforce request or file size, timeout, and concurrency limits outside Fory. -- Register only the application types the endpoint accepts and configure the reader before its - first root operation. +- In strict or xlang mode, register only the application types the endpoint accepts and configure + the reader before its first root operation. +- In non-strict native mode, use a policy to authorize every callable, class, method, state, or + reduction path that an accepted graph may contain. - Validate the deserialized value against application authorization and domain rules before use. ## Built-in safeguards Treat native-mode bytes from untrusted sources the same way you would treat untrusted pickle bytes. -Native mode can reconstruct Python objects, import modules, invoke reduction hooks, and rebuild -dynamic classes or functions when `strict=False`. +Within the configured type surface, native mode can reconstruct Python objects, import modules, +invoke reduction hooks, and rebuild classes or functions when `strict=False`. ### Production Configuration @@ -63,7 +65,7 @@ fory.register(UserModel, name="example.User") fory.register(OrderModel, name="example.Order") ``` -Use dynamic native-mode deserialization (`strict=False`) only for trusted Python-only payloads: +Use native-mode deserialization with `strict=False` only for trusted Python-only payloads: ```python import pyfory @@ -76,6 +78,13 @@ fory = pyfory.Fory( ) ``` +The first root attempt permanently freezes registration, including when that attempt fails. +`strict=False` does not permit type or serializer registration after that boundary, but its policy +may authorize module-global classes and callables resolved while reading a trusted native payload. +That resolution does not add or change an explicit registration. If a strict-mode failure requires +adding a missing registration, create and configure a new instance. An already configured reader +can process a later root after a malformed-data failure. + Received remote metadata is also limited: - `max_type_fields` limits the number of fields accepted in one received struct metadata body. @@ -99,8 +108,8 @@ schema-evolution semantics. ### DeserializationPolicy -When `strict=False` is necessary, use `DeserializationPolicy` to restrict the dynamic types and -hooks accepted during deserialization: +When `strict=False` is necessary, use `DeserializationPolicy` to restrict the types and hooks +accepted during deserialization: ```python import pyfory @@ -146,7 +155,8 @@ unchanged. ### Security Checklist - Keep `strict=True` for untrusted data. -- Register all expected application types before deserialization. +- Complete any explicit type, name, ID, or custom serializer registration before the first root + attempt. - Use `DeserializationPolicy` when `strict=False` is necessary. - Keep `max_depth` low enough to reject unexpectedly deep payloads. - Keep `max_graph_memory_bytes` at the fixed `128 MiB` default for most inputs, or set a positive diff --git a/docs/object-serialization/python/troubleshooting.md b/docs/object-serialization/python/troubleshooting.md index 11928d08c5..542d3ff2d5 100644 --- a/docs/object-serialization/python/troubleshooting.md +++ b/docs/object-serialization/python/troubleshooting.md @@ -87,26 +87,28 @@ assert result.next.next is result # Circular reference preserved ```python # Keep compatible mode enabled. This is the default. -f = pyfory.Fory() -# Version 1: Original class +# Version 1: Writer schema @dataclass -class User: +class UserV1: name: str age: pyfory.Int32 -f.register(User, name="User") -data = f.dumps(User("Alice", 30)) +writer = pyfory.Fory(xlang=True) +writer.register(UserV1, name="User") +data = writer.dumps(UserV1("Alice", 30)) # Version 2: Add new field (backward compatible) @dataclass -class User: +class UserV2: name: str age: pyfory.Int32 email: str = "unknown@example.com" # New field with default -# Can still deserialize old data -user = f.loads(data) +# Register the reader schema on a separate instance. +reader = pyfory.Fory(xlang=True) +reader.register(UserV2, name="User") +user = reader.loads(data) print(user.email) # "unknown@example.com" ``` @@ -124,6 +126,11 @@ f.register(AnotherClass, type_id=101) f = pyfory.Fory(strict=False) # Use only in trusted environments ``` +The first root serialization or deserialization attempt permanently freezes explicit registration, +including when that attempt fails. Non-strict native writes may still discover runtime types +lazily, and reads may resolve those authorized by the configured policy, without creating an +explicit registration. + ## Debug Mode Set environment variable BEFORE importing pyfory to disable Cython for debugging: @@ -153,7 +160,8 @@ try: data = fory.dumps(my_object) except TypeUnregisteredError as e: print(f"Type not registered: {e}") - # Register the type and retry + # A failed root has already frozen this instance. Configure a new one. + fory = pyfory.Fory(strict=True) fory.register(type(my_object), type_id=100) data = fory.dumps(my_object) except Exception as e: diff --git a/docs/object-serialization/python/type-registration.md b/docs/object-serialization/python/type-registration.md index 917537c767..f0d4322173 100644 --- a/docs/object-serialization/python/type-registration.md +++ b/docs/object-serialization/python/type-registration.md @@ -79,6 +79,13 @@ classes. Register application classes before serializing or deserializing payloads, and keep the same registration IDs or names on every peer that shares those payloads. +The first root serialization or deserialization attempt permanently closes +registration, including when that attempt fails. `strict=False` permits native +writes to discover runtime classes and callables and permits reads to resolve +those authorized by the configured policy. That lazy resolution does not add an explicit +registration. Explicit names, IDs, and custom serializers must be configured before the first +root; later registration attempts fail. + Compatible metadata has one data-only exception: when a remote Struct has no local registration, deserialization returns the fixed framework `pyfory.UnknownStruct` value instead of loading or generating the sender's diff --git a/docs/object-serialization/scala/configuration.md b/docs/object-serialization/scala/configuration.md index 2524eaeb0e..923f56f8c2 100644 --- a/docs/object-serialization/scala/configuration.md +++ b/docs/object-serialization/scala/configuration.md @@ -120,6 +120,13 @@ object ForyHolder { } ``` +`ForyModule` registration and Scala generated-serializer registration extensions target +`BaseFory`, so they are available on both direct and thread-safe facades. Complete registration +before the facade's first root serialization or deserialization, and before concurrent use of a +thread-safe facade begins. +Module installation is registration-only setup and must not start root serialization or +deserialization through the runtime or facade being configured. + ## Configuration All configuration options from Fory Java are available. See [Java Configuration](../java/configuration.md) for the complete list. diff --git a/docs/object-serialization/swift/type-registration.md b/docs/object-serialization/swift/type-registration.md index 6ceea8cd0a..ce96000613 100644 --- a/docs/object-serialization/swift/type-registration.md +++ b/docs/object-serialization/swift/type-registration.md @@ -92,8 +92,10 @@ Keep registration mapping consistent across peers: - Do not mix ID and name mapping for the same logical type across services - Register only one serializer for each target type on a `Fory` instance -Registration closes after the first root serialization or deserialization. -Complete all registrations before the first root operation. +Registration closes permanently when the first root serialization or +deserialization attempt begins, even if that operation fails. Complete all +registrations before the first root operation; reusing the same `Fory` instance +does not reopen registration. ## Dynamic Types and Registration diff --git a/docs/security/deserialization.md b/docs/security/deserialization.md index 0e6fcd1f31..3ca9beb3d2 100644 --- a/docs/security/deserialization.md +++ b/docs/security/deserialization.md @@ -54,7 +54,7 @@ Fory security boundaries include: - Explicit Fory policy checks, such as type, function, method, class, or registration policies that are intended to restrict what may be materialized. - Cleanup boundaries, where state created during a failed read must be released - or reset before the next root operation. + or reset before the root error escapes. Fory security boundaries do not include: @@ -615,6 +615,28 @@ that case, classify the behavior by concrete impact: - Pure strictness about whether a skipped value used one specific encoding shape is not a security issue. +## Registry Lifecycle + +The first root serialization or deserialization permanently closes explicit type and serializer +registration, including when that operation fails. Each natural registration owner keeps exactly +one authoritative frozen flag, and every later explicit registration attempt fails before changing +type, serializer, ID, name, metadata, or policy bindings. A thread-safe facade with its own public +registration surface may own its boundary flag, but must not mirror a child registry's lifecycle. +An immutable registry snapshot remains registry data and must not replace the frozen flag. Do not +add another lifecycle state, a registration commit or rollback path, or eager whole-registry +preparation solely to implement freeze. + +Registry freeze does not disable native runtime type resolution. When a mode supports unregistered +types, a root may still discover an allowed runtime type and materialize resolver-owned metadata, +serializers, or generated code. Lazy completion of an existing binding is also allowed. These +internal cache operations are not explicit registration and must not create or change an explicit +type or serializer registration, ID, name, or policy binding. + +Each explicit name and ID identifies one type-information owner. Native discovery must reject an +identity collision instead of replacing that owner. Lazy metadata or serializer completion must +preserve an explicitly selected serializer and must not retain partial state; failure leaves the +previous valid state. + ## Metadata And Type Resolution Metadata parsing is security-sensitive when it affects retained read-side state, @@ -640,6 +662,13 @@ Metadata readers should: entry so input cannot make the JVM derive an unbounded family of array classes. - Reset or release metadata state at the correct root-operation boundary. +Operation-local metadata occurrences and writer IDs from a failed root must reset before its error +escapes. Successful roots may reset before the context is reused. The reset must make prior-root +entries invisible through the current logical size and release unusual high-water backing without +adding allocation or slot-clearing work to normal roots. Bounded backing may retain inactive slot +references when the runtime-specific retention rule permits it. Runtime-specific thresholds and +reset ownership belong in the implementation guide and language guidance. + A class-resolution cache reachable from untrusted deserialization may publish an entry only from explicit trusted configuration or after the active class policy has accepted the resolved class. A cache hit therefore represents an @@ -764,6 +793,11 @@ Nested `try`/`finally` or equivalent cleanup should be added only when the outer root-operation cleanup cannot cover the state or resource owned by the nested path. +A failure object must not copy or retain the root reference table or the +materialized object graph for diagnostics. Root cleanup owns releasing that +operation-local graph, and error reporting must stay bounded independently of +the graph size. + ## Performance Requirements Security validation must preserve Fory hot-path performance. Do not add diff --git a/docs/specification/xlang_implementation_guide.md b/docs/specification/xlang_implementation_guide.md index e4c38caacf..e007969e79 100644 --- a/docs/specification/xlang_implementation_guide.md +++ b/docs/specification/xlang_implementation_guide.md @@ -79,20 +79,80 @@ not the place where nested serializers do their work. - writing and reading the root xlang header bitmap - delegating nested value encoding to `WriteContext` - delegating nested value decoding to `ReadContext` -- owning registration through `TypeResolver` -- resetting operation-local context state in a top-level `finally` +- freezing the natural registration owner before root codec work +- resetting operation-local context state at the top-level root boundary + +Explicit type and serializer registration is open only before the first root serialization or +deserialization. Starting either root sets exactly one authoritative frozen flag for the natural +registration owner before codec work and never clears it, including when the root fails. +Every explicit registration path checks that flag before mutation. A thread-safe facade with its own +public registration surface may own its boundary flag, but must not mirror a child registry's +lifecycle. Registration preparation can execute application code through custom serializer +construction, static serializer properties, or code-generation hooks. Recheck the same owner flag +after the last such callback and immediately before explicit registry publication. Do not add +another lifecycle state, a registration commit or rollback path, or eager whole-registry +preparation solely to implement freeze. + +For a Java thread-safe facade, live child registration carries the facade's existing shared-registry +check through application-controlled preparation and runs it immediately before publishing into the +child. Replaying already accepted setup into a new, unexposed thread-local child uses the child's +local resolver check; the child then adopts the shared frozen snapshot before exposure. The child +resolver must not mirror or directly enforce the facade flag because these owners govern different +boundaries. + +In JavaScript, `Fory` owns this flag. `TypeResolver` owns the registration maps but must not carry a +second lifecycle flag. + +Registry freeze does not make resolver caches immutable. Native modes that allow unregistered +runtime types may still discover an allowed type and materialize its resolver-owned metadata, +serializer, or generated code. Lazy serializer completion for an existing binding is also allowed. +These operations must not create or change an explicit type, serializer, ID, name, or policy +registration. + +Each explicit name and ID identifies one type-information owner. Native discovery must reject an +identity collision instead of replacing that owner. Lazy metadata or serializer completion must +preserve an explicitly selected serializer and must not retain partial state; failure leaves the +previous valid state. + +Java module registration remains available through `BaseFory` before the first root. Kotlin and +Scala registration extensions target `BaseFory`, so direct and thread-safe facades share the same +pre-root API. Copy operations and facade execution callbacks do not freeze registration unless they +start a root serialization or deserialization. + +`ForyModule.install` is registration-only setup. It may register nested modules and construct +serializers for the supplied child, but it must not start root serialization or deserialization +through that child or the direct or thread-safe facade installing the module. Nested serializers must not call back into root `serialize(...)` or `deserialize(...)` entry points. ### `WriteContext` and `ReadContext` hold operation-local state -`WriteContext` and `ReadContext` are prepared by `Fory` for one root operation -and reset by `Fory` in a `finally` block before reuse. +`WriteContext` and `ReadContext` are prepared by `Fory` for one root operation. A failed root resets +its operation-local state before propagating the error. Successful roots may retain bounded state +until the next root entry, but must reset it before the context is reused. A failure object must not +retain operation-local state or its materialized object graph. `prepare(...)` should only bind the active buffer and root-operation inputs. `reset()` should clear operation-local mutable state. +When MetaString and TypeMeta writer objects carry root-local dynamic IDs, each +owning table must restore those IDs and reset its own active owner count. A +bounded owner table may retain its backing storage, but only entries below that +table's current logical count participate in the next reset; otherwise prior +owners create duplicate cleanup work across roots. Implementations should +release an unusual high-water backing table. + +Java scoped meta-share `TypeInfo` occurrence tables use their logical size as the protocol +visibility boundary. A table with at most 8192 active entries resets only that size and retains its +slots; a larger table replaces its backing with eight slots. This uniform owner rule keeps normal +cleanup allocation-free and must not be specialized for particular entry counts or benchmark +shapes. JavaScript read-side metadata occurrence arrays use native replacement reset instead. Its +MetaString and TypeMeta writer owner tables retain bounded backing through 8192 active owners, reset only +their own logical size after restoring active owner IDs, and release backing above that boundary. A +failed JavaScript root resets this operation-local state before its exception escapes; state from a +successful root resets on the next root entry so the success exit remains allocation-free. + That operation-local state includes: - the current buffer @@ -216,7 +276,8 @@ The ownership split is: corresponding readable-byte, policy, and graph-memory checks before allocation - `Fory` owns root framing and operation setup/reset -- `TypeResolver` owns registration and dynamic lookup +- `TypeResolver` owns registration mappings, serializer bindings, and dynamic lookup; the natural + registry or public facade owner owns the authoritative registry lifecycle fact #### C# generated structural serializers @@ -915,11 +976,13 @@ Keep the root bitmap separate from per-object ref markers: The current root write flow is: -1. `Fory.serialize(...)` or `serializeTo(...)` prepares the target buffer. -2. `Fory` calls `writeContext.prepare(...)`. -3. `Fory` writes the root bitmap. -4. `Fory` delegates the root object to `WriteContext`. -5. `writeContext.reset()` runs in `finally`. +1. `Fory.serialize(...)` or `serializeTo(...)` permanently freezes registration. +2. `Fory` prepares the target buffer. +3. `Fory` calls `writeContext.prepare(...)`. +4. `Fory` writes the root bitmap. +5. `Fory` delegates the root object to `WriteContext`. +6. A failed write resets operation-local state before propagating its error. State retained after a + successful write resets before the next root reuses the context. For a non-null root value, `WriteContext.writeRootValue(...)` performs: @@ -941,7 +1004,7 @@ Important rules: - repeated primitive writes should go directly through the buffer - nested serializer flow should stay straight-line; do not add internal `try/finally` blocks just to clean per-operation state -- top-level `Fory.serialize(...)` owns the operation reset `finally` +- top-level `Fory.serialize(...)` owns the operation reset boundary ## Deserialization Flow @@ -949,12 +1012,14 @@ Important rules: The current root read flow mirrors the write flow: -1. `Fory.deserialize(...)` or `deserializeFrom(...)` reads the root bitmap. -2. null roots return immediately. -3. `Fory` validates xlang mode and other root framing requirements. -4. `Fory` calls `readContext.prepare(...)`. -5. `Fory` delegates to `ReadContext`. -6. `readContext.reset()` runs in `finally`. +1. `Fory.deserialize(...)` or `deserializeFrom(...)` permanently freezes registration. +2. `Fory` reads the root bitmap. +3. null roots return immediately. +4. `Fory` validates xlang mode and other root framing requirements. +5. `Fory` calls `readContext.prepare(...)`. +6. `Fory` delegates to `ReadContext`. +7. A failed read resets operation-local state before propagating its error. State retained after a + successful read resets before the next root reuses the context. ### `ReadContext` owns ref reservation and payload materialization @@ -1228,7 +1293,7 @@ Important rules: it - nested serializer flow should stay straight-line; do not add internal `try/finally` blocks just to restore operation-local state -- top-level `Fory.deserialize(...)` owns the operation reset `finally` +- top-level `Fory.deserialize(...)` owns the operation reset boundary ## Depth Tracking diff --git a/go/fory/fory.go b/go/fory/fory.go index 82c2b8c0d5..c5c116b3fa 100644 --- a/go/fory/fory.go +++ b/go/fory/fory.go @@ -33,6 +33,9 @@ import ( // ErrNoSerializer indicates no serializer is registered for a type var ErrNoSerializer = errors.New("fory: no serializer registered for type") +// ErrRegistryFrozen indicates registration was attempted after a root operation started. +var ErrRegistryFrozen = errors.New("fory: types and serializers must be registered before the first root serialization or deserialization operation") + // Public named registration accepts one dotted name; resolver primitives receive // the split wire metadata components because named TypeDefs store them separately. func splitRegisteredName(name string) (string, string, error) { @@ -201,10 +204,12 @@ func WithMaxAverageSchemaVersionsPerType(size int) Option { // Fory is the main serialization instance. // Note: Fory is NOT thread-safe. Use ThreadSafeFory for concurrent use. +// Type and serializer registration must finish before its first root operation. type Fory struct { - config Config - metaContext *MetaContext - compatibleSet bool + config Config + metaContext *MetaContext + compatibleSet bool + registryFrozen bool // Reusable contexts - avoid allocation on each SerializeWithCallback/DeserializeWithCallbackBuffers call writeCtx *WriteContext @@ -294,6 +299,14 @@ func validateUserTypeID(typeID uint32) error { return nil } +//go:noinline +func (f *Fory) checkRegistrationOpen() error { + if f.registryFrozen { + return ErrRegistryFrozen + } + return nil +} + // RegisterStruct registers a struct type with a numeric ID for cross-language serialization. // This is compatible with Java's fory.register(Class, int) method. // type_ can be either a reflect.Type or an instance of the type @@ -302,6 +315,9 @@ func validateUserTypeID(typeID uint32) error { // //go:noinline func (f *Fory) RegisterStruct(type_ any, typeID uint32) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } if err := validateUserTypeID(typeID); err != nil { return err } @@ -335,6 +351,9 @@ func (f *Fory) RegisterStruct(type_ any, typeID uint32) error { // //go:noinline func (f *Fory) RegisterUnion(type_ any, typeID uint32, serializer Serializer) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } if serializer == nil { return fmt.Errorf("RegisterUnion requires a non-nil serializer") } @@ -363,6 +382,9 @@ func (f *Fory) RegisterUnion(type_ any, typeID uint32, serializer Serializer) er // //go:noinline func (f *Fory) RegisterUnionByName(type_ any, name string, serializer Serializer) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } if serializer == nil { return fmt.Errorf("RegisterUnionByName requires a non-nil serializer") } @@ -392,6 +414,9 @@ func (f *Fory) RegisterUnionByName(type_ any, name string, serializer Serializer // //go:noinline func (f *Fory) RegisterStructByName(type_ any, name string) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } var t reflect.Type if rt, ok := type_.(reflect.Type); ok { t = rt @@ -419,6 +444,9 @@ func (f *Fory) RegisterStructByName(type_ any, name string) error { // //go:noinline func (f *Fory) RegisterEnum(type_ any, typeID uint32) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } if err := validateUserTypeID(typeID); err != nil { return err } @@ -451,6 +479,9 @@ func (f *Fory) RegisterEnum(type_ any, typeID uint32) error { // //go:noinline func (f *Fory) RegisterEnumByName(type_ any, name string) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } var t reflect.Type if rt, ok := type_.(reflect.Type); ok { t = rt @@ -483,6 +514,9 @@ func (f *Fory) RegisterEnumByName(type_ any, name string) error { // //go:noinline func (f *Fory) RegisterExtension(type_ any, typeID uint32, serializer ExtensionSerializer) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } if err := validateUserTypeID(typeID); err != nil { return err } @@ -522,6 +556,9 @@ func (f *Fory) RegisterExtension(type_ any, typeID uint32, serializer ExtensionS // //go:noinline func (f *Fory) RegisterExtensionByName(type_ any, name string, serializer ExtensionSerializer) error { + if err := f.checkRegistrationOpen(); err != nil { + return err + } var t reflect.Type if rt, ok := type_.(reflect.Type); ok { t = rt @@ -538,7 +575,7 @@ func (f *Fory) RegisterExtensionByName(type_ any, name string, serializer Extens return f.typeResolver.registerExtensionByName(t, namespace, typeName, serializer) } -// Reset clears internal state for reuse +// Reset clears root operation state for reuse. It does not reopen registration. func (f *Fory) Reset() { f.writeCtx.Reset() f.readCtx.Reset() @@ -563,6 +600,7 @@ func (f *Fory) Reset() { // // For thread-safe usage, use threadsafe.Fory which copies the data internally. func (f *Fory) Serialize(value any) ([]byte, error) { + f.registryFrozen = true defer f.resetWriteState() if !validateRootDecimal(f.writeCtx.Err(), value) { return nil, f.writeCtx.TakeError() @@ -598,6 +636,7 @@ func (f *Fory) rootRefMode() RefMode { // Deserialize deserializes data directly into the provided target value. // The target must be a pointer to the value to deserialize into. func (f *Fory) Deserialize(data []byte, v any) error { + f.registryFrozen = true defer f.resetReadState() f.readCtx.SetData(data) target := reflect.ValueOf(v).Elem() @@ -633,17 +672,30 @@ func (f *Fory) resetWriteState() { } } +func (f *Fory) restoreWriteBuffer(buffer *ByteBuffer) { + f.writeCtx.buffer = buffer + f.resetWriteState() +} + +func (f *Fory) restoreReadBuffer(buffer *ByteBuffer) { + f.readCtx.buffer = buffer + f.resetReadState() +} + // SerializeTo serializes a value and appends the bytes to the provided buffer. // This is useful when you need to write multiple serialized values to the same buffer. // Returns error if serialization fails. func (f *Fory) SerializeTo(buf *ByteBuffer, value any) error { - defer f.resetWriteState() + f.registryFrozen = true + origBuffer := f.writeCtx.buffer + // Restore the owned buffer before reset so a serializer panic cannot reset or retain the + // caller-owned buffer. + defer f.restoreWriteBuffer(origBuffer) if !validateRootDecimal(f.writeCtx.Err(), value) { return f.writeCtx.TakeError() } // Temporarily swap buffer - origBuffer := f.writeCtx.buffer f.writeCtx.buffer = buf // Write protocol header @@ -666,10 +718,8 @@ func (f *Fory) SerializeTo(buf *ByteBuffer, value any) error { typeInfo.Serializer.WriteData(f.writeCtx, elemValue) } if f.writeCtx.HasError() { - f.writeCtx.buffer = origBuffer return f.writeCtx.TakeError() } - f.writeCtx.buffer = origBuffer return nil } } @@ -677,12 +727,9 @@ func (f *Fory) SerializeTo(buf *ByteBuffer, value any) error { // Standard path - TypeMeta is written inline using streaming protocol f.writeCtx.WriteValue(rv, f.rootRefMode(), true) if f.writeCtx.HasError() { - f.writeCtx.buffer = origBuffer return f.writeCtx.TakeError() } - // Restore original buffer - f.writeCtx.buffer = origBuffer return nil } @@ -690,32 +737,29 @@ func (f *Fory) SerializeTo(buf *ByteBuffer, value any) error { // The buffer's reader index is advanced as data is read. // This is useful when reading multiple serialized values from the same buffer. func (f *Fory) DeserializeFrom(buf *ByteBuffer, v any) error { + f.registryFrozen = true // Reset contexts for each independent serialized object - defer f.resetReadState() - // Temporarily swap buffer origBuffer := f.readCtx.buffer f.readCtx.buffer = buf + // Restore the owned buffer before root cleanup so an escaping panic cannot leave a + // caller-owned buffer installed for the next operation. + defer f.restoreReadBuffer(origBuffer) target := reflect.ValueOf(v).Elem() f.readCtx.remainingGraphMemoryBytes = f.config.MaxGraphMemoryBytes f.readCtx.remainingUnbackedContainerItems = f.config.MaxUnbackedContainerItems readHeader(f.readCtx) if f.readCtx.HasError() { - f.readCtx.buffer = origBuffer return f.readCtx.TakeError() } // Deserialize the value - TypeMeta is read inline using streaming protocol f.readCtx.ReadValue(target, f.rootRefMode(), true) if f.readCtx.HasError() { - f.readCtx.buffer = origBuffer return f.readCtx.TakeError() } - // Restore original buffer - f.readCtx.buffer = origBuffer - return nil } @@ -743,6 +787,7 @@ func (f *Fory) Unmarshal(data []byte, v any) error { // If callback is provided, it will be called for each BufferObject during serialization. // Return true from callback to write in-band, false for out-of-band. func (f *Fory) SerializeWithCallback(buffer *ByteBuffer, v any, callback func(BufferObject) bool) error { + f.registryFrozen = true buf := f.writeCtx.buffer defer func() { // Reset internal state but NOT the buffer - caller manages buffer state @@ -752,11 +797,6 @@ func (f *Fory) SerializeWithCallback(buffer *ByteBuffer, v any, callback func(Bu if f.metaContext != nil { f.metaContext.Reset() } - // Set up buffer callback for out-of-band serialization - if callback != nil { - f.writeCtx.bufferCallback = nil - f.writeCtx.outOfBand = false - } }() if !validateRootDecimal(f.writeCtx.Err(), v) { return f.writeCtx.TakeError() @@ -786,17 +826,19 @@ func (f *Fory) SerializeWithCallback(buffer *ByteBuffer, v any, callback func(Bu // DeserializeWithCallbackBuffers deserializes from buffer into the provided value (for streaming/cross-language use). // The third parameter is optional external buffers for out-of-band data (can be nil). func (f *Fory) DeserializeWithCallbackBuffers(buffer *ByteBuffer, v any, buffers []*ByteBuffer) error { + f.registryFrozen = true // Use the caller buffer only for this root; later stream roots reuse the // original internal buffer. origBuffer := f.readCtx.buffer f.readCtx.buffer = buffer defer func() { + // Restore the owned buffer before reset so a cleanup panic cannot retain + // the caller-owned buffer. + f.readCtx.buffer = origBuffer f.readCtx.Reset() if f.metaContext != nil { f.metaContext.Reset() } - f.readCtx.buffer = origBuffer - f.readCtx.outOfBandBuffers = nil }() // Set up out-of-band buffers if provided if buffers != nil { @@ -919,6 +961,7 @@ func readHeaderSlow(ctx *ReadContext, bitmap byte) { // // For thread-safe usage, use threadsafe.Serialize which copies the data internally. func Serialize[T any](f *Fory, value T) ([]byte, error) { + f.registryFrozen = true defer f.resetWriteState() v := any(value) if !validateRootDecimal(f.writeCtx.Err(), v) { @@ -1075,6 +1118,7 @@ func Serialize[T any](f *Fory, value T) ([]byte, error) { // For structs, it reads directly into the struct fields. // Note: Fory instance is NOT thread-safe. Use ThreadSafeFory for concurrent use. func Deserialize[T any](f *Fory, data []byte, target *T) error { + f.registryFrozen = true // Generic roots share the same reusable read and metadata owners as the // method API, so both entry and every exit must start from a root-clean state. f.resetReadState() diff --git a/go/fory/fory_test.go b/go/fory/fory_test.go index 950ec5c1a8..bfb9dc8de7 100644 --- a/go/fory/fory_test.go +++ b/go/fory/fory_test.go @@ -274,17 +274,16 @@ func TestSerializeStructSimple(t *testing.T) { type A struct { F1 []string } - require.Nil(t, fory.RegisterStructByName(A{}, "example.A")) - serde(t, fory, A{}) - serde(t, fory, &A{}) - serde(t, fory, A{F1: []string{"str1", "", "str2"}}) - serde(t, fory, &A{F1: []string{"str1", "", "str2"}}) - type SimpleB struct { F1 []string F2 map[string]int32 } + require.Nil(t, fory.RegisterStructByName(A{}, "example.A")) require.Nil(t, fory.RegisterStructByName(SimpleB{}, "example.SimpleB")) + serde(t, fory, A{}) + serde(t, fory, &A{}) + serde(t, fory, A{F1: []string{"str1", "", "str2"}}) + serde(t, fory, &A{F1: []string{"str1", "", "str2"}}) serde(t, fory, SimpleB{}) serde(t, fory, SimpleB{ F1: []string{"str1", "", "str2"}, @@ -410,24 +409,24 @@ func newFoo() Foo { func TestSerializeStruct(t *testing.T) { for _, referenceTracking := range []bool{false, true} { fory := NewFory(WithXlang(true), WithCompatible(false), WithRefTracking(referenceTracking)) + type A struct { + F1 Bar + F2 any + } require.Nil(t, fory.RegisterStructByName(Bar{}, "example.Bar")) + require.Nil(t, fory.RegisterStructByName(A{}, "example.A")) + require.Nil(t, fory.RegisterStructByName(Foo{}, "example.Foo")) serde(t, fory, &Bar{}) bar := Bar{F1: 1, F2: "str"} serde(t, fory, bar) serde(t, fory, &bar) - type A struct { - F1 Bar - F2 any - } - require.Nil(t, fory.RegisterStructByName(A{}, "example.A")) serde(t, fory, A{}) serde(t, fory, &A{}) // Use int64 for any fields since xlang deserializes integers to int64 serde(t, fory, A{F1: Bar{F1: 1, F2: "str"}, F2: int64(-1)}) serde(t, fory, &A{F1: Bar{F1: 1, F2: "str"}, F2: int64(-1)}) - require.Nil(t, fory.RegisterStructByName(Foo{}, "example.Foo")) foo := newFoo() serde(t, fory, foo) serde(t, fory, &foo) @@ -435,8 +434,8 @@ func TestSerializeStruct(t *testing.T) { } func TestSerializeCircularReference(t *testing.T) { - fory := NewFory(WithXlang(true), WithCompatible(false), WithRefTracking(true)) { + fory := NewFory(WithXlang(true), WithCompatible(false), WithRefTracking(true)) type A struct { A1 *A } @@ -455,6 +454,7 @@ func TestSerializeCircularReference(t *testing.T) { require.Same(t, a1, a1.A1) } { + fory := NewFory(WithXlang(true), WithCompatible(false), WithRefTracking(true)) type CircularRefB struct { F1 string F2 *CircularRefB diff --git a/go/fory/fory_typed_test.go b/go/fory/fory_typed_test.go index 3a7da012ad..5e8764f386 100644 --- a/go/fory/fory_typed_test.go +++ b/go/fory/fory_typed_test.go @@ -173,9 +173,8 @@ func TestDeserializeByteSliceAcceptsUint8ArrayRootType(t *testing.T) { // TestSerializeGenericComplex tests Serialize[T]/DeserializeWithCallbackBuffers[T] with complex types. // Struct wrappers must be registered explicitly before fast serializer use. func TestSerializeGenericComplex(t *testing.T) { - f := NewFory(WithXlang(false), WithRefTracking(true), WithCompatible(false)) - t.Run("Struct", func(t *testing.T) { + f := NewFory(WithXlang(false), WithRefTracking(true), WithCompatible(false)) type TestStruct struct { Name string Value int32 @@ -195,6 +194,7 @@ func TestSerializeGenericComplex(t *testing.T) { }) t.Run("Slice", func(t *testing.T) { + f := NewFory(WithXlang(false), WithRefTracking(true), WithCompatible(false)) // Note: *[]T is not supported, use wrapper struct instead type SliceWrapper struct { Items []int32 @@ -211,6 +211,7 @@ func TestSerializeGenericComplex(t *testing.T) { }) t.Run("Map", func(t *testing.T) { + f := NewFory(WithXlang(false), WithRefTracking(true), WithCompatible(false)) // Note: *map[K]V is not supported, use wrapper struct instead type MapWrapper struct { Items map[string]int32 @@ -229,10 +230,9 @@ func TestSerializeGenericComplex(t *testing.T) { // TestSerializeDeserializeRoundTrip tests that serialized data can be correctly deserialized. func TestSerializeDeserializeRoundTrip(t *testing.T) { - f := NewFory(WithXlang(false), WithRefTracking(true), WithCompatible(false)) - // Test that SerializeWithCallback[T] uses pointer-based fast path when available t.Run("TypedSerializerPath", func(t *testing.T) { + f := NewFory(WithXlang(false), WithRefTracking(true), WithCompatible(false)) // Int32 has a registered fast path original := int32(999) data, err := Serialize(f, &original) @@ -246,12 +246,13 @@ func TestSerializeDeserializeRoundTrip(t *testing.T) { }) t.Run("FastSerializerFallbackPath", func(t *testing.T) { + f := NewFory(WithXlang(false), WithRefTracking(true), WithCompatible(false)) // Custom struct uses the fast serializer fallback path. type CustomStruct struct { ID int64 Name string } - f.RegisterStructByName(CustomStruct{}, "test.CustomStruct") + require.NoError(t, f.RegisterStructByName(CustomStruct{}, "test.CustomStruct")) original := CustomStruct{ID: 123, Name: "test"} data, err := Serialize(f, &original) diff --git a/go/fory/registry_freeze_lifecycle_test.go b/go/fory/registry_freeze_lifecycle_test.go new file mode 100644 index 0000000000..295b669958 --- /dev/null +++ b/go/fory/registry_freeze_lifecycle_test.go @@ -0,0 +1,213 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package fory + +import ( + "bytes" + "reflect" + "testing" + + "github.com/stretchr/testify/require" +) + +type registryFreezeStruct struct { + Value int32 +} + +type registryFreezeUnion struct{} + +type registryFreezeEnum int32 + +type registryFreezeExtension struct { + Value int32 +} + +type registryPanicSerializer struct{} + +func (registryPanicSerializer) WriteData(ctx *WriteContext, _ reflect.Value) { + ctx.Err().SetError(SerializationError("write failure")) + panic("write failure") +} + +func (registryPanicSerializer) ReadData(*ReadContext, reflect.Value) {} + +func TestRegistryFreezeRegistrations(t *testing.T) { + f := New(WithXlang(false), WithCompatible(false)) + _, err := f.Serialize(int32(1)) + require.NoError(t, err) + f.Reset() + + attempts := []struct { + name string + call func() error + }{ + {"struct ID", func() error { return f.RegisterStruct(registryFreezeStruct{}, 7101) }}, + {"struct name", func() error { + return f.RegisterStructByName(registryFreezeStruct{}, "test.RegistryFreezeStruct") + }}, + {"union ID", func() error { return f.RegisterUnion(registryFreezeUnion{}, 7102, nil) }}, + {"union name", func() error { + return f.RegisterUnionByName(registryFreezeUnion{}, "test.RegistryFreezeUnion", nil) + }}, + {"enum ID", func() error { return f.RegisterEnum(registryFreezeEnum(0), 7103) }}, + {"enum name", func() error { + return f.RegisterEnumByName(registryFreezeEnum(0), "test.RegistryFreezeEnum") + }}, + {"extension ID", func() error { + return f.RegisterExtension(registryFreezeExtension{}, 7104, nil) + }}, + {"extension name", func() error { + return f.RegisterExtensionByName( + registryFreezeExtension{}, "test.RegistryFreezeExtension", nil) + }}, + } + for _, attempt := range attempts { + t.Run(attempt.name, func(t *testing.T) { + require.ErrorIs(t, attempt.call(), ErrRegistryFrozen) + }) + } + + structType := reflect.TypeOf(registryFreezeStruct{}) + resolverAttempts := []func() error{ + func() error { + return f.typeResolver.RegisterStruct( + structType, f.typeResolver.structTypeID(structType, false), 7105) + }, + func() error { + return f.typeResolver.RegisterUnion( + reflect.TypeOf(registryFreezeUnion{}), 7106, nil) + }, + func() error { + return f.typeResolver.RegisterEnum( + reflect.TypeOf(registryFreezeEnum(0)), 7107) + }, + func() error { + return f.typeResolver.RegisterExtension( + reflect.TypeOf(registryFreezeExtension{}), 7108, nil) + }, + } + for _, attempt := range resolverAttempts { + require.ErrorIs(t, attempt(), ErrRegistryFrozen) + } +} + +func TestFrozenAllowsLazySerializer(t *testing.T) { + f := New(WithXlang(false), WithCompatible(false)) + _, err := f.Serialize(int32(1)) + require.NoError(t, err) + + want := map[int32]string{7: "value"} + data, err := f.Serialize(want) + require.NoError(t, err) + var got map[int32]string + require.NoError(t, f.Deserialize(data, &got)) + require.Equal(t, want, got) +} + +func TestRegistryFreezeRoots(t *testing.T) { + badDecimal := Decimal{Scale: maxDecimalScale + 1} + tests := []struct { + name string + root func(*Fory) error + }{ + {"Serialize", func(f *Fory) error { _, err := f.Serialize(badDecimal); return err }}, + {"Deserialize", func(f *Fory) error { return f.Deserialize(nil, new(int32)) }}, + {"SerializeTo", func(f *Fory) error { return f.SerializeTo(NewByteBuffer(nil), badDecimal) }}, + {"DeserializeFrom", func(f *Fory) error { + return f.DeserializeFrom(NewByteBuffer(nil), new(int32)) + }}, + {"SerializeWithCallback", func(f *Fory) error { + return f.SerializeWithCallback(NewByteBuffer(nil), badDecimal, nil) + }}, + {"DeserializeWithCallbackBuffers", func(f *Fory) error { + return f.DeserializeWithCallbackBuffers(NewByteBuffer(nil), nil, nil) + }}, + {"generic Serialize", func(f *Fory) error { _, err := Serialize(f, badDecimal); return err }}, + {"generic Deserialize", func(f *Fory) error { return Deserialize(f, nil, new(int32)) }}, + {"DeserializeFromStream", func(f *Fory) error { + return f.DeserializeFromStream(NewInputStream(bytes.NewReader(nil)), new(int32)) + }}, + {"DeserializeFromReader", func(f *Fory) error { + return f.DeserializeFromReader(bytes.NewReader(nil), new(int32)) + }}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + f := New(WithXlang(false), WithCompatible(false)) + require.Error(t, test.root(f)) + require.ErrorIs(t, f.RegisterStructByName( + registryFreezeStruct{}, "test.RegistryFreezeRoot"), ErrRegistryFrozen) + }) + } +} + +func TestBorrowedBufferPanicCleanup(t *testing.T) { + writer := New(WithXlang(false), WithCompatible(false)) + data, err := writer.Serialize(int32(7)) + require.NoError(t, err) + data = bytes.Clone(data) + + f := New(WithXlang(false), WithCompatible(false)) + borrowed := NewByteBuffer(bytes.Clone(data)) + require.Panics(t, func() { + _ = f.DeserializeFrom(borrowed, int32(0)) + }) + var value int32 + require.NoError(t, f.Deserialize(data, &value)) + require.Equal(t, int32(7), value) +} + +func TestStreamBufferPanicCleanup(t *testing.T) { + f := New(WithXlang(false), WithCompatible(false)) + owned := f.readCtx.buffer + stream := NewInputStream(bytes.NewReader(nil)) + + require.Panics(t, func() { + _ = f.DeserializeFromStream(stream, int32(0)) + }) + require.Same(t, owned, f.readCtx.buffer) +} + +func TestSerializePanicCleanup(t *testing.T) { + f := New(WithXlang(false), WithCompatible(false)) + require.NoError(t, f.RegisterExtension( + registryFreezeExtension{}, 7104, registryPanicSerializer{})) + owned := f.writeCtx.buffer + borrowed := NewByteBuffer(nil) + + require.Panics(t, func() { + _ = f.SerializeTo(borrowed, ®istryFreezeExtension{Value: 1}) + }) + require.Same(t, owned, f.writeCtx.buffer) + + data, err := f.Serialize(int32(7)) + require.NoError(t, err) + require.NotEmpty(t, data) +} + +func TestCallbackPanicCleanup(t *testing.T) { + f := New(WithXlang(false), WithCompatible(false)) + require.NoError(t, f.RegisterExtension( + registryFreezeExtension{}, 7104, registryPanicSerializer{})) + + require.Panics(t, func() { + _ = f.SerializeWithCallback( + NewByteBuffer(nil), ®istryFreezeExtension{Value: 1}, nil) + }) + require.NoError(t, f.SerializeWithCallback(NewByteBuffer(nil), int32(7), nil)) +} diff --git a/go/fory/stream.go b/go/fory/stream.go index 018f797460..5579ff7c52 100644 --- a/go/fory/stream.go +++ b/go/fory/stream.go @@ -96,15 +96,18 @@ func (is *InputStream) Shrink() { // DeserializeFromStream reads the next object from the stream into the provided value. // It preserves the stream buffer while clearing root-scoped read metadata between calls. func (f *Fory) DeserializeFromStream(is *InputStream, v any) error { + f.registryFrozen = true origBuffer := f.readCtx.buffer f.readCtx.buffer = is.buffer - target := reflect.ValueOf(v).Elem() - f.readCtx.remainingGraphMemoryBytes = f.config.MaxGraphMemoryBytes - f.readCtx.remainingUnbackedContainerItems = f.config.MaxUnbackedContainerItems defer func() { + // Restore the owned buffer before reset so caller validation panics cannot retain the + // stream-owned buffer. f.readCtx.buffer = origBuffer f.resetReadState() }() + target := reflect.ValueOf(v).Elem() + f.readCtx.remainingGraphMemoryBytes = f.config.MaxGraphMemoryBytes + f.readCtx.remainingUnbackedContainerItems = f.config.MaxUnbackedContainerItems readHeader(f.readCtx) if f.readCtx.HasError() { @@ -124,6 +127,7 @@ func (f *Fory) DeserializeFromStream(is *InputStream, v any) error { // each call, discarding any prefetched data and type metadata. // For sequential multi-object reads on the same stream, use NewInputStream instead. func (f *Fory) DeserializeFromReader(r io.Reader, v any) error { + f.registryFrozen = true defer f.resetReadState() // Always reset to enforce stateless semantics. f.readCtx.buffer.ResetWithReader(r, 0) diff --git a/go/fory/threadsafe/fory.go b/go/fory/threadsafe/fory.go index 4afdfa0de9..8aac021d60 100644 --- a/go/fory/threadsafe/fory.go +++ b/go/fory/threadsafe/fory.go @@ -24,8 +24,7 @@ import ( "github.com/apache/fory/go/fory" ) -// Fory is a thread-safe wrapper around fory.Fory using sync.Pool. -// It provides the same API as fory.Fory but is safe for concurrent use. +// Fory is a thread-safe serialization wrapper using a pool of fory.Fory instances. type Fory struct { pool sync.Pool } @@ -37,7 +36,7 @@ func New(opts ...fory.Option) *Fory { }) } -// NewWithFactory creates a new thread-safe Fory instance using a custom factory. +// NewWithFactory creates a thread-safe Fory whose factory configures each pooled instance. func NewWithFactory(factory func() *fory.Fory) *Fory { if factory == nil { panic("threadsafe.NewWithFactory requires a non-nil factory") @@ -90,13 +89,6 @@ func (f *Fory) Deserialize(data []byte, v any) error { return inner.Deserialize(data, v) } -// RegisterStructByName registers a struct type by name for cross-language serialization. -func (f *Fory) RegisterStructByName(type_ any, name string) error { - inner := f.acquire() - defer f.release(inner) - return inner.RegisterStructByName(type_, name) -} - // ============================================================================ // Generic package-level functions // ============================================================================ diff --git a/go/fory/threadsafe/fory_test.go b/go/fory/threadsafe/fory_test.go index 37b4a0ecf0..a6cdad0272 100644 --- a/go/fory/threadsafe/fory_test.go +++ b/go/fory/threadsafe/fory_test.go @@ -144,7 +144,18 @@ func TestDeserialize(t *testing.T) { type SliceWrapper struct { Items []int32 } - require.NoError(t, f.RegisterStructByName(SliceWrapper{}, "threadsafe.SliceWrapper")) + f := NewWithFactory(func() *fory.Fory { + inner := fory.New( + fory.WithXlang(false), + fory.WithRefTracking(true), + fory.WithCompatible(false), + ) + if err := inner.RegisterStructByName( + SliceWrapper{}, "threadsafe.SliceWrapper"); err != nil { + panic(err) + } + return inner + }) original := SliceWrapper{Items: []int32{1, 2, 3, 4, 5}} data, err := Serialize(f, &original) require.NoError(t, err) diff --git a/go/fory/type_resolver.go b/go/fory/type_resolver.go index ed82c94d34..93e22f4797 100644 --- a/go/fory/type_resolver.go +++ b/go/fory/type_resolver.go @@ -496,6 +496,9 @@ func validateOptionalFields(type_ reflect.Type) error { // RegisterStruct registers a type with a numeric user type ID for cross-language serialization. func (r *TypeResolver) RegisterStruct(type_ reflect.Type, typeID TypeId, userTypeID uint32) error { + if err := r.fory.checkRegistrationOpen(); err != nil { + return err + } // Check if already registered if info, ok := r.userTypeIdToTypeInfo[userTypeID]; ok { if info.Type == type_ { @@ -556,6 +559,9 @@ func (r *TypeResolver) RegisterStruct(type_ reflect.Type, typeID TypeId, userTyp // RegisterUnion registers a union type with a numeric user type ID for cross-language serialization. func (r *TypeResolver) RegisterUnion(type_ reflect.Type, userTypeID uint32, serializer Serializer) error { + if err := r.fory.checkRegistrationOpen(); err != nil { + return err + } if serializer == nil { return fmt.Errorf("RegisterUnion requires a non-nil serializer") } @@ -591,6 +597,9 @@ func (r *TypeResolver) RegisterUnion(type_ reflect.Type, userTypeID uint32, seri // RegisterEnum registers an enum type (numeric type in Go) with a user type ID. func (r *TypeResolver) RegisterEnum(type_ reflect.Type, userTypeID uint32) error { + if err := r.fory.checkRegistrationOpen(); err != nil { + return err + } // Check if already registered if info, ok := r.userTypeIdToTypeInfo[userTypeID]; ok { return fmt.Errorf("type %s with id %d has been registered", info.Type, userTypeID) @@ -791,6 +800,9 @@ func (r *TypeResolver) RegisterExtension( userTypeID uint32, userSerializer ExtensionSerializer, ) error { + if err := r.fory.checkRegistrationOpen(); err != nil { + return err + } if userTypeID > maxUserTypeID { return fmt.Errorf("typeID must be in range [0, 0xfffffffe], got %d", userTypeID) } diff --git a/go/fory/writer.go b/go/fory/writer.go index dbdcf583b4..ada20827cc 100644 --- a/go/fory/writer.go +++ b/go/fory/writer.go @@ -78,6 +78,7 @@ func (c *WriteContext) Reset() { func (c *WriteContext) ResetState() { c.refWriter.Reset() c.depth = 0 + c.err = Error{} c.bufferCallback = nil c.outOfBand = false if c.refResolver != nil { diff --git a/integration_tests/idl_tests/javascript/roundtrip.ts b/integration_tests/idl_tests/javascript/roundtrip.ts index 62dd418ffe..92a6313f28 100644 --- a/integration_tests/idl_tests/javascript/roundtrip.ts +++ b/integration_tests/idl_tests/javascript/roundtrip.ts @@ -91,9 +91,14 @@ import { import { Monster, Color, registerMonsterTypes } from "./generated/monster"; import { TreeNode, registerTreeTypes } from "./generated/tree"; -type RegisterFn = (fory: Fory) => unknown; +type RegisterFn = (fory: Fory) => Record; type AssertFn = (expected: T, actual: unknown) => void; +interface RegisteredFory { + fory: Fory; + serializers: Map; +} + function resolveCompatibleModes(): boolean[] { const value = process.env.IDL_COMPATIBLE; if (value == null || value.trim() === "") { @@ -113,18 +118,25 @@ function buildFory( compatible: boolean, ref: boolean, registerFns: ReadonlyArray, -): Fory { +): RegisteredFory { const fory = new Fory({ compatible, ref, }); + const serializers = new Map(); for (const registerFn of registerFns) { - registerFn(fory); + for (const { serializer } of Object.values(registerFn(fory))) { + serializers.set(serializer.getTypeInfo().userTypeId, serializer); + } } - return fory; + return { fory, serializers }; } -function resolveRootSerializer(fory: Fory, bytes: Uint8Array): Serializer { +function resolveRootSerializer( + registeredFory: RegisteredFory, + bytes: Uint8Array, +): Serializer { + const { fory, serializers } = registeredFory; fory.readContext.reset(bytes); const reader = fory.readContext.reader; const bitmap = reader.readUint8(); @@ -155,15 +167,14 @@ function resolveRootSerializer(fory: Fory, bytes: Uint8Array): Serializer { // registered serializer when available. const detectedSerializer = AnyHelper.detectSerializer(fory.readContext); const resolvedSerializer = - fory.typeResolver.getSerializerByTypeInfo( - detectedSerializer.getTypeInfo(), - ) ?? detectedSerializer; + serializers.get(detectedSerializer.getTypeInfo().userTypeId) ?? + detectedSerializer; return resolvedSerializer; } function runFileRoundTrip( envVar: string, - fory: Fory, + registeredFory: RegisteredFory, expected: T, assertFn: AssertFn, ): void { @@ -173,7 +184,8 @@ function runFileRoundTrip( } console.log(`Processing ${envVar}: ${filePath}`); const payload = new Uint8Array(fs.readFileSync(filePath)); - const serializer = resolveRootSerializer(fory, payload); + const serializer = resolveRootSerializer(registeredFory, payload); + const { fory } = registeredFory; const decoded = fory.deserialize(payload, serializer); assertFn(expected, decoded); const roundTripBytes = fory.serialize(decoded, serializer); diff --git a/integration_tests/idl_tests/javascript/test/roundtrip.test.ts b/integration_tests/idl_tests/javascript/test/roundtrip.test.ts index 6c1208adb4..5c0378cb46 100644 --- a/integration_tests/idl_tests/javascript/test/roundtrip.test.ts +++ b/integration_tests/idl_tests/javascript/test/roundtrip.test.ts @@ -22,7 +22,7 @@ import Fory, { BoolArray, Decimal, ForyFloat16Array, - Type, + type Serializer, } from "@apache-fory/core"; import { AddressBook, @@ -79,10 +79,12 @@ import { import { Color, Monster, registerMonsterTypes } from "../generated/monster"; import { TreeNode, registerTreeTypes } from "../generated/tree"; -type RegisterFn = (fory: Fory) => unknown; -type RegisteredTypeInfo = - | ReturnType - | ReturnType; +type RegisterFn = (fory: Fory) => Record; + +interface RegisteredFory { + fory: Fory; + serializers: Map; +} const MODES = [ { title: "schema-consistent", compatible: false }, @@ -93,38 +95,28 @@ function buildFory( compatible: boolean, ref: boolean, registerFns: ReadonlyArray, -): Fory { +): RegisteredFory { const fory = new Fory({ compatible, ref }); + const serializers = new Map(); for (const registerFn of registerFns) { - registerFn(fory); - } - return fory; -} - -function getSerializer(fory: Fory, typeInfo: RegisteredTypeInfo) { - const serializer = fory.typeResolver.getSerializerByTypeInfo(typeInfo); - if (!serializer) { - throw new Error(`Missing serializer for type id ${typeInfo.typeId}`); + for (const { serializer } of Object.values(registerFn(fory))) { + serializers.set(serializer.getTypeInfo().userTypeId, serializer); + } } - return serializer; + return { fory, serializers }; } function roundTripValue( - fory: Fory, - typeInfo: RegisteredTypeInfo, + registeredFory: RegisteredFory, + typeId: number, value: T, ): unknown { - const serializer = getSerializer(fory, typeInfo); - const bytes = fory.serialize(value, serializer); - return fory.deserialize(bytes, serializer); -} - -function roundTripStruct(fory: Fory, typeId: number, value: T): unknown { - return roundTripValue(fory, Type.struct(typeId), value); -} - -function roundTripUnion(fory: Fory, typeId: number, value: T): unknown { - return roundTripValue(fory, Type.union(typeId), value); + const serializer = registeredFory.serializers.get(typeId); + if (!serializer) { + throw new Error(`Missing serializer for type id ${typeId}`); + } + const bytes = registeredFory.fory.serialize(value, serializer); + return registeredFory.fory.deserialize(bytes, serializer); } function normalize(value: unknown): unknown { @@ -652,7 +644,7 @@ describe.each(MODES)( expectAcyclicEqual( buildAddressBook(), - roundTripStruct(fory, 103, buildAddressBook()), + roundTripValue(fory, 103, buildAddressBook()), ); const dogAnimal: Animal = { @@ -663,8 +655,8 @@ describe.each(MODES)( case: AnimalCase.CAT, value: buildCat(), }; - expectAcyclicEqual(dogAnimal, roundTripUnion(fory, 106, dogAnimal)); - expectAcyclicEqual(catAnimal, roundTripUnion(fory, 106, catAnimal)); + expectAcyclicEqual(dogAnimal, roundTripValue(fory, 106, dogAnimal)); + expectAcyclicEqual(catAnimal, roundTripValue(fory, 106, catAnimal)); }); test("round-trips auto_id messages and root wrapper unions", () => { @@ -680,14 +672,14 @@ describe.each(MODES)( value: "raw-payload", }; - expectAcyclicEqual(envelope, roundTripStruct(fory, 3022445236, envelope)); + expectAcyclicEqual(envelope, roundTripValue(fory, 3022445236, envelope)); expectAcyclicEqual( wrapperEnvelope, - roundTripUnion(fory, 1471345060, wrapperEnvelope), + roundTripValue(fory, 1471345060, wrapperEnvelope), ); expectAcyclicEqual( wrapperRaw, - roundTripUnion(fory, 1471345060, wrapperRaw), + roundTripValue(fory, 1471345060, wrapperRaw), ); }); @@ -699,23 +691,23 @@ describe.each(MODES)( expectAcyclicEqual( buildPrimitiveTypes(), - roundTripStruct(fory, 200, buildPrimitiveTypes()), + roundTripValue(fory, 200, buildPrimitiveTypes()), ); expectAcyclicEqual( buildNumericCollections(), - roundTripStruct(fory, 210, buildNumericCollections()), + roundTripValue(fory, 210, buildNumericCollections()), ); expectAcyclicEqual( buildNumericCollectionsArray(), - roundTripStruct(fory, 212, buildNumericCollectionsArray()), + roundTripValue(fory, 212, buildNumericCollectionsArray()), ); expectAcyclicEqual( buildNumericCollectionUnion(), - roundTripUnion(fory, 211, buildNumericCollectionUnion()), + roundTripValue(fory, 211, buildNumericCollectionUnion()), ); expectAcyclicEqual( buildNumericCollectionArrayUnion(), - roundTripUnion(fory, 213, buildNumericCollectionArrayUnion()), + roundTripValue(fory, 213, buildNumericCollectionArrayUnion()), ); }); @@ -728,15 +720,15 @@ describe.each(MODES)( expectAcyclicEqual( buildMonster(), - roundTripStruct(flatbufferFory, 438716985, buildMonster()), + roundTripValue(flatbufferFory, 438716985, buildMonster()), ); expectAcyclicEqual( buildContainer(), - roundTripStruct(flatbufferFory, 372413680, buildContainer()), + roundTripValue(flatbufferFory, 372413680, buildContainer()), ); expectAcyclicEqual( buildOptionalHolder(), - roundTripStruct(flatbufferFory, 122, buildOptionalHolder()), + roundTripValue(flatbufferFory, 122, buildOptionalHolder()), ); }); @@ -745,15 +737,11 @@ describe.each(MODES)( expectAcyclicEqual( buildExampleMessage(), - roundTripValue( - fory, - Type.struct({ typeId: 1500, evolving: true }), - buildExampleMessage(), - ), + roundTripValue(fory, 1500, buildExampleMessage()), ); expectAcyclicEqual( buildExampleMessageUnion(), - roundTripUnion(fory, 1501, buildExampleMessageUnion()), + roundTripValue(fory, 1501, buildExampleMessageUnion()), ); }); }, @@ -765,13 +753,13 @@ describe.each(MODES)( test("round-trips tree and preserves shared-node topology", () => { const fory = buildFory(compatible, true, [registerTreeTypes]); const tree = buildTree(); - expectTreeEqual(tree, roundTripStruct(fory, 2251833438, tree)); + expectTreeEqual(tree, roundTripValue(fory, 2251833438, tree)); }); test("round-trips graph and preserves edge/node references", () => { const fory = buildFory(compatible, true, [registerGraphTypes]); const graph = buildGraph(); - expectGraphEqual(graph, roundTripStruct(fory, 2373163777, graph)); + expectGraphEqual(graph, roundTripValue(fory, 2373163777, graph)); }); }, ); diff --git a/java/fory-core/src/main/java/org/apache/fory/AbstractThreadSafeFory.java b/java/fory-core/src/main/java/org/apache/fory/AbstractThreadSafeFory.java index 0098854119..1333a3e9df 100644 --- a/java/fory-core/src/main/java/org/apache/fory/AbstractThreadSafeFory.java +++ b/java/fory-core/src/main/java/org/apache/fory/AbstractThreadSafeFory.java @@ -19,6 +19,7 @@ package org.apache.fory; +import java.util.function.Consumer; import java.util.function.Function; import org.apache.fory.resolver.TypeChecker; import org.apache.fory.resolver.TypeResolver; @@ -26,60 +27,68 @@ import org.apache.fory.serializer.SerializerFactory; public abstract class AbstractThreadSafeFory implements ThreadSafeFory { + private void applyRegistration(Consumer registration) { + registerCallback( + (fory, checkBeforePublication) -> { + checkBeforePublication.run(); + registration.accept(fory); + }); + } + @Override public void register(Class clz) { - registerCallback(fory -> fory.register(clz)); + applyRegistration(fory -> fory.register(clz)); } @Override public void register(Class cls, int id) { - registerCallback(fory -> fory.register(cls, id)); + applyRegistration(fory -> fory.register(cls, id)); } @Override public void register(Class cls, String name) { - registerCallback(fory -> fory.register(cls, name)); + applyRegistration(fory -> fory.register(cls, name)); } @Override public void register(Class cls, String namespace, String typeName) { - registerCallback(fory -> fory.register(cls, namespace, typeName)); + applyRegistration(fory -> fory.register(cls, namespace, typeName)); } @Override public void register(String className) { - registerCallback(fory -> fory.register(className)); + applyRegistration(fory -> fory.register(className)); } @Override public void register(String className, int id) { - registerCallback(fory -> fory.register(className, id)); + applyRegistration(fory -> fory.register(className, id)); } @Override public void register(String className, String name) { - registerCallback(fory -> fory.register(className, name)); + applyRegistration(fory -> fory.register(className, name)); } @Override public void register(String className, String namespace, String typeName) { - registerCallback(fory -> fory.register(className, namespace, typeName)); + applyRegistration(fory -> fory.register(className, namespace, typeName)); } @Override public void register(ForyModule module) { - registerCallback(fory -> fory.register(module)); + applyRegistration(fory -> fory.register(module)); } public void registerUnion( Class cls, int id, org.apache.fory.serializer.Serializer serializer) { - registerCallback(fory -> fory.registerUnion(cls, id, serializer)); + applyRegistration(fory -> fory.registerUnion(cls, id, serializer)); } @Override public void registerUnion( Class cls, String name, org.apache.fory.serializer.Serializer serializer) { - registerCallback(fory -> fory.registerUnion(cls, name, serializer)); + applyRegistration(fory -> fory.registerUnion(cls, name, serializer)); } public void registerUnion( @@ -87,50 +96,58 @@ public void registerUnion( String namespace, String typeName, org.apache.fory.serializer.Serializer serializer) { - registerCallback(fory -> fory.registerUnion(cls, namespace, typeName, serializer)); + applyRegistration(fory -> fory.registerUnion(cls, namespace, typeName, serializer)); } @Override public void registerSerializer(Class type, Class serializerClass) { - registerCallback(fory -> fory.registerSerializer(type, serializerClass)); + registerCallback( + (fory, checkBeforePublication) -> + fory.registerSerializer(type, serializerClass, checkBeforePublication)); } @Override public void registerSerializer(Class type, Serializer serializer) { - registerCallback(fory -> fory.registerSerializer(type, serializer)); + applyRegistration(fory -> fory.registerSerializer(type, serializer)); } @Override public void registerSerializer( Class type, Function> serializerCreator) { - registerCallback(fory -> fory.registerSerializer(type, serializerCreator)); + registerCallback( + (fory, checkBeforePublication) -> + fory.registerSerializer(type, serializerCreator, checkBeforePublication)); } @Override public void registerSerializerAndType( Class type, Class serializerClass) { - registerCallback(fory -> fory.registerSerializerAndType(type, serializerClass)); + registerCallback( + (fory, checkBeforePublication) -> + fory.registerSerializerAndType(type, serializerClass, checkBeforePublication)); } @Override public void registerSerializerAndType(Class type, Serializer serializer) { - registerCallback(fory -> fory.registerSerializerAndType(type, serializer)); + applyRegistration(fory -> fory.registerSerializerAndType(type, serializer)); } @Override public void registerSerializerAndType( Class type, Function> serializerCreator) { - registerCallback(fory -> fory.registerSerializerAndType(type, serializerCreator)); + registerCallback( + (fory, checkBeforePublication) -> + fory.registerSerializerAndType(type, serializerCreator, checkBeforePublication)); } @Override public void registerSerializerFactory(SerializerFactory serializerFactory) { - registerCallback(fory -> fory.registerSerializerFactory(serializerFactory)); + applyRegistration(fory -> fory.registerSerializerFactory(serializerFactory)); } @Override public void setTypeChecker(TypeChecker typeChecker) { - registerCallback(fory -> fory.getTypeResolver().setTypeChecker(typeChecker)); + applyRegistration(fory -> fory.getTypeResolver().setTypeChecker(typeChecker)); } @Override diff --git a/java/fory-core/src/main/java/org/apache/fory/BaseFory.java b/java/fory-core/src/main/java/org/apache/fory/BaseFory.java index c51ac2488c..f369903121 100644 --- a/java/fory-core/src/main/java/org/apache/fory/BaseFory.java +++ b/java/fory-core/src/main/java/org/apache/fory/BaseFory.java @@ -91,10 +91,9 @@ public interface BaseFory { /** * Register a runtime module. Direct {@link Fory} instances install the module immediately; - * thread-safe runtimes install it into every underlying runtime instance. - * - *

For thread-safe runtimes, call this during setup before concurrent serialization, - * deserialization, or copy operations start. + * thread-safe runtimes install it into every current and future underlying runtime instance. + * Module installation is registration-only setup and must not start root serialization or + * deserialization through the runtime or facade being configured. */ void register(ForyModule module); diff --git a/java/fory-core/src/main/java/org/apache/fory/Fory.java b/java/fory-core/src/main/java/org/apache/fory/Fory.java index a004eef527..e43c07c333 100644 --- a/java/fory-core/src/main/java/org/apache/fory/Fory.java +++ b/java/fory-core/src/main/java/org/apache/fory/Fory.java @@ -169,13 +169,11 @@ public Fory(ForyBuilder builder, ClassLoader classLoader, SharedRegistry sharedR @Override public void register(Class cls) { - checkRegisterAllowed(); getTypeResolver().register(cls); } @Override public void register(Class cls, int id) { - checkRegisterAllowed(); getTypeResolver().register(cls, Integer.toUnsignedLong(id)); } @@ -185,48 +183,43 @@ public void register(Class cls, int id) { */ @Override public void register(Class cls, String name) { - checkRegisterAllowed(); String[] parts = splitRegistrationName(name); register(cls, parts[0], parts[1]); } public void register(Class cls, String namespace, String typeName) { - checkRegisterAllowed(); getTypeResolver().register(cls, namespace, typeName); } @Override public void register(String className) { - checkRegisterAllowed(); getTypeResolver().register(className); } @Override public void register(String className, int classId) { - checkRegisterAllowed(); getTypeResolver().register(className, Integer.toUnsignedLong(classId)); } @Override public void register(String className, String name) { - checkRegisterAllowed(); String[] parts = splitRegistrationName(name); getTypeResolver().register(className, parts[0], parts[1]); } @Override public void register(String className, String namespace, String typeName) { - checkRegisterAllowed(); getTypeResolver().register(className, namespace, typeName); } + /** Installs a module into this runtime before its first root operation. */ @Override public void register(ForyModule module) { Preconditions.checkNotNull(module); + typeResolver.checkRegistrationOpen(); if (installedModules.containsKey(module)) { return; } - checkRegisterAllowed(); installedModules.put(module, Boolean.TRUE); try { module.install(this); @@ -238,13 +231,11 @@ public void register(ForyModule module) { @Override public void registerUnion(Class cls, int id, Serializer serializer) { - checkRegisterAllowed(); getTypeResolver().registerUnion(cls, Integer.toUnsignedLong(id), serializer); } @Override public void registerUnion(Class cls, String name, Serializer serializer) { - checkRegisterAllowed(); String[] parts = splitRegistrationName(name); getTypeResolver().registerUnion(cls, parts[0], parts[1], serializer); } @@ -252,52 +243,76 @@ public void registerUnion(Class cls, String name, Serializer serializer) { @Override public void registerUnion( Class cls, String namespace, String typeName, Serializer serializer) { - checkRegisterAllowed(); getTypeResolver().registerUnion(cls, namespace, typeName, serializer); } @Override public void registerSerializer(Class type, Class serializerClass) { - checkRegisterAllowed(); getTypeResolver().registerSerializer(type, serializerClass); } + void registerSerializer( + Class type, Class serializerClass, Runnable checkBeforePublication) { + getTypeResolver().registerSerializer(type, serializerClass, checkBeforePublication); + } + @Override public void registerSerializer(Class type, Serializer serializer) { - checkRegisterAllowed(); getTypeResolver().registerSerializer(type, serializer); } @Override public void registerSerializer( Class type, Function> serializerCreator) { - checkRegisterAllowed(); - getTypeResolver().registerSerializer(type, serializerCreator.apply(typeResolver)); + registerSerializer(type, serializerCreator, typeResolver::checkRegistrationOpen); + } + + void registerSerializer( + Class type, + Function> serializerCreator, + Runnable checkBeforePublication) { + typeResolver.checkRegistrationOpen(); + Serializer serializer = serializerCreator.apply(typeResolver); + // A facade root may freeze a different child while application construction is running. The + // facade owner must be rechecked before this child publishes the prepared serializer. + checkBeforePublication.run(); + getTypeResolver().registerSerializer(type, serializer); } @Override public void registerSerializerAndType( Class type, Class serializerClass) { - checkRegisterAllowed(); getTypeResolver().registerSerializerAndType(type, serializerClass); } + void registerSerializerAndType( + Class type, Class serializerClass, Runnable checkBeforePublication) { + getTypeResolver().registerSerializerAndType(type, serializerClass, checkBeforePublication); + } + @Override public void registerSerializerAndType(Class type, Serializer serializer) { - checkRegisterAllowed(); getTypeResolver().registerSerializerAndType(type, serializer); } @Override public void registerSerializerAndType( Class type, Function> serializerCreator) { - checkRegisterAllowed(); - getTypeResolver().registerSerializerAndType(type, serializerCreator.apply(typeResolver)); + registerSerializerAndType(type, serializerCreator, typeResolver::checkRegistrationOpen); + } + + void registerSerializerAndType( + Class type, + Function> serializerCreator, + Runnable checkBeforePublication) { + typeResolver.checkRegistrationOpen(); + Serializer serializer = serializerCreator.apply(typeResolver); + checkBeforePublication.run(); + getTypeResolver().registerSerializerAndType(type, serializer); } @Override public void registerSerializerFactory(SerializerFactory serializerFactory) { - checkRegisterAllowed(); typeResolver.registerSerializerFactory(serializerFactory); } @@ -306,30 +321,30 @@ public Serializer getSerializer(Class cls) { return typeResolver.getSerializer(cls); } - private void ensureRegistrationFinished() { - if (!typeResolver.isRegistrationFinished()) { - typeResolver.finishRegistration(); - } - } - @Override public byte[] serialize(Object obj) { - MemoryBuffer buf = getBuffer(); - buf.writerIndex(0); - serialize(buf, obj, null); - byte[] bytes = buf.getBytes(0, buf.writerIndex()); - resetBuffer(); - return bytes; + typeResolver.freezeRegistration(); + try { + MemoryBuffer buf = getBuffer(); + buf.writerIndex(0); + serializeRoot(buf, obj, null); + return buf.getBytes(0, buf.writerIndex()); + } finally { + resetBuffer(); + } } @Override public byte[] serialize(Object obj, BufferCallback callback) { - MemoryBuffer buf = getBuffer(); - buf.writerIndex(0); - serialize(buf, obj, callback); - byte[] bytes = buf.getBytes(0, buf.writerIndex()); - resetBuffer(); - return bytes; + typeResolver.freezeRegistration(); + try { + MemoryBuffer buf = getBuffer(); + buf.writerIndex(0); + serializeRoot(buf, obj, callback); + return buf.getBytes(0, buf.writerIndex()); + } finally { + resetBuffer(); + } } @Override @@ -339,7 +354,23 @@ public MemoryBuffer serialize(MemoryBuffer buffer, Object obj) { @Override public MemoryBuffer serialize(MemoryBuffer buffer, Object obj, BufferCallback callback) { - ensureRegistrationFinished(); + typeResolver.freezeRegistration(); + return serializeRoot(buffer, obj, callback); + } + + @Override + public void serialize(OutputStream outputStream, Object obj) { + typeResolver.freezeRegistration(); + serializeToStream(outputStream, buf -> serializeRoot(buf, obj, null)); + } + + @Override + public void serialize(OutputStream outputStream, Object obj, BufferCallback callback) { + typeResolver.freezeRegistration(); + serializeToStream(outputStream, buf -> serializeRoot(buf, obj, callback)); + } + + private MemoryBuffer serializeRoot(MemoryBuffer buffer, Object obj, BufferCallback callback) { writeContext.prepare(buffer, callback); try { byte bitmap = headerBitmap; @@ -364,16 +395,6 @@ public MemoryBuffer serialize(MemoryBuffer buffer, Object obj, BufferCallback ca } } - @Override - public void serialize(OutputStream outputStream, Object obj) { - serializeToStream(outputStream, buf -> serialize(buf, obj, null)); - } - - @Override - public void serialize(OutputStream outputStream, Object obj, BufferCallback callback) { - serializeToStream(outputStream, buf -> serialize(buf, obj, callback)); - } - private ForyException processSerializationError(Throwable e) { if (!config.trackingRef()) { String msg = @@ -410,48 +431,33 @@ private ForyException processCopyError(Throwable e) { @Override public Object deserialize(byte[] bytes) { - return deserialize(MemoryUtils.wrap(bytes), (Iterable) null); + typeResolver.freezeRegistration(); + return deserializeRoot(MemoryUtils.wrap(bytes), (Iterable) null); } @Override public Object deserialize(ByteBuffer byteBuffer) { - return deserialize(MemoryUtils.wrap(byteBuffer)); + typeResolver.freezeRegistration(); + return deserializeRoot(MemoryUtils.wrap(byteBuffer), (Iterable) null); } @Override public T deserialize(byte[] bytes, Class type) { - return deserialize(MemoryUtils.wrap(bytes), type); + typeResolver.freezeRegistration(); + return deserializeRoot(MemoryUtils.wrap(bytes), type); } @Override public T deserialize(MemoryBuffer buffer, Class type) { - ensureRegistrationFinished(); - byte bitmap = buffer.readByte(); - if (bitmap != headerBitmap) { - checkHeaderBitmapWithoutOutOfBand(bitmap); - } - readContext.prepare(buffer, null, false); - try { - try { - jitContext.lock(); - if (readContext.getDepth() > 0) { - throwDepthDeserializationException(); - } - return deserializeByType(buffer, type); - } finally { - jitContext.unlock(); - } - } catch (Throwable t) { - throw ExceptionUtils.handleReadFailed(this, t); - } finally { - readContext.reset(); - } + typeResolver.freezeRegistration(); + return deserializeRoot(buffer, type); } @Override public T deserialize(ForyInputStream inputStream, Class type) { + typeResolver.freezeRegistration(); try { - return deserialize(inputStream.getBuffer(), type); + return deserializeRoot(inputStream.getBuffer(), type); } finally { inputStream.shrinkBuffer(); } @@ -459,17 +465,24 @@ public T deserialize(ForyInputStream inputStream, Class type) { @Override public T deserialize(ForyReadableChannel channel, Class type) { - return deserialize(channel.getBuffer(), type); + typeResolver.freezeRegistration(); + try { + return deserializeRoot(channel.getBuffer(), type); + } finally { + channel.compactBuffer(); + } } @Override public Object deserialize(byte[] bytes, Iterable outOfBandBuffers) { - return deserialize(MemoryUtils.wrap(bytes), outOfBandBuffers); + typeResolver.freezeRegistration(); + return deserializeRoot(MemoryUtils.wrap(bytes), outOfBandBuffers); } @Override public Object deserialize(MemoryBuffer buffer) { - return deserialize(buffer, (Iterable) null); + typeResolver.freezeRegistration(); + return deserializeRoot(buffer, (Iterable) null); } /** @@ -487,7 +500,64 @@ public Object deserialize(MemoryBuffer buffer) { */ @Override public Object deserialize(MemoryBuffer buffer, Iterable outOfBandBuffers) { - ensureRegistrationFinished(); + typeResolver.freezeRegistration(); + return deserializeRoot(buffer, outOfBandBuffers); + } + + @Override + public Object deserialize(ForyInputStream inputStream) { + return deserialize(inputStream, (Iterable) null); + } + + @Override + public Object deserialize(ForyInputStream inputStream, Iterable outOfBandBuffers) { + typeResolver.freezeRegistration(); + try { + return deserializeRoot(inputStream.getBuffer(), outOfBandBuffers); + } finally { + inputStream.shrinkBuffer(); + } + } + + @Override + public Object deserialize(ForyReadableChannel channel) { + return deserialize(channel, (Iterable) null); + } + + @Override + public Object deserialize(ForyReadableChannel channel, Iterable outOfBandBuffers) { + typeResolver.freezeRegistration(); + try { + return deserializeRoot(channel.getBuffer(), outOfBandBuffers); + } finally { + channel.compactBuffer(); + } + } + + private T deserializeRoot(MemoryBuffer buffer, Class type) { + byte bitmap = buffer.readByte(); + if (bitmap != headerBitmap) { + checkHeaderBitmapWithoutOutOfBand(bitmap); + } + readContext.prepare(buffer, null, false); + try { + try { + jitContext.lock(); + if (readContext.getDepth() > 0) { + throwDepthDeserializationException(); + } + return deserializeByType(buffer, type); + } finally { + jitContext.unlock(); + } + } catch (Throwable t) { + throw ExceptionUtils.handleReadFailed(t); + } finally { + readContext.reset(); + } + } + + private Object deserializeRoot(MemoryBuffer buffer, Iterable outOfBandBuffers) { byte bitmap = buffer.readByte(); boolean peerOutOfBandEnabled = false; if (bitmap != headerBitmap) { @@ -517,38 +587,12 @@ public Object deserialize(MemoryBuffer buffer, Iterable outOfBandB jitContext.unlock(); } } catch (Throwable t) { - throw ExceptionUtils.handleReadFailed(this, t); + throw ExceptionUtils.handleReadFailed(t); } finally { readContext.reset(); } } - @Override - public Object deserialize(ForyInputStream inputStream) { - return deserialize(inputStream, (Iterable) null); - } - - @Override - public Object deserialize(ForyInputStream inputStream, Iterable outOfBandBuffers) { - try { - MemoryBuffer buf = inputStream.getBuffer(); - return deserialize(buf, outOfBandBuffers); - } finally { - inputStream.shrinkBuffer(); - } - } - - @Override - public Object deserialize(ForyReadableChannel channel) { - return deserialize(channel, (Iterable) null); - } - - @Override - public Object deserialize(ForyReadableChannel channel, Iterable outOfBandBuffers) { - MemoryBuffer buf = channel.getBuffer(); - return deserialize(buf, outOfBandBuffers); - } - @SuppressWarnings("unchecked") private T deserializeByType(MemoryBuffer buffer, Class type) { // The outer root operation resets generic state after failure; balance this push here only @@ -592,7 +636,6 @@ private boolean checkHeaderBitmap(byte bitmap) { @Override public T copy(T obj) { - ensureRegistrationFinished(); try { return copyContext.copyObject(obj); } catch (Throwable e) { @@ -605,8 +648,8 @@ public T copy(T obj) { private void serializeToStream(OutputStream outputStream, Consumer function) { MemoryBuffer buf = getBuffer(); buf.writerIndex(0); - function.accept(buf); try { + function.accept(buf); byte[] bytes = buf.getHeapMemory(); if (bytes != null) { outputStream.write(bytes, 0, buf.writerIndex()); @@ -701,15 +744,6 @@ SharedRegistry getSharedRegistry() { return sharedRegistry; } - private void checkRegisterAllowed() { - if (typeResolver.isRegistrationFinished()) { - throw new ForyException( - "Cannot register class/serializer after registration has been frozen. Please register " - + "all classes before invoking top-level `serialize/deserialize/copy` methods of " - + "Fory."); - } - } - public Config getConfig() { return config; } diff --git a/java/fory-core/src/main/java/org/apache/fory/ForyModule.java b/java/fory-core/src/main/java/org/apache/fory/ForyModule.java index 0ccae89e78..eedf620f84 100644 --- a/java/fory-core/src/main/java/org/apache/fory/ForyModule.java +++ b/java/fory-core/src/main/java/org/apache/fory/ForyModule.java @@ -22,6 +22,11 @@ /** A reusable Fory runtime module installed during or after runtime construction. */ @FunctionalInterface public interface ForyModule { - /** Install this module into the concrete runtime. */ + /** + * Installs registration setup into the concrete runtime. + * + *

An installation may register nested modules and child-specific serializers, but it must not + * start a root through {@code fory} or the direct or thread-safe facade installing the module. + */ void install(Fory fory); } diff --git a/java/fory-core/src/main/java/org/apache/fory/ThreadLocalFory.java b/java/fory-core/src/main/java/org/apache/fory/ThreadLocalFory.java index 57f0d947dc..5aab882ba3 100644 --- a/java/fory-core/src/main/java/org/apache/fory/ThreadLocalFory.java +++ b/java/fory-core/src/main/java/org/apache/fory/ThreadLocalFory.java @@ -24,7 +24,7 @@ import java.util.Collections; import java.util.Map; import java.util.WeakHashMap; -import java.util.function.Consumer; +import java.util.function.BiConsumer; import java.util.function.Function; import java.util.function.Supplier; import javax.annotation.concurrent.ThreadSafe; @@ -45,14 +45,14 @@ public class ThreadLocalFory extends AbstractThreadSafeFory { private final Supplier foryFactory; private final ThreadLocal foryThreadLocal; - private Consumer factoryCallback; + private BiConsumer factoryCallback; private final Map allFory; - private final Object callbackLock = new Object(); + private final SharedRegistry sharedRegistry; public ThreadLocalFory(Function factory) { - SharedRegistry sharedRegistry = new SharedRegistry(); + sharedRegistry = new SharedRegistry(); foryFactory = () -> factory.apply(Fory.builder().withSharedRegistry(sharedRegistry)); - factoryCallback = f -> {}; + factoryCallback = (fory, checkBeforePublication) -> {}; allFory = Collections.synchronizedMap(new WeakHashMap<>()); foryThreadLocal = ThreadLocal.withInitial(this::newFory); // 1. init and warm for current thread. @@ -63,24 +63,35 @@ public ThreadLocalFory(Function factory) { } private Fory newFory() { - synchronized (callbackLock) { + synchronized (sharedRegistry) { Fory fory = foryFactory.get(); - factoryCallback.accept(fory); + // The facade may already be frozen, but this child is not exposed yet. Replay uses its local + // owner, then freezeRegistration adopts the facade's published snapshot before exposure. + factoryCallback.accept(fory, fory.getTypeResolver()::checkRegistrationOpen); + if (sharedRegistry.isRegistrationFrozen()) { + fory.getTypeResolver().freezeRegistration(); + } allFory.put(fory, null); return fory; } } private Fory currentFory() { + sharedRegistry.freezeRegistration(); return foryThreadLocal.get(); } @Internal @Override - public void registerCallback(Consumer callback) { - synchronized (callbackLock) { + public void registerCallback(BiConsumer callback) { + synchronized (sharedRegistry) { + sharedRegistry.checkRegistrationOpen(); + Runnable publicationCheck = sharedRegistry::checkRegistrationOpen; synchronized (allFory) { - allFory.keySet().forEach(callback); + for (Fory fory : allFory.keySet()) { + callback.accept(fory, publicationCheck); + sharedRegistry.checkRegistrationOpen(); + } } factoryCallback = factoryCallback.andThen(callback); } @@ -88,7 +99,11 @@ public void registerCallback(Consumer callback) { @Override public R execute(Function action) { - return action.apply(currentFory()); + Fory fory = foryThreadLocal.get(); + if (sharedRegistry.isRegistrationFrozen() && !fory.getTypeResolver().isRegistrationFrozen()) { + fory.getTypeResolver().freezeRegistration(); + } + return action.apply(fory); } @Override @@ -188,6 +203,6 @@ public Object deserialize(ForyReadableChannel channel, Iterable ou @Override public T copy(T obj) { - return currentFory().copy(obj); + return foryThreadLocal.get().copy(obj); } } diff --git a/java/fory-core/src/main/java/org/apache/fory/ThreadSafeFory.java b/java/fory-core/src/main/java/org/apache/fory/ThreadSafeFory.java index 1f2c1f3f20..407e2ba723 100644 --- a/java/fory-core/src/main/java/org/apache/fory/ThreadSafeFory.java +++ b/java/fory-core/src/main/java/org/apache/fory/ThreadSafeFory.java @@ -19,7 +19,7 @@ package org.apache.fory; -import java.util.function.Consumer; +import java.util.function.BiConsumer; import java.util.function.Function; import org.apache.fory.annotation.Internal; import org.apache.fory.resolver.TypeChecker; @@ -30,22 +30,27 @@ * *

The runtime class loader is fixed when the thread-safe serializer is built. If you need a * different class loader, build a different {@link ThreadSafeFory} instance. + * + *

Complete facade registration before concurrent serialization, deserialization, copy, or {@link + * #execute} calls begin. */ public interface ThreadSafeFory extends BaseFory { /** * Provide a context to execution operations on {@link Fory} directly and return the executed - * result. + * result. The action must not retain the runtime or register through it; use this facade's + * registration methods during setup so every underlying runtime receives the same registration. */ R execute(Function action); /** - * Set TypeChecker of serializer for current thread only. + * Set the TypeChecker for all current and future underlying runtimes before the first root. * * @param typeChecker {@link TypeChecker} for type checking */ void setTypeChecker(TypeChecker typeChecker); + /** Applies registration to current and future children using the supplied owner check. */ @Internal - void registerCallback(Consumer callback); + void registerCallback(BiConsumer callback); } diff --git a/java/fory-core/src/main/java/org/apache/fory/builder/StaticCompatibleCodecBuilder.java b/java/fory-core/src/main/java/org/apache/fory/builder/StaticCompatibleCodecBuilder.java index 1069cf571a..1cee849580 100644 --- a/java/fory-core/src/main/java/org/apache/fory/builder/StaticCompatibleCodecBuilder.java +++ b/java/fory-core/src/main/java/org/apache/fory/builder/StaticCompatibleCodecBuilder.java @@ -239,10 +239,17 @@ private String genRecordCompatibleRead() { Code.ExprCode newRecord = new Invoke(generatedObjectInstantiator(), "newInstanceWithArguments", OBJECT_TYPE, values) .genCode(ctx); + code.append("Object _f_record;\n").append("try {\n"); if (StringUtils.isNotBlank(newRecord.code())) { - code.append(newRecord.code()).append('\n'); - } - code.append("Object _f_record = ").append(newRecord.value()).append(";\n"); + code.append(indent(newRecord.code(), 2)).append('\n'); + } + code.append(" _f_record = ") + .append(newRecord.value()) + .append(";\n") + .append("} catch (Throwable _f_error) {\n") + .append(" java.util.Arrays.fill(_f_recordArgs, null);\n") + .append(" throw org.apache.fory.util.ExceptionUtils.throwException(_f_error);\n") + .append("}\n"); for (int i = 0; i < components.length; i++) { code.append("_f_recordArgs[").append(i).append("] = null;\n"); } diff --git a/java/fory-core/src/main/java/org/apache/fory/context/MapRefReader.java b/java/fory-core/src/main/java/org/apache/fory/context/MapRefReader.java index 59101b61ac..bc4a1a3ef8 100644 --- a/java/fory-core/src/main/java/org/apache/fory/context/MapRefReader.java +++ b/java/fory-core/src/main/java/org/apache/fory/context/MapRefReader.java @@ -124,11 +124,6 @@ public void setReadRef(int id, Object object) { } } - /** Exposes the resolved read-reference table for debugging and focused tests. */ - public ObjectArray getReadRefs() { - return readObjects; - } - /** Clears the current read state and keeps an approximate capacity for the next operation. */ @Override public void reset() { diff --git a/java/fory-core/src/main/java/org/apache/fory/context/MetaReadContext.java b/java/fory-core/src/main/java/org/apache/fory/context/MetaReadContext.java index e26e4ce0a3..119695aa7c 100644 --- a/java/fory-core/src/main/java/org/apache/fory/context/MetaReadContext.java +++ b/java/fory-core/src/main/java/org/apache/fory/context/MetaReadContext.java @@ -29,9 +29,24 @@ * type definitions announced by the peer remain available for later payloads. */ public class MetaReadContext { + private static final int MAX_RETAINED_TYPE_INFOS = 8192; + private static final int RESET_TYPE_INFO_CAPACITY = 8; + /** * Type infos announced by the peer, indexed by the protocol id assigned during the current or * shared meta-share session. */ public final ObjectArray readTypeInfos = new ObjectArray<>(); + + void reset() { + ObjectArray typeInfos = readTypeInfos; + int size = typeInfos.size; + // The current size is the protocol visibility boundary, so stale slots cannot be referenced + // by a later root. Keep bounded tables intact to make normal root cleanup allocation-free, and + // discard only an oversized backing array retained by an unusually metadata-heavy root. + typeInfos.size = 0; + if (size > MAX_RETAINED_TYPE_INFOS) { + typeInfos.objects = new Object[RESET_TYPE_INFO_CAPACITY]; + } + } } diff --git a/java/fory-core/src/main/java/org/apache/fory/context/ReadContext.java b/java/fory-core/src/main/java/org/apache/fory/context/ReadContext.java index 4d42074c7c..ab10f74fdf 100644 --- a/java/fory-core/src/main/java/org/apache/fory/context/ReadContext.java +++ b/java/fory-core/src/main/java/org/apache/fory/context/ReadContext.java @@ -111,9 +111,13 @@ public ReadContext( */ public void prepare( MemoryBuffer buffer, Iterable outOfBandBuffers, boolean peerOutOfBandEnabled) { + // Resolve user code before publishing root state so a failing iterator cannot leave a + // partially prepared context outside the root cleanup boundary. + Iterator outOfBandIterator = + outOfBandBuffers == null ? null : outOfBandBuffers.iterator(); this.buffer = buffer; this.peerOutOfBandEnabled = peerOutOfBandEnabled; - this.outOfBandBuffers = outOfBandBuffers == null ? null : outOfBandBuffers.iterator(); + this.outOfBandBuffers = outOfBandIterator; remainingGraphMemoryBytes = config.maxGraphMemoryBytes(); remainingUnbackedContainerItems = config.maxUnbackedContainerItems(); } @@ -300,7 +304,7 @@ public void reset() { contextObjects.clear(); } if (scopedMetaShareEnabled) { - metaReadContext.readTypeInfos.size = 0; + metaReadContext.reset(); } else { metaReadContext = null; } diff --git a/java/fory-core/src/main/java/org/apache/fory/exception/DeserializationException.java b/java/fory-core/src/main/java/org/apache/fory/exception/DeserializationException.java index 5cd154c672..82556a9a45 100644 --- a/java/fory-core/src/main/java/org/apache/fory/exception/DeserializationException.java +++ b/java/fory-core/src/main/java/org/apache/fory/exception/DeserializationException.java @@ -19,13 +19,8 @@ package org.apache.fory.exception; -import java.util.List; - /** Exception thrown when a deserialization operation fails. */ public class DeserializationException extends ForyException { - - private transient List readObjects; - public DeserializationException(String message) { super(message); } @@ -37,30 +32,4 @@ public DeserializationException(Throwable cause) { public DeserializationException(String message, Throwable cause) { super(message, cause); } - - // if `readObjects` too big, generate message lazily to avoid big string creation cost. - public DeserializationException(List readObjects, Throwable cause) { - super(cause); - this.readObjects = readObjects; - } - - @Override - public String getMessage() { - if (readObjects == null) { - return super.getMessage(); - } else { - try { - return "Deserialize failed, read objects are: " + readObjects; - } catch (Throwable e) { - StringBuilder builder = - new StringBuilder("Deserialize failed, type of read objects are: ["); - for (Object readObject : readObjects) { - builder.append(readObject == null ? null : readObject.getClass()).append(", "); - } - builder.delete(builder.length() - 2, builder.length()); - builder.append("]"); - return builder.toString(); - } - } - } } diff --git a/java/fory-core/src/main/java/org/apache/fory/io/BlockedStreamUtils.java b/java/fory-core/src/main/java/org/apache/fory/io/BlockedStreamUtils.java index deb03160c2..a8ecfa53ae 100644 --- a/java/fory-core/src/main/java/org/apache/fory/io/BlockedStreamUtils.java +++ b/java/fory-core/src/main/java/org/apache/fory/io/BlockedStreamUtils.java @@ -47,11 +47,13 @@ public class BlockedStreamUtils { private static final int MAX_CONSECUTIVE_ZERO_READS = 100; public static void serialize(Fory fory, OutputStream outputStream, Object obj) { + fory.getTypeResolver().freezeRegistration(); serializeToStream(fory, outputStream, buf -> fory.serialize(buf, obj, null)); } public static void serialize( Fory fory, OutputStream outputStream, Object obj, BufferCallback callback) { + fory.getTypeResolver().freezeRegistration(); serializeToStream(fory, outputStream, buf -> fory.serialize(buf, obj, callback)); } @@ -84,6 +86,7 @@ public static T deserialize(Fory fory, ReadableByteChannel channel, Class private static Object readFromChannel( Fory fory, ReadableByteChannel channel, Function action) { + fory.getTypeResolver().freezeRegistration(); try { MemoryBuffer buf = fory.getBuffer(); // resetBuffer may shrink the reusable buffer below the fixed frame header size. @@ -94,7 +97,7 @@ private static Object readFromChannel( readFrameBody(channel, buf, size); return action.apply(buf.slice(0, size)); } catch (Throwable t) { - throw ExceptionUtils.handleReadFailed(fory, t); + throw ExceptionUtils.handleReadFailed(t); } finally { fory.resetBuffer(); } @@ -153,12 +156,13 @@ private static void serializeToStream( private static Object deserializeFromStream( Fory fory, InputStream inputStream, Function function) { + fory.getTypeResolver().freezeRegistration(); MemoryBuffer buf = fory.getBuffer(); try { MemoryBuffer frame = readToBufferFromStream(inputStream, buf); return function.apply(frame); } catch (Throwable t) { - throw ExceptionUtils.handleReadFailed(fory, t); + throw ExceptionUtils.handleReadFailed(t); } finally { fory.resetBuffer(); } diff --git a/java/fory-core/src/main/java/org/apache/fory/io/ForyReadableChannel.java b/java/fory-core/src/main/java/org/apache/fory/io/ForyReadableChannel.java index ab15f90206..78a5f99d86 100644 --- a/java/fory-core/src/main/java/org/apache/fory/io/ForyReadableChannel.java +++ b/java/fory-core/src/main/java/org/apache/fory/io/ForyReadableChannel.java @@ -24,6 +24,7 @@ import java.nio.channels.ReadableByteChannel; import java.nio.channels.SeekableByteChannel; import javax.annotation.concurrent.NotThreadSafe; +import org.apache.fory.annotation.Internal; import org.apache.fory.exception.DeserializationException; import org.apache.fory.memory.MemoryBuffer; import org.apache.fory.platform.AndroidSupport; @@ -292,6 +293,35 @@ public MemoryBuffer getBuffer() { return memoryBuffer; } + /** Discards consumed bytes while preserving unread bytes prefetched for the next root. */ + @Internal + public void compactBuffer() { + MemoryBuffer memoryBuf = memoryBuffer; + int readerIndex = memoryBuf.readerIndex(); + if (readerIndex == 0) { + return; + } + int unreadBytes = memoryBuf.remaining(); + // Retaining a smaller consumed prefix is cheaper than copying a larger unread suffix. Once the + // prefix reaches the suffix size, compaction keeps both buffer growth and byte movement + // amortized across roots. + if (readerIndex < unreadBytes) { + return; + } + ByteBuffer byteBuf = byteBuffer; + // A read method may compute its post-fill absolute cursor before invoking fillBuffer, so moving + // bytes during a fill invalidates that pending cursor. The root cleanup boundary owns + // compaction and still preserves bytes prefetched from the following root. + int dataEnd = byteBuf.position(); + int dataStart = dataEnd - memoryBuf.size(); + byteBuf.limit(dataEnd); + byteBuf.position(dataStart + readerIndex); + byteBuf.compact(); + byteBuf.limit(unreadBytes); + memoryBuf.initByteBuffer(byteBuf, unreadBytes); + memoryBuf.readerIndex(0); + } + private void readFully(ByteBuffer dst, int length) throws IOException { int remaining = length; while (remaining > 0) { diff --git a/java/fory-core/src/main/java/org/apache/fory/pool/ThreadPoolFory.java b/java/fory-core/src/main/java/org/apache/fory/pool/ThreadPoolFory.java index 04971ae20b..9c3d2c15fd 100644 --- a/java/fory-core/src/main/java/org/apache/fory/pool/ThreadPoolFory.java +++ b/java/fory-core/src/main/java/org/apache/fory/pool/ThreadPoolFory.java @@ -24,7 +24,7 @@ import java.util.concurrent.Semaphore; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReferenceArray; -import java.util.function.Consumer; +import java.util.function.BiConsumer; import java.util.function.Function; import java.util.function.Supplier; import javax.annotation.concurrent.ThreadSafe; @@ -52,14 +52,14 @@ public class ThreadPoolFory extends AbstractThreadSafeFory { private final Fory[] pooledFory; private final Semaphore waiterSignal = new Semaphore(0); private final AtomicInteger waitingBorrowers = new AtomicInteger(); - private final Object callbackLock = new Object(); + private final SharedRegistry sharedRegistry; public ThreadPoolFory(Function foryFactory, int poolSize) { if (poolSize <= 0) { throw new IllegalArgumentException( String.format("thread safe fory pool size error, please check it, size:[%s]", poolSize)); } - SharedRegistry sharedRegistry = new SharedRegistry(); + sharedRegistry = new SharedRegistry(); Supplier factory = () -> foryFactory.apply(Fory.builder().withSharedRegistry(sharedRegistry)); this.poolSize = poolSize; @@ -73,6 +73,11 @@ public ThreadPoolFory(Function foryFactory, int poolSize) { } private PooledEntry acquire() { + sharedRegistry.freezeRegistration(); + return acquireEntry(); + } + + private PooledEntry acquireEntry() { int slotIndex = slotIndexForCurrentThread(); PooledEntry entry = tryBorrowPreferredSlots(slotIndex); if (entry != null) { @@ -146,18 +151,25 @@ private static int spread(int hash) { @Internal @Override - public void registerCallback(Consumer callback) { - synchronized (callbackLock) { + public void registerCallback(BiConsumer callback) { + synchronized (sharedRegistry) { + sharedRegistry.checkRegistrationOpen(); + Runnable publicationCheck = sharedRegistry::checkRegistrationOpen; for (Fory fory : pooledFory) { - callback.accept(fory); + callback.accept(fory, publicationCheck); + sharedRegistry.checkRegistrationOpen(); } } } @Override public R execute(Function action) { - PooledEntry entry = acquire(); + PooledEntry entry = acquireEntry(); try { + if (sharedRegistry.isRegistrationFrozen() + && !entry.fory.getTypeResolver().isRegistrationFrozen()) { + entry.fory.getTypeResolver().freezeRegistration(); + } return action.apply(entry.fory); } finally { release(entry); @@ -356,7 +368,7 @@ public Object deserialize(ForyReadableChannel channel, Iterable ou @Override public T copy(T obj) { - PooledEntry entry = acquire(); + PooledEntry entry = acquireEntry(); try { return entry.fory.copy(obj); } finally { diff --git a/java/fory-core/src/main/java/org/apache/fory/resolver/AllowListChecker.java b/java/fory-core/src/main/java/org/apache/fory/resolver/AllowListChecker.java index 235e58850e..d2e645b5c5 100644 --- a/java/fory-core/src/main/java/org/apache/fory/resolver/AllowListChecker.java +++ b/java/fory-core/src/main/java/org/apache/fory/resolver/AllowListChecker.java @@ -250,11 +250,6 @@ private void disallow(String classNameOrPrefix) { void addListener(TypeResolver resolver) { try { lock.writeLock().lock(); - if ((!disallowList.isEmpty() || !disallowListPrefix.isEmpty()) - && resolver.isRegistrationFinished()) { - throw new IllegalStateException( - "A checker with disallow entries cannot be installed after registration."); - } listeners.put(resolver, true); } finally { lock.writeLock().unlock(); @@ -272,7 +267,7 @@ void removeListener(TypeResolver resolver) { private void checkRegistrationOpen() { for (TypeResolver resolver : listeners.keySet()) { - if (resolver.isRegistrationFinished()) { + if (resolver.isRegistrationFrozen()) { throw new IllegalStateException("Classes cannot be disallowed after registration."); } } diff --git a/java/fory-core/src/main/java/org/apache/fory/resolver/ClassResolver.java b/java/fory-core/src/main/java/org/apache/fory/resolver/ClassResolver.java index b347eb3865..72e4b274d5 100644 --- a/java/fory-core/src/main/java/org/apache/fory/resolver/ClassResolver.java +++ b/java/fory-core/src/main/java/org/apache/fory/resolver/ClassResolver.java @@ -267,6 +267,7 @@ private void clearTypeInfoCache() { @Override public void initialize() { + checkRegistrationOpen(); extRegistry.objectGenericType = buildGenericType(OBJECT_TYPE); registerInternal(LambdaSerializer.ReplaceStub.class, LAMBDA_STUB_ID); registerInternal(JdkProxySerializer.ReplaceStub.class, JDK_PROXY_STUB_ID); @@ -479,6 +480,7 @@ private void registerDefaultClasses() { */ @Override public void register(Class cls) { + checkRegistrationOpen(); if (!extRegistry.registeredClassIdMap.containsKey(cls)) { while (containsUserTypeId(extRegistry.userIdGenerator)) { extRegistry.userIdGenerator++; @@ -495,6 +497,7 @@ public void register(Class cls) { */ @Override public void register(String className) { + checkRegistrationOpen(); register(loadClassFromLoader(className)); } @@ -507,6 +510,7 @@ public void register(String className) { */ @Override public void register(String className, long classId) { + checkRegistrationOpen(); register(loadClassFromLoader(className), classId); } @@ -522,6 +526,7 @@ public void register(String className, long classId) { */ @Override public void register(Class cls, long id) { + checkRegistrationOpen(); registerUserImpl(cls, toUserTypeId(id)); } @@ -532,7 +537,7 @@ public void register(Class cls, long id) { */ @Override public void register(Class cls, String namespace, String name) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkArgument(!Functions.isLambda(cls)); Preconditions.checkArgument(!ReflectionUtils.isJdkProxy(cls)); Preconditions.checkArgument(!cls.isArray()); @@ -565,7 +570,7 @@ public void register(Class cls, String namespace, String name) { @Override public void registerUnion(Class cls, long userId, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); int checkedUserId = toUserTypeId(userId); Preconditions.checkNotNull(serializer); checkRegistration(cls, checkedUserId, cls.getName(), false); @@ -585,7 +590,7 @@ public void registerUnion(Class cls, long userId, Serializer serializer) { @Override public void registerUnion(Class cls, String namespace, String name, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkNotNull(serializer); Preconditions.checkArgument(!Functions.isLambda(cls)); Preconditions.checkArgument(!ReflectionUtils.isJdkProxy(cls)); @@ -614,7 +619,7 @@ public void registerUnion(Class cls, String namespace, String name, Serialize @Override public void registerEnum(Class cls, long userId, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); int checkedUserId = toUserTypeId(userId); Preconditions.checkNotNull(serializer); checkRegistration(cls, checkedUserId, cls.getName(), false); @@ -633,7 +638,7 @@ public void registerEnum(Class cls, long userId, Serializer serializer) { @Override public void registerEnum(Class cls, String namespace, String name, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkNotNull(serializer); Preconditions.checkArgument(!Functions.isLambda(cls)); Preconditions.checkArgument(!ReflectionUtils.isJdkProxy(cls)); @@ -668,6 +673,7 @@ public void registerEnum(Class cls, String namespace, String name, Serializer * @param classes the classes to register */ public void registerInternal(Class... classes) { + checkRegistrationOpen(); for (Class cls : classes) { registerInternal(cls); } @@ -682,6 +688,7 @@ public void registerInternal(Class... classes) { * @param cls the class to register */ public void registerInternal(Class cls) { + checkRegistrationOpen(); if (!extRegistry.registeredClassIdMap.containsKey(cls)) { Preconditions.checkArgument( extRegistry.classIdGenerator < INTERNAL_NATIVE_ID_LIMIT, @@ -712,12 +719,13 @@ public void registerInternal(Class cls) { * @throws IllegalArgumentException if the ID is out of range or already in use */ public void registerInternal(Class cls, int classId) { + checkRegistrationOpen(); Preconditions.checkArgument(classId >= 0 && classId < INTERNAL_NATIVE_ID_LIMIT); registerInternalImpl(cls, classId); } private void registerInternalImpl(Class cls, int typeId) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkArgument(typeId >= 0 && typeId < INTERNAL_NATIVE_ID_LIMIT); checkRegistration(cls, typeId, cls.getName(), true); extRegistry.registeredClassIdMap.put(cls, typeId); @@ -733,7 +741,7 @@ private void registerInternalImpl(Class cls, int typeId) { } private void registerUserImpl(Class cls, int userId) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkArgument(userId != -1, "User type id 0xffffffff is reserved"); checkRegistration(cls, userId, cls.getName(), false); extRegistry.registeredClassIdMap.put(cls, userId); @@ -1216,18 +1224,36 @@ public static boolean requireJavaSerialization(Class clz) { * @param type of class */ public void registerSerializer(Class type, Class serializerClass) { - checkRegisterAllowed(); + registerSerializer(type, serializerClass, this::checkRegistrationOpen); + } + + @Override + public void registerSerializer( + Class type, Class serializerClass, Runnable checkBeforePublication) { + checkRegistrationOpen(); checkSerializerRegistration(type, serializerClass); - registerSerializerImpl(type, Serializers.newSerializer(this, type, serializerClass)); + Serializer serializer = Serializers.newSerializer(this, type, serializerClass); + checkBeforePublication.run(); + registerSerializerImpl(type, serializer); } @Override public void registerSerializer(Class type, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); checkSerializerRegistration(type, serializer.getClass()); registerSerializerImpl(type, serializer); } + @Override + public void registerSerializerAndType( + Class type, Class serializerClass, Runnable checkBeforePublication) { + checkRegistrationOpen(); + checkSerializerRegistration(type, serializerClass); + Serializer serializer = Serializers.newSerializer(this, type, serializerClass); + checkBeforePublication.run(); + registerSerializerAndType(type, serializer); + } + /** * If a serializer exists before, it will be replaced by new serializer. * @@ -1236,6 +1262,7 @@ public void registerSerializer(Class type, Serializer serializer) { */ @Override public void registerInternalSerializer(Class type, Serializer serializer) { + checkRegistrationOpen(); Integer classId = extRegistry.registeredClassIdMap.get(type); if (classId != null && !isInternalRegisteredClassId(type, classId)) { throw new IllegalArgumentException( @@ -1258,7 +1285,7 @@ public void registerInternalSerializer(Class type, Serializer serializer) } private void registerSerializerImpl(Class type, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); // Serializer registration trusts the Java name, but must not replace an existing custom name. if (extRegistry.registeredClasses.inverse().get(type) == null) { extRegistry.registeredClasses.put(type.getName(), type); diff --git a/java/fory-core/src/main/java/org/apache/fory/resolver/SharedRegistry.java b/java/fory-core/src/main/java/org/apache/fory/resolver/SharedRegistry.java index 1a95d73529..e4def43948 100644 --- a/java/fory-core/src/main/java/org/apache/fory/resolver/SharedRegistry.java +++ b/java/fory-core/src/main/java/org/apache/fory/resolver/SharedRegistry.java @@ -102,6 +102,10 @@ public final class SharedRegistry { final StaticGeneratedSerializerRegistry staticGeneratedSerializerRegistry = new StaticGeneratedSerializerRegistry(); private final Object metaStringCacheLock = new Object(); + // Thread-safe facades share this boundary across their children. A new thread-local child first + // receives the facade's fixed setup, then switches to the shared registration snapshot before it + // is exposed. + private volatile boolean registrationFrozen; private volatile int maxSchemaVersionsPerType = -1; private volatile int maxAverageSchemaVersionsPerType = -1; private final HashMap remoteTypeDefVersionsByType = new HashMap<>(); @@ -111,6 +115,27 @@ public final class SharedRegistry { public SharedRegistry() {} + public boolean isRegistrationFrozen() { + return registrationFrozen; + } + + public void freezeRegistration() { + if (!registrationFrozen) { + synchronized (this) { + registrationFrozen = true; + } + } + } + + public void checkRegistrationOpen() { + if (registrationFrozen) { + throw new ForyException( + "Cannot register class/serializer after registration has been frozen. Please register " + + "all classes before invoking top-level `serialize/deserialize` methods of " + + "ThreadSafeFory."); + } + } + public synchronized void setRemoteSchemaLimits( int maxSchemaVersionsPerType, int maxAverageSchemaVersionsPerType) { if (maxSchemaVersionsPerType <= 0) { diff --git a/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java b/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java index 60b4e7e476..5b6e9d8a5c 100644 --- a/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java +++ b/java/fory-core/src/main/java/org/apache/fory/resolver/TypeResolver.java @@ -155,7 +155,7 @@ private static final class TransformedTypeInfo { // Caches for readTypeInfo(ReadContext) - persist between calls to avoid reloading // dynamically created classes that can't be found by Class.forName private final TypeInfo[] typeInfoCache; - private boolean registrationFinished; + private boolean registrationFrozen; protected TypeResolver( Config config, @@ -193,12 +193,9 @@ public final JITContext getJITContext() { return jitContext; } - public final boolean isRegistrationFinished() { - return registrationFinished; - } - - protected final void setRegistrationFinished() { - registrationFinished = true; + @Internal + public final boolean isRegistrationFrozen() { + return registrationFrozen; } public final boolean isCrossLanguage() { @@ -229,12 +226,12 @@ public final Class getDefaultJDKStreamSerializerType() { return config.getDefaultJDKStreamSerializerType(); } - protected final void checkRegisterAllowed() { - if (registrationFinished) { + @Internal + public final void checkRegistrationOpen() { + if (isRegistrationFrozen()) { throw new ForyException( "Cannot register class/serializer after registration has been frozen. Please register " - + "all classes before invoking top-level `serialize/deserialize/copy` methods of " - + "Fory."); + + "all classes before invoking top-level `serialize/deserialize` methods of Fory."); } } @@ -265,11 +262,13 @@ protected final void checkRegisterAllowed() { /** Registers a class by name with an auto-assigned user ID. */ public void register(String className) { + checkRegistrationOpen(); register(loadClassFromLoader(className)); } /** Registers a class by name with a user-specified ID. */ public void register(String className, long classId) { + checkRegistrationOpen(); register(loadClassFromLoader(className), classId); } @@ -277,6 +276,7 @@ public void register(String className, long classId) { * Registers a class by name with a namespace and type name. The type name must not contain `.`. */ public void register(String className, String namespace, String typeName) { + checkRegistrationOpen(); register(loadClassFromLoader(className), namespace, typeName); } @@ -290,12 +290,12 @@ public void register(String className, String namespace, String typeName) { */ @Internal public final void registerRuntimeTypeAlias(Class runtimeType, Class canonicalType) { + checkRegistrationOpen(); Preconditions.checkNotNull(runtimeType, "runtimeType"); Preconditions.checkNotNull(canonicalType, "canonicalType"); if (runtimeType == canonicalType) { return; } - checkRegisterAllowed(); TypeInfo canonicalInfo = classInfoMap.get(canonicalType); Preconditions.checkArgument( canonicalInfo != null, @@ -378,6 +378,16 @@ public final ObjectInstantiator getObjectInstantiator(Class type) { public abstract void registerSerializer( Class type, Class serializerClass); + /** + * Registers a serializer class after the registration owner confirms publication is still open. + * + *

The check runs after serializer construction because constructors may execute application + * code which starts a root operation. + */ + @Internal + public abstract void registerSerializer( + Class type, Class serializerClass, Runnable checkBeforePublication); + /** * Registers a serializer for internal types (those with fixed IDs in the type system). This * method is used for built-in types like ArrayList, HashMap, etc. @@ -396,15 +406,22 @@ public abstract void registerSerializer( * later callers adopt those same maps. This method is idempotent so top-level runtime entry * points can call it defensively. */ - public final void finishRegistration() { - if (registrationFinished) { - return; + public final void freezeRegistration() { + if (!registrationFrozen) { + publishRegistrationSnapshot(); } + } + + private void publishRegistrationSnapshot() { + // A root may start through a borrowed child, so the child must close the shared facade + // boundary before freezing itself or publishing/adopting the registration snapshot. Otherwise + // a facade registration already holding that boundary could fail partway through its children. + sharedRegistry.freezeRegistration(); + registrationFrozen = true; sharedRegistry.setRegistrationIfAbsent( extRegistry.registeredClassIdMap, extRegistry.registeredClasses); - extRegistry.finishRegistration( + extRegistry.freezeRegistration( sharedRegistry.getRegisteredClassIdMap(), sharedRegistry.getRegisteredClasses()); - setRegistrationFinished(); } /** @@ -416,12 +433,13 @@ public final void finishRegistration() { */ public void registerSerializerAndType( Class type, Class serializerClass) { - if (!isRegistered(type)) { - register(type); - } - registerSerializer(type, serializerClass); + registerSerializerAndType(type, serializerClass, this::checkRegistrationOpen); } + @Internal + public abstract void registerSerializerAndType( + Class type, Class serializerClass, Runnable checkBeforePublication); + /** * Registers a type (if not already registered) and then registers the serializer instance. * @@ -429,6 +447,7 @@ public void registerSerializerAndType( * @param serializer the serializer instance to use */ public void registerSerializerAndType(Class type, Serializer serializer) { + checkRegistrationOpen(); if (!isRegistered(type)) { register(type); } @@ -2307,7 +2326,9 @@ private void buildGenericMap(Map map, GenericType genericTy } } + /** Sets the deserialization type policy before the first root operation. */ public void setTypeChecker(TypeChecker typeChecker) { + checkRegistrationOpen(); TypeChecker newChecker = typeChecker == null ? DEFAULT_TYPE_CHECKER : typeChecker; if (newChecker instanceof AllowListChecker) { ((AllowListChecker) newChecker).addListener(this); @@ -2340,6 +2361,7 @@ final void clearCheckerCache() { } public void registerSerializerFactory(SerializerFactory serializerFactory) { + checkRegistrationOpen(); extRegistry.serializerFactories.add(Preconditions.checkNotNull(serializerFactory)); } @@ -2535,7 +2557,7 @@ class ExtRegistry { codeGeneratorMap = sharedRegistry.codeGeneratorMap; } - void finishRegistration( + void freezeRegistration( IdentityHashMap, Integer> sharedRegisteredClassIdMap, BiMap> sharedRegisteredClasses) { registeredClassIdMap = sharedRegisteredClassIdMap; diff --git a/java/fory-core/src/main/java/org/apache/fory/resolver/XtypeResolver.java b/java/fory-core/src/main/java/org/apache/fory/resolver/XtypeResolver.java index 48e40ef728..c1c8b1baf4 100644 --- a/java/fory-core/src/main/java/org/apache/fory/resolver/XtypeResolver.java +++ b/java/fory-core/src/main/java/org/apache/fory/resolver/XtypeResolver.java @@ -169,6 +169,7 @@ public XtypeResolver( @Override public void initialize() { + checkRegistrationOpen(); registerDefaultTypes(); Serializers.registerDefaultSerializers(this); if (shareMeta) { @@ -204,6 +205,7 @@ protected void updateTypeInfo(Class cls, TypeInfo typeInfo) { @Override public void register(Class type) { + checkRegistrationOpen(); while (containsUserTypeId(xtypeIdGenerator)) { xtypeIdGenerator++; } @@ -212,7 +214,7 @@ public void register(Class type) { @Override public void register(Class type, long userTypeId) { - checkRegisterAllowed(); + checkRegistrationOpen(); int checkedUserTypeId = toUserTypeId(userTypeId); Preconditions.checkArgument( !containsUserTypeId(checkedUserTypeId), "Type id %s has been registered", userTypeId); @@ -265,7 +267,7 @@ public void register(Class type, long userTypeId) { @Override public void register(Class type, String namespace, String typeName) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkArgument( !typeName.isEmpty() && !typeName.contains("."), "Type name %s must be non-empty and must not contain `.` when namespace is provided", @@ -360,7 +362,7 @@ private void register( @Override public void registerUnion(Class type, long userTypeId, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkNotNull(serializer); int checkedUserTypeId = toUserTypeId(userTypeId); Preconditions.checkArgument( @@ -383,7 +385,7 @@ public void registerUnion(Class type, long userTypeId, Serializer serializ @Override public void registerUnion( Class type, String namespace, String typeName, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkNotNull(serializer); Preconditions.checkArgument( !typeName.isEmpty() && !typeName.contains("."), @@ -406,7 +408,7 @@ public void registerUnion( @Override public void registerEnum(Class type, long userTypeId, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkNotNull(serializer); int checkedUserTypeId = toUserTypeId(userTypeId); Preconditions.checkArgument( @@ -428,7 +430,7 @@ public void registerEnum(Class type, long userTypeId, Serializer serialize @Override public void registerEnum( Class type, String namespace, String typeName, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); Preconditions.checkNotNull(serializer); if (namespace == null) { namespace = ""; @@ -464,6 +466,7 @@ public void registerEnum( */ @Internal public void registerForyType(Class type, Serializer serializer, int typeId) { + checkRegistrationOpen(); Preconditions.checkArgument(typeId < MAX_TYPE_ID, "Too big type id %s", typeId); register( type, @@ -514,12 +517,20 @@ private TypeInfo newTypeInfo( } public void registerSerializer(Class type, Class serializerClass) { - checkRegisterAllowed(); - registerSerializer(type, newSerializer(type, serializerClass)); + registerSerializer(type, serializerClass, this::checkRegistrationOpen); + } + + @Override + public void registerSerializer( + Class type, Class serializerClass, Runnable checkBeforePublication) { + checkRegistrationOpen(); + Serializer serializer = newSerializer(type, serializerClass); + checkBeforePublication.run(); + registerSerializer(type, serializer); } public void registerSerializer(Class type, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); TypeInfo typeInfo = checkClassRegistration(type); checkSerializerRegistration(type, serializer.getClass()); boolean localOverride = typeInfo.serializer != null; @@ -547,6 +558,15 @@ public void registerSerializer(Class type, Serializer serializer) { } } + @Override + public void registerSerializerAndType( + Class type, Class serializerClass, Runnable checkBeforePublication) { + checkRegistrationOpen(); + Serializer serializer = newSerializer(type, serializerClass); + checkBeforePublication.run(); + registerSerializerAndType(type, serializer); + } + private void checkSerializerRegistration(Class type, Class serializerClass) { if (isCollection(type) || Collection.class.isAssignableFrom(type)) { if (!CollectionLikeSerializer.class.isAssignableFrom(serializerClass)) { @@ -568,7 +588,7 @@ private void checkSerializerRegistration(Class type, Class serializerClass @Override public void registerInternalSerializer(Class type, Serializer serializer) { - checkRegisterAllowed(); + checkRegistrationOpen(); Class unwrapped = TypeUtils.unwrap(type); if (unwrapped == char.class || unwrapped == void.class diff --git a/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java b/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java index c61d620299..c9d033bec9 100644 --- a/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java +++ b/java/fory-core/src/main/java/org/apache/fory/serializer/CompatibleSerializer.java @@ -253,9 +253,11 @@ public T read(ReadContext readContext) { readFields(readContext, fieldValues); } fieldValues = RecordUtils.remapping(recordInfo, fieldValues); - T t = objectInstantiator.newInstanceWithArguments(fieldValues); - Arrays.fill(recordInfo.getRecordComponents(), null); - return t; + try { + return objectInstantiator.newInstanceWithArguments(fieldValues); + } finally { + Arrays.fill(recordInfo.getRecordComponents(), null); + } } T targetObject = newInstance(); if (readContext.hasPreservedRefId()) { diff --git a/java/fory-core/src/main/java/org/apache/fory/serializer/CompressedArraySerializers.java b/java/fory-core/src/main/java/org/apache/fory/serializer/CompressedArraySerializers.java index 7187c67f71..bafe6c46fe 100644 --- a/java/fory-core/src/main/java/org/apache/fory/serializer/CompressedArraySerializers.java +++ b/java/fory-core/src/main/java/org/apache/fory/serializer/CompressedArraySerializers.java @@ -121,7 +121,11 @@ static void registerIfEnabled(Fory fory) { * @param fory the ThreadSafeFory instance to register serializers with */ public static void registerIfEnabled(ThreadSafeFory fory) { - fory.registerCallback(CompressedArraySerializers::registerIfEnabled); + fory.registerCallback( + (child, checkBeforePublication) -> { + checkBeforePublication.run(); + registerIfEnabled(child); + }); } /** @@ -141,7 +145,11 @@ public static void register(Fory fory) { /** Register compressed array serializers with the given Fory instance. */ public static void register(ThreadSafeFory fory) { - fory.registerCallback(CompressedArraySerializers::register); + fory.registerCallback( + (child, checkBeforePublication) -> { + checkBeforePublication.run(); + register(child); + }); } public static final class CompressedIntArraySerializer extends PrimitiveArraySerializer { diff --git a/java/fory-core/src/main/java/org/apache/fory/serializer/CopyOnlyObjectSerializer.java b/java/fory-core/src/main/java/org/apache/fory/serializer/CopyOnlyObjectSerializer.java index d686be20eb..8f3f007d5f 100644 --- a/java/fory-core/src/main/java/org/apache/fory/serializer/CopyOnlyObjectSerializer.java +++ b/java/fory-core/src/main/java/org/apache/fory/serializer/CopyOnlyObjectSerializer.java @@ -25,7 +25,7 @@ import org.apache.fory.resolver.TypeResolver; /** - * Serializer used only for copy after registration has been frozen. + * Serializer used to copy an unregistered object type that is not allowed on wire read or write. * *

Read/write keep the same security failure semantics as the normal insecure path, while copy * reuses {@link AbstractObjectSerializer}'s field-copy implementation. diff --git a/java/fory-core/src/main/java/org/apache/fory/serializer/ObjectSerializer.java b/java/fory-core/src/main/java/org/apache/fory/serializer/ObjectSerializer.java index d8d1f94aac..718d0e45e0 100644 --- a/java/fory-core/src/main/java/org/apache/fory/serializer/ObjectSerializer.java +++ b/java/fory-core/src/main/java/org/apache/fory/serializer/ObjectSerializer.java @@ -217,9 +217,11 @@ public T read(ReadContext readContext) { if (isRecord) { Object[] fields = readFields(readContext); fields = RecordUtils.remapping(recordInfo, fields); - T obj = objectInstantiator.newInstanceWithArguments(fields); - Arrays.fill(recordInfo.getRecordComponents(), null); - return obj; + try { + return objectInstantiator.newInstanceWithArguments(fields); + } finally { + Arrays.fill(recordInfo.getRecordComponents(), null); + } } T obj = newBean(); if (trackingRef) { diff --git a/java/fory-core/src/main/java/org/apache/fory/util/ExceptionUtils.java b/java/fory-core/src/main/java/org/apache/fory/util/ExceptionUtils.java index 2ffb7aae43..88a745afe1 100644 --- a/java/fory-core/src/main/java/org/apache/fory/util/ExceptionUtils.java +++ b/java/fory-core/src/main/java/org/apache/fory/util/ExceptionUtils.java @@ -20,12 +20,6 @@ package org.apache.fory.util; import java.lang.reflect.Field; -import java.util.Arrays; -import java.util.List; -import org.apache.fory.Fory; -import org.apache.fory.collection.ObjectArray; -import org.apache.fory.context.MapRefReader; -import org.apache.fory.context.ReadContext; import org.apache.fory.exception.DeserializationException; import org.apache.fory.exception.ForyException; import org.apache.fory.platform.AndroidSupport; @@ -57,17 +51,12 @@ public static StackOverflowError trySetStackOverflowErrorMessage( } } - public static RuntimeException handleReadFailed(Fory fory, Throwable t) { + // Do not attach read-reference tables to the exception. Root cleanup must release the failed + // object graph even when application code retains the exception for later inspection. + public static RuntimeException handleReadFailed(Throwable t) { if (t instanceof ForyException) { throw (ForyException) t; } - ReadContext readContext = fory.getReadContext(); - if (readContext.getRefReader() instanceof MapRefReader) { - ObjectArray readObjects = ((MapRefReader) readContext.getRefReader()).getReadRefs(); - // carry with read objects for better trouble shooting. - List objects = Arrays.asList(readObjects.objects).subList(0, readObjects.size); - throw new DeserializationException(objects, t); - } throw new DeserializationException("Failed to deserialize input", t); } diff --git a/java/fory-core/src/main/java25/org/apache/fory/memory/MemoryBuffer.java b/java/fory-core/src/main/java25/org/apache/fory/memory/MemoryBuffer.java index 2e3a834b71..9b36502407 100644 --- a/java/fory-core/src/main/java25/org/apache/fory/memory/MemoryBuffer.java +++ b/java/fory-core/src/main/java25/org/apache/fory/memory/MemoryBuffer.java @@ -205,13 +205,16 @@ private void initOffHeapBuffer(long offHeapAddress, int size, ByteBuffer offHeap checkNotNull(offHeapBuffer, "JDK25 MemoryBuffer requires a ByteBuffer owner for off-heap data"); checkArgument( offHeapBuffer.isDirect(), "Only direct ByteBuffers can back off-heap MemoryBuffer"); - this.offHeapBuffer = offHeapBuffer; - ByteBuffer nativeBuffer = offHeapBuffer.duplicate().order(NATIVE_ORDER); - // Stream readers can expand the owner buffer limit after this duplicate is created. Keep the - // absolute-access view capacity-wide so JDK25 public ByteBuffer checks match the logical buffer - // size tracked by MemoryBuffer. - nativeBuffer.clear(); - this.nativeOffHeapBuffer = nativeBuffer; + if (this.offHeapBuffer != offHeapBuffer || nativeOffHeapBuffer == null) { + this.offHeapBuffer = offHeapBuffer; + ByteBuffer nativeBuffer = offHeapBuffer.duplicate().order(NATIVE_ORDER); + // Stream readers can expand the owner buffer limit after this duplicate is created. Keep the + // absolute-access view capacity-wide so JDK25 public ByteBuffer checks match the logical + // buffer size tracked by MemoryBuffer. Reinitializing the same owner only changes its logical + // span, so retain this view instead of allocating one for every stream-root compaction. + nativeBuffer.clear(); + this.nativeOffHeapBuffer = nativeBuffer; + } this.heapMemory = null; this.address = offHeapAddress; this.addressLimit = this.address + size; diff --git a/java/fory-core/src/test/java/org/apache/fory/ForyCopyTest.java b/java/fory-core/src/test/java/org/apache/fory/ForyCopyTest.java index bd4cd7d418..525d63826b 100644 --- a/java/fory-core/src/test/java/org/apache/fory/ForyCopyTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/ForyCopyTest.java @@ -191,15 +191,6 @@ public void threadpoolCopyTest() throws InterruptedException { Assert.assertFalse(flag.get()); } - @Test - public void testCopyFinalizesRegistrationPhase() { - Fory fory = - builder().withCodegen(false).withRefCopy(true).requireClassRegistration(true).build(); - fory.register(BeanA.class); - assertEquals(fory.copy(BeanA.createBeanA(2)), BeanA.createBeanA(2)); - Assert.assertThrows(ForyException.class, () -> fory.register(BeanB.class)); - } - @Test public void testCopyOnlySerializerStillRejectsSerialize() { Fory fory = diff --git a/java/fory-core/src/test/java/org/apache/fory/ForyTest.java b/java/fory-core/src/test/java/org/apache/fory/ForyTest.java index 62df5b3dc7..5383c73a47 100644 --- a/java/fory-core/src/test/java/org/apache/fory/ForyTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/ForyTest.java @@ -27,6 +27,7 @@ import com.google.common.collect.HashBasedTable; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; +import java.io.ByteArrayOutputStream; import java.io.Serializable; import java.lang.invoke.MethodHandles; import java.math.BigDecimal; @@ -128,10 +129,20 @@ public void testRegistrationFreezesOnUse() { Fory reader = newNativeFory(); reader.deserialize(bytes); assertRegistrationFrozen(reader); + } + + @Test + public void testOutOfBandSetupFailureCleanup() { + Fory fory = newNativeFory(); + byte[] bytes = fory.serialize(7, bufferObject -> true); + Iterable failingBuffers = + () -> { + throw new IllegalStateException("iterator failed"); + }; - Fory copier = newNativeFory(); - copier.copy(1); - assertRegistrationFrozen(copier); + assertThrows(IllegalStateException.class, () -> fory.deserialize(bytes, failingBuffers)); + assertNull(fory.getReadContext().getBuffer()); + assertEquals(fory.deserialize(fory.serialize(8)), 8); } private static Fory newNativeFory() { @@ -770,34 +781,6 @@ public void testPkgAccessLevelParentClass() { serDeCheckSerializer(fory, table, "HashBasedTableSerializer"); } - @Data - static class PrintReadObject { - public PrintReadObject() { - throw new RuntimeException(); - } - - public PrintReadObject(boolean b) {} - } - - @Test - public void testPrintReadObjectsWhenFailed() { - Fory fory = - Fory.builder() - .withXlang(false) - .withRefTracking(true) - .withCodegen(false) - .requireClassRegistration(false) - .withCompatible(false) - .build(); - PrintReadObject o = new PrintReadObject(true); - try { - serDe(fory, ImmutableList.of(ImmutableList.of("a", "b"), o)); - Assert.fail(); - } catch (ForyException e) { - Assert.assertTrue(e.getMessage().contains("[a, b]")); - } - } - @Test public void testNullObjSerAndDe() { Fory fory = @@ -843,6 +826,45 @@ public void testResetBufferToSizeLimit() { assertEquals(getDefaultWriteBuffer(fory).size(), limitInBytes); } + @Test + public void testFailedWriteReleasesBuffer() { + int limitInBytes = 128; + Fory fory = + Fory.builder() + .withXlang(false) + .requireClassRegistration(false) + .withBufferSizeLimitBytes(limitInBytes) + .build(); + fory.registerSerializer(FailingWrite.class, new FailingWriteSerializer(fory.getTypeResolver())); + + assertThrows(SerializationException.class, () -> fory.serialize(new FailingWrite())); + assertEquals(getDefaultWriteBuffer(fory).size(), limitInBytes); + + assertThrows( + SerializationException.class, + () -> fory.serialize(new ByteArrayOutputStream(), new FailingWrite())); + assertEquals(getDefaultWriteBuffer(fory).size(), limitInBytes); + } + + private static final class FailingWrite {} + + private static final class FailingWriteSerializer extends Serializer { + private FailingWriteSerializer(TypeResolver typeResolver) { + super(typeResolver.getConfig(), FailingWrite.class); + } + + @Override + public void write(WriteContext writeContext, FailingWrite value) { + writeContext.getBuffer().ensure(1024); + throw new SerializationException("expected failure"); + } + + @Override + public FailingWrite read(ReadContext readContext) { + throw new UnsupportedOperationException("unused"); + } + } + private static MemoryBuffer getDefaultWriteBuffer(Fory fory) { return (MemoryBuffer) ReflectionUtils.getObjectFieldValue(fory, "buffer"); } diff --git a/java/fory-core/src/test/java/org/apache/fory/StreamTest.java b/java/fory-core/src/test/java/org/apache/fory/StreamTest.java index ec8ead87d3..abdb2938fc 100644 --- a/java/fory-core/src/test/java/org/apache/fory/StreamTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/StreamTest.java @@ -396,6 +396,57 @@ public void testStreamBufferGrowthIsGeometric() throws IOException { } } + @Test + public void testChannelPrefixCompaction() throws IOException { + Fory fory = builder().build(); + byte[] root = fory.serialize(12345); + int rootCount = 16; + byte[] roots = new byte[root.length * rootCount]; + for (int i = 0; i < rootCount; i++) { + System.arraycopy(root, 0, roots, i * root.length, root.length); + } + + try (ForyReadableChannel channel = + new ForyReadableChannel( + new ChunkedReadableByteChannel(roots, roots.length), + ByteBuffer.allocate(roots.length))) { + assertEquals(fory.deserialize(channel), 12345); + assertEquals(channel.getBuffer().readerIndex(), root.length); + assertEquals(channel.getBuffer().remaining(), root.length * (rootCount - 1)); + + for (int i = 1; i < rootCount / 2; i++) { + assertEquals(fory.deserialize(channel), 12345); + } + assertEquals(channel.getBuffer().readerIndex(), 0); + assertEquals(channel.getBuffer().remaining(), root.length * (rootCount / 2)); + + for (int i = rootCount / 2; i < rootCount; i++) { + assertEquals(fory.deserialize(channel), 12345); + } + assertEquals(channel.getBuffer().remaining(), 0); + } + + try (ForyReadableChannel channel = + new ForyReadableChannel( + new ChunkedReadableByteChannel(roots, roots.length), + ByteBuffer.allocateDirect(roots.length))) { + assertEquals(fory.deserialize(channel, Integer.class), 12345); + assertEquals(channel.getBuffer().readerIndex(), root.length); + assertEquals(channel.getBuffer().remaining(), root.length * (rootCount - 1)); + + for (int i = 1; i < rootCount / 2; i++) { + assertEquals(fory.deserialize(channel, Integer.class), 12345); + } + assertEquals(channel.getBuffer().readerIndex(), 0); + assertEquals(channel.getBuffer().remaining(), root.length * (rootCount / 2)); + + for (int i = rootCount / 2; i < rootCount; i++) { + assertEquals(fory.deserialize(channel, Integer.class), 12345); + } + assertEquals(channel.getBuffer().remaining(), 0); + } + } + private static void assertGeometricGrowth(MemoryBuffer buffer, int numBytes, String label) { int growCount = 0; Object lastBacking = backingBuffer(buffer); diff --git a/java/fory-core/src/test/java/org/apache/fory/ThreadSafeForyTest.java b/java/fory-core/src/test/java/org/apache/fory/ThreadSafeForyTest.java index 530a5429f3..ebc8b1080b 100644 --- a/java/fory-core/src/test/java/org/apache/fory/ThreadSafeForyTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/ThreadSafeForyTest.java @@ -31,8 +31,11 @@ import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicReference; +import java.util.function.Consumer; import lombok.Data; +import org.apache.fory.context.CopyContext; import org.apache.fory.context.MetaReadContext; import org.apache.fory.context.MetaWriteContext; import org.apache.fory.context.ReadContext; @@ -519,6 +522,48 @@ public Foo read(ReadContext readContext) { } } + public static class ReentrantFooSerializer extends FooSerializer { + private static Consumer constructionAction; + + public ReentrantFooSerializer(TypeResolver typeResolver, Class type) { + super(typeResolver, type); + if (constructionAction != null) { + constructionAction.accept(typeResolver); + } + } + } + + private static final class BlockingCopyValue {} + + private static final class BlockingCopySerializer extends Serializer { + private final CountDownLatch copyStarted; + private final CountDownLatch finishCopy; + + private BlockingCopySerializer( + TypeResolver resolver, CountDownLatch copyStarted, CountDownLatch finishCopy) { + super(resolver.getConfig(), BlockingCopyValue.class); + this.copyStarted = copyStarted; + this.finishCopy = finishCopy; + } + + @Override + public void write(WriteContext writeContext, BlockingCopyValue value) { + throw new UnsupportedOperationException("unused"); + } + + @Override + public BlockingCopyValue read(ReadContext readContext) { + throw new UnsupportedOperationException("unused"); + } + + @Override + public BlockingCopyValue copy(CopyContext copyContext, BlockingCopyValue value) { + copyStarted.countDown(); + awaitUnchecked(finishCopy); + return new BlockingCopyValue(); + } + } + public static class CustomClassLoader extends ClassLoader { public CustomClassLoader(ClassLoader parent) { super(parent); @@ -578,6 +623,103 @@ public void testSerializerRegister() { }); } + @Test + public void testExecuteConcurrency() throws InterruptedException { + for (ThreadSafeFory fory : newThreadSafeRuntimes()) { + CountDownLatch entered = new CountDownLatch(2); + CountDownLatch release = new CountDownLatch(1); + AtomicReference error = new AtomicReference<>(); + Thread first = new Thread(() -> runBlockingExecute(fory, entered, release, error)); + Thread second = new Thread(() -> runBlockingExecute(fory, entered, release, error)); + + first.start(); + second.start(); + boolean concurrent = entered.await(10, TimeUnit.SECONDS); + release.countDown(); + first.join(); + second.join(); + + assertTrue(concurrent); + assertNull(error.get()); + fory.register(BeanA.class); + } + } + + private static void runBlockingExecute( + ThreadSafeFory fory, + CountDownLatch entered, + CountDownLatch release, + AtomicReference error) { + try { + fory.execute( + child -> { + entered.countDown(); + awaitUnchecked(release); + return null; + }); + } catch (Throwable t) { + error.compareAndSet(null, t); + } + } + + @Test + public void testCopyConcurrency() throws InterruptedException { + for (ThreadSafeFory fory : newThreadSafeRuntimes()) { + CountDownLatch entered = new CountDownLatch(2); + CountDownLatch release = new CountDownLatch(1); + AtomicReference error = new AtomicReference<>(); + fory.registerSerializer( + BlockingCopyValue.class, + resolver -> new BlockingCopySerializer(resolver, entered, release)); + Thread first = new Thread(() -> runBlockingCopy(fory, error)); + Thread second = new Thread(() -> runBlockingCopy(fory, error)); + + first.start(); + second.start(); + boolean concurrent = entered.await(10, TimeUnit.SECONDS); + release.countDown(); + first.join(); + second.join(); + + assertTrue(concurrent); + assertNull(error.get()); + fory.register(BeanA.class); + } + } + + private static void runBlockingCopy(ThreadSafeFory fory, AtomicReference error) { + try { + fory.copy(new BlockingCopyValue()); + } catch (Throwable t) { + error.compareAndSet(null, t); + } + } + + @Test + public void testFrozenThreadLocalCreatesChild() throws InterruptedException { + ThreadSafeFory fory = + Fory.builder() + .withXlang(false) + .requireClassRegistration(true) + .withCompatible(false) + .buildThreadLocalFory(); + fory.register(Foo.class); + fory.serialize("freeze"); + AtomicReference error = new AtomicReference<>(); + Thread thread = + new Thread( + () -> { + try { + fory.serialize(new Foo()); + } catch (Throwable t) { + error.set(t); + } + }); + thread.start(); + thread.join(); + assertNull(error.get()); + } + @Test public void testRegisterAfterSerializeThrows() { ThreadSafeFory fory = @@ -617,6 +759,189 @@ public void testPoolRegisterAfterSerializeThrows() { Assert.assertThrows(ForyException.class, () -> fory.register(BeanB.class)); } + @Test + public void testFailedRootFreezesRegistration() { + for (ThreadSafeFory fory : newThreadSafeRuntimes()) { + Assert.assertThrows(RuntimeException.class, () -> fory.deserialize(new byte[0])); + Assert.assertThrows(ForyException.class, () -> fory.register(BeanB.class)); + } + } + + @Test + public void testExecuteRootFreezesFacade() { + for (ThreadSafeFory fory : newThreadSafeRuntimes()) { + fory.execute(child -> child.serialize("value")); + AtomicInteger callbacks = new AtomicInteger(); + + Assert.assertThrows( + ForyException.class, + () -> + fory.registerCallback( + (child, checkBeforePublication) -> callbacks.incrementAndGet())); + assertEquals(callbacks.get(), 0); + } + } + + @Test + public void testReentrantRootStopsRegistration() throws InterruptedException { + for (int registrationKind = 0; registrationKind < 3; registrationKind++) { + for (ThreadSafeFory fory : newThreadSafeRuntimes()) { + AtomicReference otherChild = new AtomicReference<>(); + Thread childThread = + new Thread( + () -> + fory.execute( + child -> { + otherChild.set(child); + return null; + })); + childThread.start(); + childThread.join(); + + TypeResolver rootResolver = fory.execute(Fory::getTypeResolver); + if (fory instanceof ThreadLocalFory) { + Assert.assertNotSame(otherChild.get().getTypeResolver(), rootResolver); + } + AtomicReference lateResolver = new AtomicReference<>(); + AtomicInteger roots = new AtomicInteger(); + Consumer startRoot = + resolver -> { + if (resolver != rootResolver && roots.compareAndSet(0, 1)) { + lateResolver.set(resolver); + fory.serialize("freeze"); + } + }; + Class registeredSerializer; + + if (registrationKind == 1) { + registeredSerializer = ReentrantFooSerializer.class; + ReentrantFooSerializer.constructionAction = startRoot; + try { + Assert.assertThrows( + ForyException.class, + () -> fory.registerSerializer(Foo.class, ReentrantFooSerializer.class)); + } finally { + ReentrantFooSerializer.constructionAction = null; + } + } else if (registrationKind == 2) { + registeredSerializer = ReentrantFooSerializer.class; + ReentrantFooSerializer.constructionAction = startRoot; + try { + Assert.assertThrows( + ForyException.class, + () -> fory.registerSerializerAndType(Foo.class, ReentrantFooSerializer.class)); + } finally { + ReentrantFooSerializer.constructionAction = null; + } + } else { + registeredSerializer = FooSerializer.class; + Assert.assertThrows( + ForyException.class, + () -> + fory.registerSerializer( + Foo.class, + resolver -> { + startRoot.accept(resolver); + return new FooSerializer(resolver, Foo.class); + })); + } + + assertEquals(roots.get(), 1); + Assert.assertFalse(lateResolver.get().isRegistered(Foo.class)); + Assert.assertNotEquals( + lateResolver.get().getSerializer(Foo.class).getClass(), registeredSerializer); + Assert.assertThrows(ForyException.class, () -> fory.register(BeanB.class)); + + if (fory instanceof ThreadLocalFory) { + AtomicReference> serializerType = new AtomicReference<>(); + Thread futureChild = + new Thread( + () -> + fory.execute( + child -> { + serializerType.set(child.getSerializer(Foo.class).getClass()); + return null; + })); + futureChild.start(); + futureChild.join(); + Assert.assertNotEquals(serializerType.get(), registeredSerializer); + } + } + } + } + + @Test + public void testChildFreezeWaitsForFacade() throws InterruptedException { + SharedRegistry sharedRegistry = new SharedRegistry(); + Fory child = + Fory.builder() + .withXlang(false) + .requireClassRegistration(true) + .withCompatible(false) + .withSharedRegistry(sharedRegistry) + .build(); + AtomicReference rootError = new AtomicReference<>(); + Thread rootThread = + new Thread( + () -> { + try { + child.serialize("value"); + } catch (Throwable t) { + rootError.set(t); + } + }); + + synchronized (sharedRegistry) { + rootThread.start(); + awaitThreadBlocked(rootThread); + Assert.assertFalse(child.getTypeResolver().isRegistrationFrozen()); + } + rootThread.join(); + assertNull(rootError.get()); + assertTrue(child.getTypeResolver().isRegistrationFrozen()); + } + + private static void awaitThreadBlocked(Thread thread) throws InterruptedException { + long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(30); + while (thread.getState() != Thread.State.BLOCKED && System.nanoTime() < deadline) { + Thread.sleep(1); + } + assertEquals(thread.getState(), Thread.State.BLOCKED); + } + + @Test + public void testNonRootKeepsRegistrationOpen() { + Fory direct = + Fory.builder() + .withXlang(false) + .requireClassRegistration(true) + .withCompatible(false) + .build(); + direct.copy("value"); + direct.register(BeanA.class); + + for (ThreadSafeFory fory : newThreadSafeRuntimes()) { + fory.copy("value"); + fory.execute(child -> child.getConfig()); + fory.register(BeanA.class); + } + } + + private static ThreadSafeFory[] newThreadSafeRuntimes() { + return new ThreadSafeFory[] { + Fory.builder() + .withXlang(false) + .requireClassRegistration(true) + .withCompatible(false) + .buildThreadLocalFory(), + Fory.builder() + .withXlang(false) + .requireClassRegistration(true) + .withCompatible(false) + .buildThreadSafeForyPool(2) + }; + } + private void assertConcurrentRoundTrip(ThreadSafeFory fory, BeanA beanA) throws InterruptedException { ExecutorService executorService = Executors.newFixedThreadPool(12); diff --git a/java/fory-core/src/test/java/org/apache/fory/builder/StaticCompatibleCodecBuilderTest.java b/java/fory-core/src/test/java/org/apache/fory/builder/StaticCompatibleCodecBuilderTest.java index 8bfc9fcf45..b22d0cc2b9 100644 --- a/java/fory-core/src/test/java/org/apache/fory/builder/StaticCompatibleCodecBuilderTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/builder/StaticCompatibleCodecBuilderTest.java @@ -47,10 +47,13 @@ import org.apache.fory.platform.GraalvmSupport; import org.apache.fory.reflect.TypeRef; import org.apache.fory.resolver.TypeResolver; +import org.apache.fory.serializer.CompatibleSerializer; import org.apache.fory.serializer.FieldGroups.FieldCodecCategory; +import org.apache.fory.serializer.ObjectSerializer; import org.apache.fory.serializer.Serializer; import org.apache.fory.serializer.StaticGeneratedStructSerializer; import org.apache.fory.serializer.StaticGeneratedStructSerializer.RemoteFieldInfo; +import org.apache.fory.util.record.RecordInfo; import org.testng.Assert; import org.testng.SkipException; import org.testng.annotations.DataProvider; @@ -254,33 +257,178 @@ public void testStaticCompatibleRecordSerializerConvertsRemoteField() throws Exc @Test public void testInaccessibleRecordInstantiator() throws Exception { assumeRecordSupport(); + String simpleName = "StaticCompatibleHiddenRecordFailure"; CompilationResult writerResult = compile( - "test.StaticCompatibleHiddenRecordPayload", - "package test;\n" - + "public class StaticCompatibleHiddenRecordPayload {\n" + "writer." + simpleName, + "package writer;\n" + + "public class " + + simpleName + + " {\n" + " public String id;\n" - + " public StaticCompatibleHiddenRecordPayload() {}\n" + + " public " + + simpleName + + "() {}\n" + "}\n"); + String readerName = "org.apache.fory.builder." + simpleName; CompilationResult readerResult = compile( - "test.StaticCompatibleHiddenRecordPayload", - "package test;\n" + "record StaticCompatibleHiddenRecordPayload(int id) {}\n"); + readerName, + "package org.apache.fory.builder;\n" + + "record " + + simpleName + + "(int id) {\n" + + " public static boolean fail;\n" + + " " + + simpleName + + " {\n" + + " if (fail) throw new IllegalStateException(\"expected\");\n" + + " }\n" + + "}\n"); Assert.assertTrue(writerResult.success, writerResult.diagnostics()); Assert.assertTrue(readerResult.success, readerResult.diagnostics()); - try (URLClassLoader writerLoader = writerResult.classLoader(); - URLClassLoader readerLoader = readerResult.classLoader()) { - Class writerType = writerLoader.loadClass("test.StaticCompatibleHiddenRecordPayload"); - Class readerType = readerLoader.loadClass("test.StaticCompatibleHiddenRecordPayload"); - Fory writer = compatibleFory(writerLoader, writerType, false, "hidden-record-writer"); - Fory reader = compatibleFory(readerLoader, readerType, false, "hidden-record-reader"); + try (URLClassLoader writerLoader = writerResult.classLoader()) { + Class writerType = writerLoader.loadClass("writer." + simpleName); + Class readerType = defineTestClass(readerResult, readerName); + Fory writer = compatibleFory(writerLoader, writerType, true, "hidden-record-writer"); + Fory reader = + compatibleFory( + StaticCompatibleCodecBuilderTest.class.getClassLoader(), + readerType, + true, + "hidden-record-reader"); TypeDef remoteTypeDef = TypeDef.buildTypeDef(writer.getTypeResolver(), writerType); String generatedSource = new StaticCompatibleCodecBuilder(TypeRef.of(readerType), reader, remoteTypeDef).genCode(); Assert.assertTrue(generatedSource.contains("newInstanceWithArguments")); Assert.assertTrue(generatedSource.contains("Object[] _f_recordArgs = this._f_recordArgs")); + Assert.assertTrue(generatedSource.contains("java.util.Arrays.fill(_f_recordArgs, null)")); + Assert.assertFalse(generatedSource.contains("finally")); + Assert.assertTrue(generatedSource.contains("_f_recordArgs[0] = null")); Assert.assertFalse( - generatedSource.contains("return new test.StaticCompatibleHiddenRecordPayload")); + generatedSource.contains("return new org.apache.fory.builder." + simpleName)); + + Class staticSerializerClass = + CodecUtils.loadOrGenStaticCompatibleCodecClass( + reader.getTypeResolver(), cast(readerType), remoteTypeDef); + // Construction installs this TypeDef-specific serializer in the reader resolver. The + // deserialization below verifies that the installed instance owns record-argument cleanup. + staticSerializerClass + .getConstructor(TypeResolver.class, Class.class, TypeDef.class) + .newInstance(reader.getTypeResolver(), readerType, remoteTypeDef); + + Object writerValue = writerType.getConstructor().newInstance(); + setField(writerType, writerValue, "id", "73"); + writer.setMetaWriteContext(new MetaWriteContext()); + byte[] bytes = writer.serialize(writerValue); + + setField(readerType, null, "fail", true); + MetaReadContext metaReadContext = new MetaReadContext(); + reader.setMetaReadContext(metaReadContext); + Assert.assertThrows(RuntimeException.class, () -> reader.deserialize(bytes)); + + Serializer serializer = metaReadContext.readTypeInfos.get(0).getSerializer(); + Assert.assertTrue( + serializer instanceof GeneratedStaticCompatibleSerializer, + serializer.getClass().getName()); + Field recordArgsField = serializer.getClass().getDeclaredField("_f_recordArgs"); + recordArgsField.setAccessible(true); + Assert.assertEquals(recordArgsField.get(serializer), new Object[] {null}); + + setField(readerType, null, "fail", false); + reader.setMetaReadContext(new MetaReadContext()); + Assert.assertEquals(invoke(readerType, reader.deserialize(bytes), "id"), 73); + } + } + + @Test + public void testCompatibleRecordClearsArgs() throws Exception { + assumeRecordSupport(); + CompilationResult writerResult = + compile( + "test.CompatibleFailingRecord", + "package test;\n" + + "public class CompatibleFailingRecord {\n" + + " public String value;\n" + + " public CompatibleFailingRecord() {}\n" + + "}\n"); + CompilationResult readerResult = + compile( + "test.CompatibleFailingRecord", + "package test;\n" + + "public record CompatibleFailingRecord(String value) {\n" + + " public static boolean fail;\n" + + " public CompatibleFailingRecord {\n" + + " if (fail) throw new IllegalStateException(\"expected\");\n" + + " }\n" + + "}\n"); + Assert.assertTrue(writerResult.success, writerResult.diagnostics()); + Assert.assertTrue(readerResult.success, readerResult.diagnostics()); + try (URLClassLoader writerLoader = writerResult.classLoader(); + URLClassLoader readerLoader = readerResult.classLoader()) { + Class writerType = writerLoader.loadClass("test.CompatibleFailingRecord"); + Class readerType = readerLoader.loadClass("test.CompatibleFailingRecord"); + Fory writer = compatibleFory(writerLoader, writerType, false, "failing-record-writer", false); + Fory reader = compatibleFory(readerLoader, readerType, false, "failing-record-reader", false); + Object writerValue = writerType.getConstructor().newInstance(); + setField(writerType, writerValue, "value", "retained-value"); + writer.setMetaWriteContext(new MetaWriteContext()); + byte[] bytes = writer.serialize(writerValue); + + setField(readerType, null, "fail", true); + MetaReadContext metaReadContext = new MetaReadContext(); + reader.setMetaReadContext(metaReadContext); + Assert.assertThrows(RuntimeException.class, () -> reader.deserialize(bytes)); + + Serializer serializer = metaReadContext.readTypeInfos.get(0).getSerializer(); + Assert.assertTrue(serializer instanceof CompatibleSerializer); + Field recordInfoField = CompatibleSerializer.class.getDeclaredField("recordInfo"); + recordInfoField.setAccessible(true); + RecordInfo recordInfo = (RecordInfo) recordInfoField.get(serializer); + Assert.assertEquals(recordInfo.getRecordComponents(), new Object[] {null}); + } + } + + @Test + public void testRecordFailureClearsArgs() throws Exception { + assumeRecordSupport(); + CompilationResult result = + compile( + "test.FailingRecord", + "package test;\n" + + "public record FailingRecord(String value) {\n" + + " public static boolean fail;\n" + + " public FailingRecord {\n" + + " if (fail) throw new IllegalStateException(\"expected\");\n" + + " }\n" + + "}\n"); + Assert.assertTrue(result.success, result.diagnostics()); + try (URLClassLoader loader = result.classLoader()) { + Class type = loader.loadClass("test.FailingRecord"); + Fory fory = + Fory.builder() + .withClassLoader(loader) + .withXlang(false) + .withRefTracking(true) + .withCodegen(false) + .requireClassRegistration(false) + .build(); + Object value = type.getConstructor(String.class).newInstance("retained-value"); + byte[] bytes = fory.serialize(value); + + setField(type, null, "fail", true); + Assert.assertThrows(RuntimeException.class, () -> fory.deserialize(bytes, cast(type))); + + Serializer serializer = fory.getTypeResolver().getSerializer(type); + Assert.assertTrue(serializer instanceof ObjectSerializer); + Field recordInfoField = ObjectSerializer.class.getDeclaredField("recordInfo"); + recordInfoField.setAccessible(true); + RecordInfo recordInfo = (RecordInfo) recordInfoField.get(serializer); + Assert.assertEquals(recordInfo.getRecordComponents(), new Object[] {null}); + + setField(type, null, "fail", false); + Assert.assertEquals( + invoke(type, fory.deserialize(bytes, cast(type)), "value"), "retained-value"); } } @@ -774,6 +922,16 @@ private static CompilationResult compile(String typeName, String source) throws } } + private static Class defineTestClass(CompilationResult result, String typeName) + throws Exception { + Path classFile = result.classRoot.resolve(typeName.replace('.', '/') + ".class"); + byte[] classBytes = Files.readAllBytes(classFile); + java.lang.reflect.Method defineClass = + java.lang.invoke.MethodHandles.Lookup.class.getMethod("defineClass", byte[].class); + return (Class) + defineClass.invoke(java.lang.invoke.MethodHandles.lookup(), (Object) classBytes); + } + private static void assumeRecordSupport() { if (javaSpecificationVersion() < 16) { throw new SkipException("Record source tests require JDK 16 or newer"); diff --git a/java/fory-core/src/test/java/org/apache/fory/context/MetaReadContextTest.java b/java/fory-core/src/test/java/org/apache/fory/context/MetaReadContextTest.java new file mode 100644 index 0000000000..1d228de3a1 --- /dev/null +++ b/java/fory-core/src/test/java/org/apache/fory/context/MetaReadContextTest.java @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.fory.context; + +import static org.testng.Assert.assertEquals; +import static org.testng.Assert.assertNotSame; +import static org.testng.Assert.assertNull; +import static org.testng.Assert.assertSame; + +import org.apache.fory.resolver.TypeInfo; +import org.testng.annotations.Test; + +public class MetaReadContextTest { + @Test + public void testRetainedOccurrenceTableReset() { + MetaReadContext context = new MetaReadContext(); + TypeInfo typeInfo = new TypeInfo(Object.class, null); + for (int i = 0; i < 8192; i++) { + context.readTypeInfos.add(typeInfo); + } + Object[] objects = context.readTypeInfos.objects; + + context.reset(); + + assertEquals(context.readTypeInfos.size, 0); + assertSame(context.readTypeInfos.objects, objects); + assertSame(objects[0], typeInfo); + } + + @Test + public void testLargeOccurrenceTableReset() { + MetaReadContext context = new MetaReadContext(); + TypeInfo typeInfo = new TypeInfo(Object.class, null); + for (int i = 0; i < 8193; i++) { + context.readTypeInfos.add(typeInfo); + } + Object[] objects = context.readTypeInfos.objects; + + context.reset(); + + assertEquals(context.readTypeInfos.size, 0); + assertEquals(context.readTypeInfos.objects.length, 8); + assertNotSame(context.readTypeInfos.objects, objects); + assertNull(context.readTypeInfos.objects[0]); + } +} diff --git a/java/fory-core/src/test/java/org/apache/fory/io/BlockedStreamUtilsTest.java b/java/fory-core/src/test/java/org/apache/fory/io/BlockedStreamUtilsTest.java index 7e73c6a274..d4466499f1 100644 --- a/java/fory-core/src/test/java/org/apache/fory/io/BlockedStreamUtilsTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/io/BlockedStreamUtilsTest.java @@ -24,12 +24,14 @@ import java.io.ByteArrayInputStream; import java.io.ByteArrayOutputStream; import java.io.IOException; +import java.io.InputStream; import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.nio.channels.ReadableByteChannel; import org.apache.fory.Fory; import org.apache.fory.ForyTestBase; import org.apache.fory.exception.DeserializationException; +import org.apache.fory.exception.ForyException; import org.apache.fory.memory.MemoryBuffer; import org.apache.fory.test.bean.Foo; import org.testng.annotations.Test; @@ -105,6 +107,57 @@ public void testPersistentChannelZeroRead() { } } + @Test + public void testReadFreezesBeforeIo() throws IOException { + Fory streamFory = builder().withCodegen(false).build(); + boolean[] streamRegistrationRejected = {false}; + InputStream inputStream = + new InputStream() { + @Override + public int read() { + return -1; + } + + @Override + public int read(byte[] bytes, int offset, int length) { + expectThrows(ForyException.class, () -> streamFory.register(LateType.class)); + streamRegistrationRejected[0] = true; + return -1; + } + }; + assertThrows( + RuntimeException.class, () -> BlockedStreamUtils.deserialize(streamFory, inputStream)); + assertTrue(streamRegistrationRejected[0]); + + Fory channelFory = builder().withCodegen(false).build(); + boolean[] channelRegistrationRejected = {false}; + try (ReadableByteChannel channel = + new ReadableByteChannel() { + private boolean open = true; + + @Override + public int read(ByteBuffer dst) { + expectThrows(ForyException.class, () -> channelFory.register(LateType.class)); + channelRegistrationRejected[0] = true; + return -1; + } + + @Override + public boolean isOpen() { + return open; + } + + @Override + public void close() { + open = false; + } + }) { + assertThrows( + RuntimeException.class, () -> BlockedStreamUtils.deserialize(channelFory, channel)); + } + assertTrue(channelRegistrationRejected[0]); + } + @Test public void testSmallBufferStreamReuse() { Fory writerFory = builder().withCodegen(false).build(); @@ -162,6 +215,8 @@ private static byte[] frameHeader(int size) { return ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(size).array(); } + private static final class LateType {} + private static final class ChunkedReadableByteChannel implements ReadableByteChannel { private final byte[] data; private final int chunkSize; diff --git a/java/fory-core/src/test/java/org/apache/fory/memory/MemoryBufferTest.java b/java/fory-core/src/test/java/org/apache/fory/memory/MemoryBufferTest.java index 99fe14686b..281dad004d 100644 --- a/java/fory-core/src/test/java/org/apache/fory/memory/MemoryBufferTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/memory/MemoryBufferTest.java @@ -105,6 +105,22 @@ public void testDirectBufferRejectsHeap() { () -> MemoryBuffer.fromDirectByteBuffer(ByteBuffer.allocate(8), 8, null)); } + @Test + public void testDirectReinitRetainsView() { + if (JdkVersion.MAJOR_VERSION < 25) { + throw new SkipException("The retained direct view is specific to JDK 25+"); + } + ByteBuffer owner = ByteBuffer.allocateDirect(8); + MemoryBuffer buffer = MemoryBuffer.fromDirectByteBuffer(owner, 8, null); + ByteBuffer nativeView = TestUtils.getFieldValue(buffer, "nativeOffHeapBuffer"); + + buffer.initByteBuffer(owner, 4); + Assert.assertSame(TestUtils.getFieldValue(buffer, "nativeOffHeapBuffer"), nativeView); + + buffer.initByteBuffer(ByteBuffer.allocateDirect(8), 4); + Assert.assertNotSame(TestUtils.getFieldValue(buffer, "nativeOffHeapBuffer"), nativeView); + } + @Test public void testBackingRangeChecks() { requireRootMemoryBuffer(); diff --git a/java/fory-core/src/test/java/org/apache/fory/resolver/AllowListCheckerTest.java b/java/fory-core/src/test/java/org/apache/fory/resolver/AllowListCheckerTest.java index a2d5aecbf0..c1bde12559 100644 --- a/java/fory-core/src/test/java/org/apache/fory/resolver/AllowListCheckerTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/resolver/AllowListCheckerTest.java @@ -28,6 +28,7 @@ import org.apache.fory.Fory; import org.apache.fory.ThreadSafeFory; import org.apache.fory.config.Language; +import org.apache.fory.exception.ForyException; import org.apache.fory.exception.InsecureException; import org.apache.fory.logging.LogLevel; import org.apache.fory.logging.LoggerFactory; @@ -193,8 +194,7 @@ public void testDisallowSetupOnly() { AllowListChecker lateChecker = new AllowListChecker(AllowListChecker.CheckLevel.WARN); lateChecker.disallowClass("org.apache.fory.missing.Type"); - assertThrows( - IllegalStateException.class, () -> fory.getTypeResolver().setTypeChecker(lateChecker)); + assertThrows(ForyException.class, () -> fory.getTypeResolver().setTypeChecker(lateChecker)); AllowListChecker disabledChecker = new AllowListChecker(AllowListChecker.CheckLevel.DISABLE); disabledChecker.disallowClass("org.apache.fory.missing.Type"); @@ -230,9 +230,13 @@ public void testCheckerReplacement() { .requireClassRegistration(false) .withTypeChecker(oldChecker) .build(); - fory.serialize("value"); - fory.getTypeResolver().setTypeChecker(new AllowListChecker(AllowListChecker.CheckLevel.WARN)); + fory.serialize("value"); + assertThrows( + ForyException.class, + () -> + fory.getTypeResolver() + .setTypeChecker(new AllowListChecker(AllowListChecker.CheckLevel.WARN))); oldChecker.disallowClass(missingClass); assertThrows(InsecureException.class, () -> oldChecker.checkType(null, missingClass)); } diff --git a/java/fory-core/src/test/java/org/apache/fory/resolver/ClassResolverTest.java b/java/fory-core/src/test/java/org/apache/fory/resolver/ClassResolverTest.java index ac48a604d9..edbb5f4723 100644 --- a/java/fory-core/src/test/java/org/apache/fory/resolver/ClassResolverTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/resolver/ClassResolverTest.java @@ -857,8 +857,8 @@ public void testSharedRegistryCachesFieldDescriptorsAndDescriptorGrouper() { ClassResolver resolver1 = (ClassResolver) fory1.getTypeResolver(); ClassResolver resolver2 = (ClassResolver) fory2.getTypeResolver(); - resolver1.finishRegistration(); - resolver2.finishRegistration(); + resolver1.freezeRegistration(); + resolver2.freezeRegistration(); List descriptors1 = resolver1.getFieldDescriptors(BeanB.class, true); List descriptors2 = resolver2.getFieldDescriptors(BeanB.class, true); @@ -893,8 +893,8 @@ public void testSharedRegistryCachesTypeDefDescriptorsAndDescriptorGrouperBySema ClassResolver resolver1 = (ClassResolver) fory1.getTypeResolver(); ClassResolver resolver2 = (ClassResolver) fory2.getTypeResolver(); - resolver1.finishRegistration(); - resolver2.finishRegistration(); + resolver1.freezeRegistration(); + resolver2.freezeRegistration(); TypeDef canonicalTypeDef = resolver1.getTypeDef(BeanB.class, true); MemoryBuffer buffer1 = MemoryBuffer.newHeapBuffer(256); @@ -968,7 +968,7 @@ public void testIdRegistrationAcceptsJavaName() { } @Test - public void testFinishRegisterPublishesAndAdoptsSharedRegistration() { + public void testFreezePublishesRegistration() { ForyBuilder builder = Fory.builder().withXlang(false).requireClassRegistration(true).withCompatible(false); finishBuilder(builder); @@ -984,11 +984,11 @@ public void testFinishRegisterPublishesAndAdoptsSharedRegistration() { assertNull(resolver2.getRegisteredClassId(BeanB.class)); assertNull(resolver2.getRegisteredClass("ns.C1")); - resolver1.finishRegistration(); + resolver1.freezeRegistration(); assertEquals(sharedRegistry.registeredClassIdMap.get(BeanB.class), Integer.valueOf(1)); assertEquals(sharedRegistry.registeredClasses.get("ns.C1"), C1.class); - resolver2.finishRegistration(); + resolver2.freezeRegistration(); assertEquals(resolver2.getRegisteredClassId(BeanB.class), Integer.valueOf(1)); assertEquals(resolver2.getRegisteredClass("ns.C1"), C1.class); } @@ -1495,8 +1495,8 @@ public void testShareableSerializerSharedAcrossRuntimes() { resolver2.register(Foo.class, 101); resolver1.registerSerializer(Foo.class, ShareableFooSerializer.class); resolver2.registerSerializer(Foo.class, ShareableFooSerializer.class); - resolver1.finishRegistration(); - resolver2.finishRegistration(); + resolver1.freezeRegistration(); + resolver2.freezeRegistration(); Serializer serializer1 = resolver1.getSerializer(Foo.class); TypeInfo sharedTypeInfo = sharedRegistry.registeredTypeInfoCache.get(Foo.class); diff --git a/java/fory-core/src/test/java/org/apache/fory/serializer/RegisterTest.java b/java/fory-core/src/test/java/org/apache/fory/serializer/RegisterTest.java index be7b718a1e..91cd6fef74 100644 --- a/java/fory-core/src/test/java/org/apache/fory/serializer/RegisterTest.java +++ b/java/fory-core/src/test/java/org/apache/fory/serializer/RegisterTest.java @@ -19,7 +19,11 @@ package org.apache.fory.serializer; +import java.io.ByteArrayInputStream; +import java.nio.ByteBuffer; +import java.util.Arrays; import java.util.concurrent.atomic.AtomicBoolean; +import java.util.function.Consumer; import org.apache.fory.Fory; import org.apache.fory.ForyModule; import org.apache.fory.ForyTestBase; @@ -27,6 +31,8 @@ import org.apache.fory.context.ReadContext; import org.apache.fory.context.WriteContext; import org.apache.fory.exception.ForyException; +import org.apache.fory.io.ForyInputStream; +import org.apache.fory.memory.MemoryBuffer; import org.apache.fory.resolver.TypeResolver; import org.testng.Assert; import org.testng.annotations.Test; @@ -197,6 +203,42 @@ public void testFrozenFacadeRegistration() { Assert.assertFalse(creatorCalled.get()); } + @Test + public void testStreamRootFreezesBeforeBuffer() { + Fory reader = Fory.builder().requireClassRegistration(false).build(); + byte[] bytes = Fory.builder().requireClassRegistration(false).build().serialize("value"); + AtomicBoolean registrationRejected = new AtomicBoolean(); + ForyInputStream inputStream = + new ForyInputStream(new ByteArrayInputStream(bytes)) { + @Override + public MemoryBuffer getBuffer() { + try { + reader.register(MyExt.class); + } catch (ForyException e) { + registrationRejected.set(true); + } + return super.getBuffer(); + } + }; + + Assert.assertEquals(reader.deserialize(inputStream, String.class), "value"); + Assert.assertTrue(registrationRejected.get()); + } + + @Test + public void testInvalidRootFreezes() { + for (Consumer root : + Arrays.>asList( + fory -> fory.deserialize((byte[]) null), + fory -> fory.deserialize((ByteBuffer) null), + fory -> fory.deserialize((byte[]) null, Object.class), + fory -> fory.deserialize((byte[]) null, (Iterable) null))) { + Fory fory = Fory.builder().build(); + Assert.assertThrows(NullPointerException.class, () -> root.accept(fory)); + Assert.assertThrows(ForyException.class, () -> fory.register(MyExt.class)); + } + } + public static class MyExtSerializer extends Serializer { public MyExtSerializer(TypeResolver typeResolver) { super(typeResolver.getConfig(), MyExt.class); diff --git a/javascript/packages/core/lib/context.ts b/javascript/packages/core/lib/context.ts index d6875f0ffb..6ffea84063 100644 --- a/javascript/packages/core/lib/context.ts +++ b/javascript/packages/core/lib/context.ts @@ -276,7 +276,9 @@ export class RefReader { } export class MetaStringWriter { - private disposeMetaStringBytes: MetaStringBytes[] = []; + private static readonly MAX_RETAINED_META_STRING_OWNERS = 8192; + + private metaStringOwners: MetaStringBytes[] = []; private dynamicNameId = 0; private namespaceEncoder = new MetaStringEncoder(".", "_"); private typenameEncoder = new MetaStringEncoder("$", "_"); @@ -285,9 +287,10 @@ export class MetaStringWriter { if (bytes.dynamicWriteStringId !== -1) { writer.writeVarUInt32(((bytes.dynamicWriteStringId + 1) << 1) | 1); } else { - bytes.dynamicWriteStringId = this.dynamicNameId; + const index = this.dynamicNameId; + bytes.dynamicWriteStringId = index; this.dynamicNameId += 1; - this.disposeMetaStringBytes.push(bytes); + this.metaStringOwners[index] = bytes; const len = bytes.bytes.getBytes().byteLength; writer.writeVarUInt32(len << 1); if (len !== 0) { @@ -306,9 +309,16 @@ export class MetaStringWriter { } reset() { - this.disposeMetaStringBytes.forEach((item) => { - item.dynamicWriteStringId = -1; - }); + const owners = this.metaStringOwners; + const size = this.dynamicNameId; + for (let i = 0; i < size; i++) { + owners[i].dynamicWriteStringId = -1; + } + // These owners remain serializer-owned. Keep bounded backing without making old entries + // protocol-visible, and release only an unusual root's oversized owner table. + if (size > MetaStringWriter.MAX_RETAINED_META_STRING_OWNERS) { + this.metaStringOwners = []; + } this.dynamicNameId = 0; } } @@ -356,11 +366,13 @@ export class MetaStringReader { } export class WriteContext { + private static readonly MAX_RETAINED_TYPE_META_OWNERS = 8192; + readonly writer: BinaryWriter; readonly refWriter: RefWriter; readonly metaStringWriter: MetaStringWriter; - private disposeTypeMetaOwners: Array<{ dynamicTypeId: number }> = []; + private typeMetaOwners: Array<{ dynamicTypeId: number }> = []; private dynamicTypeId = 0; constructor( @@ -376,10 +388,16 @@ export class WriteContext { this.writer.reset(); this.refWriter.reset(); this.metaStringWriter.reset(); - this.disposeTypeMetaOwners.forEach((owner) => { - owner.dynamicTypeId = -1; - }); - this.disposeTypeMetaOwners = []; + const owners = this.typeMetaOwners; + const size = this.dynamicTypeId; + for (let i = 0; i < size; i++) { + owners[i].dynamicTypeId = -1; + } + // The logical size is the current root's visibility boundary. Reuse bounded backing and + // release only an unusual root's oversized owner table. + if (size > WriteContext.MAX_RETAINED_TYPE_META_OWNERS) { + this.typeMetaOwners = []; + } this.dynamicTypeId = 0; } @@ -423,7 +441,7 @@ export class WriteContext { const index = this.dynamicTypeId; owner.dynamicTypeId = index; this.dynamicTypeId += 1; - this.disposeTypeMetaOwners.push(owner); + this.typeMetaOwners[index] = owner; this.writer.writeVarUInt32(index << 1); this.writer.buffer(bytes); } @@ -590,12 +608,6 @@ export class ReadContext { this.remainingUnbackedContainerItems = this.maxUnbackedContainerItems; } - resetReadDepth() { - // Root reads call this in finally; nested readers retain depth when a child throws. - this._depth = 0; - this.remainingUnbackedContainerItems = 0; - } - reserveGraphMemory(bytes: number) { const remaining = this.remainingGraphMemoryBytes - bytes; if (remaining >= 0 && bytes >= 0 && (bytes | 0) === bytes) { diff --git a/javascript/packages/core/lib/fory.ts b/javascript/packages/core/lib/fory.ts index 8eff8df042..73a3a384aa 100644 --- a/javascript/packages/core/lib/fory.ts +++ b/javascript/packages/core/lib/fory.ts @@ -40,13 +40,15 @@ const DEFAULT_MAX_SCHEMA_VERSIONS_PER_TYPE = 10 as const; const DEFAULT_MAX_AVERAGE_SCHEMA_VERSIONS_PER_TYPE = 3 as const; const DEFAULT_MAX_GRAPH_MEMORY_BYTES = 128 * 1024 * 1024; const DEFAULT_MAX_UNBACKED_CONTAINER_ITEMS = 8192 as const; +const EMPTY_BYTES = new Uint8Array(0); export default class Fory { - readonly typeResolver: TypeResolver; + private readonly typeResolver: TypeResolver; readonly anySerializer: Serializer; readonly config: Config; readonly writeContext: WriteContext; readonly readContext: ReadContext; private readonly rootSerializers = new WeakMap PlatformBuffer>(); + private registrationFrozen = false; private readonly rootDeserializers = new WeakMap any>(); @@ -147,22 +149,36 @@ export default class Fory { deserialize(bytes: Uint8Array): InstanceType | null; }; register(constructor: any, customSerializer?: CustomSerializer) { + this.ensureRegistrationOpen(); + // Codegen hooks can start a root operation. Recheck the facade-owned flag before every + // serializer publication performed by this explicit registration graph. + const ensureRegistrationOpen = () => this.ensureRegistrationOpen(); let serializer: Serializer; if (constructor.prototype?.[ForyTypeInfoSymbol]) { const typeInfo: TypeInfo = (constructor.prototype[ForyTypeInfoSymbol] as WithForyClsInfo) .structTypeInfo; typeInfo.freeze(); - serializer = new Gen(this.typeResolver, { - creator: constructor, - customSerializer, - }).generateSerializer(typeInfo); + serializer = new Gen( + this.typeResolver, + { + creator: constructor, + customSerializer, + }, + ensureRegistrationOpen, + ).generateSerializer(typeInfo); + this.ensureRegistrationOpen(); this.typeResolver.registerSerializer(typeInfo, serializer); } else { const typeInfo = constructor; typeInfo.freeze(); - serializer = new Gen(this.typeResolver, { - customSerializer, - }).generateSerializer(typeInfo); + serializer = new Gen( + this.typeResolver, + { + customSerializer, + }, + ensureRegistrationOpen, + ).generateSerializer(typeInfo); + this.ensureRegistrationOpen(); this.typeResolver.registerSerializer(typeInfo, serializer); } return { @@ -172,17 +188,25 @@ export default class Fory { }; } + private ensureRegistrationOpen() { + if (this.registrationFrozen) { + throw new Error("types and serializers must be registered before the first root operation"); + } + } + deserialize(bytes: Uint8Array, serializer: Serializer = this.anySerializer): T | null { - this.readContext.reset(bytes); + this.registrationFrozen = true; try { + this.readContext.reset(bytes); const reader = this.readContext.reader; const bitmap = reader.readUint8(); if (bitmap !== ConfigFlags.isCrossLanguageFlag) { this.throwInvalidRootHeader(bitmap); } return serializer.readRef(); - } finally { - this.readContext.resetReadDepth(); + } catch (error) { + this.readContext.reset(EMPTY_BYTES); + throw error; } } @@ -206,11 +230,18 @@ export default class Fory { const writer = writeContext.writer; const rootHeader = ConfigFlags.isCrossLanguageFlag; rootSerializer = (data: any) => { + this.registrationFrozen = true; + // The entry reset releases state from the previous root before this context is reused. writeContext.reset(); - writer.writeUint8(rootHeader); - writer.reserve(serializer.fixedSize); - serializer.writeRef(data); - return writer.dump(); + try { + writer.writeUint8(rootHeader); + writer.reserve(serializer.fixedSize); + serializer.writeRef(data); + return writer.dump(); + } catch (error) { + writeContext.reset(); + throw error; + } }; this.rootSerializers.set(serializer, rootSerializer); return rootSerializer; @@ -228,15 +259,17 @@ export default class Fory { : this.anySerializer; const rootHeader = ConfigFlags.isCrossLanguageFlag; rootDeserializer = (bytes: Uint8Array) => { - readContext.reset(bytes); + this.registrationFrozen = true; try { + readContext.reset(bytes); const bitmap = reader.readUint8(); if (bitmap !== rootHeader) { this.throwInvalidRootHeader(bitmap); } return rootSerializer.readRef(); - } finally { - readContext.resetReadDepth(); + } catch (error) { + readContext.reset(EMPTY_BYTES); + throw error; } }; this.rootDeserializers.set(serializer, rootDeserializer); @@ -244,6 +277,15 @@ export default class Fory { } serialize(data: T, serializer: Serializer = this.anySerializer) { - return this.getRootSerializer(serializer)(data); + this.registrationFrozen = true; + let rootSerializer; + try { + rootSerializer = this.getRootSerializer(serializer); + } catch (error) { + // Serializer lookup is part of the root attempt and can fail before the cached root owns it. + this.writeContext.reset(); + throw error; + } + return rootSerializer(data); } } diff --git a/javascript/packages/core/lib/gen/index.ts b/javascript/packages/core/lib/gen/index.ts index 9e62446722..f6ceefba33 100644 --- a/javascript/packages/core/lib/gen/index.ts +++ b/javascript/packages/core/lib/gen/index.ts @@ -65,6 +65,7 @@ export class Gen { constructor( private typeResolver: TypeResolver, private regOptions: { [key: string]: any } = {}, + private ensureRegistrationOpen?: () => void, ) {} private generate(typeInfo: TypeInfo): Serializer { @@ -104,6 +105,7 @@ export class Gen { } private register(typeInfo: TypeInfo, serializer?: Serializer) { + this.ensureRegistrationOpen?.(); this.typeResolver.registerSerializer(typeInfo, serializer); } diff --git a/javascript/packages/core/lib/writer/index.ts b/javascript/packages/core/lib/writer/index.ts index f08f6bb5ec..7a4b18b339 100644 --- a/javascript/packages/core/lib/writer/index.ts +++ b/javascript/packages/core/lib/writer/index.ts @@ -99,6 +99,8 @@ export class BinaryWriter { } this.cursor = 0; this.reserved = 0; + // Successful dumps already release a large buffer; this also covers aborted roots. + this.tryFreePool(); } bool(bool: boolean) { diff --git a/javascript/test/array.test.ts b/javascript/test/array.test.ts index 5637d0f78b..9ef09290a8 100644 --- a/javascript/test/array.test.ts +++ b/javascript/test/array.test.ts @@ -284,7 +284,21 @@ describe("array", () => { values: Type.uint16Array(), }, ); + const float16Type = Type.struct( + { typeName: "example.float16array" }, + { + values: Type.float16Array(), + }, + ); + const bfloat16Type = Type.struct( + { typeName: "example.bfloat16array" }, + { + values: Type.bfloat16Array(), + }, + ); const uint16Serializer = fory.register(uint16Type).serializer; + const float16Serializer = fory.register(float16Type).serializer; + const bfloat16Serializer = fory.register(bfloat16Type).serializer; const uint16Bytes = fory.serialize( { values: new Uint16Array([0x1234, 0xabcd]), @@ -293,13 +307,6 @@ describe("array", () => { ); expect(containsBytes(uint16Bytes, [0x34, 0x12, 0xcd, 0xab])).toBe(true); - const float16Type = Type.struct( - { typeName: "example.float16array" }, - { - values: Type.float16Array(), - }, - ); - const float16Serializer = fory.register(float16Type).serializer; const float16Bytes = fory.serialize( { values: new ForyFloat16Array([1, -2]), @@ -308,13 +315,6 @@ describe("array", () => { ); expect(containsBytes(float16Bytes, [0x00, 0x3c, 0x00, 0xc0])).toBe(true); - const bfloat16Type = Type.struct( - { typeName: "example.bfloat16array" }, - { - values: Type.bfloat16Array(), - }, - ); - const bfloat16Serializer = fory.register(bfloat16Type).serializer; const bfloat16Bytes = fory.serialize( { values: new BFloat16Array([1, -2]), diff --git a/javascript/test/crossLanguage.test.ts b/javascript/test/crossLanguage.test.ts index 6b700c7e99..0409b5fc25 100644 --- a/javascript/test/crossLanguage.test.ts +++ b/javascript/test/crossLanguage.test.ts @@ -261,28 +261,23 @@ describe("bool", () => { } const bfs = []; + const typeResolver = (fory as any).typeResolver; // Serialize each deserialized item back for (let index = 0; index < deserializedData.length; index++) { const item = deserializedData[index]; let serializedData; if (index === 11) { - serializedData = fory.serialize(item, fory.typeResolver.getSerializerById(TypeId.FLOAT32)); + serializedData = fory.serialize(item, typeResolver.getSerializerById(TypeId.FLOAT32)); } else if (index === 12) { - serializedData = fory.serialize(item, fory.typeResolver.getSerializerById(TypeId.FLOAT64)); + serializedData = fory.serialize(item, typeResolver.getSerializerById(TypeId.FLOAT64)); } else if (index === 14) { - serializedData = fory.serialize(item, fory.typeResolver.getSerializerById(TypeId.DATE)); + serializedData = fory.serialize(item, typeResolver.getSerializerById(TypeId.DATE)); } else if (index === 15) { - serializedData = fory.serialize( - item, - fory.typeResolver.getSerializerById(TypeId.TIMESTAMP), - ); + serializedData = fory.serialize(item, typeResolver.getSerializerById(TypeId.TIMESTAMP)); } else if (index === 16) { - serializedData = fory.serialize( - item, - fory.typeResolver.getSerializerById(TypeId.BOOL_ARRAY), - ); + serializedData = fory.serialize(item, typeResolver.getSerializerById(TypeId.BOOL_ARRAY)); } else if (index === 17) { - serializedData = fory.serialize(item, fory.typeResolver.getSerializerById(TypeId.BINARY)); + serializedData = fory.serialize(item, typeResolver.getSerializerById(TypeId.BINARY)); } else if (index === 26) { serializedData = colorSerialize(item); } else { diff --git a/javascript/test/decimal.test.ts b/javascript/test/decimal.test.ts index b6a169cc8e..48ac75260c 100644 --- a/javascript/test/decimal.test.ts +++ b/javascript/test/decimal.test.ts @@ -176,8 +176,15 @@ describe("decimal", () => { const roundTrip = fory.deserialize(fory.serialize(value)) as Decimal; expect(roundTrip.equals(value)).toBe(true); } else { + const writer = (fory as any).writeContext.writer; + const bodyBefore = Array.from( + writer.getPlatformBuffer().subarray(bodyOffset, bodyOffset + 5), + ); expect(() => fory.serialize(value)).toThrow(/Decimal scale/); - expect((fory as any).writeContext.writer.writeGetCursor()).toBe(bodyOffset); + expect(writer.writeGetCursor()).toBe(0); + expect(Array.from(writer.getPlatformBuffer().subarray(bodyOffset, bodyOffset + 5))).toEqual( + bodyBefore, + ); } const payload = decimalPayload(scale); @@ -186,7 +193,7 @@ describe("decimal", () => { expect(decoded.equals(value)).toBe(true); } else { expect(() => fory.deserialize(payload.bytes)).toThrow(/Decimal scale/); - expect((fory as any).readContext.reader.readGetCursor()).toBe(payload.scaleEnd); + expect((fory as any).readContext.reader.readGetCursor()).toBe(0); } } }); @@ -210,7 +217,7 @@ describe("decimal", () => { writer.getPlatformBuffer().subarray(bodyOffset, bodyOffset + 5), ); expect(() => fory.serialize(value)).toThrow(/Decimal magnitude/); - expect(writer.writeGetCursor()).toBe(bodyOffset); + expect(writer.writeGetCursor()).toBe(0); expect(Array.from(writer.getPlatformBuffer().subarray(bodyOffset, bodyOffset + 5))).toEqual( bodyBefore, ); @@ -222,7 +229,7 @@ describe("decimal", () => { expect(decoded.equals(value)).toBe(true); } else { expect(() => fory.deserialize(payload.bytes)).toThrow(/Decimal magnitude length/); - expect((fory as any).readContext.reader.readGetCursor()).toBe(payload.magnitudeOffset); + expect((fory as any).readContext.reader.readGetCursor()).toBe(0); } } }); diff --git a/javascript/test/depthLimit.test.ts b/javascript/test/depthLimit.test.ts index 5a0eb029fb..7a2ce28039 100644 --- a/javascript/test/depthLimit.test.ts +++ b/javascript/test/depthLimit.test.ts @@ -275,6 +275,11 @@ describe("depth-limit", () => { readerFory.register(readerChild); const writer = writerFory.register(writerRoot); const reader = readerFory.register(readerRoot); + const shallowType = Type.struct(7403, { + value: Type.int32().setId(1), + }); + const shallowWriter = writerFory.register(shallowType); + const shallowReader = readerFory.register(shallowType); const malformedDepth = writer.serialize({ child: { grandchild: { value: "7" }, @@ -288,18 +293,13 @@ describe("depth-limit", () => { ); expect(readerFory.readContext.depth).toBe(0); - const shallowType = Type.struct(7403, { - value: Type.int32().setId(1), - }); - const shallowWriter = writerFory.register(shallowType); - const shallowReader = readerFory.register(shallowType); expect(shallowReader.deserialize(shallowWriter.serialize({ value: 10 }))).toEqual({ value: 10, }); expect(readerFory.readContext.depth).toBe(0); }); - test("should reset depth at start of each deserialization", () => { + test("resets depth between roots", () => { const fory = new Fory({ compatible: false, maxDepth: 50 }); const typeInfo = Type.struct( { @@ -313,7 +313,6 @@ describe("depth-limit", () => { const { serialize, deserialize } = fory.register(typeInfo); deserialize(serialize({ a: 1 })); - // Depth will be reset at the start of resetRead() call expect(fory.readContext.depth).toBe(0); deserialize(serialize({ a: 2 })); diff --git a/javascript/test/enum.test.ts b/javascript/test/enum.test.ts index b9852e8803..3ca3a6d75c 100644 --- a/javascript/test/enum.test.ts +++ b/javascript/test/enum.test.ts @@ -79,13 +79,6 @@ describe("enum", () => { const enumSerializer = fory.register(enumType); expect(enumSerializer.serializer.needToWriteRef()).toBe(false); - const rootBytes = enumSerializer.serialize(Foo.first); - const reader = new BinaryReader({}); - reader.reset(rootBytes); - expect(reader.readUint8()).toBe(ConfigFlags.isCrossLanguageFlag); - expect(reader.readInt8()).toBe(RefFlags.NotNullValueFlag); - expect(reader.readUint8()).toBe(TypeId.ENUM); - const nodeType = Type.struct(102, { value: Type.int32(), }); @@ -96,6 +89,13 @@ describe("enum", () => { second: nodeType.clone().setTrackingRef(true).setId(3), }), ); + const rootBytes = enumSerializer.serialize(Foo.first); + const reader = new BinaryReader({}); + reader.reset(rootBytes); + expect(reader.readUint8()).toBe(ConfigFlags.isCrossLanguageFlag); + expect(reader.readInt8()).toBe(RefFlags.NotNullValueFlag); + expect(reader.readUint8()).toBe(TypeId.ENUM); + const shared = { value: 7 }; const result = sequenceSerializer.deserialize( sequenceSerializer.serialize({ diff --git a/javascript/test/fory.test.ts b/javascript/test/fory.test.ts index 50b9b51948..37d9ae55b2 100644 --- a/javascript/test/fory.test.ts +++ b/javascript/test/fory.test.ts @@ -84,6 +84,60 @@ describe("fory", () => { testTypeInfo(typeinfo8, "123"); }); + test.each(["serialize", "deserialize"] as const)("freezes on failed %s", (operation) => { + const fory = new Fory({ compatible: false }); + + if (operation === "serialize") { + expect(() => fory.serialize(Symbol("unsupported"))).toThrow(); + } else { + expect(() => fory.deserialize(new Uint8Array([0]))).toThrow(); + } + + expect(() => fory.register(Type.struct(8102, {}))).toThrow(); + }); + + test("freezes before serializer lookup", () => { + const fory = new Fory({ compatible: false }); + + expect(() => fory.serialize(1, null as any)).toThrow(); + expect(() => fory.register(Type.struct(8104, {}))).toThrow(); + }); + + test("freezes during serializer generation", () => { + let armed = false; + let fory: Fory; + const typeInfo = Type.struct(8105, { value: Type.int32() }); + fory = new Fory({ + compatible: false, + hooks: { + afterCodeGenerated(code) { + if (armed) { + fory.serialize(null); + } + return code; + }, + }, + }); + armed = true; + + expect(() => fory.register(typeInfo)).toThrow(); + const serializer = (fory as any).typeResolver.getSerializerByTypeInfo(typeInfo); + expect(serializer._initialized).toBe(false); + }); + + test.each(["serialize", "deserialize"] as const)("freezes after %s", (operation) => { + const fory = new Fory({ compatible: false }); + + if (operation === "serialize") { + fory.serialize(1); + } else { + const bytes = new Fory({ compatible: false }).serialize(1); + fory.deserialize(bytes); + } + + expect(() => fory.register(Type.struct(8103, {}))).toThrow(); + }); + function testTypeInfo(typeinfo: TypeInfo, input: any, expected?: any) { const fory = new Fory({ compatible: false }); const serialize = fory.register(typeinfo); diff --git a/javascript/test/protocol/struct.test.ts b/javascript/test/protocol/struct.test.ts index 95cb26e2ca..edaad7ef66 100644 --- a/javascript/test/protocol/struct.test.ts +++ b/javascript/test/protocol/struct.test.ts @@ -60,7 +60,6 @@ describe("protocol", () => { }, ); const nonNullableSer = fory.register(nonNullable); - expect(() => nonNullableSer.serialize({ a: null })).toThrow(/Field "a" is not nullable/); // 2) nullable not specified => keep old behavior (null allowed) const nullableUnspecified = Type.struct( @@ -72,6 +71,7 @@ describe("protocol", () => { }, ); const { serialize, deserialize } = fory.register(nullableUnspecified); + expect(() => nonNullableSer.serialize({ a: null })).toThrow(/Field "a" is not nullable/); expect(deserialize(serialize({ a: null }))).toEqual({ a: null }); }); diff --git a/javascript/test/rootCleanup.test.ts b/javascript/test/rootCleanup.test.ts new file mode 100644 index 0000000000..2d293ad67b --- /dev/null +++ b/javascript/test/rootCleanup.test.ts @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +import Fory, { Type } from "../packages/core/index"; +import { TypeMeta } from "../packages/core/lib/meta/TypeMeta"; +import { describe, expect, test } from "@jest/globals"; + +function expectRootStateCleared(readContext: any) { + expect(readContext.refReader.readObjects).toHaveLength(0); + expect(readContext.metaStringReader.names).toHaveLength(0); + expect(readContext.typeMeta).toHaveLength(0); +} + +function populateLogicalTables(readContext: any, typeMeta: TypeMeta) { + readContext.metaStringReader.names.push("stale"); + readContext.typeMeta.push(typeMeta); +} + +describe.each([ + { + name: "Fory.deserialize", + invoke: (fory: Fory, registered: ReturnType, bytes: Uint8Array) => + fory.deserialize(bytes, registered.serializer), + }, + { + name: "registered deserialize", + invoke: (_fory: Fory, registered: ReturnType, bytes: Uint8Array) => + registered.deserialize(bytes), + }, +])("$name root cleanup", ({ invoke }) => { + test("restores generated root state after failure", () => { + const writerFory = new Fory({ compatible: true, ref: true }); + const readerFory = new Fory({ compatible: true, ref: true }); + const writer = writerFory.register( + Type.struct(7601, { + value: Type.int32().setId(1), + }), + ); + const reader = readerFory.register( + Type.struct(7601, { + value: Type.int32().setId(1), + }), + ); + const bytes = writer.serialize({ value: 7 }); + const input = bytes.subarray(0, bytes.length - 1); + + expect(() => invoke(readerFory, reader, input)).toThrow(); + expectRootStateCleared(readerFory.readContext); + expect(invoke(readerFory, reader, bytes)).toEqual({ value: 7 }); + }); + + test("restores logical tables after failure", () => { + const fory = new Fory({ compatible: true, ref: true }); + const registered = fory.register(Type.struct(7602, {})); + const readContext = (fory as any).readContext; + const typeMeta = TypeMeta.fromTypeInfo(Type.struct(7603, {})); + const headerHash = typeMeta.getHash(); + readContext.typeMetaCache.set(headerHash, typeMeta); + populateLogicalTables(readContext, typeMeta); + + registered.serializer.readRef = () => { + expectRootStateCleared(readContext); + populateLogicalTables(readContext, typeMeta); + throw new Error("root read failed"); + }; + + const read = () => invoke(fory, registered, new Uint8Array([1])); + expect(read).toThrow(); + expectRootStateCleared(readContext); + registered.serializer.readRef = () => { + expectRootStateCleared(readContext); + return 7; + }; + expect(read()).toBe(7); + + expect(readContext.typeMetaCache.get(headerHash)).toBe(typeMeta); + }); + + test("clears retained state when input binding fails", () => { + const fory = new Fory({ compatible: true, ref: true }); + const registered = fory.register(Type.struct(7615, {})); + const readContext = (fory as any).readContext; + const typeMeta = TypeMeta.fromTypeInfo(Type.struct(7616, {})); + + registered.serializer.readRef = () => { + readContext.refReader.reference({}); + populateLogicalTables(readContext, typeMeta); + return 7; + }; + expect(invoke(fory, registered, new Uint8Array([1]))).toBe(7); + + const invalidInput = new Uint8Array([1]); + Object.defineProperty(invalidInput, "buffer", { + get() { + throw new Error("input binding failed"); + }, + }); + expect(() => invoke(fory, registered, invalidInput)).toThrow("input binding failed"); + expect(readContext.reader.platformBuffer).toHaveLength(0); + expectRootStateCleared(readContext); + }); +}); + +test("restores root write state after failure", () => { + const fory = new Fory({ compatible: true, ref: true }); + const registered = fory.register(Type.struct(7606, {})); + const writeContext = (fory as any).writeContext; + const typeMeta = TypeMeta.fromTypeInfo(Type.struct(7607, {})); + const name = writeContext.metaStringWriter.encodeTypeName("FailedRoot"); + const value = {}; + + registered.serializer.writeRef = () => { + writeContext.refWriter.writeRef(value); + writeContext.metaStringWriter.writeBytes(writeContext.writer, name); + writeContext.writeTypeMeta(typeMeta, typeMeta.toBytes()); + throw new Error("root write failed"); + }; + + expect(() => registered.serialize(value)).toThrow(); + expect(writeContext.refWriter.writeObjects.size).toBe(0); + expect(name.dynamicWriteStringId).toBe(-1); + expect(typeMeta.dynamicTypeId).toBe(-1); + expect(fory.serialize(7)).toBeDefined(); +}); + +test("clears write state before serializer lookup", () => { + const fory = new Fory({ compatible: true, ref: true }); + const registered = fory.register(Type.struct(7613, {})); + const writeContext = (fory as any).writeContext; + const typeMeta = TypeMeta.fromTypeInfo(Type.struct(7614, {})); + const name = writeContext.metaStringWriter.encodeTypeName("PreviousRoot"); + const value = {}; + + registered.serializer.writeRef = () => { + writeContext.refWriter.writeRef(value); + writeContext.metaStringWriter.writeBytes(writeContext.writer, name); + writeContext.writeTypeMeta(typeMeta, typeMeta.toBytes()); + }; + expect(registered.serialize(value)).toBeDefined(); + + expect(() => fory.serialize(1, null as any)).toThrow(); + expect(writeContext.refWriter.writeObjects.size).toBe(0); + expect(name.dynamicWriteStringId).toBe(-1); + expect(typeMeta.dynamicTypeId).toBe(-1); +}); + +test("reuses root write metastring owners", () => { + const fory = new Fory({ compatible: true }); + const registered = fory.register(Type.struct(7609, {})); + const writeContext = (fory as any).writeContext; + const name = writeContext.metaStringWriter.encodeTypeName("RootName"); + + registered.serializer.writeRef = () => { + writeContext.metaStringWriter.writeBytes(writeContext.writer, name); + }; + + expect(registered.serialize({})).toBeDefined(); + const owners = writeContext.metaStringWriter.metaStringOwners; + expect(owners).toHaveLength(1); + expect(name.dynamicWriteStringId).toBe(0); + + expect(registered.serialize({})).toBeDefined(); + expect(writeContext.metaStringWriter.metaStringOwners).toBe(owners); + expect(owners).toHaveLength(1); + expect(name.dynamicWriteStringId).toBe(0); +}); + +test("reuses write metadata owners", () => { + const fory = new Fory({ compatible: true }); + const registered = fory.register(Type.struct(7610, {})); + const writeContext = (fory as any).writeContext; + const typeMeta = TypeMeta.fromTypeInfo(Type.struct(7611, {})); + + registered.serializer.writeRef = () => { + writeContext.writeTypeMeta(typeMeta, typeMeta.toBytes()); + }; + + expect(registered.serialize({})).toBeDefined(); + const owners = writeContext.typeMetaOwners; + expect(owners).toHaveLength(1); + expect(typeMeta.dynamicTypeId).toBe(0); + + expect(registered.serialize({})).toBeDefined(); + expect(writeContext.typeMetaOwners).toBe(owners); + expect(owners).toHaveLength(1); + expect(typeMeta.dynamicTypeId).toBe(0); +}); + +test.each([8192, 8193])("bounds %s metastring owners", (ownerCount) => { + const fory = new Fory({ compatible: true }); + const writeContext = (fory as any).writeContext; + const metaStringWriter = writeContext.metaStringWriter; + + for (let i = 0; i < ownerCount; i++) { + const owner = metaStringWriter.encodeTypeName(`name-${i}`); + metaStringWriter.writeBytes(writeContext.writer, owner); + } + const owners = metaStringWriter.metaStringOwners; + + writeContext.reset(); + if (ownerCount === 8192) { + expect(metaStringWriter.metaStringOwners).toBe(owners); + } else { + expect(metaStringWriter.metaStringOwners).not.toBe(owners); + expect(metaStringWriter.metaStringOwners).toHaveLength(0); + } + const nextOwner = metaStringWriter.encodeTypeName("next-root"); + metaStringWriter.writeBytes(writeContext.writer, nextOwner); + expect(nextOwner.dynamicWriteStringId).toBe(0); +}); + +test.each([8192, 8193])("bounds %s type metadata owners", (ownerCount) => { + const fory = new Fory({ compatible: true }); + const writeContext = (fory as any).writeContext; + const typeMetaOwners = Array.from({ length: ownerCount }, () => ({ dynamicTypeId: -1 })); + const bytes = new Uint8Array(); + + for (const owner of typeMetaOwners) { + writeContext.writeTypeMeta(owner, bytes); + } + const owners = writeContext.typeMetaOwners; + + writeContext.reset(); + expect(typeMetaOwners.every((owner) => owner.dynamicTypeId === -1)).toBe(true); + if (ownerCount === 8192) { + expect(writeContext.typeMetaOwners).toBe(owners); + } else { + expect(writeContext.typeMetaOwners).not.toBe(owners); + expect(writeContext.typeMetaOwners).toHaveLength(0); + } + const nextOwner = { dynamicTypeId: -1 }; + writeContext.writeTypeMeta(nextOwner, bytes); + expect(nextOwner.dynamicTypeId).toBe(0); +}); + +test("releases failed write buffer", () => { + const fory = new Fory({ compatible: true }); + const registered = fory.register(Type.struct(7608, {})); + const writer = (fory as any).writeContext.writer; + + registered.serializer.writeRef = () => { + writer.buffer(new Uint8Array(4 * 1024 * 1024)); + throw new Error("root write failed"); + }; + + expect(() => registered.serialize({})).toThrow(); + expect(writer.getPlatformBuffer().byteLength).toBeLessThan(4 * 1024 * 1024); + expect(fory.serialize(7)).toBeDefined(); +}); + +test("clears failed read refs", () => { + const fory = new Fory({ compatible: false, ref: true }); + const registered = fory.register(Type.struct(7612, {})); + const refReader = (fory as any).readContext.refReader; + + registered.serializer.readRef = () => { + refReader.reference({}); + throw new Error("root read failed"); + }; + expect(() => registered.deserialize(new Uint8Array([1]))).toThrow(); + expect(refReader.readObjects).toHaveLength(0); + + registered.serializer.readRef = () => { + expect(refReader.readObjects).toHaveLength(0); + return {}; + }; + expect(registered.deserialize(new Uint8Array([1]))).toEqual({}); +}); diff --git a/javascript/test/typemeta.test.ts b/javascript/test/typemeta.test.ts index effd162278..bd6afe1ac4 100644 --- a/javascript/test/typemeta.test.ts +++ b/javascript/test/typemeta.test.ts @@ -337,7 +337,6 @@ describe("typemeta", () => { const writer = writerFory.register(writerRoot); const reader = readerFory.register(readerRoot); const childTypeMeta = TypeMeta.fromTypeInfo(writerChild, (writerFory as any).typeResolver); - const rootTypeMeta = TypeMeta.fromTypeInfo(writerRoot, (writerFory as any).typeResolver); const value = { child: { value: 9 } }; const valid = writer.serialize(value); const overwritten = replaceFirstBytes( @@ -347,11 +346,7 @@ describe("typemeta", () => { ); const readContext = (readerFory as any).readContext; - expect(() => reader.deserialize(overwritten)).toThrow( - "Invalid new TypeMeta index 0; expected 1", - ); - expect(readContext.typeMeta).toHaveLength(1); - expect(readContext.typeMeta[0].getHash()).toBe(rootTypeMeta.getHash()); + expect(() => reader.deserialize(overwritten)).toThrow(); expect(readContext.typeMetaCache.has(childTypeMeta.getHash())).toBe(false); expect(reader.deserialize(valid)).toEqual(value); }); @@ -1166,8 +1161,7 @@ describe("typemeta", () => { ); const readContext = (readerFory as any).readContext; - expect(() => reader.deserialize(wrongBytes)).toThrow("Compatible TypeMeta owner mismatch"); - expect(readContext.typeMeta).toHaveLength(1); + expect(() => reader.deserialize(wrongBytes)).toThrow(); expect(readContext.typeMetaCache.has(writerChildMeta.getHash())).toBe(false); expect(readContext.compatibleReadSerializers.has(writerChildMeta.getHash())).toBe(false); @@ -1175,8 +1169,7 @@ describe("typemeta", () => { value: 8, }); expect(readContext.typeMetaCache.has(writerChildMeta.getHash())).toBe(false); - expect(() => reader.deserialize(wrongBytes)).toThrow("Compatible TypeMeta owner mismatch"); - expect(readContext.typeMeta).toHaveLength(1); + expect(() => reader.deserialize(wrongBytes)).toThrow(); expect(readContext.compatibleReadSerializers.has(writerChildMeta.getHash())).toBe(false); const localChildType = Type.struct(readerChildId, { @@ -1222,12 +1215,7 @@ describe("typemeta", () => { first: { value: 1 }, second: { value: 2 }, }); - const readContext = (readerFory as any).readContext; - - expect(() => reader.deserialize(wrongBytes)).toThrow("Compatible TypeMeta owner mismatch"); - expect(readContext.typeMeta).toHaveLength(2); - expect(() => reader.deserialize(wrongBytes)).toThrow("Compatible TypeMeta owner mismatch"); - expect(readContext.typeMeta).toHaveLength(2); + expect(() => reader.deserialize(wrongBytes)).toThrow(); localWriterFory.register(Type.struct(writerChildId, childProps)); localWriterFory.register(Type.struct(readerChildId, childProps)); @@ -2177,16 +2165,18 @@ describe("typemeta", () => { expect(Array.from(result.values as Int32Array)).toEqual([0, 1, -1]); + const taggedWriterFory = new Fory({ compatible: true }); + const taggedReaderFory = new Fory({ compatible: true }); const taggedWriterType = Type.struct(7219, { values: Type.list(Type.int64({ encoding: "tagged" })).setId(1), }); const taggedReaderType = Type.struct(7219, { values: Type.int64Array().setId(1), }); - const taggedBytes = writerFory.register(taggedWriterType).serialize({ + const taggedBytes = taggedWriterFory.register(taggedWriterType).serialize({ values: [0n, 1n, -1n], }); - const taggedResult = readerFory.register(taggedReaderType).deserialize(taggedBytes); + const taggedResult = taggedReaderFory.register(taggedReaderType).deserialize(taggedBytes); expect(Array.from(taggedResult.values as BigInt64Array)).toEqual([0n, 1n, -1n]); }); diff --git a/kotlin/fory-kotlin/src/main/java/org/apache/fory/serializer/kotlin/KotlinSerializers.java b/kotlin/fory-kotlin/src/main/java/org/apache/fory/serializer/kotlin/KotlinSerializers.java index d8edd694dc..663c72f5f7 100644 --- a/kotlin/fory-kotlin/src/main/java/org/apache/fory/serializer/kotlin/KotlinSerializers.java +++ b/kotlin/fory-kotlin/src/main/java/org/apache/fory/serializer/kotlin/KotlinSerializers.java @@ -19,10 +19,7 @@ package org.apache.fory.serializer.kotlin; -import java.util.Collections; -import java.util.Map; import java.util.Objects; -import java.util.WeakHashMap; import kotlin.*; import kotlin.UByteArray; import kotlin.UIntArray; @@ -35,9 +32,11 @@ import kotlin.uuid.Uuid; import org.apache.fory.Fory; import org.apache.fory.ThreadSafeFory; +import org.apache.fory.annotation.Internal; import org.apache.fory.codegen.GeneratedClassNames; import org.apache.fory.config.Config; import org.apache.fory.exception.ForyException; +import org.apache.fory.kotlin.ForyKotlin; import org.apache.fory.resolver.TypeResolver; import org.apache.fory.serializer.EnumSerializer; import org.apache.fory.serializer.Serializer; @@ -49,129 +48,119 @@ @SuppressWarnings({"rawtypes", "unchecked"}) public class KotlinSerializers { private static final String XLANG_GENERATED_SERIALIZER_SUFFIX = "_ForySerializer"; - private static final Map INSTALLED_FORY = - Collections.synchronizedMap(new WeakHashMap<>()); public static void registerSerializers(ThreadSafeFory fory) { - fory.register(KotlinSerializers::registerSerializers); + fory.register(ForyKotlin.INSTANCE); } public static void registerSerializers(Fory fory) { - synchronized (INSTALLED_FORY) { - if (INSTALLED_FORY.containsKey(fory)) { - return; - } - INSTALLED_FORY.put(fory, Boolean.TRUE); - } - try { - DefaultValueUtils.setKotlinDefaultValueSupport(new KotlinDefaultValueSupport()); - TypeResolver resolver = fory.getTypeResolver(); - if (resolver.isCrossLanguage()) { - return; - } - Config config = resolver.getConfig(); - - // UByte - Class ubyteClass = KotlinToJavaClass.INSTANCE.getUByteClass(); - registerIfAbsent(resolver, ubyteClass); - resolver.registerSerializer(ubyteClass, new UByteSerializer(config)); - - // UShort - Class ushortClass = KotlinToJavaClass.INSTANCE.getUShortClass(); - registerIfAbsent(resolver, ushortClass); - resolver.registerSerializer(ushortClass, new UShortSerializer(config)); - - // UInt - Class uintClass = KotlinToJavaClass.INSTANCE.getUIntClass(); - registerIfAbsent(resolver, uintClass); - resolver.registerSerializer(uintClass, new UIntSerializer(config)); - - // ULong - Class ulongClass = KotlinToJavaClass.INSTANCE.getULongClass(); - registerIfAbsent(resolver, ulongClass); - resolver.registerSerializer(ulongClass, new ULongSerializer(config)); - - // EmptyList - Class emptyListClass = KotlinToJavaClass.INSTANCE.getEmptyListClass(); - registerIfAbsent(resolver, emptyListClass); - resolver.registerSerializer( - emptyListClass, new CollectionSerializers.EmptyListSerializer(resolver, emptyListClass)); - - // EmptySet - Class emptySetClass = KotlinToJavaClass.INSTANCE.getEmptySetClass(); - registerIfAbsent(resolver, emptySetClass); - resolver.registerSerializer( - emptySetClass, new CollectionSerializers.EmptySetSerializer(resolver, emptySetClass)); - - // EmptyMap - Class emptyMapClass = KotlinToJavaClass.INSTANCE.getEmptyMapClass(); - registerIfAbsent(resolver, emptyMapClass); - resolver.registerSerializer( - emptyMapClass, new MapSerializers.EmptyMapSerializer(resolver, emptyMapClass)); - - // Non-Java collection implementation in kotlin stdlib. - Class arrayDequeClass = KotlinToJavaClass.INSTANCE.getArrayDequeClass(); - registerIfAbsent(resolver, arrayDequeClass); - resolver.registerSerializer( - arrayDequeClass, new KotlinArrayDequeSerializer(resolver, arrayDequeClass)); - - // Unsigned array classes: UByteArray, UShortArray, UIntArray, ULongArray. - registerIfAbsent(resolver, UByteArray.class); - resolver.registerSerializer(UByteArray.class, new UByteArraySerializer(resolver)); - registerIfAbsent(resolver, UShortArray.class); - resolver.registerSerializer(UShortArray.class, new UShortArraySerializer(resolver)); - registerIfAbsent(resolver, UIntArray.class); - resolver.registerSerializer(UIntArray.class, new UIntArraySerializer(resolver)); - registerIfAbsent(resolver, ULongArray.class); - resolver.registerSerializer(ULongArray.class, new ULongArraySerializer(resolver)); - - // Ranges and Progressions. - registerIfAbsent(resolver, kotlin.ranges.CharRange.class); - registerIfAbsent(resolver, kotlin.ranges.CharProgression.class); - registerIfAbsent(resolver, kotlin.ranges.IntRange.class); - registerIfAbsent(resolver, kotlin.ranges.IntProgression.class); - registerIfAbsent(resolver, kotlin.ranges.LongRange.class); - registerIfAbsent(resolver, kotlin.ranges.LongProgression.class); - registerIfAbsent(resolver, kotlin.ranges.UIntRange.class); - registerIfAbsent(resolver, kotlin.ranges.UIntProgression.class); - registerIfAbsent(resolver, kotlin.ranges.ULongRange.class); - registerIfAbsent(resolver, kotlin.ranges.ULongProgression.class); - - // Built-in classes. - registerIfAbsent(resolver, kotlin.Pair.class); - registerIfAbsent(resolver, kotlin.Triple.class); - registerIfAbsent(resolver, kotlin.Result.class); - registerIfAbsent(resolver, Result.Failure.class); - - // kotlin.random - registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRandomDefaultClass()); - registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRandomInternalClass()); - registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRandomSerializedClass()); - - // kotlin.text - registerIfAbsent(resolver, Regex.class); - registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRegexSerializedClass()); - registerIfAbsent(resolver, RegexOption.class); - registerIfAbsent(resolver, CharCategory.class); - registerIfAbsent(resolver, CharDirectionality.class); - registerIfAbsent(resolver, HexFormat.class); - registerIfAbsent(resolver, MatchGroup.class); - - // kotlin.time - registerIfAbsent(resolver, DurationUnit.class); - registerIfAbsent(resolver, Duration.class); - resolver.registerSerializer(Duration.class, new DurationSerializer(config)); - registerIfAbsent(resolver, TimedValue.class); - - // kotlin.uuid - registerIfAbsent(resolver, Uuid.class); - resolver.registerSerializer(Uuid.class, new UuidSerializer(config)); - } catch (RuntimeException | Error e) { - synchronized (INSTALLED_FORY) { - INSTALLED_FORY.remove(fory); - } - throw e; + fory.register(ForyKotlin.INSTANCE); + } + + @Internal + public static void installSerializers(Fory fory) { + DefaultValueUtils.setKotlinDefaultValueSupport(new KotlinDefaultValueSupport()); + TypeResolver resolver = fory.getTypeResolver(); + if (resolver.isCrossLanguage()) { + return; } + Config config = resolver.getConfig(); + + // UByte + Class ubyteClass = KotlinToJavaClass.INSTANCE.getUByteClass(); + registerIfAbsent(resolver, ubyteClass); + resolver.registerSerializer(ubyteClass, new UByteSerializer(config)); + + // UShort + Class ushortClass = KotlinToJavaClass.INSTANCE.getUShortClass(); + registerIfAbsent(resolver, ushortClass); + resolver.registerSerializer(ushortClass, new UShortSerializer(config)); + + // UInt + Class uintClass = KotlinToJavaClass.INSTANCE.getUIntClass(); + registerIfAbsent(resolver, uintClass); + resolver.registerSerializer(uintClass, new UIntSerializer(config)); + + // ULong + Class ulongClass = KotlinToJavaClass.INSTANCE.getULongClass(); + registerIfAbsent(resolver, ulongClass); + resolver.registerSerializer(ulongClass, new ULongSerializer(config)); + + // EmptyList + Class emptyListClass = KotlinToJavaClass.INSTANCE.getEmptyListClass(); + registerIfAbsent(resolver, emptyListClass); + resolver.registerSerializer( + emptyListClass, new CollectionSerializers.EmptyListSerializer(resolver, emptyListClass)); + + // EmptySet + Class emptySetClass = KotlinToJavaClass.INSTANCE.getEmptySetClass(); + registerIfAbsent(resolver, emptySetClass); + resolver.registerSerializer( + emptySetClass, new CollectionSerializers.EmptySetSerializer(resolver, emptySetClass)); + + // EmptyMap + Class emptyMapClass = KotlinToJavaClass.INSTANCE.getEmptyMapClass(); + registerIfAbsent(resolver, emptyMapClass); + resolver.registerSerializer( + emptyMapClass, new MapSerializers.EmptyMapSerializer(resolver, emptyMapClass)); + + // Non-Java collection implementation in kotlin stdlib. + Class arrayDequeClass = KotlinToJavaClass.INSTANCE.getArrayDequeClass(); + registerIfAbsent(resolver, arrayDequeClass); + resolver.registerSerializer( + arrayDequeClass, new KotlinArrayDequeSerializer(resolver, arrayDequeClass)); + + // Unsigned array classes: UByteArray, UShortArray, UIntArray, ULongArray. + registerIfAbsent(resolver, UByteArray.class); + resolver.registerSerializer(UByteArray.class, new UByteArraySerializer(resolver)); + registerIfAbsent(resolver, UShortArray.class); + resolver.registerSerializer(UShortArray.class, new UShortArraySerializer(resolver)); + registerIfAbsent(resolver, UIntArray.class); + resolver.registerSerializer(UIntArray.class, new UIntArraySerializer(resolver)); + registerIfAbsent(resolver, ULongArray.class); + resolver.registerSerializer(ULongArray.class, new ULongArraySerializer(resolver)); + + // Ranges and Progressions. + registerIfAbsent(resolver, kotlin.ranges.CharRange.class); + registerIfAbsent(resolver, kotlin.ranges.CharProgression.class); + registerIfAbsent(resolver, kotlin.ranges.IntRange.class); + registerIfAbsent(resolver, kotlin.ranges.IntProgression.class); + registerIfAbsent(resolver, kotlin.ranges.LongRange.class); + registerIfAbsent(resolver, kotlin.ranges.LongProgression.class); + registerIfAbsent(resolver, kotlin.ranges.UIntRange.class); + registerIfAbsent(resolver, kotlin.ranges.UIntProgression.class); + registerIfAbsent(resolver, kotlin.ranges.ULongRange.class); + registerIfAbsent(resolver, kotlin.ranges.ULongProgression.class); + + // Built-in classes. + registerIfAbsent(resolver, kotlin.Pair.class); + registerIfAbsent(resolver, kotlin.Triple.class); + registerIfAbsent(resolver, kotlin.Result.class); + registerIfAbsent(resolver, Result.Failure.class); + + // kotlin.random + registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRandomDefaultClass()); + registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRandomInternalClass()); + registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRandomSerializedClass()); + + // kotlin.text + registerIfAbsent(resolver, Regex.class); + registerIfAbsent(resolver, KotlinToJavaClass.INSTANCE.getRegexSerializedClass()); + registerIfAbsent(resolver, RegexOption.class); + registerIfAbsent(resolver, CharCategory.class); + registerIfAbsent(resolver, CharDirectionality.class); + registerIfAbsent(resolver, HexFormat.class); + registerIfAbsent(resolver, MatchGroup.class); + + // kotlin.time + registerIfAbsent(resolver, DurationUnit.class); + registerIfAbsent(resolver, Duration.class); + resolver.registerSerializer(Duration.class, new DurationSerializer(config)); + registerIfAbsent(resolver, TimedValue.class); + + // kotlin.uuid + registerIfAbsent(resolver, Uuid.class); + resolver.registerSerializer(Uuid.class, new UuidSerializer(config)); } private static void registerIfAbsent(TypeResolver resolver, Class cls) { @@ -216,6 +205,8 @@ public static void registerType(Fory fory, Class cls, String namespace, Strin } public static void register(Fory fory, Class cls) { + // Generated construction resolves the registered STRUCT TypeInfo. Publish that identity first; + // registerSerializer rechecks freeze after construction before installing the serializer. fory.register(cls); registerSerializer(fory, cls); } @@ -237,8 +228,11 @@ public static void register(Fory fory, Class cls, String namespace, String ty public static void registerSerializer(Fory fory, Class cls) { TypeResolver resolver = fory.getTypeResolver(); + resolver.checkRegistrationOpen(); Serializer serializer = newGeneratedSerializer(resolver, cls); + resolver.checkRegistrationOpen(); if (resolver.isRegistered(cls)) { + // Preserve the registered STRUCT TypeInfo; registerSerializer would reclassify it as EXT. resolver.setSerializer(cls, serializer); } else { resolver.registerSerializer(cls, serializer); @@ -247,40 +241,50 @@ public static void registerSerializer(Fory fory, Class cls) { public static void registerEnum(Fory fory, Class cls, long typeId) { TypeResolver resolver = fory.getTypeResolver(); - resolver.registerEnum(cls, typeId, new EnumSerializer(resolver.getConfig(), enumClass(cls))); + resolver.checkRegistrationOpen(); + Serializer serializer = new EnumSerializer(resolver.getConfig(), enumClass(cls)); + resolver.registerEnum(cls, typeId, serializer); } public static void registerEnum(Fory fory, Class cls, String namespace, String typeName) { checkTypeName(typeName); TypeResolver resolver = fory.getTypeResolver(); - resolver.registerEnum( - cls, namespace, typeName, new EnumSerializer(resolver.getConfig(), enumClass(cls))); + resolver.checkRegistrationOpen(); + Serializer serializer = new EnumSerializer(resolver.getConfig(), enumClass(cls)); + resolver.registerEnum(cls, namespace, typeName, serializer); } public static void registerEnum(Fory fory, Class cls, String name) { TypeResolver resolver = fory.getTypeResolver(); + resolver.checkRegistrationOpen(); String[] parts = splitName(name); - resolver.registerEnum( - cls, parts[0], parts[1], new EnumSerializer(resolver.getConfig(), enumClass(cls))); + Serializer serializer = new EnumSerializer(resolver.getConfig(), enumClass(cls)); + resolver.registerEnum(cls, parts[0], parts[1], serializer); } public static void registerUnion(Fory fory, Class cls, long typeId) { TypeResolver resolver = fory.getTypeResolver(); - resolver.registerUnion(cls, typeId, newGeneratedSerializer(resolver, cls)); + resolver.checkRegistrationOpen(); + Serializer serializer = newGeneratedSerializer(resolver, cls); + resolver.registerUnion(cls, typeId, serializer); registerCaseAliases(fory, cls); } public static void registerUnion(Fory fory, Class cls, String namespace, String typeName) { checkTypeName(typeName); TypeResolver resolver = fory.getTypeResolver(); - resolver.registerUnion(cls, namespace, typeName, newGeneratedSerializer(resolver, cls)); + resolver.checkRegistrationOpen(); + Serializer serializer = newGeneratedSerializer(resolver, cls); + resolver.registerUnion(cls, namespace, typeName, serializer); registerCaseAliases(fory, cls); } public static void registerUnion(Fory fory, Class cls, String name) { TypeResolver resolver = fory.getTypeResolver(); + resolver.checkRegistrationOpen(); String[] parts = splitName(name); - resolver.registerUnion(cls, parts[0], parts[1], newGeneratedSerializer(resolver, cls)); + Serializer serializer = newGeneratedSerializer(resolver, cls); + resolver.registerUnion(cls, parts[0], parts[1], serializer); registerCaseAliases(fory, cls); } diff --git a/kotlin/fory-kotlin/src/main/kotlin/org/apache/fory/kotlin/ForyKotlin.kt b/kotlin/fory-kotlin/src/main/kotlin/org/apache/fory/kotlin/ForyKotlin.kt index 9d26b250b3..41061483da 100644 --- a/kotlin/fory-kotlin/src/main/kotlin/org/apache/fory/kotlin/ForyKotlin.kt +++ b/kotlin/fory-kotlin/src/main/kotlin/org/apache/fory/kotlin/ForyKotlin.kt @@ -28,6 +28,6 @@ public object ForyKotlin : ForyModule { @JvmStatic public fun builder(): ForyBuilder = Fory.builder().withModule(this) override fun install(fory: Fory) { - KotlinSerializers.registerSerializers(fory) + KotlinSerializers.installSerializers(fory) } } diff --git a/python/README.md b/python/README.md index b26faf5195..afa4c46f39 100644 --- a/python/README.md +++ b/python/README.md @@ -680,21 +680,17 @@ class Fory: ### ThreadSafeFory Class -Thread-safe serialization interface using thread-local storage: +Thread-safe serialization interface using an instance pool: ```python class ThreadSafeFory: - def __init__( - self, - xlang: bool = True, - ref: bool = False, - strict: bool = True, - compatible: bool | None = None, - max_depth: int = 50 - ) + def __init__(self, fory_factory=None, **kwargs) ``` -`ThreadSafeFory` provides thread-safe serialization by maintaining a pool of `Fory` instances protected by a lock. When a thread needs to serialize/deserialize, it gets an instance from the pool, uses it, and returns it. All type registrations must be done before any serialization to ensure consistency across all instances. +`ThreadSafeFory` provides thread-safe serialization by maintaining a pool of `Fory` instances protected by a lock. When a thread needs to serialize/deserialize, it gets an instance from the pool, uses it, and returns it. Complete explicit type and serializer registration before the first serialization or deserialization attempt. + +When supplied, `fory_factory` creates each pooled `Fory` instance. Otherwise, `**kwargs` are passed +to the normal `Fory` constructor. **Thread Safety Example:** @@ -728,8 +724,10 @@ for t in threads: t.join() - **Instance Pool**: Maintains a pool of `Fory` instances protected by a lock for thread safety - **Shared Configuration**: All registrations must be done upfront and are applied to all instances -- **Same API**: Drop-in replacement for `Fory` class with identical methods -- **Registration Safety**: Prevents registration after first use to ensure consistency +- **Shared Root API**: Provides the same root serialization and deserialization operations as + `Fory` +- **Registration Safety**: Prevents explicit registration after the first root serialization or + deserialization attempt **When to Use:** @@ -739,15 +737,21 @@ for t in threads: t.join() **Parameters:** -- **`xlang`** (`bool`, default=`True`): Use xlang mode. Set `False` for Python native mode supporting Python-specific objects. -- **`ref`** (`bool`, default=`False`): Enable reference tracking for shared/circular references. Disable for better performance if your data has no shared references. -- **`strict`** (`bool`, default=`True`): Require type registration for security. **Highly recommended** for production. Only disable in trusted environments. -- **`compatible`** (`bool | None`, default `None`): Enable schema evolution. `None` enables compatible mode in both xlang and native mode. Set `False` only when every reader and writer always uses the same Python class schema and you want faster serialization and smaller size. -- **`max_depth`** (`int`, default=`50`): Maximum deserialization depth for security, preventing stack overflow attacks. +- **`fory_factory`** (`Callable | None`, default=`None`): No-argument factory for configured `Fory` + instances. +- **`**kwargs`**: Normal `Fory` construction options, used when `fory_factory` is not supplied. **Key Methods:** ```python +fory = pyfory.ThreadSafeFory(xlang=True) + +# Complete registration before the first root operation. Choose one form: +fory.register(MyClass, type_id=123) +# fory.register(MyClass, name="my.package.MyClass") +# fory.register(MyClass, type_id=123, serializer=MySerializer) +# fory.register(MyClass, name="my.package.MyClass", serializer=MySerializer) + # Serialization (serialize/deserialize are identical to dumps/loads) data: bytes = fory.serialize(obj) obj = fory.deserialize(data) @@ -755,14 +759,6 @@ obj = fory.deserialize(data) # Alternative API (aliases) data: bytes = fory.dumps(obj) obj = fory.loads(data) - -# Type registration by id -fory.register(MyClass, type_id=123) -fory.register(MyClass, type_id=123, serializer=custom_serializer) - -# Type registration by name -fory.register(MyClass, name="my.package.MyClass") -fory.register(MyClass, name="my.package.MyClass", serializer=custom_serializer) ``` ### Xlang And Native Mode Comparison @@ -857,7 +853,12 @@ assert result.children[0].parent is result # Reference preserved In strict mode, Fory loads and instantiates only registered application types. Compatible metadata for an unregistered remote Struct returns the fixed data-only `pyfory.UnknownStruct` carrier; it does not load or generate the -sender-named class. This prevents arbitrary class materialization: +sender-named class. This prevents arbitrary class materialization. + +The first root serialization or deserialization attempt permanently freezes explicit type and +serializer registration for that `Fory` instance, including when the attempt fails. Non-strict +native writes may still discover runtime types lazily, and reads may resolve those authorized by +the configured policy. That discovery does not add an explicit registration. ```python import pyfory @@ -903,22 +904,22 @@ class Foo: f2: str class FooSerializer(Serializer): - def __init__(self, fory, cls): - super().__init__(fory, cls) + def __init__(self, type_resolver, cls): + super().__init__(type_resolver, cls) - def write(self, buffer, obj: Foo): + def write(self, write_context, obj: Foo): # Custom serialization logic - buffer.write_varint32(obj.f1) - buffer.write_string(obj.f2) + write_context.write_varint32(obj.f1) + write_context.write_string(obj.f2) - def read(self, buffer): + def read(self, read_context): # Custom deserialization logic - f1 = buffer.read_varint32() - f2 = buffer.read_string() + f1 = read_context.read_varint32() + f2 = read_context.read_string() return Foo(f1, f2) f = pyfory.Fory(xlang=False) -f.register(Foo, type_id=100, serializer=FooSerializer(f, Foo)) +f.register(Foo, type_id=100, serializer=FooSerializer(f.type_resolver, Foo)) # Now Foo uses your custom serializer data = f.dumps(Foo(42, "hello")) @@ -1008,7 +1009,7 @@ fory.register(MyClass, type_id=100) fory.register(MyClass, name="com.example.MyClass") # Pattern 3: With custom serializer -fory.register(MyClass, type_id=100, serializer=MySerializer(fory, MyClass)) +fory.register(MyClass, type_id=100, serializer=MySerializer(fory.type_resolver, MyClass)) # Pattern 4: Batch registration type_id = 100 @@ -1031,7 +1032,8 @@ try: data = fory.dumps(my_object) except TypeUnregisteredError as e: print(f"Type not registered: {e}") - # Register the type and retry + # A failed root has already frozen this instance. Configure a new one. + fory = pyfory.Fory(strict=True) fory.register(type(my_object), type_id=100) data = fory.dumps(my_object) except Exception as e: @@ -1245,26 +1247,26 @@ import pyfory # Now uses pure Python implementation ```python # A: Xlang mode defaults to compatible schema evolution. -f = pyfory.Fory(xlang=True) - -# Version 1: Original class +# Version 1: Writer schema @dataclass -class User: +class UserV1: name: str age: int -f.register(User, name="User") -data = f.dumps(User("Alice", 30)) +writer = pyfory.Fory(xlang=True) +writer.register(UserV1, name="User") +data = writer.dumps(UserV1("Alice", 30)) -# Version 2: Add new field (backward compatible) +# Version 2: Reader schema with a new field @dataclass -class User: +class UserV2: name: str age: int email: str = "unknown@example.com" # New field with default -# Can still deserialize old data -user = f.loads(data) +reader = pyfory.Fory(xlang=True) +reader.register(UserV2, name="User") +user = reader.loads(data) print(user.email) # "unknown@example.com" ``` diff --git a/python/pyfory/_fory.py b/python/pyfory/_fory.py index a8ff222759..a0c25ffa63 100644 --- a/python/pyfory/_fory.py +++ b/python/pyfory/_fory.py @@ -88,9 +88,9 @@ class Fory: objects and cross-language reference metadata; Python native mode handles the broader Python object graph surface, including circular Python objects. - In Python native mode (xlang=False), Fory can serialize all Python objects - including dataclasses, classes with custom serialization methods, and local - functions/classes, making it a drop-in replacement for pickle. + In Python native mode (xlang=False), Fory can serialize Python objects including + dataclasses, classes with custom serialization methods, and local functions/classes. + With strict mode disabled, policy-authorized types are discovered lazily. In xlang mode, the default, Fory serializes objects in a format that can be deserialized by other Fory-supported languages (Java, Go, Rust, C++, etc.). @@ -152,11 +152,11 @@ def __init__( Args: xlang: Enable xlang mode. When False, uses - Python native mode supporting all Python objects (dataclasses, __reduce__, - local functions/classes). With ref=True and strict=False, serves as a - drop-in replacement for pickle. When True, uses the xlang wire format - compatible with other Fory languages (Java, Go, Rust, etc), but Python- - specific features like functions and __reduce__ methods are not supported. + Python native mode supporting Python objects such as dataclasses, + __reduce__, and local functions/classes. When True, uses the xlang wire + format compatible with other Fory languages (Java, Go, Rust, etc), but + Python-specific features like functions and __reduce__ methods are not + supported. ref: Enable reference tracking for shared references and Python native-mode circular references. When enabled, duplicate objects are stored once. @@ -165,9 +165,11 @@ def __init__( strict: Require registration before loading or instantiating application classes (default: True). Compatible metadata for an unregistered remote Struct uses the fixed data-only UnknownStruct carrier instead of loading - or generating the sender-named class. When strict mode is disabled, - dynamic application types can be deserialized, which may be insecure if - malicious code exists in __new__/__init__/__eq__/__hash__ methods. + or generating the sender-named class. Disabling strict mode authorizes + lazy native type discovery. Policy-authorized module globals may be + resolved while reading trusted native payloads. Dynamic application types + can be insecure if malicious code exists in + __new__/__init__/__eq__/__hash__ methods. **WARNING**: Only disable in trusted environments. When disabling strict mode, you should provide a custom `policy` parameter to control which types are allowed. We are not responsible for security risks when this option @@ -300,6 +302,7 @@ def register( >>> fory.register(Person, type_id=100) >>> >>> # Register with name (more flexible) + >>> fory = Fory(xlang=True) >>> fory.register(Person, name="com.example.Person") >>> >>> # Python native mode (no cross-language matching needed) @@ -346,6 +349,7 @@ def register_type( >>> fory.register_type(Person, type_id=100) >>> >>> # Register with name (more flexible) + >>> fory = Fory(xlang=True) >>> fory.register_type(Person, name="com.example.Person") >>> >>> # Python native mode (no cross-language matching needed) @@ -390,7 +394,9 @@ def register_serializer(self, cls: type, serializer): Example: >>> fory = Fory(xlang=False) - >>> fory.register_serializer(MyClass, MyCustomSerializer()) + >>> fory.register_type(MyClass) + >>> serializer = MyCustomSerializer(fory.type_resolver, MyClass) + >>> fory.register_serializer(MyClass, serializer) """ self.type_resolver.register_serializer(cls, serializer) @@ -421,6 +427,8 @@ def dump(self, obj, stream): the passed object (or a view of it) is unsupported. If your sink needs retention, copy bytes inside ``write``. """ + if not self.type_resolver._registry_frozen: + self.type_resolver._freeze_registry() try: self.buffer.set_writer_index(0) output_stream = Buffer.wrap_output_stream(stream) @@ -476,6 +484,8 @@ def serialize( >>> print(type(data)) """ + if not self.type_resolver._registry_frozen: + self.type_resolver._freeze_registry() try: write_buffer = self._serialize( obj, @@ -553,6 +563,8 @@ def deserialize( >>> print(obj) {'key': 'value'} """ + if not self.type_resolver._registry_frozen: + self.type_resolver._freeze_registry() try: return self._deserialize(buffer, buffers, unsupported_objects) finally: @@ -610,7 +622,8 @@ def reset(self): Reset both write and read state. Clears all per-operation state including buffers and reference tracking. - Use this to ensure a clean state before reusing a Fory instance. + Use this to ensure clean operation state before reusing a Fory instance. + Reset does not reopen registration after the first root attempt. """ self.reset_write() self.reset_read() @@ -625,18 +638,14 @@ class ThreadSafeFory: needs to serialize or deserialize data, it acquires an instance from the pool, uses it, and returns it for reuse by other threads. - All type registrations must be performed before any serialization operations to ensure - consistency across all pooled instances. Attempting to register types after the first - serialization will raise a RuntimeError. + All type registrations must be performed before the first root serialization or + deserialization attempt to ensure consistency across all pooled instances. Registration + remains closed even when that first operation fails. Args: - xlang (bool): Whether to enable xlang mode. Defaults to True. - ref (bool): Whether to enable reference tracking. Defaults to False. - strict (bool): Whether to require type registration. Defaults to True. - compatible (bool): Whether to enable compatible mode. Defaults to compatible mode - in both xlang and Python native mode. Set False only when every reader and - writer always uses the same Python class schema and smaller payloads matter. - max_depth (int): Maximum depth for deserialization. Defaults to 50. + fory_factory: Optional no-argument factory that returns a configured Fory + instance. + **kwargs: Fory construction options used when fory_factory is not supplied. Example: >>> import pyfory >>> import threading @@ -685,13 +694,13 @@ def __init__(self, fory_factory=None, **kwargs): self._fory_class = CythonFory else: self._fory_class = Fory - self._instances_created = False + self._registry_frozen = False def _get_fory(self): with self._lock: + self._registry_frozen = True if self._pool: return self._pool.pop() - self._instances_created = True if self._fory_factory is not None: fory = self._fory_factory() else: @@ -706,10 +715,8 @@ def _return_fory(self, fory): def _register_callback(self, callback): with self._lock: - if self._instances_created: - raise RuntimeError( - "Cannot register types after Fory instances have been created. Please register all types before calling serialize/deserialize." - ) + if self._registry_frozen: + raise RuntimeError("Cannot register types after the first root serialization or deserialization operation has started.") self._callbacks.append(callback) def register( diff --git a/python/pyfory/registry.py b/python/pyfory/registry.py index 9cf00a4432..1c6391807c 100644 --- a/python/pyfory/registry.py +++ b/python/pyfory/registry.py @@ -372,6 +372,7 @@ class TypeResolver: "meta_share", "_internal_py_serializer_map", "_actual_type_resolver", + "_registry_frozen", ) def __init__(self, config, *, shared_registry): @@ -412,6 +413,14 @@ def __init__(self, config, *, shared_registry): self.meta_share = config.meta_share self._internal_py_serializer_map = {} self._actual_type_resolver = self + self._registry_frozen = False + + def _check_registry_mutable(self): + if self._actual_type_resolver._registry_frozen: + raise RuntimeError("Cannot register types or serializers after the first root operation has started") + + def _freeze_registry(self): + self._registry_frozen = True def _set_actual_resolver(self, type_resolver): # Cython mode injects the compiled companion before initialize() so all @@ -578,6 +587,7 @@ def register_type( name: str = None, serializer=None, ): + self._check_registry_mutable() namespace, typename = _split_registration_name(name) return self._register_type( cls, @@ -595,7 +605,16 @@ def register_union( name: str = None, serializer=None, ): + self._check_registry_mutable() + cls = normalize_fory_type(cls) namespace, typename = _split_registration_name(name) + registration_id = type_id if type_id not in {0, None} else NO_USER_TYPE_ID + self._check_registration_identity( + cls, + namespace=namespace, + typename=typename, + user_type_id=registration_id, + ) if serializer is None: raise TypeError("register_union requires a serializer") if serializer is not None and not isinstance(serializer, Serializer): @@ -604,6 +623,7 @@ def register_union( self._actual_type_resolver, cls, ) + self._check_registry_mutable() if typename is not None and type_id is not None: raise TypeError(f"type name {typename} and id {type_id} should not be set at the same time") if typename is None and type_id is None: @@ -650,6 +670,8 @@ def _register_type( self._actual_type_resolver, cls, ) + if not internal: + self._check_registry_mutable() if ( cls in self._types_info and type_id is None @@ -660,12 +682,17 @@ def _register_type( ): return self._types_info[cls] n_params = len({typename, type_id, None}) - 1 - if n_params == 0 and typename is None: - type_id = self._next_type_id() if n_params == 2: raise TypeError(f"type name {typename} and id {type_id} should not be set at the same time") - if cls in self._types_info: - raise TypeError(f"{cls} registered already") + registration_id = type_id if not internal and type_id is not None else user_type_id + self._check_registration_identity( + cls, + namespace=namespace, + typename=typename, + user_type_id=registration_id, + ) + if n_params == 0 and typename is None: + type_id = self._next_type_id() return self._register_xtype( cls, type_id=type_id, @@ -746,6 +773,23 @@ def __register_type( internal: bool = False, ): dynamic_type = type_id is not None and type_id < 0 + if typename is not None: + if namespace is None: + splits = typename.rsplit(".", 1) + if len(splits) == 2: + namespace, typename = splits + else: + namespace = "" + else: + namespace = namespace or "" + if not typename: + raise ValueError("type name must not be empty") + self._check_registration_identity( + cls, + namespace=namespace, + typename=typename, + user_type_id=user_type_id, + ) # In metashare mode, for struct types, we want to keep serializer=None # so that _set_type_info will be called to create the TypeDef-based serializer # This applies to both types registered by name and by ID @@ -753,20 +797,14 @@ def __register_type( if should_create_serializer: serializer = self._create_serializer(cls) + if not internal: + self._check_registry_mutable() if serializer is not None and type_id in _NO_REF_NUMERIC_TYPE_IDS: serializer.need_to_write_ref = False if typename is None: typeinfo = TypeInfo(cls, type_id, user_type_id, serializer, None, None, dynamic_type) else: - if namespace is None: - splits = typename.rsplit(".", 1) - if len(splits) == 2: - namespace, typename = splits - else: - namespace = "" # Use empty string for consistency with lookup - if not typename: - raise ValueError("type name must not be empty") ns_metastr = self.namespace_encoder.encode(namespace or "") ns_meta_bytes = self.shared_registry.get_encoded_meta_string(ns_metastr) type_metastr = self.typename_encoder.encode(typename) @@ -776,10 +814,6 @@ def __register_type( self._ns_type_to_type_info[(ns_meta_bytes, type_meta_bytes)] = typeinfo self._types_info[cls] = typeinfo if type_id is not None and type_id != 0: - if needs_user_type_id(type_id) and user_type_id not in {None, NO_USER_TYPE_ID}: - existing = self._user_type_id_to_type_info.get(user_type_id) - if existing is not None and existing.cls is not cls: - raise TypeError(f"user_type_id {user_type_id} already registered for {existing.cls}") if needs_user_type_id(type_id) and user_type_id not in {None, NO_USER_TYPE_ID}: if user_type_id not in self._user_type_id_to_type_info or not internal: self._user_type_id_to_type_info[user_type_id] = typeinfo @@ -787,10 +821,28 @@ def __register_type( elif not TypeId.is_namespaced_type(type_id): if type_id not in self._type_id_to_type_info or not internal: self._type_id_to_type_info[type_id] = typeinfo - self._types_info[cls] = typeinfo self._index_python_type(cls) return typeinfo + def _check_registration_identity( + self, + cls, + *, + namespace, + typename, + user_type_id, + ): + if cls in self._types_info: + raise TypeError(f"{cls} registered already") + if typename is not None: + existing = self._named_type_to_type_info.get((namespace, typename)) + if existing is not None and existing.cls is not cls: + raise TypeError(f"type name {(namespace, typename)!r} already registered for {existing.cls}") + if user_type_id not in {None, NO_USER_TYPE_ID}: + existing = self._user_type_id_to_type_info.get(user_type_id) + if existing is not None and existing.cls is not cls: + raise TypeError(f"user_type_id {user_type_id} already registered for {existing.cls}") + def _index_python_type(self, cls): if self._python_name_to_type is None or not isinstance(cls, type): return @@ -813,11 +865,13 @@ def _next_type_id(self): return type_id def register_serializer(self, cls, serializer): + self._check_registry_mutable() cls = normalize_fory_type(cls) assert isinstance(cls, type) or type(cls) is int, cls if cls not in self._types_info: raise TypeUnregisteredError(f"{cls} not registered") typeinfo = self._types_info[cls] + self._check_registry_mutable() prev_type_id = typeinfo.type_id prev_user_type_id = typeinfo.user_type_id if needs_user_type_id(prev_type_id) and prev_user_type_id not in {None, NO_USER_TYPE_ID}: @@ -831,6 +885,7 @@ def register_serializer(self, cls, serializer): else: typeinfo.type_id = TypeId.EXT typeinfo.serializer = serializer + typeinfo.type_def = None if needs_user_type_id(typeinfo.type_id) and typeinfo.user_type_id not in {None, NO_USER_TYPE_ID}: self._user_type_id_to_type_info[typeinfo.user_type_id] = typeinfo else: @@ -855,30 +910,35 @@ def get_type_info(self, cls, create=True): return type_info elif not create: return None + # Unknown remote enums use this internal placeholder; it is not an explicit registration. if cls is NonExistEnum: return self._get_nonexist_enum_type_info() if self.require_registration and not issubclass(cls, Enum): raise TypeUnregisteredError(f"{cls} not registered") logger.info("Type %s not registered", cls) + return self._create_inferred_type_info(cls) + + def _create_inferred_type_info(self, cls): serializer = self._create_serializer(cls) - type_id = None - if not self.xlang: - if isinstance(serializer, EnumSerializer): - type_id = TypeId.NAMED_ENUM - elif isinstance(serializer, (ObjectSerializer, StatefulSerializer)): - type_id = TypeId.NAMED_EXT - elif self._internal_py_serializer_map.get(type(serializer)) is not None: - type_id = self._internal_py_serializer_map.get(type(serializer))[1] - if not self.require_registration: + native_registration = self._internal_py_serializer_map.get(type(serializer)) + if native_registration is not None: + type_id = native_registration[1] + elif not self.xlang and isinstance(serializer, EnumSerializer): + type_id = TypeId.NAMED_ENUM + elif not self.xlang and isinstance(serializer, (ObjectSerializer, StatefulSerializer)): + type_id = TypeId.NAMED_EXT + else: + type_id = None + if not self.xlang and not self.require_registration: from pyfory import struct as struct_module data_class_types = tuple( - cls - for cls in ( + serializer_type + for serializer_type in ( getattr(struct_module, "DataClassSerializer", None), getattr(struct_module, "DataClassStubSerializer", None), ) - if cls is not None + if serializer_type is not None ) if data_class_types and isinstance(serializer, data_class_types): type_id = TypeId.NAMED_STRUCT @@ -890,39 +950,49 @@ def get_type_info(self, cls, create=True): namespace=cls.__module__, typename=cls.__qualname__, serializer=serializer, + internal=True, ) def _set_type_info(self, typeinfo): serializer_type_resolver = self._actual_type_resolver type_id = typeinfo.type_id - if is_struct_type(type_id): - from pyfory.struct import DataClassSerializer, DataClassStubSerializer - - # Set a stub serializer FIRST to break recursion for self-referencing types. - # get_type_info() only calls _set_type_info when serializer is None, - # so setting stub first prevents re-entry for circular type references. - typeinfo.serializer = DataClassStubSerializer(serializer_type_resolver, typeinfo.cls) - - if self.meta_share: - type_def = encode_typedef(serializer_type_resolver, typeinfo.cls) - if type_def is not None: - typeinfo.serializer = type_def.create_serializer(serializer_type_resolver) - typeinfo.type_def = type_def - else: - typeinfo.serializer = DataClassSerializer(serializer_type_resolver, typeinfo.cls) + previous_serializer = typeinfo.serializer + previous_type_def = typeinfo.type_def + try: + if is_struct_type(type_id): + from pyfory.struct import DataClassSerializer, DataClassStubSerializer + + if typeinfo.serializer is None or isinstance(typeinfo.serializer, DataClassStubSerializer): + # Recursive construction needs the stub to be temporarily visible. Restore the + # prior owner if later TypeDef or serializer construction fails. + typeinfo.serializer = DataClassStubSerializer(serializer_type_resolver, typeinfo.cls) + if self.meta_share: + type_def = encode_typedef(serializer_type_resolver, typeinfo.cls) + if type_def is not None: + typeinfo.serializer = type_def.create_serializer(serializer_type_resolver) + typeinfo.type_def = type_def + else: + typeinfo.serializer = DataClassSerializer(serializer_type_resolver, typeinfo.cls) + else: + typeinfo.serializer = DataClassSerializer(serializer_type_resolver, typeinfo.cls) + elif self.meta_share and typeinfo.type_def is None and TypeId.is_type_share_meta(type_id): + typeinfo.type_def = encode_typedef(serializer_type_resolver, typeinfo.cls) else: - typeinfo.serializer = DataClassSerializer(serializer_type_resolver, typeinfo.cls) - else: - typeinfo.serializer = self._create_serializer(typeinfo.cls) - if ( - self.meta_share - and typeinfo.type_def is None - and ( - TypeId.is_namespaced_type(type_id) - or (needs_user_type_id(type_id) and typeinfo.user_type_id is not None and typeinfo.user_type_id != NO_USER_TYPE_ID) - ) - ): - typeinfo.type_def = encode_typedef(serializer_type_resolver, typeinfo.cls) + if typeinfo.serializer is None: + typeinfo.serializer = self._create_serializer(typeinfo.cls) + if ( + self.meta_share + and typeinfo.type_def is None + and ( + TypeId.is_namespaced_type(type_id) + or (needs_user_type_id(type_id) and typeinfo.user_type_id is not None and typeinfo.user_type_id != NO_USER_TYPE_ID) + ) + ): + typeinfo.type_def = encode_typedef(serializer_type_resolver, typeinfo.cls) + except Exception: + typeinfo.serializer = previous_serializer + typeinfo.type_def = previous_type_def + raise return typeinfo @@ -1057,6 +1127,18 @@ def _load_metabytes_to_type_info(self, ns_metabytes, type_metabytes): if self.strict: name = ns + "." + typename if ns else typename raise TypeUnregisteredError(f"{name} not registered") + if not ns and "." in typename: + split_ns, split_typename = typename.rsplit(".", 1) + typeinfo = self._named_type_to_type_info.get((split_ns, split_typename)) + if typeinfo is not None: + self._cache_wire_type_info(ns_metabytes, type_metabytes, typeinfo) + return typeinfo + typename = split_typename + ns = split_ns + if typename: + matches = [info for (reg_ns, reg_typename), info in self._named_type_to_type_info.items() if reg_typename == typename] + if len(matches) == 1: + return matches[0] cls = load_class(ns + "#" + typename, policy=self.policy) typeinfo = self.get_type_info(cls) self._cache_wire_type_info(ns_metabytes, type_metabytes, typeinfo) @@ -1107,35 +1189,10 @@ def read_type_info(self, read_context, expected_typeinfo=None): ) ns_metabytes = read_context.meta_string_reader.read_encoded_meta_string(buffer) type_metabytes = read_context.meta_string_reader.read_encoded_meta_string(buffer) - typeinfo = self._ns_type_to_type_info.get((ns_metabytes, type_metabytes)) - if typeinfo is None: - ns = ns_metabytes.decode(self.namespace_decoder) - typename = type_metabytes.decode(self.typename_decoder) - typeinfo = self._named_type_to_type_info.get((ns, typename)) - if typeinfo is None and self.strict: - name = ns + "." + typename if ns else typename - raise TypeUnregisteredError(f"{name} not registered") - if typeinfo is None and typename: - alt_typename = typename[0].upper() + typename[1:] - typeinfo = self._named_type_to_type_info.get((ns, alt_typename)) - if typeinfo is not None: - self._cache_wire_type_info(ns_metabytes, type_metabytes, typeinfo) - return typeinfo - if not ns and "." in typename: - split_ns, split_typename = typename.rsplit(".", 1) - typeinfo = self._named_type_to_type_info.get((split_ns, split_typename)) - if typeinfo is not None: - self._cache_wire_type_info(ns_metabytes, type_metabytes, typeinfo) - return typeinfo - typename = split_typename - ns = split_ns - if typename and not self.strict: - matches = [info for (reg_ns, reg_typename), info in self._named_type_to_type_info.items() if reg_typename == typename] - if len(matches) == 1: - return matches[0] - name = ns + "." + typename if ns else typename - raise TypeUnregisteredError(f"{name} not registered") - return typeinfo + return self._load_metabytes_to_type_info( + ns_metabytes, + type_metabytes, + ) if type_id in {TypeId.ENUM, TypeId.STRUCT, TypeId.EXT, TypeId.TYPED_UNION}: user_type_id = buffer.read_var_uint32() return self.get_type_info_by_id(type_id, user_type_id=user_type_id) diff --git a/python/pyfory/serialization.pyx b/python/pyfory/serialization.pyx index 169e42fd9d..43b873caff 100644 --- a/python/pyfory/serialization.pyx +++ b/python/pyfory/serialization.pyx @@ -257,6 +257,7 @@ cdef class TypeResolver: cdef readonly bint strict cdef readonly bint compatible cdef readonly bint field_nullable + cdef readonly bint _registry_frozen cdef readonly object policy cdef readonly bint meta_share cdef readonly dict _types_info @@ -292,6 +293,7 @@ cdef class TypeResolver: self.strict = resolver.strict self.compatible = resolver.compatible self.field_nullable = resolver.field_nullable + self._registry_frozen = False self.policy = resolver.policy self.meta_share = resolver.meta_share self._types_info = resolver._types_info @@ -348,11 +350,11 @@ cdef class TypeResolver: cdef TypeInfo typeinfo cdef uint8_t previous_type_id cdef uint32_t previous_user_type_id + self.resolver._check_registry_mutable() typeinfo = self.resolver.get_type_info(cls) previous_type_id = typeinfo.type_id previous_user_type_id = typeinfo.user_type_id self.resolver.register_serializer(cls, serializer) - typeinfo = self.resolver.get_type_info(cls) if previous_type_id != typeinfo.type_id or previous_user_type_id != typeinfo.user_type_id: if ( previous_type_id == TypeId.ENUM @@ -1224,6 +1226,7 @@ cdef class Fory: ) def dump(self, obj, stream): + self.type_resolver._registry_frozen = True try: self.buffer.set_writer_index(0) self.buffer.bind_output_stream(Buffer.wrap_output_stream(stream)) @@ -1247,6 +1250,7 @@ cdef class Fory: def serialize(self, obj, Buffer buffer=None, buffer_callback=None, unsupported_callback=None): cdef Buffer write_buffer + self.type_resolver._registry_frozen = True try: write_buffer = self._serialize( obj, @@ -1286,6 +1290,7 @@ cdef class Fory: return buffer def deserialize(self, buffer, buffers=None, unsupported_objects=None): + self.type_resolver._registry_frozen = True try: return self._deserialize( buffer, diff --git a/python/pyfory/tests/test_function.py b/python/pyfory/tests/test_function.py index 46ab15b55f..57934cfe7d 100644 --- a/python/pyfory/tests/test_function.py +++ b/python/pyfory/tests/test_function.py @@ -27,14 +27,8 @@ def test_lambda_functions_serialization(): ) test_input = 5 - # Register the necessary types - fory.register_type(tuple) - fory.register_type(list) - # dict is already registered by default with MapSerializer - # Simple lambda simple_lambda = lambda x: x * 2 # noqa: E731 - fory.register_type(type(simple_lambda)) serialized = fory.serialize(simple_lambda) deserialized = fory.deserialize(serialized) assert simple_lambda(test_input) == deserialized(test_input) @@ -64,16 +58,10 @@ def complex_function(a, b, c=10): return a * b + c # Test regular function - fory.register_type(type(add_one)) serialized = fory.serialize(add_one) deserialized = fory.deserialize(serialized) assert add_one(test_input) == deserialized(test_input) - # Register the necessary types for complex functions - fory.register_type(tuple) - fory.register_type(list) - # dict is already registered by default with MapSerializer - # Test complex function serialized = fory.serialize(complex_function) deserialized = fory.deserialize(serialized) @@ -88,11 +76,6 @@ def test_nested_functions_serialization(): compatible=False, ) - # Register the necessary types - fory.register_type(tuple) - fory.register_type(list) - # dict is already registered by default with MapSerializer - def outer_function(x): def inner_function(y): return x + y @@ -101,7 +84,6 @@ def inner_function(y): # Create a nested function nested_func = outer_function(10) - fory.register_type(type(nested_func)) serialized = fory.serialize(nested_func) deserialized = fory.deserialize(serialized) @@ -117,11 +99,6 @@ def test_local_class_serialization(): compatible=False, ) - # Register the necessary types - fory.register_type(tuple) - fory.register_type(list) - # dict is already registered by default with MapSerializer - def create_local_class(): from dataclasses import dataclass @@ -133,7 +110,6 @@ class LocalClass: return LocalClass(42, "test") local_obj = create_local_class() - fory.register_type(type(local_obj)) serialized = fory.serialize(local_obj) deserialized = fory.deserialize(serialized) diff --git a/python/pyfory/tests/test_serializer.py b/python/pyfory/tests/test_serializer.py index 8815610988..22113c556a 100644 --- a/python/pyfory/tests/test_serializer.py +++ b/python/pyfory/tests/test_serializer.py @@ -792,6 +792,24 @@ def __init__(self, f1=None): self.f1 = f1 +class FrozenRegistration: + pass + + +class RejectedRegistration: + pass + + +@dataclass +class FrozenChild: + value: int + + +@dataclass +class FrozenParent: + child: FrozenChild + + def test_register_py_serializer(): fory = Fory(xlang=False, ref=True, strict=False, compatible=False) @@ -800,11 +818,9 @@ def write(self, write_context, value): write_context.write_int32(value.f1) def read(self, read_context): - a = A() - a.f1 = read_context.read_int32() - return a + return RegisterClass(read_context.read_int32()) - fory.register_type(A, serializer=Serializer(fory.type_resolver, RegisterClass)) + fory.register_type(RegisterClass, serializer=Serializer(fory.type_resolver, RegisterClass)) assert fory.deserialize(fory.serialize(RegisterClass(100))).f1 == 100 @@ -828,17 +844,26 @@ def read(self, read_context): self.read_count += 1 return Value(read_context.read_int32() - 17) - fory = Fory(xlang=True, ref=False, compatible=False) + fory = Fory(xlang=True, ref=False, compatible=True) if registration == "id": fory.register_type(Value, type_id=701) else: fory.register_type(Value, name="test.ReplacedValue") + assert fory.type_resolver.get_serializer(Value) is not None + assert fory.type_resolver.get_type_info(Value).type_def is not None replacement = ReplacementSerializer(fory.type_resolver) fory.register_serializer(Value, replacement) - assert fory.type_resolver.get_serializer(Value) is replacement + type_info = fory.type_resolver.get_type_info(Value) + assert type_info.serializer is replacement + assert type_info.type_def is None + if registration == "id": + assert fory.type_resolver._user_type_id_to_type_info[701] is type_info + else: + assert fory.type_resolver.get_type_info_by_name("test", "ReplacedValue") is type_info assert fory.deserialize(fory.serialize(Value(25))) == Value(25) + assert type_info.serializer is replacement assert (replacement.write_count, replacement.read_count) == (1, 1) @@ -902,6 +927,208 @@ def test_register_type_name_exclusive(): fory.register_type(A, type_id=100, name="example.A") +@pytest.mark.parametrize("identity", ["id", "name"]) +def test_registration_identity_atomic(identity): + fory = Fory(xlang=True, compatible=False) + options = {"type_id": 100} if identity == "id" else {"name": "test.SharedName"} + type_info = fory.register_type(A, **options) + + with pytest.raises(TypeError): + fory.register_type(RejectedRegistration, **options) + + assert fory.type_resolver.get_type_info(RejectedRegistration, create=False) is None + if identity == "id": + assert fory.type_resolver.get_type_info_by_id(type_info.type_id, 100) is type_info + else: + assert fory.type_resolver.get_type_info_by_name("test", "SharedName") is type_info + + +def test_union_registration_atomic(): + fory = Fory(xlang=True, compatible=True) + serializer = BarSerializer(fory.type_resolver, RejectedRegistration) + type_info = fory.register_union( + RejectedRegistration, + name="test.FirstUnion", + serializer=serializer, + ) + + with pytest.raises(TypeError, match="registered already"): + fory.register_union( + RejectedRegistration, + name="test.SecondUnion", + serializer=serializer, + ) + + assert fory.type_resolver.get_type_info(RejectedRegistration) is type_info + assert fory.type_resolver.get_type_info_by_name("test", "FirstUnion") is type_info + assert fory.type_resolver.get_type_info_by_name("test", "SecondUnion") is None + + +def test_lazy_type_keeps_explicit_name(): + explicit_type = type("SharedType", (), {"__module__": "registry_owner"}) + lazy_type = type("SharedType", (), {"__module__": "registry_owner"}) + fory = Fory(xlang=False, strict=False, compatible=False) + type_info = fory.register_type(explicit_type, name="registry_owner.SharedType") + fory.serialize(None) + + with pytest.raises(TypeError, match="type name"): + fory.serialize(lazy_type()) + + assert fory.type_resolver.get_type_info(lazy_type, create=False) is None + assert fory.type_resolver.get_type_info_by_name("registry_owner", "SharedType") is type_info + + +@pytest.mark.parametrize("root", ["serialize", "deserialize", "dump"]) +def test_registry_freezes_at_root(root): + fory = Fory(xlang=True, compatible=False) + fory.register_type(FrozenRegistration, type_id=701) + + if root == "serialize": + fory.serialize(None) + elif root == "deserialize": + with pytest.raises(Exception): + fory.deserialize(b"") + else: + fory.dump(None, io.BytesIO()) + + registrations = ( + lambda: fory.register(RejectedRegistration, type_id=702), + lambda: fory.register_type(RejectedRegistration, name="test.Rejected"), + lambda: fory.register_union( + RejectedRegistration, + name="test.RejectedUnion", + serializer=object(), + ), + lambda: fory.register_serializer(FrozenRegistration, object()), + lambda: fory.type_resolver.register_type(RejectedRegistration, type_id=702), + lambda: fory.type_resolver.register_union( + RejectedRegistration, + name="test.RejectedUnion", + serializer=object(), + ), + lambda: fory.type_resolver.register_serializer(FrozenRegistration, object()), + ) + for registration in registrations: + with pytest.raises(RuntimeError): + registration() + assert fory.type_resolver.get_type_info(RejectedRegistration, create=False) is None + assert fory.type_resolver.get_type_info(FrozenRegistration).type_id == TypeId.STRUCT + + +def test_factory_root_freeze(): + fory = Fory(xlang=True, compatible=False) + + def factory(type_resolver, cls): + fory.serialize(None) + return BarSerializer(type_resolver, cls) + + with pytest.raises(RuntimeError): + fory.register_type(FrozenRegistration, serializer=factory) + assert fory.type_resolver.get_type_info(FrozenRegistration, create=False) is None + + +def test_type_setup_freeze(): + fory = Fory(xlang=False, strict=False, compatible=False) + fory.register_type(FrozenRegistration) + armed = False + + class RootDuringMro(type): + def __getattribute__(cls, name): + nonlocal armed + if armed and name == "__mro__": + armed = False + fory.serialize(None) + return super().__getattribute__(name) + + class Reduced(metaclass=RootDuringMro): + def __reduce__(self): + return Reduced, () + + armed = True + with pytest.raises(RuntimeError): + fory.register_type(Reduced) + assert fory.type_resolver.get_type_info(Reduced, create=False) is None + + +def test_serializer_registration_freeze(): + fory = Fory(xlang=False, strict=False, compatible=False) + armed = False + + class RootDuringHash(type): + def __hash__(cls): + nonlocal armed + if armed: + armed = False + fory.serialize(None) + return super().__hash__() + + class Registered(metaclass=RootDuringHash): + pass + + fory.register_type(Registered, name="test.Registered") + typeinfo = fory.type_resolver.get_type_info(Registered) + serializer = typeinfo.serializer + armed = True + with pytest.raises(RuntimeError): + fory.register_serializer( + Registered, + BarSerializer(fory.type_resolver, Registered), + ) + assert typeinfo.serializer is serializer + + +def test_registered_types_build_lazily(): + fory = Fory(xlang=True, compatible=True) + parent_info = fory.register_type(FrozenParent, name="test.FrozenParent") + child_info = fory.register_type(FrozenChild, name="test.FrozenChild") + assert parent_info.serializer is None + assert child_info.serializer is None + + value = FrozenParent(FrozenChild(7)) + data = fory.serialize(value) + + assert parent_info.serializer is not None + assert child_info.serializer is not None + assert parent_info.type_def is not None + assert child_info.type_def is not None + assert fory.deserialize(data) == value + + +def test_lazy_completion_is_atomic(monkeypatch): + from pyfory import registry + + fory = Fory(xlang=True, compatible=True) + type_info = fory.register_type(FrozenChild, name="test.AtomicChild") + encode_typedef = registry.encode_typedef + + def fail_typedef(*_args, **_kwargs): + raise ValueError("TypeDef failure") + + monkeypatch.setattr(registry, "encode_typedef", fail_typedef) + with pytest.raises(ValueError, match="TypeDef failure"): + fory.serialize(FrozenChild(7)) + assert type_info.serializer is None + assert type_info.type_def is None + + monkeypatch.setattr(registry, "encode_typedef", encode_typedef) + value = FrozenChild(7) + assert fory.deserialize(fory.serialize(value)) == value + + +def test_lazy_dataclass_serializer(): + from pyfory.struct import DataClassStubSerializer + + fory = Fory(xlang=False, strict=False, compatible=False) + type_info = fory.register_type(FrozenChild) + assert isinstance(type_info.serializer, DataClassStubSerializer) + + value = FrozenChild(7) + data = fory.serialize(value) + + assert not isinstance(type_info.serializer, DataClassStubSerializer) + assert fory.deserialize(data) == value + + def test_np_types(): fory = Fory(xlang=False, ref=True, strict=False, compatible=False) o1 = [1, True, np.dtype(np.int32)] diff --git a/python/pyfory/tests/test_thread_safe.py b/python/pyfory/tests/test_thread_safe.py index 3bfb7a0d9b..d7111283cd 100644 --- a/python/pyfory/tests/test_thread_safe.py +++ b/python/pyfory/tests/test_thread_safe.py @@ -18,6 +18,7 @@ import threading from dataclasses import dataclass +import pytest from pyfory import ThreadSafeFory @@ -188,8 +189,5 @@ def test_thread_safe_fory_register_after_use(): person = Person(name="Alice", age=30) fory.serialize(person) - try: + with pytest.raises(RuntimeError): fory.register(Address) - assert False, "Should raise RuntimeError" - except RuntimeError as e: - assert "Cannot register types after Fory instances have been created" in str(e) diff --git a/scala/fory-scala/src/main/java/org/apache/fory/serializer/scala/ScalaSerializers.java b/scala/fory-scala/src/main/java/org/apache/fory/serializer/scala/ScalaSerializers.java index 595d2c7885..1330e825b6 100644 --- a/scala/fory-scala/src/main/java/org/apache/fory/serializer/scala/ScalaSerializers.java +++ b/scala/fory-scala/src/main/java/org/apache/fory/serializer/scala/ScalaSerializers.java @@ -19,15 +19,14 @@ package org.apache.fory.serializer.scala; -import java.util.Collections; -import java.util.Map; import java.util.Objects; -import java.util.WeakHashMap; import org.apache.fory.Fory; import org.apache.fory.ThreadSafeFory; import org.apache.fory.annotation.Internal; import org.apache.fory.config.Config; import org.apache.fory.resolver.TypeResolver; +import org.apache.fory.scala.ForyScala$; +import org.apache.fory.serializer.Serializer; import scala.collection.immutable.NumericRange; import scala.collection.immutable.Range; @@ -35,32 +34,27 @@ import static org.apache.fory.serializer.scala.ToFactorySerializers.MapToFactoryClass; public class ScalaSerializers { - private static final Map INSTALLED_FORY = - Collections.synchronizedMap(new WeakHashMap<>()); - public static void registerSerializers(ThreadSafeFory fory) { - fory.register(ScalaSerializers::registerSerializers); + fory.register(ForyScala$.MODULE$); } public static void registerSerializers(Fory fory) { - synchronized (INSTALLED_FORY) { - if (INSTALLED_FORY.containsKey(fory)) { - return; - } - INSTALLED_FORY.put(fory, Boolean.TRUE); - } + fory.register(ForyScala$.MODULE$); + } + + @Internal + public static void installSerializers(Fory fory) { TypeResolver resolver = fory.getTypeResolver(); - try { - fory.registerSerializerFactory(new ScalaSerializerFactory()); - if (resolver.isCrossLanguage()) { - return; - } - Config config = resolver.getConfig(); + fory.registerSerializerFactory(new ScalaSerializerFactory()); + if (resolver.isCrossLanguage()) { + return; + } + Config config = resolver.getConfig(); - resolver.registerSerializer( - IterableToFactoryClass, new ToFactorySerializers.IterableToFactorySerializer(config)); - resolver.registerSerializer( - MapToFactoryClass, new ToFactorySerializers.MapToFactorySerializer(config)); + resolver.registerSerializer( + IterableToFactoryClass, new ToFactorySerializers.IterableToFactorySerializer(config)); + resolver.registerSerializer( + MapToFactoryClass, new ToFactorySerializers.MapToFactorySerializer(config)); // Seq resolver.register(scala.collection.immutable.Seq.class); @@ -181,19 +175,15 @@ public static void registerSerializers(Fory fory) { resolver.register(scala.collection.mutable.Queue$.class); resolver.register(scala.collection.mutable.Stack.class); resolver.register(scala.collection.mutable.Stack$.class); - resolver.register(scala.collection.mutable.BitSet.class); - resolver.register(scala.collection.mutable.BitSet$.class); - } catch (RuntimeException | Error e) { - synchronized (INSTALLED_FORY) { - INSTALLED_FORY.remove(fory); - } - throw e; - } + resolver.register(scala.collection.mutable.BitSet.class); + resolver.register(scala.collection.mutable.BitSet$.class); } public static void registerEnum(Fory fory, Class cls, long typeId) { TypeResolver resolver = fory.getTypeResolver(); - resolver.registerEnum(cls, typeId, new ScalaEnumSerializer(resolver, cls)); + resolver.checkRegistrationOpen(); + Serializer serializer = new ScalaEnumSerializer(resolver, cls); + resolver.registerEnum(cls, typeId, serializer); registerEnumRuntimeAliases(fory, cls); } @@ -221,15 +211,19 @@ private static void checkTypeName(String typeName) { public static void registerEnum(Fory fory, Class cls, String name) { TypeResolver resolver = fory.getTypeResolver(); + resolver.checkRegistrationOpen(); String[] parts = splitName(name); - resolver.registerEnum(cls, parts[0], parts[1], new ScalaEnumSerializer(resolver, cls)); + Serializer serializer = new ScalaEnumSerializer(resolver, cls); + resolver.registerEnum(cls, parts[0], parts[1], serializer); registerEnumRuntimeAliases(fory, cls); } public static void registerEnum(Fory fory, Class cls, String namespace, String typeName) { checkTypeName(typeName); TypeResolver resolver = fory.getTypeResolver(); - resolver.registerEnum(cls, namespace, typeName, new ScalaEnumSerializer(resolver, cls)); + resolver.checkRegistrationOpen(); + Serializer serializer = new ScalaEnumSerializer(resolver, cls); + resolver.registerEnum(cls, namespace, typeName, serializer); registerEnumRuntimeAliases(fory, cls); } diff --git a/scala/fory-scala/src/main/scala-3/org/apache/fory/scala/ForySerializer.scala b/scala/fory-scala/src/main/scala-3/org/apache/fory/scala/ForySerializer.scala index bf839fc1bb..87a1d34c8b 100644 --- a/scala/fory-scala/src/main/scala-3/org/apache/fory/scala/ForySerializer.scala +++ b/scala/fory-scala/src/main/scala-3/org/apache/fory/scala/ForySerializer.scala @@ -106,11 +106,15 @@ object ForySerializer { @Internal def registerSerializer[T](fory: Fory, cls: Class[T])(using serializer: ForySerializer[T]): Unit = { + val resolver = fory.getTypeResolver + resolver.checkRegistrationOpen() if serializer.isUnion then { throw new IllegalArgumentException("Use ForySerializer.register for Scala union serializers") } - val resolver = fory.getTypeResolver - resolver.setSerializer(cls, serializer.createSerializer(resolver)) + resolver.checkRegistrationOpen() + val runtimeSerializer = serializer.createSerializer(resolver) + resolver.checkRegistrationOpen() + resolver.setSerializer(cls, runtimeSerializer) } private def register[T]( @@ -123,11 +127,15 @@ object ForySerializer { checkTypeName(typeName) } val resolver = fory.getTypeResolver + resolver.checkRegistrationOpen() serializer match { case _ if serializer.isUnion => - val unionSerializer = serializer.createSerializer(resolver) + resolver.checkRegistrationOpen() + val runtimeSerializer = serializer.createSerializer(resolver) + resolver.checkRegistrationOpen() + val runtimeClasses = serializer.handledRuntimeClasses(cls) if typeId != null then { - resolver.registerUnion(cls, typeId.longValue(), unionSerializer) + resolver.registerUnion(cls, typeId.longValue(), runtimeSerializer) } else { val unionNamespace = if namespace != null then namespace else Option(cls.getPackage).map(_.getName).orNull @@ -136,14 +144,19 @@ object ForySerializer { cls, if unionNamespace == null then "" else unionNamespace, unionTypeName, - unionSerializer) + runtimeSerializer) } - serializer.handledRuntimeClasses(cls).foreach { runtimeClass => + runtimeClasses.foreach { runtimeClass => ScalaSerializers.registerRuntimeTypeAlias(fory, runtimeClass, cls) } case _ => + // Generated TypeDef construction resolves this registered STRUCT identity. Publish the + // identity first, then recheck freeze after construction before installing the serializer. registerType(fory, cls, typeId, namespace, typeName) - resolver.setSerializer(cls, serializer.createSerializer(resolver)) + val runtimeSerializer = serializer.createSerializer(resolver) + resolver.checkRegistrationOpen() + // Preserve the registered STRUCT TypeInfo; registerSerializer would reclassify it as EXT. + resolver.setSerializer(cls, runtimeSerializer) } } diff --git a/scala/fory-scala/src/main/scala/org/apache/fory/scala/ForyScala.scala b/scala/fory-scala/src/main/scala/org/apache/fory/scala/ForyScala.scala index f891272bcb..c71a966a77 100644 --- a/scala/fory-scala/src/main/scala/org/apache/fory/scala/ForyScala.scala +++ b/scala/fory-scala/src/main/scala/org/apache/fory/scala/ForyScala.scala @@ -27,5 +27,5 @@ import org.apache.fory.serializer.scala.ScalaSerializers object ForyScala extends ForyModule { def builder(): ForyBuilder = Fory.builder().withModule(this) - override def install(fory: Fory): Unit = ScalaSerializers.registerSerializers(fory) + override def install(fory: Fory): Unit = ScalaSerializers.installSerializers(fory) } diff --git a/swift/Sources/Fory/Fory.swift b/swift/Sources/Fory/Fory.swift index d243ca9936..eb7b00c6a9 100644 --- a/swift/Sources/Fory/Fory.swift +++ b/swift/Sources/Fory/Fory.swift @@ -363,7 +363,7 @@ public final class Fory { private func serializeRoot( _ body: (WriteContext) throws -> Void ) throws -> Data { - try typeResolver.finishRegistration() + typeResolver.freezeRegistration() let context = writeContext context.buffer.clear() defer { @@ -379,7 +379,7 @@ public final class Fory { to output: inout Data, _ body: (WriteContext) throws -> Void ) throws { - try typeResolver.finishRegistration() + typeResolver.freezeRegistration() let context = writeContext context.buffer.clear() defer { @@ -395,7 +395,7 @@ public final class Fory { data: Data, _ body: (ReadContext) throws -> R ) throws -> R { - try typeResolver.finishRegistration() + typeResolver.freezeRegistration() return try withReusableReadContext(data: data) { context in try readHead(buffer: context.buffer) let value = try body(context) @@ -411,7 +411,7 @@ public final class Fory { from buffer: ByteBuffer, _ body: (ReadContext) throws -> R ) throws -> R { - try typeResolver.finishRegistration() + typeResolver.freezeRegistration() readContext.buffer.swapState(with: buffer) readContext.remainingGraphMemoryBytes = Int(self.config.maxGraphMemoryBytes) readContext.remainingUnbackedContainerItems = self.config.maxUnbackedContainerItems diff --git a/swift/Sources/Fory/ReadContext.swift b/swift/Sources/Fory/ReadContext.swift index 2f59cf7570..82debbe5cd 100644 --- a/swift/Sources/Fory/ReadContext.swift +++ b/swift/Sources/Fory/ReadContext.swift @@ -455,6 +455,17 @@ public final class ReadContext { return try requireCompatibleOwner(cached, for: localTypeInfo) } + // Ref and checked header-cache hits must not complete local metadata. Build it only after + // both miss, then compare the received protocol identity before parsing the remote body. + if localTypeInfo.typeDefBytes == nil { + try localTypeInfo.ensureTypeMeta(resolver: typeResolver) + } + if headerHash == localTypeInfo.typeDefHeaderHash { + try buffer.skip(bodySize) + compatibleTypeDefTypeInfos.push(localTypeInfo) + return localTypeInfo + } + let cachedTypeInfo = try readTypeInfoBody( start: typeMetaStart, headerHash: headerHash, diff --git a/swift/Sources/Fory/Serializer.swift b/swift/Sources/Fory/Serializer.swift index 6e8359f5df..7980f99182 100644 --- a/swift/Sources/Fory/Serializer.swift +++ b/swift/Sources/Fory/Serializer.swift @@ -66,7 +66,7 @@ public protocol StructSerializer: Serializer { static func foryFieldsInfo(trackRef: Bool) -> [TypeMeta.FieldInfo] /// Builds field metadata after all serializer registrations are visible to the resolver. - /// Serialization and deserialization hot paths must use finalized TypeInfo metadata instead. + /// The registered TypeInfo completes this metadata lazily on first use. static func foryFieldsInfo( trackRef: Bool, resolveSerializerTypeId: (Any.Type) throws -> TypeId diff --git a/swift/Sources/Fory/TypeResolver.swift b/swift/Sources/Fory/TypeResolver.swift index 3f90359b32..cd2f4edb24 100644 --- a/swift/Sources/Fory/TypeResolver.swift +++ b/swift/Sources/Fory/TypeResolver.swift @@ -271,7 +271,7 @@ public final class TypeInfo: @unchecked Sendable { let evolving: Bool let namespace: MetaString let typeName: MetaString - /// Finalized local metadata. Generated compatible readers use this for local field comparison; + /// Local metadata prepared on demand. Generated compatible readers use it for field comparison; /// remote metadata remains exposed through `compatibleTypeMeta`. public private(set) var typeMeta: TypeMeta? public var compatibleTypeMeta: TypeMeta? { remoteCompatibleTypeMeta ?? typeMeta } @@ -466,12 +466,19 @@ public final class TypeInfo: @unchecked Sendable { context.writeStaticTypeInfo(wireTypeID) switch wireTypeID { case .compatibleStruct, .namedCompatibleStruct: + // Generic type lookup must not prepare metadata; this wire owner does so only on a miss. + if typeDefBytes == nil { + try ensureTypeMeta(resolver: context.typeResolver) + } guard typeDefBytes != nil else { throw ForyError.invalidData("missing compatible type definition for \(typeID)") } try context.writeTypeMeta(self) case .namedEnum, .namedStruct, .namedExt, .namedUnion: if context.compatible { + if typeDefBytes == nil { + try ensureTypeMeta(resolver: context.typeResolver) + } guard typeDefBytes != nil else { throw ForyError.invalidData("missing compatible type definition for \(typeID)") } @@ -501,8 +508,8 @@ public final class TypeInfo: @unchecked Sendable { } @inline(never) - func finalizeTypeMeta(resolver: TypeResolver) throws { - guard typeDefBytes == nil, let typeMetaFieldsBuilder else { + func ensureTypeMeta(resolver: TypeResolver) throws { + guard let typeMetaFieldsBuilder else { return } let fields = try typeMetaFieldsBuilder(resolver) @@ -516,7 +523,7 @@ public final class TypeInfo: @unchecked Sendable { ) let typeDefBytes = try typeMeta.encode() let typeDefHeaderHash = try encodedTypeDefHeaderHash(typeDefBytes) - self.typeMeta = try TypeMeta( + let resolvedTypeMeta = try TypeMeta( typeID: compatibleWireTypeID.rawValue, userTypeID: registerByName ? nil : userTypeID, namespace: namespace, @@ -525,6 +532,9 @@ public final class TypeInfo: @unchecked Sendable { fields: fields, headerHash: typeDefHeaderHash ) + // Publish only after all fallible work succeeds. A failed lazy build keeps its builder and + // can be retried without exposing partial metadata. + self.typeMeta = resolvedTypeMeta self.typeDefBytes = typeDefBytes self.typeDefHeaderHash = typeDefHeaderHash self.typeDefHasUserTypeFields = encodedTypeDefHasUserTypeFields(fields) @@ -622,19 +632,17 @@ final class TypeResolver { private static let maxRemoteTypeMetaKeys = 8192 private let trackRef: Bool - private var registrationFinished = false + private var registryFrozen = false private var bySerializerType = UInt64Map(initialCapacity: 64) private var byTargetType = UInt64Map(initialCapacity: 64) private var byUserTypeID = UInt64Map(initialCapacity: 64) private var byTypeName: [TypeNameKey: TypeInfo] = [:] - private var registeredTypeInfos: [TypeInfo] = [] private var builtinTypeInfoByID: [TypeInfo?] = [] // Never key this cache by the complete header: its low 12 framing bits may vary on a hit. private var typeInfoByHeaderHash = UInt64Map(initialCapacity: 64) private var remoteSchemaVersionsByType: [String: Int] = [:] private var totalAcceptedSchemaVersions = 0 - init(trackRef: Bool = false) { self.trackRef = trackRef seedBuiltinTypeInfos() @@ -832,19 +840,11 @@ final class TypeResolver { } @inline(__always) - func finishRegistration() throws { - if registrationFinished { + func freezeRegistration() { + if registryFrozen { return } - try finishRegistrationSlow() - } - - @inline(never) - private func finishRegistrationSlow() throws { - for typeInfo in registeredTypeInfos { - try typeInfo.finalizeTypeMeta(resolver: self) - } - registrationFinished = true + registryFrozen = true } func register(_ type: T.Type, id: UInt32) throws { @@ -900,6 +900,8 @@ final class TypeResolver { bodyReader: registeredBodyReader(for: T.self) ) + // Static serializer witnesses above are application code and may start a root. Recheck + // after the last witness and before either returning or publishing the TypeInfo. if let existing = bySerializerType.value( for: UInt64(UInt(bitPattern: serializerTypeID))), existing.matches( @@ -910,9 +912,11 @@ final class TypeResolver { typeName: (namespace: "", name: "") ) { + try ensureRegistrationAllowed() return } + try ensureRegistrationAllowed() store(typeInfo, userTypeID: id) } @@ -972,6 +976,8 @@ final class TypeResolver { bodyReader: registeredBodyReader(for: T.self) ) + // Static serializer witnesses above are application code and may start a root. Recheck + // after the last witness and before either returning or publishing the TypeInfo. if let existing = bySerializerType.value( for: UInt64(UInt(bitPattern: serializerTypeID))), existing.matches( @@ -982,9 +988,11 @@ final class TypeResolver { typeName: (namespace: namespace, name: typeName) ) { + try ensureRegistrationAllowed() return } + try ensureRegistrationAllowed() store(typeInfo, typeNameKey: TypeNameKey(namespace: namespace, typeName: typeName)) } @@ -1037,6 +1045,7 @@ final class TypeResolver { if let cached = typeInfoByHeaderHash.value(for: headerHash) { return cached } + try localTypeInfo.ensureTypeMeta(resolver: self) if localTypeInfo.typeDefHeaderHash == headerHash { // A validated 52-bit hash is the complete schema identity. The local metadata bytes // may use different current-frame low bits, so byte equality must not decide ownership. @@ -1044,7 +1053,7 @@ final class TypeResolver { return localTypeInfo } guard let localTypeMeta = localTypeInfo.typeMeta else { - throw ForyError.invalidData("local type metadata for \(localTypeInfo.typeID) is not finalized") + throw ForyError.invalidData("local type metadata for \(localTypeInfo.typeID) is unavailable") } let canonicalTypeMeta = try typeMeta.assigningFieldIDs(from: localTypeMeta) // Failed compatibility checks must not consult or mutate persistent remote accounting. @@ -1124,7 +1133,6 @@ final class TypeResolver { if let typeNameKey { byTypeName[typeNameKey] = typeInfo } - registeredTypeInfos.append(typeInfo) } @inline(never) @@ -1323,7 +1331,7 @@ final class TypeResolver { } private func ensureRegistrationAllowed() throws { - guard !registrationFinished else { + guard !registryFrozen else { throw ForyError.invalidData( "cannot register more types after top-level serialize/deserialize has frozen registration" ) diff --git a/swift/Tests/ForyTests/CollectionSerializerTests.swift b/swift/Tests/ForyTests/CollectionSerializerTests.swift index 814478b829..cb8eb9fea4 100644 --- a/swift/Tests/ForyTests/CollectionSerializerTests.swift +++ b/swift/Tests/ForyTests/CollectionSerializerTests.swift @@ -792,7 +792,6 @@ func generatedReadProgress() throws { let fory = Fory(config: Config(trackRef: false, compatible: true)) try fory.register(AdvancingReadStruct.self, id: 9705) - try fory.typeResolver.finishRegistration() let local = try fory.typeResolver.requireTypeInfo(for: AdvancingReadStruct.self) let emptyMeta = try TypeMeta( typeID: TypeId.compatibleStruct.rawValue, diff --git a/swift/Tests/ForyTests/DecoderStateTests.swift b/swift/Tests/ForyTests/DecoderStateTests.swift index 624374e2dd..74cc278bf1 100644 --- a/swift/Tests/ForyTests/DecoderStateTests.swift +++ b/swift/Tests/ForyTests/DecoderStateTests.swift @@ -143,7 +143,6 @@ func remoteSchemaLogicalKeyLimitPersists() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() let localTypeInfo = try resolver.requireTypeInfo(for: Person.self) func remoteTypeMeta( diff --git a/swift/Tests/ForyTests/ExternalTypeSerializationTests.swift b/swift/Tests/ForyTests/ExternalTypeSerializationTests.swift index a37a23859b..bed6ebf62b 100644 --- a/swift/Tests/ForyTests/ExternalTypeSerializationTests.swift +++ b/swift/Tests/ForyTests/ExternalTypeSerializationTests.swift @@ -1203,3 +1203,14 @@ func registrationFreezesAtFirstRoot() throws { try fory.register(UserSerializer.self, id: 131) } } + +@Test +func failedRootFreezesRegistration() throws { + let fory = Fory() + #expect(throws: ForyError.self) { + let _: Int32 = try fory.deserialize(Data()) + } + #expect(throws: ForyError.self) { + try fory.register(UserSerializer.self, id: 132) + } +} diff --git a/swift/Tests/ForyTests/ForySwiftTests.swift b/swift/Tests/ForyTests/ForySwiftTests.swift index d7478d1a5e..439f7965f6 100644 --- a/swift/Tests/ForyTests/ForySwiftTests.swift +++ b/swift/Tests/ForyTests/ForySwiftTests.swift @@ -26,6 +26,40 @@ struct Address: Equatable { var zip: Int32 } +private enum IDRegistrationProbe: Serializer { + case value + + nonisolated(unsafe) static let resolver = TypeResolver(config: Config()) + + static var staticTypeId: TypeId { + resolver.freezeRegistration() + return .ext + } + + static func defaultValue(_: ReadContext) throws -> IDRegistrationProbe { .value } + + static func writeData(_: IDRegistrationProbe, _: WriteContext) throws {} + + static func readData(_: ReadContext) throws -> IDRegistrationProbe { .value } +} + +private enum NameRegistrationProbe: Serializer { + case value + + nonisolated(unsafe) static let resolver = TypeResolver(config: Config()) + + static var staticTypeId: TypeId { + resolver.freezeRegistration() + return .ext + } + + static func defaultValue(_: ReadContext) throws -> NameRegistrationProbe { .value } + + static func writeData(_: NameRegistrationProbe, _: WriteContext) throws {} + + static func readData(_: ReadContext) throws -> NameRegistrationProbe { .value } +} + @ForyStruct struct Person: Equatable { var id: Int64 @@ -641,7 +675,6 @@ func schemaLimitTracksStructTypesSeparately() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() func remoteTypeMeta(userTypeID: UInt32, fieldName: String) throws -> TypeMeta { try TypeMeta( @@ -687,7 +720,6 @@ func nonStructTypeMetaUsesSchemaLimit() throws { let config = Config(maxSchemaVersionsPerType: 1) let resolver = TypeResolver(config: config) try resolver.register(SparseStatus.self, name: "example.SharedEnum") - try resolver.finishRegistration() let namespace = try MetaStringEncoder.namespace.encode("example") let typeName = try MetaStringEncoder.typeName.encode("SharedEnum") @@ -728,8 +760,8 @@ func localNonStructMetaBypassesLimit() throws { let config = Config(compatible: true, maxSchemaVersionsPerType: 1) let resolver = TypeResolver(config: config) try resolver.register(SparseStatus.self, name: "example.SharedEnum") - try resolver.finishRegistration() let localTypeInfo = try resolver.requireTypeInfo(for: SparseStatus.self) + try localTypeInfo.ensureTypeMeta(resolver: resolver) let namespace = try MetaStringEncoder.namespace.encode("example") let typeName = try MetaStringEncoder.typeName.encode("SharedEnum") @@ -763,7 +795,7 @@ func localNonStructMetaBypassesLimit() throws { } @Test -func typeMetaUsesFinalRegistration() throws { +func typeMetaUsesAllRegistrations() throws { func holderTypeDefBytes(registerFieldTypeFirst: Bool) throws -> [UInt8] { let resolver = TypeResolver(config: Config(compatible: true)) if registerFieldTypeFirst { @@ -773,8 +805,10 @@ func typeMetaUsesFinalRegistration() throws { try resolver.register(LateMetaHolder.self, name: "example.LateMetaHolder") try resolver.register(LateMetaExt.self, name: "example.LateMetaExt") } - try resolver.finishRegistration() - return try resolver.requireTypeInfo(for: LateMetaHolder.self).typeDefBytes! + resolver.freezeRegistration() + let typeInfo = try resolver.requireTypeInfo(for: LateMetaHolder.self) + try typeInfo.ensureTypeMeta(resolver: resolver) + return typeInfo.typeDefBytes! } let fieldFirst = try holderTypeDefBytes(registerFieldTypeFirst: true) @@ -792,7 +826,6 @@ func failedSchemaDoesNotConsumeLimit() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() func remoteTypeMeta(fieldName: String, fieldType: TypeMeta.FieldType) throws -> TypeMeta { try TypeMeta( @@ -853,7 +886,6 @@ func staticTypeRejectsWrongMetaOwner() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() let wrongTypeMeta = try TypeMeta( typeID: TypeId.compatibleStruct.rawValue, userTypeID: 901, @@ -893,7 +925,6 @@ func cachedMetaChecksConcreteOwner() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() let remote = try TypeMeta( typeID: TypeId.compatibleStruct.rawValue, userTypeID: 901, @@ -941,7 +972,6 @@ func failedStaticMetaDoesNotCount() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() func typeMeta(userTypeID: UInt32, fieldName: String) throws -> TypeMeta { try TypeMeta( @@ -1180,6 +1210,25 @@ func registrationIsRejectedAfterFirstTopLevelUse() throws { } } +@Test +func registrationRechecksBeforeStore() throws { + let idResolver = IDRegistrationProbe.resolver + #expect(throws: ForyError.self) { + try idResolver.register(IDRegistrationProbe.self, id: 901) + } + #expect(throws: ForyError.self) { + _ = try idResolver.requireTypeInfo(for: IDRegistrationProbe.self) + } + + let nameResolver = NameRegistrationProbe.resolver + #expect(throws: ForyError.self) { + try nameResolver.register(NameRegistrationProbe.self, name: "probe.name") + } + #expect(throws: ForyError.self) { + _ = try nameResolver.requireTypeInfo(for: NameRegistrationProbe.self) + } +} + @Test func serializeToAppendsRoots() throws { let fory = Fory() diff --git a/swift/Tests/ForyTests/GraphMemoryBudgetTests.swift b/swift/Tests/ForyTests/GraphMemoryBudgetTests.swift index d7b28cb1f3..65939f6fbf 100644 --- a/swift/Tests/ForyTests/GraphMemoryBudgetTests.swift +++ b/swift/Tests/ForyTests/GraphMemoryBudgetTests.swift @@ -716,7 +716,6 @@ func unknownCaseChargesDynamicHeapBox() throws { let config = Config(compatible: false) let resolver = TypeResolver(config: config) try resolver.register(DynamicBoxBudgetV1.self, id: 9821) - try resolver.finishRegistration() let value = DynamicBoxBudgetV1(first: 1, second: 2, third: 3, fourth: 4) let buffer = ByteBuffer() let writeContext = WriteContext( diff --git a/swift/Tests/ForyTests/TypeMetaDepthTests.swift b/swift/Tests/ForyTests/TypeMetaDepthTests.swift index cc92b3733f..f389a03080 100644 --- a/swift/Tests/ForyTests/TypeMetaDepthTests.swift +++ b/swift/Tests/ForyTests/TypeMetaDepthTests.swift @@ -61,7 +61,6 @@ func remoteTypeMetaUsesFixedDepth() throws { let config = Config(compatible: true, maxDepth: 2) let resolver = TypeResolver(config: config) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() func context(_ encoded: [UInt8]) -> ReadContext { let buffer = ByteBuffer() @@ -129,7 +128,6 @@ func cachedMetaUsesHeaderHash() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() let remote = try TypeMeta( typeID: TypeId.compatibleStruct.rawValue, userTypeID: 901, @@ -189,10 +187,11 @@ func localMetaUsesHeaderHash() throws { let resolver = TypeResolver(config: config) try resolver.register(Person.self, id: 901) try resolver.register(Address.self, id: 902) - try resolver.finishRegistration() let firstTypeInfo = try resolver.requireTypeInfo(for: Person.self) + try firstTypeInfo.ensureTypeMeta(resolver: resolver) let firstBytes = try #require(firstTypeInfo.typeDefBytes) let localTypeInfo = try resolver.requireTypeInfo(for: Address.self) + try localTypeInfo.ensureTypeMeta(resolver: resolver) let headerHash = try #require(localTypeInfo.typeDefHeaderHash) let currentBody: [UInt8] = [0xD1, 0xD2, 0xD3] let currentHeader = (headerHash << 12) | UInt64(currentBody.count)