feat(provider-tck): add Java conformance suite for OpenFeature providers

[aepfli]

Closes #1829

Draft — opened for review of the approach, not because it is unfinished. The suite is green and the module passes the full codequality,deploy gate. See Open questions for the decisions I would most like challenged.

What this is

A conformance suite any OpenFeature provider can adopt to verify it implements the provider contract of the specification. OpenFeature's central promise is that swapping providers does not change application behaviour — nothing verifies that today, and every provider tests differently.

New module tools/provider-tck carries the canonical Gherkin, the Cucumber step definitions, and an abstract JUnit Platform Suite that owns the entire test lifecycle. It also carries a standardised backend control API (OpenAPI) and the canonical flag set, both derived from what flagd-testbed's launchpad already implements.

flagd is the first adopter. flagd-testbed is not modified, and the existing flagd e2e suites are untouched.

Full design rationale is in #1829; this description covers what a reviewer needs.

The adoption surface

Four methods with no default, everything else a convention. The flagd adoption in full — 48 lines of code:

public class FlagdTckTest extends AbstractProviderTckTest {

    @Override public File composeFile() {
        return new File("src/test/resources/tck/docker-compose.yaml");
    }

    @Override public List<Integer> backendPorts() {
        return Collections.singletonList(RPC_PORT);
    }

    @Override public FeatureProvider createProvider(BackendEndpoint endpoint) {
        return new FlagdProvider(FlagdOptions.builder()
                .resolverType(Config.Resolver.RPC)
                .host(endpoint.host())
                .port(endpoint.port(RPC_PORT))
                .deadline(1000).retryGracePeriod(2).retryBackoffMs(500)
                .build());
    }

    @Override public FeatureProvider createUnavailableProvider() { /* localhost:9999 */ }
}

Plus a 15-line compose file and a one-line META-INF/services registration. createProvider is a factory rather than a field because external ports are dynamically mapped and unknown until the stack is running.

The structure follows tools/flagd-api-testkit, which already solved this adoption problem: features packaged in the JAR and selected with @SelectClasspathResource, so consumers need no git submodule; normally test-scoped deps promoted to compile; ServiceLoader discovery.

The two normative requirements

Both are in openapi/control-api.yaml and both are easy to get wrong.

Containers are never stopped or restarted mid-suite. Backend unavailability is simulated inside the running stack — process kill, proxy toxic, socket block. This is portability, not preference: Testcontainers cannot reliably preserve dynamically mapped host ports across a container restart, and which bindings preserve them differs by language. A TCK that restarts containers works in one language and mysteriously fails in another. flagd's testbed already does the right thing by killing the flagd process inside a container that keeps running.

/start resets flag state; /restart preserves it. An outage must be observable as a change in availability, never as a change in flag values. Scenario isolation depends on that split.

Findings from the first real run

1. The flagd provider silently narrows a float flag to an integer. Evaluating float-flag (0.5) through getIntegerDetails returns 0 with no error code at all — not TYPE_MISMATCH with the code default. The application sees a plausible value and no indication anything went wrong.

Handled by not declaring Capability.STRICT_NUMERIC_TYPING in FlagdTckTest, so the scenario reports as skipped with the reason printed rather than silently passing. Needs its own issue against the flagd provider.

2. Cucumber parallelism silently corrupts the suite. providers/flagd's junit-platform.properties sets cucumber.execution.parallel.enabled=true, which the TCK inherited just by being on that classpath. Scenarios raced each other's control-API calls — one scenario's /start restarted the backend underneath another's disconnect assertion — and the symptom looked like a flaky provider, not a broken test. AbstractProviderTckTest now pins serial execution via @ConfigurationParameter, which overrides the consuming module's properties file. Enforced rather than documented, because the failure mode is so misleading.

Verification

Check	Result
mvn --projects tools/provider-tck -P codequality,deploy clean verify	passes — checkstyle, PMD, SpotBugs, Spotless/Palantir, javadoc failOnWarnings=true
Compiles under maven.compiler.release=11	yes — Testcontainers 2.x targets Java 8 bytecode, so it links on the compile classpath
mvn --projects providers/flagd -P e2e test -Dtest=FlagdTckTest	15 scenarios: 14 passed, 1 skipped, 0 failed
Capability gating reports skips visibly	yes — reason appears in the surefire report, never a silent pass
Compose stack started exactly once per suite	yes
Existing flagd e2e suites	RunFileTest 151, RunInProcessTest 223, RunRpcTest 213 — all green

Testcontainers is pinned to 2.0.4, matching what providers/flagd already uses, so two Testcontainers majors never share a test classpath.

Open questions

Where I would most value pushback:

1. Is @strict-numeric-typing acceptable as a capability? It models a spec violation as an optional feature, which puts it on the same footing as "this provider has no streaming transport". It is pragmatic — a provider with the defect can adopt today and see the gap reported explicitly instead of being unable to adopt at all — but a separate "known deviations" concept might be more honest.
2. Should the Gherkin, control API spec and canonical flag set live in open-feature/spec? They are language-agnostic contract definitions, not Java artifacts, and every language's TCK must agree on them byte for byte or "conformance" means nothing. The POM carries a comment describing the migration; consumers would see no difference. Cross-language tracking issue to follow.
3. Is STATIC the right expected reason for a flag with no targeting rules, universally across providers, or does it need to be a per-provider expectation?
4. Scenario scope. This is a representative subset covering each architectural mechanism once. Is the boundary in the right place?

Known gaps

• Context passthrough is unverifiable. The suite builds evaluation contexts but cannot assert one reached the backend — that needs an echo operation on the control API. A provider that silently drops the context passes today.
• Targeting/bucketing (out of scope by design — backend logic), caching, hooks, and flag metadata scenarios are not covered.
• TckRuntime is static, so one TCK suite runs per JVM fork at a time.
• flagd in-process mode is not covered — it is a copy of FlagdTckTest with a different resolver, left out to keep the reviewable surface small.

---

Update: both flagd resolver modes, expanded type-mismatch matrix

Both modes now covered

flagd resolves flags two quite different ways — RPC evaluates remotely over gRPC, in-process syncs the ruleset and evaluates locally. Both are now under the TCK.

This exposed a leak in the adoption surface and I fixed the base class rather than absorbing the boilerplate. Harness discovery went through ServiceLoader, which becomes ambiguous the moment a provider registers a second harness; resolving it needed a system property plus one Surefire execution per mode in every adopter's POM. That is exactly the kind of leak this design is supposed to avoid.

A concrete suite class already is a ProviderTckHarness, so the TCK now asks the JUnit Platform which suite is running: TckSuiteListener, a TestExecutionListener auto-registered from this JAR, reports the executing suite class and TckRuntime instantiates it.

Consequences:

• Adding a mode is one class and nothing else — no registration file, no system property, no build configuration.
• Adoption drops to a single file; the META-INF/services registration is gone, retained only as a documented fallback for launchers that disable listener auto-registration.
• flagd's two modes are a shared AbstractFlagdTckTest plus a four-line subclass each.

One genuinely per-mode difference worth noting for other providers: in-process needs a longer initialisation deadline than RPC, because it syncs the whole ruleset before reporting ready. The unavailable provider keeps a short deadline so the init-failure scenarios still assert promptness rather than just eventual failure.

Type-mismatch coverage expanded

From one case to the full non-numeric matrix — string, boolean, integer, float and object each requested as every incompatible type, 15 cases. Each asserts the full three-part contract: the code default is returned, TYPE_MISMATCH is reported, and nothing is thrown.

All 15 pass in both modes, including structured-flag-as-scalar.

Numeric coercion stays separate under @strict-numeric-typing, because "is 0.5 an integer?" has a defensible wrong answer whereas "is a string a boolean?" does not.

Finding refined

Both resolvers narrow float to int identically — 0, no error code. That places the defect in the shared provider layer rather than in either transport, so the capability is withheld on the shared base class rather than per mode. It is the only scenario either mode does not pass.

Verification

Suite	Result
FlagdRpcTckTest	29 scenarios — 28 passed, 1 skipped, 0 failed
FlagdInProcessTckTest	29 scenarios — 28 passed, 1 skipped, 0 failed
RunFileTest / RunInProcessTest / RunRpcTest	151 / 223 / 213 — all green, unaffected
mvn -P codequality,deploy clean verify on the module	passes

Both suites run in one JVM fork, sequentially, each with its own Compose stack, with correct harness selection and no configuration.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bb5687a0-c9b9-45e1-a0a1-d53781ec2757

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}