Roadmap: cloud-native, composable data products #1

🤖 Generated by the Agentic Engineer

Evidence

• The maintainer wants to turn the completed data-product thesis into a new cloud-native portfolio product that can provision or integrate data sources, expose useful query APIs, compose products into larger products, and participate in data spaces.
• The archived thesis proved a schema-driven product with inputs, outputs, REST, GraphQL, ownership, telemetry, catalog integration, and a dashboard, but packaged those capabilities as one coupled application.
• The Open Data Mesh Data Product Descriptor Specification provides a technology-neutral descriptor intended for discovery, access, deployment, and composition.
• W3C DCAT 3 standardizes catalogs, datasets, distributions, and data services. OpenAPI and AsyncAPI provide machine-readable contracts for request/response and event-driven interfaces.
• The Eclipse Dataspace Protocol provides an interoperability path for catalog publication, usage-policy negotiation, and governed data transfer between autonomous participants.

Audience and problem

Data owners need a repeatable way to turn a new or existing source into a self-describing, operable data product. Data consumers need to discover, query, and compose those products without adopting the producer's implementation or central UI. Platform operators need declarative lifecycle and health signals without storing application data in the Kubernetes API.

Hypothesis

A Kubernetes controller with a small DataProduct API, standard interface descriptors and composable product references will let teams publish independently deployed data products that other products and applications can consume through their published APIs. Consumers own their UIs and transport; public metadata does not grant data access. Any client library is optional, and no product UI runtime is required for integration.

The maintainer chose API-only integration on 2026-10-14. #390 tracks the migration, beginning with the registry workspace in #391 and PR #393, followed by versioned contract/runtime retirement in #392 and Platform adoption. Legacy embedding work is historical delivery; its rollout acceptance is not a prerequisite for the new direction.

Success signal

Within 30 days of the foundation release:

• one product backed by a platform-provisioned source and one product integrating an existing source are both discoverable through the public registry;
• a third product composes at least one interface from each without copying their data or product-specific UI code;
• the catalog discovers all three, and a consumer-owned application combines their published interfaces without importing publisher UI code;
• an independent client consumes the same named output and standard API contract without a required proprietary library;
• every product reports actionable readiness, interface, ownership, and dependency status while credentials remain in Kubernetes Secrets rather than CR fields.

Guardrails: no application data or credentials are stored in Kubernetes CRs; the registry does not execute publisher code or query product data; interfaces are allow-listed and machine-described; cross-product dependencies are explicit; and no data-space publication bypasses authentication, policy, or usage-control decisions. Legacy compatibility controls remain enforced until their coordinated retirement; source merge, release publication and installed acceptance remain separate evidence.

Smallest useful change

Ship and host a v1alpha1 foundation containing the DataProduct CRD, composition-aware readiness, a descriptor/registry API, a default-off descriptor-driven UI, one independently described demo product, an OCI image and Helm chart, and a production Platform deployment with public smoke tests.

Roadmap

1. Bootstrap and host the v1alpha1 controller/API/UI foundation.
2. Provision new data sources through pluggable Kubernetes resource references.
3. Integrate existing sources and expose standard query contracts.
4. Compose products with dependency, schema, and lineage status.
5. Complete API-only integration, retire legacy UI declarations/runtime and verify independent consumer use (Make data product integration API-only #390).
6. Add DCAT and Dataspace Protocol interoperability with policy-aware exchange.

Acceptance criteria

• The roadmap is decomposed into first-class sub-issues with independent acceptance criteria.
• The v1alpha1 API, compatibility guarantees, composition model, and UI trust boundary are documented under docs/adr/.
• Controller, API, UI, image, and chart have automated tests and supply-chain checks.
• The initial product is reachable on the Platform and verified through its real public user path.
• At least one independent client consumes a named output through its standard API contract without adopting a product UI runtime or importing controller-specific code.
• Success is measured after release and the roadmap is revised from evidence.

Rough size: XL, delivered as independently releasable children.