test(delta-harness): add harness-core framework and scenarios by mkuchenbecker · Pull Request #682 · linkedin/openhouse

mkuchenbecker · 2026-08-13T20:08:33Z

Summary

This PR adds the delta-harness execution framework and a fixed 34-case foundation
catalog: 10 scalar data-type cases, 12 representative DML cases, and 12 DML-validation
cases. Catalog.scala (Catalog object, not ScenarioCatalog) is the sole catalog
assembly point; it composes foundationContributions (this PR's three named case
lists) with each child layer's one-line extensionContributions entry.

Stack

Layer	Branch	Parent	Contribution	Combined	PR
Core foundation	`delta-harness-oss`	`main`	34	34	this PR
Standard DML	`delta-harness-standard-dml`	Core foundation	96	130	#715
Schema and types	`delta-harness-schema-types`	Core foundation	60	94	#717
Catalog constraints	`delta-harness-catalog-constraints`	Core foundation	24	58	#718
Column defaults	`delta-harness-column-defaults`	Core foundation	4	38	#720

Direct children of this PR: Standard DML, Schema and types, Catalog constraints, and
Column defaults.

Scope

This layer owns table naming and ownership-gated cleanup (Framework.scala), the
portable retry/parallel execution contract (Runner.scala), the typed scenario kit
(ScenarioKit.scala), and Catalog.scala's composition point. It owns exactly three
named contributions and excludes every other capability; each is reviewed as its own
PR.

Contribution	Cases	Coverage
`dataTypeCases` (`ScenarioDataType.scala`)	10	Round trip of long, int, double, decimal, and string; all-null, NaN/Infinity, boundary-value, and unicode/empty-string inserts.
`dmlCases` (`ScenarioDml.scala`)	12	Six representative operations (`read.projection`, `insert.into`, `insert.overwrite`, `delete.byPredicate`, `update.byPredicate`, `merge.upsert`) on each columnar format.
`dmlValidationCases` (`ScenarioDmlValidation.scala`)	12	Rejected DELETE on an undeclared column, nondeterministic DELETE/UPDATE predicates, short-value INSERT, and MERGE clauses that assign or match a target row twice.

Validation

Check	Result
`git diff --check` against `main` (three-dot, merge-base `ba400b38`)	Clean, 18 files, +2093/-6.
Added lines	ASCII only.
Embedded catalog	34 combined cases: 34 passed, 0 skipped, 0 failed.
`CaseCatalogTest`	Pins the exact ordered 34-case foundation and checks catalog-wide uniqueness.
`TableLifecycleTest`	Ownership-gated cleanup and cleanup-failure suppression.
`RunnerTest` / `RowGeneratorTest`	Retry, terminal-failure, and deterministic row-generation checks.
`verifyOpenHouseFoundation` (attached to `check`)	Runs the fixed 34-case foundation so representative embedded behavior cannot drift unnoticed.

Run a local slice with:

export JAVA_HOME=$(/usr/libexec/java_home -v 17)
./gradlew --no-daemon \
  :integrations:spark:openhouse-spark-delta-harness_2.12:runOpenHouse \
  --args='read.projection'

Adds a self-contained Scala behavioral test harness that characterizes OpenHouse + Apache Iceberg table behavior end-to-end. The harness crosses a large matrix of table layouts (partitioning, MoR/CoW, ordered writes, nested types) with DDL, DML, maintenance, branching/WAP, streaming, and negative-path operations, asserting deltas against observed pre-state so each case holds under any layout. It runs locally against a real embedded OpenHouse catalog (harness/openhouse/Env.scala boots OpenHouseLocalServer + the OpenHouse Spark catalog; see run-openhouse.sh and HARNESS-GUIDE.md). The scenario and framework sources are also structured as a publishable Gradle library module (openhouse-spark-delta-harness_2.12) that excludes the embedded-only Env so downstream environments can supply their own adapter. Genuine product or upstream bugs are tagged in Plan.knownBugs with a prose explanation and skipped rather than silently passed, so the suite stays green while documenting the defect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds TESTING-MATRIX.md, a living reference that explains the harness as a cross product of independent axes (operation family, data file format, partitioning, write mode, schema, preparation lineage, and reference routing). Documents how a case id reads, the CoreTable/NestedTypesTable/TypesTable schemas, the table layouts, the preparation lineages, and each operation family including the DDL sub-families. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Move each test's preparation, action, and assertions into its scenario file so the complete behavior is readable in one place. Keep reusable preparation recipes while creating a fresh table for every case. Preserve the exact 2,574-case catalog, ordering, and known-bug behavior with regression tests for the catalog fingerprint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Explain the scenario-owned test structure, immutable preparations, and fresh-table isolation used by the localized test cases. Describe the matrix as living documentation for the current harness architecture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-08-26T16:20:56Z

+          TablePreparation(
+            layout.label,
+            createAndSeed(layout, 3)
+              .sql("ddl")(table => s"ALTER TABLE $table ADD COLUMN cc int")(),


what ddl does is unclear here. ITs a function.

mkuchenbecker · 2026-08-26T16:21:21Z

+          layout.label.endsWith("/orc"))
+      .flatMap { layout =>
+        val preparations = List(
+          TablePreparation(


Each preparation needs a description.

mkuchenbecker · 2026-08-26T16:21:46Z

+        layout.label.endsWith("/parquet") ||
+          layout.label.endsWith("/orc"))


Why not just ahve a list of layouts?

mkuchenbecker · 2026-08-26T16:22:15Z

+            "ddlConsume:writeOrder."),
+          TablePreparation(
+            layout.label,
+            createAndSeed(layout, 3)


Create and seed should be separate calls. What is seeded should be visible or standard.

Separate table preparations from DML operations so each case shows its starting state, mutation, and relative assertions in one place. Keep feature-owned scenarios in removable RTAS, merge-on-read, and branch layers while preserving the exact ordered 2,572-case catalog. Run the same published sources through the local Gradle task and the acceptance adapter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the standard branch focused on copy-on-write behavior, shared table preparations, bespoke DDL coverage, and the local execution framework. Remove RTAS, merge-on-read, branch, and WAP scenario ownership from this layer. Pin the resulting ordered standard catalog at 1,181 cases so each child branch can add one reviewable feature delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-08-26T23:05:10Z

+    // (b) toJson takes no format-version argument, so the key serializes the same regardless of format version.
+    // (c) The value round-trips through fromJson then toJson.
+    val reparsed = org.apache.iceberg.SchemaParser.fromJson(json)
+    val json2 = org.apache.iceberg.SchemaParser.toJson(reparsed)


Just import. Apply this across all of these deep calls.

unaddressed

Keep runtime case metadata limited to stable identifiers and execution state. Put preparation and test explanations beside their Scala behavior so reviewers can read each case without tracing string registries. Generate a fresh UUID for every table and begin cleanup only after the preparation creates it, which preserves any pre-existing table on a name conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Extract the owned-table cleanup state machine behind a package-private boundary so its failure paths can be tested without starting Spark. Pin conflict preservation, successful cleanup, and suppression of cleanup failure behind the primary test failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-01T19:09:45Z

+    deleteByNullCondition)
+
+  /**
+   * DELETE WHERE datepartition = '2024-01-01-00' removes the rows in that partition value, keeps


datepartition is a smell. Instead of that, its a date column that is partitioned. Adding the name partition to the col sows confusion.

mkuchenbecker · 2026-09-01T19:13:57Z

+// size, and the compaction plan. These behaviors have no catalog SQL surface of their own, so a
+// case reaches them through the Iceberg API or a Spark configuration and asserts the result a
+// caller can observe.
+trait ForkScenarios extends ScenarioKit {


forkscenarios are Iceberg tests. they should all run agains the fork, so its unclear why this is a file vs denormalized into other features.

mkuchenbecker · 2026-09-01T19:14:53Z

+   * Reflectively builds an optional int NestedField carrying the given initial default. Returns
+   * None when the builder API is absent, so a caller can assert that absence directly.
+   */
+  private def buildDefaultedIntField(id: Int, name: String, dflt: Int): Option[org.apache.iceberg.types.Types.NestedField] = {


move default values to its own PR.

mkuchenbecker · 2026-09-01T19:16:34Z

+   * DataFrame into a 4-partition table under each mode therefore yields at least as many files
+   * under the default as under HASH. The file format is the parameter.
+   */
+  private def forkPartitionDistDefault(fmt: String)(ctx: Ctx): Unit = {


this test doesn't seem useful. is it? a write distribution mode test makes sense but not a None test. To me this is purely spark and nothing specific to iceberg being tested. That may be fine depending on how we phrase it.

mkuchenbecker · 2026-09-01T19:22:35Z

+  val typesLayouts: List[Layout] =
+    List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table =>


constructing this list literal is suspect vs getting a standard "AllFileTypes". this is a nit for now, but a standard list is perferred.

Reflow harness documentation to the repository's 120-column target and explain the DML operation and preparation matrix at its source. Name the reusable date column independently from partitioning so layouts, not column identifiers, express partition choices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-01T19:55:02Z

+object Main {
+  def main(args: Array[String]): Unit = {


Main seems to ahve nothing to do with env.

mkuchenbecker · 2026-09-01T19:57:13Z

+    }
+
+  /** The rejected DML statements, on the preparedCoreFormats preparations. */
+  val negativeCases: List[Plan.Case] =


compositions seem like they should be at teh top or bottom (top is preferred IMO). We can organize by public and private.

Replace provenance and consequence buckets with capability-owned scenario files whose public contribution surfaces explain the catalog at a glance. Separate local runner code from the publishable harness, make preparations show creation and standard seeding explicitly, and reindex generic case IDs. Use generated table names and failure-preserving ownership boundaries for every case-owned table, view, registration, rename, and lock lifecycle. Move column-default coverage out of the standard layer for a dedicated follow-up PR while pinning the remaining 1,177-case catalog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-02T00:58:05Z

@@ -0,0 +1,219 @@
+package harness


This should fail when you attempt to do so across RTAS to a previous table. Make sure RTAS PR capturs that.

mkuchenbecker · 2026-09-02T00:58:49Z

@@ -0,0 +1,43 @@
+package harness
+
+/**


Stack on top unless tags need RTAS testing.

mkuchenbecker · 2026-09-02T00:59:12Z

@@ -0,0 +1,137 @@
+package harness


can be stacked on top

mkuchenbecker · 2026-09-02T00:59:46Z

+import java.nio.file.{Files, Paths}
+
+/**
+ * Encryption: the OSS build writes table data in plaintext, because OpenHouse delegates table-data encryption to an


can be stacked on top unless there is the chance RTAS silently drops encryption.

mkuchenbecker · 2026-09-02T01:01:15Z

+
+  /** The plaintext data-file case, on the standard seeded Parquet table. */
+  lazy val encryptionCases: List[Plan.Case] =
+    List(dataFilePlaintextCase(preparedStandardTable("parquet")))


orc should be the default test choice or both orc and parquet. Both is better. I would jsut standardize to do both.

mkuchenbecker · 2026-09-02T01:05:59Z

@@ -0,0 +1,61 @@
+package harness


rtas should test that after an RTAS sort order can be changed or dropped if its not already

mkuchenbecker · 2026-09-02T01:06:33Z

@@ -0,0 +1,213 @@
+package harness


can be stacked on top

mkuchenbecker · 2026-09-02T01:07:04Z

@@ -0,0 +1,98 @@
+package harness


RTAS definately needs time travel testing as a negative test case.

mkuchenbecker · 2026-09-02T01:07:24Z

@@ -0,0 +1,161 @@
+package harness


can be stacked on top

mkuchenbecker · 2026-09-02T01:07:37Z

@@ -0,0 +1,50 @@
+package harness


can be stacked on top

Name every scenario source and trait ScenarioFoo so scenario files group together and the framework files remain visually distinct. Preserve the catalog contributions, IDs, ordering, count, and fingerprint unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-02T01:37:51Z

+   * here rather than derived from `Plan.contributions`, so adding, dropping, renaming or reordering a capability fails
+   * this test until the intended catalog shape is restated.
+   */
+  private val expectedContributionNames = List(


this test is a tautaulogy.

mkuchenbecker · 2026-09-02T01:38:04Z

+ */
+final class CaseCatalogTest {
+  private val expectedCaseCount = 1177
+  private val expectedCatalogSha256 =


does this regenerate automatically?

Keep PR 682 focused on reusable DDL and DML coverage while moving orthogonal capabilities to extension branches. - retain 642 Parquet and ORC foundation cases - extract reusable changelog and concurrency support - preserve Plan and Scenarios consumer compatibility - add extension-stable catalog and support contract tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker · 2026-09-02T16:23:20Z

+    readTestCases ++ deleteTestCases ++ updateTestCases
+
+  /**
+   * Every DML case, with the partition-predicate DELETE marked as a known bug: the Spark and Iceberg rewrite crashes on


"Marked as a known bug"

This is a stateful comment. Either there is a test that is disabled with a reason why (i.e. currently broken) or the test enabled.

So if there is a crash, there should be a disabled test that crashes.

mkuchenbecker · 2026-09-02T16:24:26Z

+        assert(
+          selected == before.rows.map(_.get(Core.long0)).filter(_ >= 2).sorted,
+          s"filtered read returned $selected")
+        assert(after == before, "a read leaves the rows and the snapshot count unchanged")


I don't think asserting no mutation is useful.

mkuchenbecker · 2026-09-02T16:25:16Z

+        val after = table.state
+
+        assert(
+          selected == before.rows.map(_.get(Core.long0)).filter(_ >= 2).sorted,


this assumes there is data in the table. This assumption should be called out.

Keep only generated ID uniqueness and the ownership and cleanup failure behaviors that can fail independently. Scenario definitions are validated by running their behavior cases instead of restating source declarations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Place scenario definitions, suite composition, and scenario-facing tests in dedicated directories while preserving the harness package and public class names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep source documentation aligned with behavior-based validation and the new framework and scenario directory boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the foundation focused on the execution framework and one exact read, append, overwrite, delete, update, and merge contract across Parquet and ORC. Move matrix breadth to capability-owned child branches so reviewers can validate the harness before reviewing generated scale. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove the general HTTP helper from the core framework. The locking capability owns its generated Tables client in the later compatibility layer where the control-plane operations are exercised. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the core context and runner limited to the Spark session and table namespace they execute. The compatibility layer introduces the Tables client and lock lifecycle where locking is first exercised. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename the shared catalog and demonstrate framework composition with DML, data-type, and validation scenarios. Make the foundation catalog part of the standard check lifecycle and cover retry, cleanup, seed, naming, and configuration contracts with focused tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the root catalog contract stable when descendant layers register additional scenario contributions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Describe the scalar cases at the level their assertions prove and update the catalog-test classpath rationale. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Retry only transient session creation failures before a case starts, strengthen scalar and MERGE rejection assertions, and pin the portable jar boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep the foundation fixture surface limited to table primitives used by its 34 cases. Rename the DML scenario contributors so their scope and rejection semantics are explicit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker and others added 5 commits August 13, 2026 12:47

docs(delta-harness): move guides to stack

4064251

Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker changed the title ~~delta-harness: OpenHouse Iceberg behavioral test harness + guide~~ delta-harness: add OpenHouse behavioral test matrix Aug 25, 2026

mkuchenbecker commented Aug 26, 2026

View reviewed changes

Comment thread integrations/spark/delta-harness/src/main/scala/harness/openhouse/DmlScenarios.scala Outdated

mkuchenbecker and others added 2 commits August 26, 2026 12:57

This was referenced Aug 26, 2026

test(delta-harness): add merge-on-read matrix mkuchenbecker/openhouse#38

Closed

test(delta-harness): add RTAS matrix mkuchenbecker/openhouse#39

Closed

test(delta-harness): add branch and WAP matrix mkuchenbecker/openhouse#40

Closed

mkuchenbecker commented Aug 26, 2026

View reviewed changes

mkuchenbecker and others added 3 commits August 27, 2026 18:02

test(delta-harness): cover cleanup failure

808fcc9

Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mkuchenbecker changed the title ~~delta-harness: add OpenHouse behavioral test matrix~~ test(delta-harness): add standard matrix Sep 1, 2026

mkuchenbecker commented Sep 1, 2026

View reviewed changes

mkuchenbecker commented Sep 2, 2026

View reviewed changes

mkuchenbecker and others added 10 commits September 2, 2026 12:40

refactor(delta-harness): separate scenario sources

ac7ae9d

Place scenario definitions, suite composition, and scenario-facing tests in dedicated directories while preserving the harness package and public class names. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

docs(delta-harness): describe scenario ownership

0ac34dd

Keep source documentation aligned with behavior-based validation and the new framework and scenario directory boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

test(delta-harness): pin foundation cases

694ee01

Keep the root catalog contract stable when descendant layers register additional scenario contributions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

docs(delta-harness): align coverage descriptions

7c523fe

Describe the scalar cases at the level their assertions prove and update the catalog-test classpath rationale. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

test(delta-harness): make failure oracles trustworthy

8fc5c4b

Retry only transient session creation failures before a case starts, strengthen scalar and MERGE rejection assertions, and pin the portable jar boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This was referenced Sep 4, 2026

test(delta-harness): add standard DML scenarios #715

Closed

test(delta-harness): add schema and type scenarios #717

Closed

test(delta-harness): add catalog constraint scenarios #718

Closed

test(delta-harness): add column default scenarios #720

Closed

mkuchenbecker changed the title ~~test(delta-harness): add standard matrix~~ test(delta-harness): add harness-core framework and scenarios Sep 4, 2026

mkuchenbecker closed this Sep 8, 2026

		layout.label.endsWith("/parquet") \|\|
		layout.label.endsWith("/orc"))

		val typesLayouts: List[Layout] =
		List("parquet", "orc", "avro").map(format => Layout(s"types-unpartitioned/$format", table =>

Uh oh!

Conversation

mkuchenbecker commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Stack

Scope

Validation

Uh oh!

mkuchenbecker Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mkuchenbecker commented Aug 13, 2026 •

edited

Loading

mkuchenbecker Aug 26, 2026 •

edited

Loading