test: add robust testing to streaming ingestion integration tests - #20060
test: add robust testing to streaming ingestion integration tests#20060zhang-arvin wants to merge 2 commits into
Conversation
Adds FaultyStreamEventStreamGenerator that can inject faulty data (invalid JSON, null/empty fields, multi-row data, empty strings) at a configurable ratio into the streaming ingestion pipeline. Adds KafkaStreamIngestionRobustnessTest that verifies the supervisor and indexing tasks remain healthy and continue to ingest valid data even when some stream records are malformed. Closes apache#10971
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 3 |
| P3 | 0 |
| Total | 3 |
Reviewed 2 of 2 changed files.
The review found three P2 test-correctness issues involving premature health checks, nondeterministic fault coverage, and an invalid multi-row payload.
This is an automated review by Codex GPT-5.6-Luna(max)
| cluster.callApi().postSupervisor(supervisor); | ||
|
|
||
| // Verify supervisor is healthy | ||
| verifySupervisorIsRunningHealthy(supervisor.getId()); |
There was a problem hiding this comment.
[P2] Health is checked before malformed records are consumed
The supervisor is checked before malformed records are consumed; later assertions only wait for valid rows. A supervisor can become unhealthy afterward and the test still passes. Check health after input processing completes.
| if (variant == DataVariant.ALL_VALID) { | ||
| return false; | ||
| } | ||
| return ThreadLocalRandom.current().nextDouble() < faultyRatio; |
There was a problem hiding this comment.
[P2] Fault injection can emit zero faulty records
Random sampling can emit zero faulty records, while the tests never assert fault coverage. Use deterministic selection or an injected RNG and assert the expected faulty count.
| return ("[" | ||
| + "{\"timestamp\":\"2021-01-01T00:00:00Z\",\"page\":\"Multi1\",\"language\":\"en\",\"user\":\"test\",\"unpatrolled\":\"true\",\"newPage\":\"true\",\"robot\":\"false\",\"anonymous\":\"false\",\"namespace\":\"article\",\"continent\":\"North America\",\"country\":\"United States\",\"region\":\"Bay Area\",\"city\":\"San Francisco\",\"added\":1,\"deleted\":0,\"delta\":1}," | ||
| + "{\"timestamp\":\"2021-01-01T00:00:01Z\",\"page\":\"Multi2\",\"language\":\"en\",\"user\":\"test\",\"unpatrolled\":\"true\",\"newPage\":\"true\",\"robot\":\"false\",\"anonymous\":\"false\",\"namespace\":\"article\",\"continent\":\"North America\",\"country\":\"United States\",\"region\":\"Bay Area\",\"city\":\"San Francisco\",\"added\":2,\"deleted\":0,\"delta\":2}" | ||
| + "\"]") |
There was a problem hiding this comment.
[P2] MULTI_ROW payload is invalid and unsupported
The MULTI_ROW payload has an invalid JSON suffix, so it is rejected as malformed. Kafka's non-line-splittable JsonReader also treats a top-level array as one value, not separate rows. The test only waits for valid rows, so it can pass without testing multi-row ingestion.
Description
Adds more robust testing to streaming ingestion integration tests as described in #10971.
Changes
New
FaultyStreamEventStreamGenerator— ASyntheticStreamGeneratorsubclass that can inject faulty data at a configurable ratio:INVALID_JSON— Malformed JSON bytes that cannot be parsed (e.g.,{"broken": })NULL_FIELDS— Valid JSON objects where all field values arenullEMPTY_JSON— Empty JSON objects ({})MULTI_ROW— JSON arrays containing multiple objectsEMPTY_STRING— Completely empty stringsALL_VALID— No faults injected (baseline)New
KafkaStreamIngestionRobustnessTest— Six integration tests that verify the streaming ingestion pipeline handles faulty data gracefully:test_supervisorHandlesInvalidJsonGracefully— Supervisor stays healthy with 20% invalid JSONtest_supervisorHandlesNullFieldsGracefully— Supervisor stays healthy with 20% null-field recordstest_supervisorHandlesEmptyJsonGracefully— Supervisor stays healthy with 20% empty JSON objectstest_supervisorHandlesMultiRowDataGracefully— Supervisor stays healthy with 20% multi-row JSON arraystest_supervisorHandlesEmptyStringGracefully— Supervisor stays healthy with 20% empty stringstest_supervisorHandlesAllValidDataCorrectly— Baseline test with all valid dataEach test verifies that:
RUNNINGstate)Closes #10971.
This PR has: