Skip to content

test: add robust testing to streaming ingestion integration tests - #20060

Open
zhang-arvin wants to merge 2 commits into
apache:masterfrom
zhang-arvin:fix/10971-streaming-ingestion-tests
Open

test: add robust testing to streaming ingestion integration tests#20060
zhang-arvin wants to merge 2 commits into
apache:masterfrom
zhang-arvin:fix/10971-streaming-ingestion-tests

Conversation

@zhang-arvin

Copy link
Copy Markdown

Description

Adds more robust testing to streaming ingestion integration tests as described in #10971.

Changes

  1. New FaultyStreamEventStreamGenerator — A SyntheticStreamGenerator subclass that can inject faulty data at a configurable ratio:

    • INVALID_JSON — Malformed JSON bytes that cannot be parsed (e.g., {"broken": })
    • NULL_FIELDS — Valid JSON objects where all field values are null
    • EMPTY_JSON — Empty JSON objects ({})
    • MULTI_ROW — JSON arrays containing multiple objects
    • EMPTY_STRING — Completely empty strings
    • ALL_VALID — No faults injected (baseline)
  2. New KafkaStreamIngestionRobustnessTest — Six integration tests that verify the streaming ingestion pipeline handles faulty data gracefully:

    • test_supervisorHandlesInvalidJsonGracefully — Supervisor stays healthy with 20% invalid JSON
    • test_supervisorHandlesNullFieldsGracefully — Supervisor stays healthy with 20% null-field records
    • test_supervisorHandlesEmptyJsonGracefully — Supervisor stays healthy with 20% empty JSON objects
    • test_supervisorHandlesMultiRowDataGracefully — Supervisor stays healthy with 20% multi-row JSON arrays
    • test_supervisorHandlesEmptyStringGracefully — Supervisor stays healthy with 20% empty strings
    • test_supervisorHandlesAllValidDataCorrectly — Baseline test with all valid data

Each test verifies that:

  • The supervisor remains healthy (RUNNING state)
  • Valid records are ingested successfully
  • The row count matches the expected valid records

Closes #10971.


This PR has:

  • been self-reviewed.

Adds FaultyStreamEventStreamGenerator that can inject faulty data
(invalid JSON, null/empty fields, multi-row data, empty strings) at a
configurable ratio into the streaming ingestion pipeline.

Adds KafkaStreamIngestionRobustnessTest that verifies the supervisor
and indexing tasks remain healthy and continue to ingest valid data
even when some stream records are malformed.

Closes apache#10971

@FrankChen021 FrankChen021 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Severity Findings
P0 0
P1 0
P2 3
P3 0
Total 3

Reviewed 2 of 2 changed files.

The review found three P2 test-correctness issues involving premature health checks, nondeterministic fault coverage, and an invalid multi-row payload.


This is an automated review by Codex GPT-5.6-Luna(max)

cluster.callApi().postSupervisor(supervisor);

// Verify supervisor is healthy
verifySupervisorIsRunningHealthy(supervisor.getId());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Health is checked before malformed records are consumed

The supervisor is checked before malformed records are consumed; later assertions only wait for valid rows. A supervisor can become unhealthy afterward and the test still passes. Check health after input processing completes.

if (variant == DataVariant.ALL_VALID) {
return false;
}
return ThreadLocalRandom.current().nextDouble() < faultyRatio;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Fault injection can emit zero faulty records

Random sampling can emit zero faulty records, while the tests never assert fault coverage. Use deterministic selection or an injected RNG and assert the expected faulty count.

return ("["
+ "{\"timestamp\":\"2021-01-01T00:00:00Z\",\"page\":\"Multi1\",\"language\":\"en\",\"user\":\"test\",\"unpatrolled\":\"true\",\"newPage\":\"true\",\"robot\":\"false\",\"anonymous\":\"false\",\"namespace\":\"article\",\"continent\":\"North America\",\"country\":\"United States\",\"region\":\"Bay Area\",\"city\":\"San Francisco\",\"added\":1,\"deleted\":0,\"delta\":1},"
+ "{\"timestamp\":\"2021-01-01T00:00:01Z\",\"page\":\"Multi2\",\"language\":\"en\",\"user\":\"test\",\"unpatrolled\":\"true\",\"newPage\":\"true\",\"robot\":\"false\",\"anonymous\":\"false\",\"namespace\":\"article\",\"continent\":\"North America\",\"country\":\"United States\",\"region\":\"Bay Area\",\"city\":\"San Francisco\",\"added\":2,\"deleted\":0,\"delta\":2}"
+ "\"]")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] MULTI_ROW payload is invalid and unsupported

The MULTI_ROW payload has an invalid JSON suffix, so it is rejected as malformed. Kafka's non-line-splittable JsonReader also treats a top-level array as one value, not separate rows. The test only waits for valid rows, so it can pass without testing multi-row ingestion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add more robust testing to streaming ingestion integration tests

2 participants