Add end to end tests - #160
Open
aagbsn wants to merge 71 commits into
Open
Conversation
add these services to docker-compose.yml
FIXME: parametize this
diff --git a/tests/integration/tests/conftest.py b/tests/integration/tests/conftest.py index 36b7ca6..8ca007e 100644 --- a/tests/integration/tests/conftest.py +++ b/tests/integration/tests/conftest.py @@ -98,7 +98,7 @@ def params_since_and_until_with_two_days(): @pytest.fixture def params_since_and_until_with_ten_days(): - return set_since_and_until_params(since="2026-01-01", until="2026-01-02") + return set_since_and_until_params(since="2026-01-01", until="2026-01-10") def set_since_and_until_params(since, until): diff --git a/tests/integration/tests/test_oonidata_aggregate_analysis.py b/tests/integration/tests/test_oonidata_aggregate_analysis.py index 23762bd..6bad311 100644 --- a/tests/integration/tests/test_oonidata_aggregate_analysis.py +++ b/tests/integration/tests/test_oonidata_aggregate_analysis.py @@ -3,7 +3,7 @@ import os route = "/api/v1/aggregation/analysis" since = "2026-01-01" -until = "2026-01-02" +until = "2026-01-10" def test_oonidata_aggregation_analysis(client): diff --git a/tests/integration/tests/test_oonidata_list_analysis.py b/tests/integration/tests/test_oonidata_list_analysis.py index 1be6fce..64af92e 100644 --- a/tests/integration/tests/test_oonidata_list_analysis.py +++ b/tests/integration/tests/test_oonidata_list_analysis.py @@ -2,7 +2,7 @@ import pytest route = "/api/v1/analysis" since = "2026-01-01" -until = "2026-01-02" +until = "2026-01-10" def test_oonidata_list_analysis(client):
--- a/tests/integration/docker-compose.yml
+++ b/tests/integration/docker-compose.yml
@@ -57,7 +57,7 @@ services:
environment:
CLICKHOUSE_URL: "http://testuser:testuser@clickhouse:9000/ooni"
command: >
- bash -c "./run_fastpath --debug --clickhouse-url $CLICKHOUSE_URL --stdout"
+ bash -c "./run_fastpath --debug --clickhouse-url $CLICKHOUSE_URL --keep-s3-cache --write-to-disk --stdout --start-day 2026-01-01 --end-day 2026-01-10 --noapi"
depends_on:
clickhouse:
condition: service_healthy
fastpath doesn't seem to injest these from disk and this seems to create a double dependency of the downloader service, so it gets invoked twice
variable expansion works slightly differently in docker compose than podman compose; so the CLICKHOSUE_URL was not being expanded correctly
aagbsn
force-pushed
the
add_end_to_end_tests
branch
from
April 10, 2026 08:35
e4092e4 to
3cf400e
Compare
This reverts commit eb52b75.
this can be brought up independently with its dependencies clickhouse by docker compose up fastpath2
aagbsn
force-pushed
the
add_end_to_end_tests
branch
from
April 20, 2026 11:27
2ef34e1 to
6d94632
Compare
aagbsn
force-pushed
the
add_end_to_end_tests
branch
from
April 20, 2026 13:55
ba5f89a to
2abfea4
Compare
aagbsn
added a commit
to ooni/devops
that referenced
this pull request
Aug 18, 2026
Extends the harness with a second, additional CI job that verifies the actual OONI data pipeline -- not just synthetic seed data -- survives the production upgrade path. Design per direct instruction: download real data exactly once per run, re-verify integrity + query correctness at each hop, and walk PRODUCTION_HOPS (the real 4-hop runbook) rather than every diagnostic bisection waypoint. - harness/real_data.py: new module. Snapshots real-data tables (row count + order-independent cityHash64 checksum) across all 3 nodes, loads real OONI measurements once via ooni/data#160's downloader/fastpath containers, takes a golden snapshot, then per hop: upgrades all 3 nodes (reusing scenarios.upgrade_node_step unmodified), diffs against the golden snapshot, and re-runs ooni/data's own pytest suite against the real api-oonimeasurements service. - docker-compose.real-data.yml: new overlay adapting ooni/data#160's tests/integration stack onto this project's existing 3-node replicated cluster instead of a single throwaway node. Pinned to the add_end_to_end_tests branch since #160 is still unmerged -- update the ref once it merges. - sql/001_schema.sql: adds fingerprints_dns, fingerprints_http, obs_web_ctrl, obs_http_middlebox, obs_openvpn -- the tables oonipipeline's observations workflow and fastpath actually write, with per-table provenance notes and manual signedness fixes where oonipipeline's own DDL generator is known-wrong (Int32/Int8 vs the already-verified obs_web table's UInt32/UInt8 convention). - harness/compose.py: multi-compose-file support (files= param throughout) plus inspect_exit_code()/run_oneoff() for polling one-shot containers and running a fresh `verify` pytest pass per hop. - harness/scenarios.py: extracted apply_schema() from load_schema_and_seed() so the real-data scenario can reuse the schema load without the synthetic seed rows; generalized step_ok() to cover every non-upgrade-node step shape via a single top-level "ok" fallback. - harness/report.py: render_ci_step() now handles the real-data scenario's step shapes (setup-real-data, load-real-data, golden-snapshot, verify-e2e, real-data-hop). - ci_step.py: 6 new subcommands (setup-real-data, load-real-data, golden-snapshot, verify-e2e, real-data-hop, teardown-real-data) wiring harness/real_data.py into discrete, individually pass/fail-able CI steps, matching the existing setup/upgrade-node/verify-ddl pattern. - .github/workflows/clickhouse_upgrade_test.yml: new real-data-upgrade job (checkout ooni/data at add_end_to_end_tests, sanity-check PRODUCTION_HOPS alignment, setup/load/snapshot/4-hop-loop/report/ teardown). Deliberately not run on pull_request -- depends on an unmerged external branch and real network downloads -- only on push to main and workflow_dispatch (scenario: real-data or all). - README.md: documents the new job's design, the load-once/verify-per-hop rationale, the external-branch caveat, and marks PR #477 review point 4 ("run the target version against the real API + data pipeline") as addressed. Verified offline (no Docker daemon available in this sandbox, consistent with every prior patch in this project): all Python modules compile, all YAML/compose files parse and interpolate, SQL is structurally sound (balanced parens/backticks, unique table names), and every new step shape round-trips correctly through step_ok()/render_ci_step(). Real validation deferred to an actual GitHub Actions run.
hellais
requested changes
Aug 18, 2026
hellais
left a comment
hellais
left a comment
Member
There was a problem hiding this comment.
Once the changes to the API land we can merge this
|
|
||
| def __exit__(self, exc_type, exc_val, exc_tb): | ||
| """Context manager exit""" | ||
| self.close() |
Member
There was a problem hiding this comment.
We should just use the standard pytest fastapi client which supports all these methods already.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.