Skip to content

fix(ci): repair end-to-end and integration tests - #50

Merged
lucataco merged 8 commits into
mainfrom
lcatacora/fix-ci
Nov 5, 2026
Merged

lucataco merged 8 commits into
mainfrom
lcatacora/fix-ci

Conversation

@lucataco

@lucataco lucataco commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Why

CI on main has been red, so a red check tells nobody anything. This PR makes end-to-end-test and integration-test pass again by fixing each cause at its source. No test is skipped or loosened.

What changed

  • Dead test image. The GCS bucket behind fast-car.jpg now returns 403. The tests read philips-pm5544.768x576.retro-tv-test.png from multimedia-example-files.replicate.dev, the host the fuzzer already uses. The AI prompts and sizes match the new file.
  • Parallel Cog runs shared a directory. Cog 0.20 and later keeps build state in .cog/build under the project. Two cog push runs in one fixture directory delete each other's state. Each e2e test now works in its own copy of the fixture.
  • Ruff changed its defaults. Ruff 0.16 raised the default rules from 59 to 413. lint_file then rejected idiomatic code such as image: Path = Input(...) (B008) in any model without a ruff config. The repo's own ruff.toml hid this in CI. lint_file now selects E4,E7,E9,F, the old defaults.
  • Unstable output shape. For one model version, Replicate returns a single URI output as a bare string on some predictions and as a one-item list on others. CheckOutputsMatch then failed with "The types of the outputs don't match". predict now unwraps a one-item list when the Output schema is not an array.
  • Cleanup 500s. Deleting a test model version sometimes returns an empty-body 500 after the test passed. The e2e cleanup retries 5xx responses and treats 404 as done.
  • CI prints the installed Cog version.

Scope

  • Covers the e2e and integration failures seen on the latest runs. Logs for the 2025 main failures have expired, so I could not check that they had the same causes.
  • Leaves the actions/checkout@v3 Node 20 warning and the Cog ERROR: ... no access allowed to dir "context" build noise alone. Builds succeed with the noise.

Tradeoffs

  • Committing the old car photo and serving it from raw.githubusercontent.com keeps the content but needs a commit-pinned URL. The Replicate example host needs no pinning.

Blast Radius

The two product changes affect users of the tool. lint_file is more lenient than ruff's new defaults and equal to the old ones. predict changes only a one-item list under a non-array schema. Everything else is test code and CI. Leaving main red keeps hiding real regressions.

Verification

  • Unit tests and lint pass locally. The two new unit tests failed before their fixes: test_lint_file_accepts_idiomatic_cog_defaults and the one-item list case of test_predict_output_follows_output_schema.
  • Reproduced the .cog/build race locally with Cog 0.23.1. A cog build that started while a cog predict finished in the same directory failed 3 of 3 times. In separate copies of the fixture it passed 2 of 2 times.
  • CI on the final head is green on all four jobs, with end-to-end-test at 8 passed and integration-test at 93 passed: https://github.com/replicate/cog-safe-push/actions/runs/42064940453. The previous head also passed twice in a row, in attempt 1 and attempt 2 of https://github.com/replicate/cog-safe-push/actions/runs/42061494182.

Cog 0.2x keeps build state in .cog/build inside the project directory.
Parallel cog runs in one directory delete each other's state and fail
with "open .cog/build/Dockerfile: no such file or directory". The e2e
tests run under xdist and share fixtures such as base and image-base.
The test image lived in a GCS bucket that now returns 403. Use the
PM5544 test pattern from multimedia-example-files.replicate.dev, the
host the fuzzer already depends on. Update the AI prompts and the
dimensions to match the new file.
The e2e job installs the latest Cog release. Printing the version makes
a Cog-caused failure easy to spot in the log.
Fails with ruff 0.16 and later. Their default rule set flags B008 on
`image: Path = Input(...)`.
Fails when Replicate wraps a single URI output in a one-item list.
Ruff 0.16 expanded its default rules from 59 to 413. Models without a
ruff config now fail the predict.py lint on idiomatic Cog code such as
`image: Path = Input(...)`. Select the pre-0.16 defaults explicitly.
User ignores and per-file ignores still apply.
Replicate returns a single URI output as a bare string on some
predictions and as a one-item list on others, for the same version.
That made CheckOutputsMatch fail with "The types of the outputs don't
match" at random. Unwrap the list when the Output schema type is not
array.
Cleanup in two e2e runs failed with an empty-body 500 from
DELETE /models/{owner}/{name}/versions/{id} after the test body had
passed. Retry with a delay, and treat 404 as already deleted.
@lucataco
lucataco merged commit f2e2f22 into main Nov 5, 2026
4 checks passed
@lucataco
lucataco deleted the lcatacora/fix-ci branch November 5, 2026 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant