Skip to content

feat: push monorepo Actors by uploading the repository root with actorPath - #1505

Open
DaveHanns wants to merge 87 commits into
masterfrom
feat/push-monorepo-upload
Open

DaveHanns wants to merge 87 commits into
masterfrom
feat/push-monorepo-upload

Conversation

@DaveHanns

@DaveHanns DaveHanns commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

apify push of a monorepo Actor (one whose dockerContextDir lies outside its folder) now uploads the Git repository root, or the Docker context root outside Git, and tells the platform where the Actor lies in it through the actorPath version field, instead of refusing with exit code 255. Both Actors of apify/actor-monorepo-example can then build from a push, as they do from a Git source.

Closes #1525, part of the monorepo Epic apify/apify-core#28685. Needs apify/apify-core#31479 and apify/apify-worker#2249 deployed first (see Merge gate).

Builds on the preflight (#1502), the actor.json path checks (#1503), multi-Actor selection (#1504), the ignore rules that the Actor folder's file list uses (#1537), and how createActZip writes the zip for every push (#1549), all on master now.

Merge gate

Merge only after both platform PRs are deployed:

  1. apify/apify-core#31479 accepts actorPath on SOURCE_FILES and TARBALL versions. Before it, the API rejects the key with a 400, so every monorepo push fails.
  2. apify/apify-worker#2249 makes the build honor actorPath. Before it, the worker ignores the field and builds from the upload root, which is the repository root, so the build can't find the Actor or builds the wrong image, after the push has already replaced the version's source.

Changes

  • In context mode the upload root is the resolver's sourceRoot and the file list comes from getContextFilePaths (Git clone semantics, / separators); outside Git it is the Docker context root with the ordinary list. Git's output is read whole, past the default 1 MiB buffer, so a large repository never falls back to the non-Git list, and a file name starting with a space keeps the space. The Actor folder itself goes as an ordinary push of it would send it, so its .gitignore and .actorignore apply as apify push --help says, negations such as !dist/ included.
  • Works with multi-Actor selection: --all, the picker, and --dir choose the Actors, and each one is pushed on its own: a monorepo Actor as the repository root with its own actorPath, any other Actor as its folder, as before.
  • An Actor in an ignored folder can still be picked or named with --dir, as feat: let apify push select which Actors to push in a repository with several #1504 decided. A clone has at most the folder's tracked files, so the rest join the upload with the Actor folder, and the platform finds the Actor at actorPath. Which files go depends on fix: leave out every gitignored file when apify push collects files #1537. Without it, an ignored folder goes with every file in it, including a .env that its own .gitignore lists, and one that holds a tracked file loses its untracked files. An ordinary push of such a folder on master does the same today.
  • actorPath carries the Actor folder's names as stored on disk. A --dir typed in another letter case on macOS or Windows still gives the right path, and the Actor folder doesn't end up in the upload twice when Git lists it in another form: composed (NFC) while it is stored decomposed (NFD), or in its old case after a case-only rename. Other files in a folder on the Actor folder's way, such as packages/lib next to packages/actor, go under the same spelling as actorPath, so the platform doesn't write that folder twice. A link named like such a folder in another case, on a case-sensitive file system, stays a folder of its own. A dockerContextDir that leaves the Actor folder through a symlink leading back to it exits 5 instead of sending an empty actorPath, outside Git or when the Actor folder is the repository root.
  • The actor.json path check runs with that upload root and the Docker context, so a dockerfile outside the context exits 5 before any platform call.
  • An absolute dockerContextDir, or one with a backslash, exits 5 before any upload, as such paths to files in actor.json do: the platform resolves it from .actor/ on Linux. On master, such a value inside the Actor folder pushed, then failed the build.
  • Under MAX_MULTIFILE_BYTES the version gets SOURCE_FILES with names relative to the upload root; otherwise the existing zip upload to the actor-<id>-source store and TARBALL with the (signed) tarballUrl. Push also zips, for every push, when a file name is over the platform's 255-character limit, which a deep path from the repository root can reach, or when the content as sent is over the limit although the files on disk aren't: base64 makes a binary file a third larger, and the platform counts each character beyond the Basic Multilingual Plane, such as an emoji, as five bytes. On master, both cases fail the version update with a 400. The monorepo zip has the repository layout from its top level, where apify/apify-worker#2249 resolves actorPath. The monorepo zip goes under a key of its own for each actorPath, version-<version>-monorepo-<first 16 hex digits of the SHA-256 of actorPath>.zip, so a version update that fails afterwards leaves alone the record an existing version builds from, whether that version has no actorPath or another one, such as the Actor folder's name before a rename. Every version create or update payload of such a push, including the inline version of a new Actor, carries actorPath. Ordinary pushes send no such key, and the platform clears a stored actorPath on any update that sends sourceType, sourceFiles, or tarballUrl without it, so a later push of the Actor folder resets an old path.
  • The temporary zip now goes to a unique file under os.tmpdir() for every push that zips, not temp_file.zip in the working directory, and is removed in a finally.
  • The "newer changes on the platform" check scans the upload root and, in context mode, also runs on the zip path.
  • A one-line warning on stderr when the repository has uncommitted changes, also before its first commit. During an unresolved merge, a conflicted file goes once, not once per index stage.
  • A symlink to a file the upload holds goes as that file's content at the link's path, a symlink to a directory inside the repository as copies of the uploaded files under its target, directory links there included, with a link back to a target on the way ending a cycle (as the ordinary push's file walk does), and a dangling link, a link to a file the upload leaves out (such as a workspace package's .env linking to the gitignored root .env, or a file in .git), or one pointing outside the repository is left out. A link to a file that only the Actor folder's upload adds, such as its gitignored dist/ that its .actorignore takes back, goes too, and so does a directory link to such files, under the Actor folder's spelling in Git when Git spells it differently from the disk. Outside Git, where the file list follows folder links, so are the files under a link to a folder outside the upload. The actor.json path check accepts a reference to a link that is uploaded and rejects one to a link that is left out. Before this, a tracked directory symlink crashed the push with a raw EISDIR or ENOENT error.

Tests

  • test/local/commands/push-monorepo.test.ts (mocked client, no network):
    • Upload: source files with repository-relative names and actorPath, the --json envelope, the zip path (store record version-0.0-monorepo-<hash>.zip with application/zip, TARBALL with a signed tarballUrl, no temp_file.zip left in the working directory), a separate record after a rename whose version update fails, the zip's repository layout from its top level, the zip for a file name over 255 characters, for binary content over the limit in base64, and for emoji the platform counts over the limit, no zip left in the OS temp directory when the store call fails, and the uncommitted-changes warning.
    • New Actor: actorPath on the inline version of the Actor create and on the version update.
    • Files: the no-Git fallback with .actorignore, the Actor's own node_modules and storage left out outside Git, the Actor folder's .actorignore in Git (an exclusion and a !dist/ negation), an ignored Actor folder named with --dir, an Actor folder name stored decomposed, an Actor folder renamed by case only, which Git lists in its old case (on case-insensitive file systems only), and the siblings of the Actor folder under actorPath's spelling after a case-only rename of its parent (on case-insensitive file systems only) and in a parent stored decomposed (on macOS only).
    • Symlinks: a directory symlink, a directory link inside a linked directory and a cycle, a directory link to files only the Actor folder adds, also after a case-only rename of the Actor folder, and next to a tracked file that differs in case only (on case-sensitive file systems only), a link named like a folder of the Actor folder in another case (on case-sensitive file systems only), a dangling link, a link pointing outside the repository, a link to a gitignored .env and one into .git left out while a link to a tracked file goes, a link to a file only the Actor folder adds, links that spell the Actor folder composed while it is stored decomposed (on macOS only) or in its old case after a case-only rename (on case-insensitive file systems only), the files under a folder link leading out of the Docker context outside Git, a linked file zipped as its content, a schema reached through a link inside the repository (accepted) and through one pointing outside (exit 5), a dockerfile in a Docker context reached through a link inside the repository, a link whose target names the repository in another letter case (on case-insensitive file systems only), and a --dir that reaches the Actor through a symlink.
    • Paths and selection: a dockerfile outside the Docker context, a --dir typed in another case (on case-insensitive file systems only), --all from the repository root over two monorepo Actors and a plain one, and the cases kept from the preflight (feat: detect dockerContextDir in apify push and refuse monorepo Actors before any write #1502).
  • test/local/lib/docker-context.test.ts: an absolute dockerContextDir, one with backslashes, and one that leads back to the Actor folder through a symlink, outside Git and in Git with the Actor folder at the repository root.
  • test/local/lib/docker-context-files.test.ts: a conflicted path listed once during an unresolved merge, a file list over 1 MiB, and a name starting with a space.
  • test/local/lib/docker-context-state.test.ts: a repository with no commits counts as having uncommitted changes.
  • test/local/lib/docker-context-realpath.test.ts: the resolver and the path check fall back to the JS realpath where the native call fails.
  • pnpm run lint, pnpm run format, tsc --noEmit, pnpm run build, and pnpm run test:local pass, except the two [python] suites (python_support.test.ts, works-with-spaces-in-path-to-actor.test.ts), which fail the same way on master because the local Python is 3.9.
  • Not run: test:api, which needs a platform token, and the [api] case the issue asks for (see below). Run with Node.js 26.4.0. The cases for case-sensitive file systems ran on a case-sensitive APFS disk image.

Notes for reviewers

  • Nothing on the CLI side can detect a platform or worker without actorPath support, hence the merge gate.
  • No apify-client release needed: the CLI spreads actorPath into the version payload itself.
  • The Actor folder follows the ordinary push rules and the rest of the repository follows a clone. So the Actor folder's own .actorignore applies, while a .actorignore at the repository root applies only outside Git, and there only outside the Actor folder.
  • The command description now says what a monorepo push uploads: the Actor folder as before, and everything else that Git tracks or doesn't ignore, without .actorignore; outside Git, the whole Docker context. docs/ is regenerated with pnpm run update-docs.
  • The real paths come from the native realpath, which gives names in their case on disk. Where it fails, as on Windows RAM or virtual disks, the push falls back to the JS implementation, which master uses.
  • The monorepo zip relies on apify/apify-worker#2249 resolving actorPath from the archive's top level, which that PR tests ("in a flat archive with only directories at its top level").
  • Symlinks: getContextFilePaths keeps them as a Git clone does. The push sends a linked file as its content in both the multi-file and the zip upload, so it resolves them as described under Changes. A link whose target lies outside the repository, or is a file the clone list lacks, is left out too, as a clone couldn't resolve it, unless the Actor folder's upload adds that file. This holds for links outside the Actor folder; links inside it follow the ordinary push, which reads them through.
  • Follow-up: the token-gated [api] case in test/api/commands/push.test.ts pushing a two-Actor fixture, once the platform change is deployed.

🤖 Generated with Claude Code

…ths' into feat/push-monorepo-upload

# Conflicts:
#	src/lib/docker-context.ts
push reads through a symlink and uploads the content at the link's own
path, so a link to a file outside the upload root builds on the platform.
The realpath escape rule refused it; the uploaded files rule already
rejects anything push does not send.
The worker copies inputSchema over input and outputSchema over output
before resolving, so a schema path under the alias was never checked.
The worker treats an empty string as an unset field, so an empty value
was wrongly reported as the .actor folder being a directory. A backslash
passes path.resolve on Windows but is a filename character on the Linux
worker, so it is refused before any resolution.
The lib test now builds its fixture through useTempPath, as the other
lib tests do, adds a changelog file so the case-insensitive match is
checked for all three listed fields, and covers the reason given when
the uploaded file differs only in case. The command test restores the
real process.exitCode next to the mocked copy.
checkActorPaths resolves the roots and builds the posix set of uploaded
files once and hands them to every field, instead of each field
recomputing them.
archiver writes a symlink as a link entry, which the platform unzips as a link
to a file the upload may lack. The multi-file upload already sends the
content, so a push that passes the path checks no longer depends on its size.
The zip was written before the source store call and outside the try, so an
error there left it in the OS temp directory. The store is now fetched first,
and the zip is created inside the try whose finally removes it.
The zip went to the record an existing TARBALL version builds from, before the
version update that adds actorPath. When that update failed, as on a platform
without the field, the version was left building from the repository root.
The guard matched any 400 that named actorPath, so a path the platform rejects
as too long got the refusal that says monorepo pushes are not supported. It
now matches only the unknown-key error of a platform without the field.
…r case

The upload root comes from the native realpath, with names in their case on
disk, but the Actor folder's files were checked against it with the JS
realpath, which keeps the case of the link's text. On a case-insensitive file
system such a link inside the repository was dropped without a message.
After a case-only rename without git mv on a case-insensitive file system, the
index keeps the old case, so the clone list kept the folder's tracked files
next to the ordinary-push list under actorPath. That doubled the Actor and
bypassed its .actorignore for tracked files. The folder is now matched without
regard to case or normalization and confirmed on disk.
@DaveHanns

Copy link
Copy Markdown
Contributor Author

Don't merge yet. This PR depends on:

  1. apify/apify-core#31479, merged and deployed.
  2. apify/apify-worker#2249, merged and deployed.

Until then, CI passes, but monorepo pushes don't work. Without the core change, the API rejects actorPath, and the push ends with the same refusal as on master (exit code 255). With the core change but without the worker change, the push replaces the version's source with the repository root, and the build runs from that root instead of the Actor folder, so it fails or builds the wrong image.

…upload

# Conflicts:
#	src/lib/utils.ts
#	test/local/commands/push-monorepo.test.ts
…th a space included

Past 1 MiB of names, git's output hit execFileSync's default buffer, and the
push silently fell back to the non-Git file list. Trimming the output also
cut the space off a first name that starts with one.
…as its root

The worker roots a zip with no file at its top level at its last folder, as
it does a GitHub archive, and resolves actorPath from there. An upload root
with only folders at its top level then failed the build. Ordinary zips
keep their layout.
…push uploads anything

The platform resolves the path from .actor/ on Linux, so such a value can't
build there. The push uploaded the whole repository first, then the build
failed.
… upload

Outside Git, the file list follows folder links, and only the links themselves
were checked, so the files under a link to an outside folder were uploaded.
Each folder's real path is now checked once.
Git state reading stopped at the missing HEAD, so the warning never showed,
though every file a monorepo push uploads is uncommitted there.
A committed link to a gitignored file, such as a workspace package's .env
pointing at the root .env, uploaded the file's content, and a link into .git
uploaded the repository's config. A file link now goes only when its target
is in the file list, as folder links already did.
apify/apify-worker#2249 now resolves actorPath from the archive's top level
whenever the path is there, and this push ships after it. The source/ folder
that worked around the worker's guess of the root goes, and createActZip
takes no root folder again.
…der adds

Links outside the Actor folder resolve as a clone of the repository would,
so one to a file the clone lacks is left out. A file the Actor folder's
upload adds, such as a gitignored dist that its .actorignore takes back,
now counts as held too. It is matched by the target's native real path, as
the Actor folder's files have their names as stored on disk, so nothing
else changes: links elsewhere still resolve against the clone list as
before, which tests now pin for a composed link to a decomposed Actor folder
and for one in the folder's old case after a case-only rename.
…upload

# Conflicts:
#	test/local/lib/utils.test.ts
- Give the monorepo zip a key per actorPath, so a failed version update
  after a rename leaves the record the version builds from intact.
- Zip instead of sending multiple files when a name is over 255
  characters, or when the content as sent (base64, emoji counted as
  five bytes) is over the multi-file limit.
- Copy files through directory links inside linked directories, with a
  cycle guard, and through directory links to files only the Actor
  folder adds, under Git's spelling of the Actor folder.
- Send the files next to the Actor folder under actorPath's spelling,
  so a case-only rename or a decomposed name doesn't split a folder in
  two on the platform; a symlink named like such a folder stays apart.
…e help

- Remove the guard that turned a 400 for an unknown actorPath key into
  the old refusal (exit 255) and deleted a just-created Actor. The
  platform accepts actorPath now, so the guard can't trigger; a
  platform error now surfaces as for any other push. The refusal
  message and its tracking-issue URL go with it.
- Replace the guard's tests with one that creates a new Actor and
  checks actorPath on its inline version and on the version update.
- The help text said .gitignore and .actorignore exclude files, which
  is wrong outside the Actor folder of a monorepo push. It now says
  what goes: the Actor folder as before, and everything else that Git
  tracks or doesn't ignore, without .actorignore; outside Git, the
  whole Docker context. docs/ regenerated.
The tests skip the system Git config, where a Windows runner enables
core.longpaths, so Git skipped the folder over 260 characters and the
file never reached the upload.
@DaveHanns DaveHanns self-assigned this Nov 9, 2026
@DaveHanns
DaveHanns marked this pull request as ready for review November 9, 2026 10:20
@apify-service-account apify-service-account added tested Temporary label used only programatically for some analytics. t-builders Issues owned by the Builders team. labels Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-builders Issues owned by the Builders team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

apify push: upload monorepo Actors with the repository root and actorPath

2 participants