Skip to content

fix(git): bound team-repo pull, fetch and push with the block timeout and end the git process tree - #1777

Merged
SaulMoro merged 1 commit into
Tencent:mainfrom
RClaudiuM:fix/1749-git-block-timeout
Nov 6, 2026
Merged

SaulMoro merged 1 commit into
Tencent:mainfrom
RClaudiuM:fix/1749-git-block-timeout

Conversation

@RClaudiuM

Copy link
Copy Markdown
Contributor

Summary

A team repo whose ssh:// remote stalls after the channel opens hung teamai pull, teamai push's pre-pull and the SessionStart pass forever, and left git pull → git fetch → ssh running (#1749). As the triage and the maintainer decision set out, the team-repo pull, fetch and push now run on the existing createGitWithBlockTimeout / gitBlockTimeoutMs seam with --progress, the factory ends git's whole process tree when the timeout or the caller's abort fires, and the error names the git command that stalled.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature causing existing behavior to change)
  • Documentation only
  • Refactor / internal cleanup

Change

src/utils/git.ts:

  • createGitWithBlockTimeout(basePath?, abort?) takes the optional abort that createGit takes, so pullRepo and src/team-ref.ts keep the git-hook pull's 5 s cap.
  • When the block timeout or the abort fires, simple-git ends git through the child's kill. The factory wraps that kill to call the new endProcessTree(pid) first:
    • On macOS/Linux it lists the tree with ps -A -o pid=,ppid= before signalling anything, because a killed parent's children move to init and can no longer be found. It then sends SIGTERM to each process. git handles SIGTERM by removing its lock files, and ssh exits on it.
    • On Windows it runs taskkill /PID <pid> /T /F.
    • If ps cannot list the tree, only git is signalled, as before.
  • simple-git hands the output handler the child's stdout but not the process, so the factory finds the process through Node's child_process diagnostics channel. The handler takes the child whose stdout is the stream simple-git just passed in. Both happen in the same tick as the spawn.
  • A command the timeout ended fails with GitStalledError, for example git pull --ff-only --progress stalled: no output for 30s, so teamai ended it and the processes it started. Set TEAMAI_INIT_PUSH_TIMEOUT_MS (milliseconds) to wait longer. It extends simple-git's GitError, which simple-git rethrows unchanged. Any other error class would come back wrapped in a plain GitError that keeps only its text. An abort keeps simple-git's own error, which pull.ts already recognises through cap.aborted.
  • These calls move to the block-timeout factory with --progress:
    • pullRepo: the pull and its fetch for the fetch + reset recovery.
    • pullRepoFastForward.
    • The push in pushRepoDirectly.
    • The reuse fetch and both pushes in pushRepoBranch.
  • pullRepo rethrows a GitStalledError without trying the fetch + reset recovery, since a fetch from the same remote would stall again and double the wait.
  • The local steps (add, commit, checkout, reset, status) stay on createGit unless initPush already put them under the timeout.
  • The "Not global" doc comment on gitBlockTimeoutMs now says what the timeout covers: remote calls, run with --progress. Local commands of the team repo stay unbounded.

src/team-ref.ts: the fetch --prune of branches and tags uses the block-timeout factory with the hook's signal. src/utils/reports-branch.ts: fetchReportsOriginFile's ls-remote and fetch use it too.

autoPushViaMR is untouched (#1760 owns it). It reaches the bound through pushRepoBranch.

Test Plan

  • npx tsc --noEmit passes (npm run typecheck)
  • npm run lint passes
  • npx vitest run passes: 738 files pass. dashboard-collector, env-loader (zsh cases) and token-usage fail with 8 tests, identically on unmodified main on this macOS machine, and are unrelated. In an earlier full run, one workflow-skill-driver test failed once and passed when rerun alone.
  • Added/updated tests for the change:
    • src/__tests__/git-stalled-remote.test.ts (POSIX) uses real git against an ssh:// origin through a core.sshCommand stand-in. While the stand-in serves, pullRepo still reports 1 file(s) changed, so --progress does not break the summary. Once it accepts and never writes (and ignores SIGINT), three cases run:

      • A stalled pullRepo rejects with a GitStalledError naming git pull --ff-only --progress in one timeout.
      • An aborted pullRepo rejects.
      • A side-branch fetchTrackingRef rejects naming its fetch.

      Each of the three then checks that the stand-in and the git that started it are gone. On main all three fail: the pull and the side-branch fetch time out at 10 s, and the abort leaves git fetch and the stand-in alive (expected [ 97237, 97236 ] to deeply equal []).

    • src/__tests__/git-process-tree.test.ts checks that endProcessTree ends a node → node → node tree. It is added to WINDOWS_TESTS, so the Windows CI job runs the taskkill branch. On main it fails (endProcessTree is not a function).

    • git-kind-reports.test.ts: the stalled-origin write now has to name git push --progress origin teamai-reports … stalled. git.test.ts expects the new --progress arguments. git.test.ts and init.test.ts keep simple-git's real exports in their mock, because GitStalledError extends GitError.

Real-CLI record, after npm run build. Setup:

  • A sandbox HOME (env -i, XDG dirs inside it) and a workspace initialized with teamai init ssh://git@stall.invalid/<sandbox>/team.git --provider git.
  • core.sshCommand is a /bin/sh stand-in with ssh.variant=simple. It serves the local bare repo by running the requested git-upload-pack / git-receive-pack. Once a flag file exists, it accepts, never writes, and ignores stdin EOF and SIGINT. No network host is contacted.
  • Each command runs under a 60 s watchdog. 2 s after the command returns, ps -A -E lists every process that carries the sandbox HOME or path, plus its children.
  • TEAMAI_INIT_PUSH_TIMEOUT_MS=5000 lowers the timeout. The last run uses the default 30 s.
  • members list reads the teamai-reports side branch through the reports worktree refresh.
  • Run on 2026-11-07, macOS 27.2, Node v24.14.1, git 2.54.0 (Apple Git-157).

main at 72a492a:

--- ssh stand-in serving
$ teamai pull
exit 0 after 1.1s
  | ✔ [workspace] Team repo: already up to date
--- ssh stand-in stalled
$ teamai pull   (TEAMAI_INIT_PUSH_TIMEOUT_MS=5000)
no return in 60s; watchdog killed the node process
  | - [workspace] Pulling team repo...
processes left alive 2s later:
  33030     1 01:02 git pull --ff-only
  33032 33030 01:02 git fetch --update-head-ok
  33033 33032 01:02 /bin/sh …/stall-ssh git@stall.invalid git-upload-pack '…/team.git'
$ teamai -v members list   (TEAMAI_INIT_PUSH_TIMEOUT_MS=5000)
no return in 60s; watchdog killed the node process
processes left alive 2s later:
  33364     1 01:02 git -c http.lowSpeedLimit=1 -c http.lowSpeedTime=6 fetch --progress origin +refs/heads/teamai-reports:refs/remotes/origin/teamai-reports
  33365 33364 01:02 /bin/sh …/stall-ssh git@stall.invalid git-upload-pack '…/team.git'

this head (0f726f7):

--- ssh stand-in serving
$ teamai pull
exit 0 after 1.6s
  | ✔ [workspace] Team repo: already up to date
--- ssh stand-in stalled
$ teamai pull   (TEAMAI_INIT_PUSH_TIMEOUT_MS=5000)
exit 1 after 5.7s
  | ✖ [workspace] Pull failed: git pull --ff-only --progress stalled: no output for 5s, so teamai ended it and the processes it started. Set TEAMAI_INIT_PUSH_TIMEOUT_MS (milliseconds) to wait longer.
processes left alive 2s later: (none)
$ teamai -v members list   (TEAMAI_INIT_PUSH_TIMEOUT_MS=5000)
exit 0 after 10.3s
  |   [debug] [reports] fetch failed, using the local copy: git fetch --progress origin +refs/heads/teamai-reports:refs/remotes/origin/teamai-reports stalled: no output for 5s, so teamai ended it and the processes it started. …
  | Team members (1):
  |   Sandbox — Sandbox (you)
processes left alive 2s later: (none)
$ teamai pull   (default timeout)
exit 1 after 30.8s
  | ✖ [workspace] Pull failed: git pull --ff-only --progress stalled: no output for 30s, so teamai ended it and the processes it started. Set TEAMAI_INIT_PUSH_TIMEOUT_MS (milliseconds) to wait longer.
processes left alive 2s later: (none)

members list takes two timeouts on this head: the ls-remote that asks whether the branch is on origin, which is already on the block-timeout factory and swallowed as "unknown", and then the fetch named above.

Related Issues

Fixes #1749

Notes for Reviewers

  • Not verified on Windows. I have no Windows machine. taskkill /PID <pid> /T /F is untested by me. git-process-tree.test.ts is in WINDOWS_TESTS, so the Windows CI job will run it against a node process tree, but not against git or Windows OpenSSH. The reporter's actual stall (Windows OpenSSH 9.5p2 in session 0) was not reproduced, only its symptom. HTTPS stalls and the SessionStart background pass through a real agent were not rerun either. The SessionStart pass goes through the same transitionTeamRepo → pullRepo path as teamai pull.

  • Why the diagnostics channel. simple-git 4 has no public hook that exposes the spawned process, and spawnOptions accepts only uid/gid, so git cannot be started in its own process group. The alternative was reaching into simple-git's private plugin store, which a minor release could break. Node's built-in child_process channel is documented. It is marked experimental among the built-in channels. I checked it publishes on Node 20.10 and 24.14. The match is by stdout identity, so a child from any other spawn is never taken for git. If the channel ever stops publishing, the factory falls back to the old behaviour (git alone is signalled).

  • SIGTERM, not SIGKILL, on POSIX. A tree member that ignores SIGTERM would survive, but git cleans its lock files on SIGTERM and would not on SIGKILL. That matters for the abort, which can land while the pull is writing. Windows has no equivalent: /F is required for console processes, and simple-git already used TerminateProcess there for git itself.

  • Why the e2e and test stand-in ignores SIGINT. On macOS, git fetch forwards the SIGINT that simple-git sends it to its ssh and waits for it. So with an ssh that exits on SIGINT, a plain side-branch fetch already ended on main, and only the git pull → git fetch chain and the abort were left behind. The stand-in ignores SIGINT to stand for an ssh that git cannot interrupt. The teamai pull result on main is the same either way.

  • Residual risk from the maintainer's choice to bound pull and push. A team repo hook that runs inside the bounded command can now be ended if it prints nothing for the whole timeout:

    • a post-merge hook during git pull;
    • a pre-push hook during git push;
    • most likely in single-repo mode, where the team repo is the business repo.

    Hook output resets the timer, and TEAMAI_INIT_PUSH_TIMEOUT_MS raises it. The env var keeps its name, although it now covers more than init pushes.

  • Remote calls left on createGit, to keep this to the calls the triage listed:

    • remoteBranchExists, pushLearningToOrigin and withDefaultBranchPreview's fetches in src/utils/git.ts and reports-branch.ts;
    • the fetch in withKnowledgeWorktree;
    • session-flow/archive.ts, workflow/cache.ts, and maintenance/promote.ts / promotion-publish.ts.

    Each could move the same way in a follow-up.

  • The optional teamai doctor warning is not part of this PR. No doc or skill text describes these timeouts. docs/, skill-data/ and the READMEs were grepped for TEAMAI_INIT_PUSH_TIMEOUT_MS, block timeout and lowSpeed.

… and end the git process tree

pullRepo, pullRepoFastForward, the push in pushRepoDirectly and
pushRepoBranch, pushRepoBranch's fetch, the team-ref fetch and
fetchReportsOriginFile now run on createGitWithBlockTimeout, with
--progress so a transfer that still moves keeps resetting the timeout.

When the block timeout or the caller's abort fires, the factory ends git
and every process under it (ps + SIGTERM, taskkill /T /F on Windows), so
a git fetch a pull started and the ssh under it no longer survive and hold
the CLI open. A stalled command fails with GitStalledError naming it, and
pullRepo no longer retries a stalled pull as a fetch + reset.

Fixes Tencent#1749
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

No confident findings in the requested diff.

The PR description documents sufficient testing: build, lint, unit tests, and representative real-CLI runs at 0f726f7, including timeout errors and process-tree cleanup. Windows coverage is explicitly deferred to CI and is not blocking.

Review was read-only; I did not run, build, or install anything from the PR.

@SaulMoro SaulMoro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was generated by AI during triage.

Approving 0f726f70. It fixes #1749. We built main (72a492ac, also the merge-base) and this head, and used a stand-in ssh command that serves a local repo and can be made to go silent:

case main this head
pull, silent remote hangs; git pull, git fetch and ssh stay alive after node is killed exit 1 after about 5.5 s, names git pull --ff-only --progress
pull, default timeout hangs exit 1 after 31 s
push, silent at the push step hangs exit 1 after 5.7 s, names git push --progress …
members list, silent reports fetch hangs exit 0 from the local copy
slow but moving transfer (about 6 MB over 26 s, 3 s timeout) pull and push succeed pull and push succeed

After every stalled run on this head, no git or ssh process was left 2 s later and there was no .lock file. TEAMAI_INIT_PUSH_TIMEOUT_MS, which the error names, works: against a remote silent for 7 s, 5000 fails and 15000 succeeds. From the code, endProcessTree signals only descendants of the git child. The diagnostics-channel lookup cannot pick another child when several git calls run at once. The listener is registered once, and POSIX uses SIGTERM, so git removes its locks.

Non-blocking:

  • [P2 non-blocking] The new ps call at src/utils/git.ts:396 has no windowsHide: true. #1775 adds a lint rule that requires it on every child-process launch, so whichever of the two lands second fails npm run lint until that option is added. It is one line.
  • [P3 nit] A git hook, or a password or host-key prompt, that stays silent for the whole timeout is now ended. The body discloses this, and the environment variable raises the limit.

The CI review (#1777 (comment)) reports no findings. We agree. The interaction with #1775 is outside this diff.

@SaulMoro
SaulMoro merged commit b4e8ed5 into Tencent:main Nov 6, 2026
18 checks passed
SaulMoro added a commit to SaulMoro/teamai-cli that referenced this pull request Nov 7, 2026
…n place on a failed e2e setup

Tencent#1777 added a spawnSync('ps') without windowsHide, which the new
teamai/windows-hide rule rejects once this branch sits on main.

Case (b) of the Windows lifetime e2e moved ~/.teamai aside before its
setup, outside the try/finally that restores it. Move it aside last.
SaulMoro pushed a commit that referenced this pull request Nov 8, 2026
The status fetch, the promotion fetch and push, and remoteBranchExists's
ls-remote still ran on plain createGit after #1777. A stalled ssh origin
held teamai status, the sync lock around recall promote, and the pending
push prune forever. Bound them with createGitWithBlockTimeout and
--progress; local status/add/commit stay on the plain instance.
SaulMoro added a commit to SaulMoro/teamai-cli that referenced this pull request Nov 9, 2026
…n place on a failed e2e setup

Tencent#1777 added a spawnSync('ps') without windowsHide, which the new
teamai/windows-hide rule rejects once this branch sits on main.

Case (b) of the Windows lifetime e2e moved ~/.teamai aside before its
setup, outside the try/finally that restores it. Move it aside last.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Team-repo git calls (pull, push's pre-pull, SessionStart) have no block timeout: a stalled ssh:// remote hangs them and orphans git/ssh

3 participants