Skip to content

Recurring infrastructure CI failures

Jump to bottom
Richard Lau edited this page Sep 2, 2026 · 4 revisions

This page is designed to log problems that occur periodically in the CI which are unrelated to individual test case failures along with solutions.

Merge conflicts

This often occurs in the Makefile but can also affect other files. It does not necessarily happen across multiple machines at the same time, although it can do. It tends to happen on node-test-commit-linux. It happens during the git rebase operation for the test branch and something in the _jenkins_local_branch which is out of sync with what the rebase operation can handle. It fails even when there is no rebase operation in place (we already do a git rebase --abort beforehand). I have a copy of a workspace from one of the machines that shows the problem which can be used for further debugging (Memo to self: workspace-rebase-in-progress.tar and the problem can be triggered with git rebase --committer-date-is-author-date abb365abea80915022c81908bea42d321d8377f4)

We do not currently have a good solution to this (although I did some experiments, nothing has quite worked so far) so the best solution when this occurs is to remove the workspace on the affected machine as it will be re-cloned automatically on the next run

rm -rf /home/iojs/build/workspace/node

The sequence of events that causes this is not yet clear although proximity to git network glitches or runs of `node-stress-stest is a possibility.

git-rebase failures from fanned jobs

This can manifest itself like this:

09:14:26 + git push binary_tmp@67.158.54.159:binary_tmp.git +jenkins-node-test-commit-arm-fanned-7774d81562a3e59125fe911942acf92e271d7a7b
09:14:26 remote: fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-afca0a912d48b57060873eb43afd878e2aa84697        
09:14:26 fatal: bad object refs/heads/jenkins-node-test-commit-windows-fanned-afca0a912d48b57060873eb43afd878e2aa84697
09:14:26 To 67.158.54.159:binary_tmp.git
09:14:26  ! [remote rejected]         jenkins-node-test-commit-arm-fanned-7774d81562a3e59125fe911942acf92e271d7a7b -> jenkins-node-test-commit-arm-fanned-7774d81562a3e59125fe911942acf92e271d7a7b (missing necessary objects)
09:14:26 error: failed to push some refs to '67.158.54.159:binary_tmp.git'
09:14:26 Build step 'Execute shell' marked build as failure

As noted in issue 4408 the binary_tmp directory can get quite large (last time I saw this it was over 80GB) so we perhaps need to look at the clean_binary_tmp.sh script and see if we can improve it. The fix is to remove the directory and reclone in a manner equivalent to the ansible scripts:

git clone --bare https://github.com/nodejs/node /home/binary_tmp/binary_tmp.git && chown -R binary_tmp /home/binary_tmp/binary_tmp.git

Another (faster) option is to clear up the missing object as follows:

git update-ref -d refs/heads/jenkins-node-test-commit-windows-fanned-00e420719573c648bff88a847b6247c89e1b77ba-bin-win-vs2022_clang-arm64

openjdk being updated while jenkins agent is running

This is a recurring problem that causes an error message which is typically caused by unattended updates which update java without restarting the jenkins agent. It results in messages such as this:

17:52:21  > git init /home/iojs/build/workspace/node # timeout=10
17:52:21 ERROR: Error cloning remote repo 'origin'
17:52:21 hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node

The solution is to restart the jenkins agent, typically with systemctl restart jenkins

We could look at blocking the openjdk code from updating and having a maintenance window to update it in a controlled manner when we can do an agent restart: https://github.com/nodejs/build/issues/3962

On Debian-based systems you can check for recent updates that have occurred with:

grep -A1 openjdk /var/log/apt/history.log
zgrep -A1 openjdk /var/log/apt/history.log.1.gz

jenkins-workspace index.lock': File exists.

This is a side effect of colocating the workspaces on the build machines. It causes a conflict on the workspace machines in /home/iojs/build/workspace/node if multiple jobs run concurrently. Likely visible on any jobs in https://ci.nodejs.org/job/node-test-commit-linux/buildTimeTrend which stopped in a few seconds.

Issue: https://github.com/nodejs/build/issues/3897#issuecomment-2451883311

06:57:20 ERROR: Error fetching remote repo 'origin'
06:57:20 hudson.plugins.git.GitException: Failed to fetch from git@github.com:nodejs/node.git
06:57:20 	at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:998)
06:57:20 	at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
06:57:20 	at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
06:57:20 	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
06:57:20 	at hudson.scm.SCM.checkout(SCM.java:540)
06:57:20 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1250)
06:57:20 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:648)
06:57:20 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:84)
06:57:20 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
06:57:20 	at hudson.model.Run.execute(Run.java:1843)
06:57:20 	at PluginClassLoader for matrix-project//hudson.matrix.MatrixBuild.run(MatrixBuild.java:331)
06:57:20 	at hudson.model.ResourceController.execute(ResourceController.java:97)
06:57:20 	at hudson.model.Executor.run(Executor.java:456)
06:57:20 Caused by: hudson.plugins.git.GitException: Command "git reset --hard" returned status code 128:
06:57:20 stdout: 
06:57:20 stderr: fatal: Unable to create '/home/iojs/build/workspace/node/.git/index.lock': File exists.
06:57:20 
06:57:20 Another git process seems to be running in this repository, e.g.
06:57:20 an editor opened by 'git commit'. Please make sure all processes
06:57:20 are terminated then try again. If it still fails, a git process
06:57:20 may have crashed in this repository earlier:
06:57:20 remove the file manually to continue.
06:57:20

request-ci workflow

If this needs to be disabled for any reason during problematic periods this can be done via https://github.com/nodejs/node/actions/workflows/auto-start-ci.yml

Clone this wiki locally