Feature request: Skip NodejsNpmEsbuildBuilder:NpmUpdate step when using --build-in-source in a js/ts "monorepo"

Describe your idea/feature/enhancement

We are currently making great use of the new --build-in-source flag for sharing js/ts code between several AWS resources (also written in js/ts). Our project is set up in a "monorepo" structure, something like:

packages/
  package-a/
    package.json
endpoints/
  endpoint-a/
    package.json
package.json

We use npm workspaces in the root package.json to allow resources like endpoint-a to specify modules like package-a as a dependency. The built-in SAM esbuild options can then find and bundle the code when using --build-in-source.

We've noticed, though, that overall build times have increased by quite a bit because of (what seems to be) a redundant NodejsNpmEsbuildBuilder:NpmUpdate step for each resource. In our case, all of our required dependencies have already been installed before we run sam build. This is true for all resources, because they are all also specified in the workspace's package.json.

The second step of the build for each resource (NodejsNpmEsbuildBuilder:EsbuildBundle ) represents less than 5% of the build time. If we could intelligently or explicitly skip the NpmUpdate step, our stack builds would speed up considerably.

Proposal

Option A: An explicit flag like --skip-dependencies to turn off SAM's management/installation of dependencies. Potentially also requiring --build-in-source. (It feels like that might be the only case where this new flag makes sense.)
Option B: Skip redundant installs that can be asserted based on the contents of a root package.json. This seems a bit antithetical to the way SAM thinks about resources and handles building, but may be technically possible.

Somewhat related: this would also resolve our issue in #6491

@cdriesler

Hi @cdriesler, thanks for the feature request. I will bring this up with the team and PM's.

We've noticed, though, that overall build times have increased by quite a bit because of (what seems to be) a redundant

This is interesting as npm caching should be able to reduce the build times. Would you be able to provide a sample project for us to investigate why build times are increasing?

@aws-sdk

We are in the middle of updating our SAM project and we've ended up with the same project structure. We also use npm workspaces and we also have shared custom dependencies like package-a that is used by our lambda endpoints.
Using the --build-in-source flag is mandatory in this case otherwise package-a cannot be resolved when esbuild tries to create the bundle. It runs an npm install but since package-a is not on the registry, it fails.

Anyway, we have found that for each endpoint an npm update is executed with the --install-links flag which makes the build really slow. For example the esbuild bundle command is executed in 1-2 seconds, but the npm update takes more than a minute. And it is executed for each lambda function one-by-one.
We have checked the logs during the npm update and it looked like it was running on the ROOT project instead of the lambda. We guess this is because the workspaces do not have a package-lock.json file.

Note: We have noticed that the only way to get rid of the built in @aws-sdk v3 to be deployed with the lambda is to set the `External˙ build properties of the lambdas to an empty array []. With this setting we can achieve around 200ms cold start times. Every other settings resulted in much longer cold starts 600ms+. We strongly believe this is caused by the presence of the default aws sdk.
This also prevents us to use layers because those dependencies must be listed in the Externals property otherwise the esbuild failes. But if we add them as externals (eq. the Externals is not an empty array) then the default aws-sdk is also deployed. We definiately want to avoid that.

Note 2: We have a workaround but it is not ideal at all. We do not use the --build-in-source flag and we perform additional tasks via lerna on each lambda before sam build:

In each lambda package.json we have a script to bundle the code: esbuild ./ --bundle --platform=node --outfile=bundle.js --minify. We use this script to bundle the code.
After the bundle is created we remove the dependencies block completely from the package.json file.

During the sam build the esbuild is still performed but at that point the bundle is already created and it has no additional dependencies so it completes without error.

Another AWS product that doesn't understand that there are more package managers in the NodeJS space than npm, and then also doesn't provide any way to work around these annoying issues. Working with any software product out of AWS is such an endless pain.

The real workaround is to drop SAM ASAP and implement a real build.

@hnnasit

@hnnasit is there any update on this?

I’d like to revive this issue because I believe the current build behavior undermines security, reliability, and developer productivity.

Like others here, I’m working with a monorepo setup using NPM workspaces, where each Lambda function lives in its own package alongside shared library packages. I typically run sam build --build-in-source to build.

With this setup, I’m running into several recurring problems:

npm update vs. npm install vs. none of them

Running npm update during builds is problematic. It introduces nondeterministic outcomes and unnecessary supply chain risk, especially in light of recent NPM security incidents. As a developer, I want to control when my dependencies update. During builds, installs should resolve consistently and deterministically. Ideally, builds should not trigger updates or reinstalls at all.

Because SAM reinstalls or updates dependencies during each build, my editor often lags behind, flagging modules as missing until the process settles. This introduces avoidable bad DX.

Non-deterministic build failures

My SAM builds frequently fail multiple times in a row before succeeding. A typical failure looks like this:

Build Failed
Error: NodejsNpmEsbuildBuilder:NpmUpdate - NPM Failed: npm warn Unknown cli config "--unsafe-perm". This will stop working in the next major version of npm.
npm warn config production Use `--omit=dev` instead.
npm warn workspaces @function/name in filter set, but no workspace folder present
npm error code ENOTEMPTY
npm error syscall rename
npm error path /.../monorepo/node_modules/@aws-sdk/client-s3
npm error dest /.../monorepo/node_modules/@aws-sdk/.client-s3-0OHxCkCC
npm error errno -66
npm error ENOTEMPTY: directory not empty, rename '/.../monorepo/node_modules/@aws-sdk/client-s3' -> '/.../monorepo/node_modules/@aws-sdk/.client-s3-0OHxCkCC'
npm error A complete log of this run can be found in: ~/.npm/_logs/...

The only workaround I’ve found is to run npm ci and then retry sam build --build-in-source multiple times until it eventually works. This is so tedious, sometimes I forget what I was doing in the first place.

If there's anything I can do to help make this happen, I'm here for it.

@ffraenz

@ffraenz By now we have a quite reliable and working process. I will try to outline our project structure here so you can adopt it if you want.

In our root package.json we define our workspaces like this:
"workspaces": [ "src/**/module" ]

We create the following structure for our lambdas and utility modules:

modules/
  module-1/
    module/
      package.json
      index.js
lambdas/
  lambda-1/
    module/
      package.json
      index.js
package.json

We also have our unit test scripts inside the module directories but that's not important right now.

In the package.json of every module we set the following parameters:

  "type": "module",
  "exports": "./index.js",

And we have a script in the package.json of the lambdas to bundle the code:

  "scripts": {
    "bundle": "esbuild ./index.js --bundle --format=cjs --main-fields=module,main --platform=node --outdir=bundle --minify",
    "bundle:dev": "esbuild ./index.js --bundle --format=cjs --main-fields=module,main --platform=node --outdir=bundle"
  }

This creates a bundle directory in each module and puts the bundled index.js file there.

We use lerna to bundle the modules.

lerna.json:

{
  "$schema": "node_modules/lerna/schemas/lerna-schema.json",
  "version": "0.0.0"
}

nx.json:

{
  "namedInputs": {
    "source": ["{projectRoot}/**/*.js", "!{projectRoot}/bundle/index.js"],
    "modules": ["{workspaceRoot}/src/modules/**/*.js", "!{workspaceRoot}/src/modules/**/tests/*.js"]
  },
  "targetDefaults": {
    "bundle": {
      "cache": true,
      "inputs": ["source", "modules"],
      "outputs": ["{projectRoot}/bundle/index.js"]
    },
    "bundle:dev": {
      "cache": true,
      "inputs": ["source", "modules"],
      "outputs": ["{projectRoot}/bundle/index.js"]
    }
  }
}

In the SAM templates we define the following properties:

  Lambda1Function
    Type: AWS::Serverless::Function
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        Format: cjs
        Minify: false
        External: []
    Properties:
      CodeUri: ../../src/lambdas/lambda-1/module/bundle
      FunctionName: Lambda1

This setup uses the bundled index.js as the lambda code. The Metadata part is needed to ensure that the default aws-sdk is not added to the deployment package.
The CodeUri is relative so you need to set it according to the location of your sam template file.

We have scripts in our root package.json to bundle all lambdas:

    "bundle": "lerna run bundle",
    "bundle:dev": "lerna run bundle:dev",

After the bundle is ready we can simply run sam build and then sam deploy. We do NOT use --build-in-source

We have tried out lots of options and settled on this one eventually. We are using this for more than a year by now without any issue. Please let me know if you tried it or if you have any question. This is quite a complex stuff so maybe I missed out something.

@RobertMarton1985

Thanks for the detailed reports here — especially @RobertMarton1985 and @ffraenz, whose descriptions turned out to be exactly right about the mechanism. I reproduced this end to end and dug into where the time goes. Summary: this is reproducible on SAM CLI 1.166.2, and there are three distinct problems underneath the feature request, one of which is a correctness issue rather than a performance one.

Reproduction

npm workspaces monorepo, two esbuild functions, one shared local package (@mono/shared), all dependencies already installed at the root. SAM CLI 1.166.2, node 24.21.0, npm 11.19.0.

sam build --build-in-source runs NodejsNpmEsbuildBuilder:NpmUpdate once per function, as reported.
Without --build-in-source the build fails with npm error 404 Not Found - GET https://registry.npmjs.org/@mono%2fshared, so --build-in-source is indeed mandatory for this layout (confirming @RobertMarton1985).
The install step emits npm warn workspaces @mono/endpoint-a in filter set, but no workspace folder present — npm resolves up to the workspace root, so the work is root-scoped, and it is repeated once per function. That is why npm's cache does not help: the cost is resolution/reconciliation of the whole monorepo tree, not package downloads.

Problem 1: the install step ignores your lockfile (correctness)

For build-in-source, get_install_action returns NodejsNpmUpdateAction, which runs:

npm update --no-audit --no-save --omit=dev --no-package-lock --install-links

--no-package-lock means the lockfile is not consulted. Measured on a fixture whose root package-lock.json pinned lodash@4.17.20 with a spec of ^4.17.20:

before sam build: root node_modules/lodash = 4.17.20
after sam build: root node_modules/lodash = 4.18.1

So a build silently upgrades dependencies in the developer's own source tree, despite a committed lockfile. This is @ffraenz's point, and it is a real defect: sam build is not supposed to be a dependency-update command. It is also the mechanism behind the intermittent ENOTEMPTY … rename failures — npm is renaming packages inside the shared root node_modules while an editor/tsserver (or another build) has the tree open — and behind #6491, since --omit=dev prunes devDependencies from the source tree.

Relevant code: aws_lambda_builders/workflows/nodejs_npm/workflow.py:264-265 and aws_lambda_builders/workflows/nodejs_npm/actions.py (NodejsNpmUpdateAction).

Problem 2: `UseNpmCi: true` cannot be reached here — and would be harmful if it were

get_install_action looks for the lockfile in the function's directory (workflow.py:247-248). With npm workspaces the lockfile only exists at the repo root, so UseNpmCi: true silently falls through to NpmUpdate. Verified: setting it in BuildProperties changes nothing, the log still shows NpmUpdate.

It is worth saying that simply making it reachable would be the wrong fix. Running npm ci --install-links from a workspace package operates on the root project and removes the root devDependencies — in my fixture it deleted node_modules/.bin/esbuild, i.e. the very binary the esbuild workflow resolves through npm root. So npm ci is not the escape hatch for this layout.

Problem 3: `--cached` cannot skip the install either

The mechanism that already exists for this (download_dependencies=False, used by the incremental build strategy) is unreachable under --build-in-source. The esbuild workflow returns early when building in source and never runs MoveDependenciesAction (aws_lambda_builders/workflows/nodejs_npm_esbuild/workflow.py:146-148), so .aws-sam/deps/<uuid> is never populated; IncrementalBuildStrategy then sees a missing dependencies directory on every run and sets download_dependencies=True again. Verified: three consecutive sam build --build-in-source --cached runs each ran NpmUpdate, and .aws-sam/deps/ stayed empty. sam sync reads the same flag.

What the numbers say about the fix

Each command below was run in the function directory of a pristine copy of the same fixture (root lockfile pinning lodash@4.17.20, esbuild as a root devDependency):

command	lockfile honored	root devDependencies	prunes a dep removed between builds (plain layout / workspaces)	wall time
`npm update --no-audit --no-save --omit=dev --no-package-lock --install-links` (current)	no — 4.17.20 → 4.18.1	kept	yes / no	582 ms
`npm ci --install-links` (`UseNpmCi`)	yes	esbuild removed	–	528 ms
`npm install --no-audit --no-save --omit=dev --install-links`	yes, no drift	kept	yes / no	290 ms

Two things follow. First, the motivation for #579 (npm install will not remove a dependency that was deleted from package.json between two builds) no longer holds on current npm: plain npm install prunes the removed package in the flat build-in-source layout, and in a workspaces monorepo neither command prunes it, so update buys nothing there while still bypassing the lockfile. Second, none of these commands modify or create a lockfile when --no-save is passed, which I checked separately.

Proposal

Use a lockfile-respecting install for build-in-source instead of npm update — drop --no-package-lock and use install. This makes builds deterministic, stops mutating the developer's dependency versions, keeps the pruning behaviour Fix stringifying λ environment variables when using Python2 #579 wanted, and is roughly 2x faster in a trivial fixture (much more in a real monorepo, since an up-to-date tree becomes a near no-op). I am preparing a PR against aws-lambda-builders for this.
Let --cached actually short-circuit the install under --build-in-source for the nodejs/esbuild workflows, so repeat builds skip dependency work entirely when the manifest has not changed. This is the fix that addresses the original request without any new user-facing surface.
Then consider the explicit opt-out from the description (Option A — e.g. a SkipDependencyInstall build property, or a CLI flag valid with --build-in-source), wired to the existing internal download_dependencies=False. This is still worth having for people who own their own build step, which is the ask in Feature request: Skip npm install #7847 — but after 1 and 2 it stops being the only way to get a reasonable build time.

Keeping this issue open as the tracking item for that work. #7847 is the same request from the "I already build my own bundles" angle and I'll close it as a duplicate of this one.

fix: honour the lockfile when building nodejs dependencies in source

Now Claude is on the job 🫡

fix: honour the lockfile when building nodejs dependencies in source

Two corrections to my comment above, plus measured numbers now that the PR exists.

1. The devDependencies column is npm 11 behaviour only. My fixture ran npm 11.19.0. On npm 10 and older, --omit=dev run from a workspace package prunes the workspace root's devDependencies for npm install and npm update alike — that is #6491's actual mechanism, and it is why my prose attributed #6491 to the current command while my table showed devDependencies "kept": both are true, on different npm versions. Corrected table cells: "kept (npm 11) / pruned (npm ≤ 10)" for both the current command and the proposed npm install. The switch fixes the lockfile drift; #6491 remains on npm ≤ 10 either way and needs its own fix.

2. Proposal 3 needs one more wire than I said. The esbuild workflow rejects download_dependencies=False unless a dependencies_dir is provided (it raises EsbuildExecutionError in nodejs_npm_esbuild/workflow.py), even though the build-in-source path never reads that directory. So an explicit skip-install option can't be wired directly to the existing flag for esbuild functions without relaxing that gate.

Status and numbers. The PR from proposal 1 is open: aws/aws-lambda-builders#931. Measured on an npm-workspaces monorepo with disjoint per-function dependencies (~300 packages, esbuild functions, build-in-source, whole sam build equivalent):

	8 functions	20 functions
today (`npm update` per function)	21.9s	57.8s
#931 (`npm install` per function)	6.5s	16.7s
install once per workspace root (design in progress)	1.5s	3.0s

So the install switch alone is ~3.4x here (not the 2x of the trivial fixture), and deduplicating the root-scoped install to run once per workspace — which is what the root-scoped npm warn workspaces observation above implies — is the rest.

fix: honour the lockfile when building nodejs dependencies in source

feat: install nodejs workspace dependencies once per monorepo root wi…

test: end-to-end esbuild monorepo build off one shared install

Navigation Menu

Repository navigation

Feature request: Skip `NodejsNpmEsbuildBuilder:NpmUpdate` step when using `--build-in-source` in a js/ts "monorepo" #6567

Describe your idea/feature/enhancement

Proposal

Reproduction

Problem 1: the install step ignores your lockfile (correctness)

Problem 2: `UseNpmCi: true` cannot be reached here — and would be harmful if it were

Problem 3: `--cached` cannot skip the install either

What the numbers say about the fix

Proposal

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature request: Skip NodejsNpmEsbuildBuilder:NpmUpdate step when using --build-in-source in a js/ts "monorepo" #6567

Description

Describe your idea/feature/enhancement

Proposal

Activity

cdriesler commented on Jan 17, 2024

hnnasit commented on Jan 19, 2024

hnnasit commented on Jan 22, 2024

RobertMarton1985 commented on Jan 31, 2024

adesso-os commented on Jun 7, 2024

RobertMarton1985 commented on Oct 29, 2024

ffraenz commented on Oct 3, 2025

RobertMarton1985 commented on Oct 3, 2025

bnusunny commented on Sep 27, 2026

Reproduction

Problem 1: the install step ignores your lockfile (correctness)

Problem 2: UseNpmCi: true cannot be reached here — and would be harmful if it were

Problem 3: --cached cannot skip the install either

What the numbers say about the fix

Proposal

2 remaining items

adesso-os commented on Sep 28, 2026

bnusunny commented on Sep 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Feature request: Skip `NodejsNpmEsbuildBuilder:NpmUpdate` step when using `--build-in-source` in a js/ts "monorepo" #6567

Problem 2: `UseNpmCi: true` cannot be reached here — and would be harmful if it were

Problem 3: `--cached` cannot skip the install either