Skip to content

perf(docker): reuse runtime layers and scope Maven builds #3194

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lokidundun wants to merge 1 commit into
base: master
Choose a base branch
from
+25 −18
Open

perf(docker): reuse runtime layers and scope Maven builds #3194

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions hugegraph-pd/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,14 @@ ARG MAVEN_ARGS
ARG SOURCE_REVISION=local

RUN --mount=type=cache,id=hugegraph-maven-${SOURCE_REVISION},target=/root/.m2,sharing=locked \
mvn install $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
mvn install -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \
-am $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
&& rm ./hugegraph-server/*.tar.gz ./hugegraph-pd/*.tar.gz ./hugegraph-store/*.tar.gz

# 2nd stage: runtime env
# Note: ZGC (The Z Garbage Collector) is only supported on ARM-Mac with java > 13
FROM eclipse-temurin:11-jre-jammy

COPY --from=build /pkg/hugegraph-pd/apache-hugegraph-pd-*/ /hugegraph-pd/
LABEL maintainer="HugeGraph Docker Maintainers <dev@hugegraph.apache.org>"

# TODO: use g1gc or zgc as default
Expand All @@ -58,6 +58,8 @@ RUN apt-get -q update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

COPY --from=build /pkg/hugegraph-pd/apache-hugegraph-pd-*/ /hugegraph-pd/

# 2. Init docker script
COPY hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh .
RUN chmod 755 ./docker-entrypoint.sh
Expand Down
12 changes: 7 additions & 5 deletions hugegraph-server/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,14 @@ ARG MAVEN_ARGS
ARG SOURCE_REVISION=local

RUN --mount=type=cache,id=hugegraph-maven-${SOURCE_REVISION},target=/root/.m2,sharing=locked \
mvn install $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
mvn install -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \
-am $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
Comment on lines +31 to +32

@bitflicker64 bitflicker64 Sep 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 The module list is hardcoded ahead of $MAVEN_ARGS, which is the only Maven knob docker/bake.hcl exposes. Maven accumulates repeated -pl values rather than letting a later one replace an earlier one, so MAVEN_ARGS can still add modules or drop them with a ! prefix, but it can no longer set the scope outright. The PR description notes that the fork experiments "enabled the same module selection through MAVEN_ARGS"; that route closes here.

Drift between the four copies is already covered by the docker-bake-check job in .github/workflows/docker-build-ci.yml, so this is only about overridability.

Suggested change: hoist the list into a build arg beside the existing ARG MAVEN_ARGS and pass it through _common.args in docker/bake.hcl.

ARG MAVEN_PROJECTS="hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist"

with the command becoming mvn install -pl "$MAVEN_PROJECTS" -am $MAVEN_ARGS .... All four files stay byte-identical, so the CI identity check still passes.

&& rm ./hugegraph-server/*.tar.gz ./hugegraph-pd/*.tar.gz ./hugegraph-store/*.tar.gz

@bitflicker64 bitflicker64 Sep 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 The reactor is scoped but the build context is not, so the caching win is smaller than the benchmark suggests. .dockerignore at head excludes only build output, archives, IDE and OS files, .git, .github, **/*.md and the compose files, so hugegraph-test, hg-pd-test, hg-store-test, hugegraph-cluster-test/**, hugegraph-example, hg-pd-cli, hg-store-cli and install-dist all still land in the context and still feed the COPY . . cache key on line 25. Editing any of them therefore invalidates this shared build stage and pays for a full scoped Maven run, for modules the build no longer compiles.

A follow-up rather than a change here, but worth recording. One caveat for whoever picks it up: these directories cannot be ignored wholesale, because the root pom lists them in <modules> and Maven fails when a listed module's pom.xml is absent, so only their src/ subtrees can be excluded.


# 2nd stage: runtime env
# Note: ZGC (The Z Garbage Collector) is only supported on ARM-Mac with java > 13
FROM eclipse-temurin:11-jre-jammy

COPY --from=build /pkg/hugegraph-server/apache-hugegraph-server-*/ /hugegraph-server/
LABEL maintainer="HugeGraph Docker Maintainers <dev@hugegraph.apache.org>"

# TODO: use g1gc or zgc as default
Expand All @@ -46,7 +46,7 @@ ENV JAVA_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseContainerSupport -XX:Max

WORKDIR /hugegraph-server/

# 1. Install runtime dependencies and configure server
# 1. Install runtime dependencies
# Note: iproute2 provides `ss`, which the bin/util.sh port preflight needs. The
# jammy base image ships neither ss nor netstat, so without it the preflight is
# permanently inconclusive and a duplicate start is no longer caught. It
Expand All @@ -59,8 +59,10 @@ RUN apt-get -q update \
iproute2 \
vim \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties
&& rm -rf /var/lib/apt/lists/*

COPY --from=build /pkg/hugegraph-server/apache-hugegraph-server-*/ /hugegraph-server/

@bitflicker64 bitflicker64 Sep 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ With the apt layer now ahead of this COPY, its cache key no longer includes the application source, so the layer is reused until the eclipse-temurin:11-jre-jammy digest changes. What is lost is the per-source-change reinstall: rebuilding the same commit already hit cache before, but any source change used to force a fresh apt-get install, and now nothing short of a base image update does.

This only bites on a registry-cache flow, and that flow is out of tree: docker/bake.hcl gates every cache-to behind EXPORT_CACHE, which defaults to false, and no workflow here runs bake to build or push. The PR's own numbers show the effect, with eight runtime layers restored in 2.0-9.3 s "without rerunning package installation".

Could you add a documented way to force a reinstall, for example ARG RUNTIME_DEPS_EPOCH=1 immediately before the apt block in all four files, bumped when packages need refreshing? --no-cache-filter is not an option as things stand, since the runtime stage has no AS name. A short note on the cache behaviour in docker/README.md would help whoever operates the publish flow.

RUN sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties

# 2. Init docker script
COPY hugegraph-server/hugegraph-dist/docker/scripts/remote-connect.groovy ./scripts
Expand Down
19 changes: 10 additions & 9 deletions hugegraph-server/Dockerfile-hstore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,18 +28,14 @@ ARG MAVEN_ARGS
ARG SOURCE_REVISION=local

RUN --mount=type=cache,id=hugegraph-maven-${SOURCE_REVISION},target=/root/.m2,sharing=locked \
mvn install $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
mvn install -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \
-am $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
&& rm ./hugegraph-server/*.tar.gz ./hugegraph-pd/*.tar.gz ./hugegraph-store/*.tar.gz

# 2nd stage: runtime env
# Note: ZGC (The Z Garbage Collector) is only supported on ARM-Mac with java > 13
FROM eclipse-temurin:11-jre-jammy

COPY --from=build /pkg/hugegraph-server/apache-hugegraph-server-*/ /hugegraph-server/
# remove hugegraph.properties and rename hstore.properties.template for default hstore backend
RUN cd /hugegraph-server/conf/graphs \
&& rm hugegraph.properties && mv hstore.properties.template hugegraph.properties

LABEL maintainer="HugeGraph Docker Maintainers <dev@hugegraph.apache.org>"

# TODO: use g1gc or zgc as default
Expand All @@ -48,7 +44,7 @@ ENV JAVA_OPTS="-XX:+UnlockExperimentalVMOptions -XX:+UseContainerSupport -XX:Max

WORKDIR /hugegraph-server/

# 1. Install runtime dependencies and configure server
# 1. Install runtime dependencies
# Note: iproute2 provides `ss`, which the bin/util.sh port preflight needs. The
# jammy base image ships neither ss nor netstat, so without it the preflight is
# permanently inconclusive and a duplicate start is no longer caught. It
Expand All @@ -61,8 +57,13 @@ RUN apt-get -q update \
iproute2 \
vim \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties
&& rm -rf /var/lib/apt/lists/*

COPY --from=build /pkg/hugegraph-server/apache-hugegraph-server-*/ /hugegraph-server/
# remove hugegraph.properties and rename hstore.properties.template for default hstore backend
RUN cd /hugegraph-server/conf/graphs \
&& rm hugegraph.properties && mv hstore.properties.template hugegraph.properties
RUN sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties
Comment on lines +64 to +66

@bitflicker64 bitflicker64 Sep 5, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Both of these RUNs edit only the tree copied on line 62, so they are invalidated together and the split just costs an image layer. This is the one file where that layer is avoidable: the plain server Dockerfile also gained a standalone sed layer, but it has a single edit with nothing to fold into.

Suggested change
RUN cd /hugegraph-server/conf/graphs \
&& rm hugegraph.properties && mv hstore.properties.template hugegraph.properties
RUN sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties
RUN cd /hugegraph-server/conf/graphs \
&& rm hugegraph.properties && mv hstore.properties.template hugegraph.properties \
&& cd /hugegraph-server \
&& sed -i "s/^restserver.url.*$/restserver.url=http:\/\/0.0.0.0:8080/g" ./conf/rest-server.properties

The comment on line 63 then covers only part of what the merged RUN does, so it is worth widening at the same time.


# 2. Init docker script
COPY hugegraph-server/hugegraph-dist/docker/scripts/remote-connect.groovy ./scripts
Expand Down
6 changes: 4 additions & 2 deletions hugegraph-store/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,14 @@ ARG MAVEN_ARGS
ARG SOURCE_REVISION=local

RUN --mount=type=cache,id=hugegraph-maven-${SOURCE_REVISION},target=/root/.m2,sharing=locked \
mvn install $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
mvn install -pl hugegraph-server/hugegraph-dist,hugegraph-pd/hg-pd-dist,hugegraph-store/hg-store-dist \
-am $MAVEN_ARGS -e -B -ntp -Dmaven.test.skip=true -Dmaven.javadoc.skip=true \
&& rm ./hugegraph-server/*.tar.gz ./hugegraph-pd/*.tar.gz ./hugegraph-store/*.tar.gz

# 2nd stage: runtime env
# Note: ZGC (The Z Garbage Collector) is only supported on ARM-Mac with java > 13
FROM eclipse-temurin:11-jre-jammy

COPY --from=build /pkg/hugegraph-store/apache-hugegraph-store-*/ /hugegraph-store/
LABEL maintainer="HugeGraph Docker Maintainers <dev@hugegraph.apache.org>"

# TODO: use g1gc or zgc as default
Expand All @@ -58,6 +58,8 @@ RUN apt-get -q update \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

COPY --from=build /pkg/hugegraph-store/apache-hugegraph-store-*/ /hugegraph-store/

# 2. Init docker script
COPY hugegraph-store/hg-store-dist/docker/docker-entrypoint.sh .
RUN chmod 755 ./docker-entrypoint.sh
Expand Down
Loading