From 05c54be0dc4595ec3b43f264f54a687319ee5bce Mon Sep 17 00:00:00 2001 From: Esteban Zimanyi Date: Wed, 29 Jul 2026 00:56:32 +0200 Subject: [PATCH] Generalize the refresh hub from JVM consumers to any binding refresh-jvm-chain.sh refreshed only the JVM consumers. Every binding shares the same chain: MobilityDB -> provision-meos.sh (catalog, and for native/FFI bindings libmeos) -> the binding's own generate + build. refresh-binding.sh is that generic entry point, a superset of the JVM hub: BUILD_LIBMEOS (native vs pure-catalog), a conf-settable MDB_REF (bindings that pin a MEOS ref), a configurable CATALOG_DEST (generators that take the catalog path as an argument set it empty), and the JMEOS-jar leg made conditional on JMEOS_COORDS. refresh-jvm-chain.sh becomes a thin alias for it, so the JVM consumers keep working unchanged; a JVM consumer is just a binding whose refresh.conf sets JMEOS_COORDS. --- GENERATION.md | 31 ++++--- tools/refresh-binding.sh | 179 +++++++++++++++++++++++++++++++++++++ tools/refresh-jvm-chain.sh | 172 ++--------------------------------- 3 files changed, 205 insertions(+), 177 deletions(-) create mode 100755 tools/refresh-binding.sh diff --git a/GENERATION.md b/GENERATION.md index 1b29304..91bc1d9 100644 --- a/GENERATION.md +++ b/GENERATION.md @@ -71,30 +71,33 @@ Each binding then regenerates from that file through its own entry point — for substrate, `JMEOS/tools/regen-from-catalog.sh `, which also builds the jar the JVM consumers bind. See each repo's `GENERATION.md`. -## Refreshing a JVM consumer end to end +## Refreshing a binding end to end -`tools/refresh-jvm-chain.sh` runs the whole JVM chain from one command, so a MobilityFlink, -MobilitySpark or MobilityKafka developer refreshes their facades against the latest MEOS API -without walking the four repos by hand: +`tools/refresh-binding.sh` runs the whole chain for **any** binding from one command, so a +GoMEOS, meos-rs, PyMEOS, MEOS.NET or JVM developer refreshes their generated surface against the +latest MEOS API without walking the repos by hand: ``` -MobilityDB -> provision-meos.sh (catalog + libmeos) -> JMEOS jar -> consumer facades +MobilityDB -> provision-meos.sh (catalog [+ libmeos]) -> the binding's generate + build ``` -It composes the per-leg scripts above — `provision-meos.sh`, then `JMEOS/regen-from-catalog.sh`, -then the consumer's own Maven build — and adds no derivation of its own, only the sequencing and -the sibling checkouts. Each consumer carries the tiny wrapper `tools/refresh-from-master.sh` that -locates this script and calls it, so the developer only ever runs, from their checkout: +It composes the per-leg scripts above — `provision-meos.sh`, then the binding's own generator and +build — and adds no derivation of its own, only the sequencing and the sibling checkouts. Each +binding carries the tiny wrapper `tools/refresh-from-master.sh` that locates this script and calls +it, so the developer only ever runs, from their checkout: ```bash tools/refresh-from-master.sh ``` -Siblings default to their latest upstream default branch; `--mdb ` / `--jmeos ` (or -`$MDB` / `$JMEOS`) point at an existing checkout on any branch instead, to refresh against -a MEOS branch still under development. The consumer's last leg is a few lines in its -`tools/refresh.conf` — `ENGINE`, the Maven `BUILD_DIR`, the `JMEOS_COORDS` the build resolves the -jar as, and the `BUILD_CMD`; `refresh-jvm-chain.sh --help` documents the contract. +MobilityDB defaults to its latest master; `--mdb ` (or `$MDB`) points at an existing checkout +on any branch instead, to refresh against a MEOS branch still under development, and a binding that +pins a MEOS ref sets `MDB_REF` in its config. The binding's last leg is a few lines in its +`tools/refresh.conf` — `ENGINE`, `BUILD_DIR`, `BUILD_LIBMEOS` (native/FFI bindings build libmeos; +pure-catalog ones do not), `CATALOG_DEST`, the JVM-only `JMEOS_COORDS`, and the `BUILD_CMD`; +`refresh-binding.sh --help` documents the contract. The JVM consumers are the special case whose +config sets `JMEOS_COORDS`, which inserts the JMEOS-jar leg; `refresh-jvm-chain.sh` is a thin +compatibility alias for `refresh-binding.sh`. ## Sequencing several bindings diff --git a/tools/refresh-binding.sh b/tools/refresh-binding.sh new file mode 100755 index 0000000..6703bfa --- /dev/null +++ b/tools/refresh-binding.sh @@ -0,0 +1,179 @@ +#!/usr/bin/env bash +# refresh-binding.sh — refresh ONE MobilityDB binding against the latest MEOS API, end to end, +# with one command. The single generic entry point for every binding (Go, Rust, Python, C#, +# and the JVM consumers): it composes the pieces that already define each leg and adds no +# derivation logic of its own, only the sequencing and the sibling checkouts, so it cannot +# drift from CI: +# 1. tools/provision-meos.sh derives the catalog (and, for native/FFI bindings, +# builds libmeos) from a MobilityDB checkout; +# 2. [JVM only] JMEOS regen-from-catalog.sh builds the jar the JVM consumers bind; +# 3. the binding's own generate + build regenerates its surface from the catalog. +# +# By default MobilityDB is taken at its latest master, so the refresh tracks the newest MEOS +# API. Point --mdb (or $MDB) at an existing checkout to refresh against in-flight work on any +# branch; a binding that pins a MEOS ref sets MDB_REF in its tools/refresh.conf. +# +# The binding describes its last leg in tools/refresh.conf (sourced). All keys are optional +# except BUILD_CMD: +# ENGINE=go # informational label +# BUILD_DIR=. # dir BUILD_CMD runs from, relative to the repo ("." = root) +# BUILD_LIBMEOS=true # true for native/FFI bindings; false for pure-catalog +# MDB_REF=master # MobilityDB ref to derive from (pinned bindings override) +# FAMILIES=-DALL=ON # cmake family flags for the libmeos build +# CATALOG_DEST=tools/meos-idl.json # where to stage the catalog, relative to the repo +# JMEOS_COORDS=org.jmeos:meos:1.0 # JVM only: build+install the JMEOS jar as these coords +# BUILD_CMD='' # run from / +# BUILD_CMD runs with $PREFIX (libmeos install prefix, empty when BUILD_LIBMEOS=false), +# $CATALOG (absolute catalog path), $JAR (JVM only) exported; $SKIP_TESTS set with --skip-tests. +# +# Usage: +# tools/refresh-binding.sh --binding [options] +# +# Options: +# --binding The binding repo to refresh (has tools/refresh.conf). Default: $PWD. +# (--consumer is accepted as an alias.) +# --work-dir Scratch for sibling clones and the libmeos prefix. Default: /.meos-chain. +# --mdb Existing MobilityDB checkout (any branch), used as-is. Env: MDB. +# --jmeos Existing JMEOS checkout (JVM bindings), used as-is. Env: JMEOS. +# --meos-api MEOS-API repo (holds provision-meos.sh). Default: this script's repo. +# --mdb-ref Override the MobilityDB ref (else refresh.conf MDB_REF, else master). +# --jmeos-ref JMEOS ref when --jmeos is not given. Default: main. +# --families Override the cmake family flags (else refresh.conf FAMILIES, else -DALL=ON). +# --skip-tests Build without running the binding's test suite (SKIP_TESTS=1 for BUILD_CMD). +# --force Rebuild libmeos even when the MobilityDB commit is unchanged. +# -h, --help +set -euo pipefail + +usage() { sed -n '2,/^set -euo/{/^set -euo/d;s/^# \{0,1\}//;p;}' "$0"; } + +BINDING="$PWD" +WORK_DIR="" +MDB="${MDB:-}" +JMEOS="${JMEOS:-}" +MEOSAPI="${MEOSAPI:-$(cd "$(dirname "$0")/.." && pwd)}" +MDB_REF_CLI="" +JMEOS_REF="main" +FAMILIES_CLI="" +SKIP_TESTS=0 +FORCE=0 + +while [ $# -gt 0 ]; do + case "$1" in + --binding|--consumer) BINDING="$2"; shift 2 ;; + --work-dir) WORK_DIR="$2"; shift 2 ;; + --mdb) MDB="$2"; shift 2 ;; + --jmeos) JMEOS="$2"; shift 2 ;; + --meos-api) MEOSAPI="$(cd "$2" && pwd)"; shift 2 ;; + --mdb-ref) MDB_REF_CLI="$2"; shift 2 ;; + --jmeos-ref) JMEOS_REF="$2"; shift 2 ;; + --families) FAMILIES_CLI="$2"; shift 2 ;; + --skip-tests) SKIP_TESTS=1; shift ;; + --force) FORCE=1; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "refresh-binding.sh: unknown argument '$1'" >&2; usage >&2; exit 2 ;; + esac +done + +step() { echo; echo "==> $*" >&2; } + +BINDING="$(cd "$BINDING" && pwd)" +[ -f "$BINDING/tools/refresh.conf" ] || { + echo "refresh-binding.sh: '$BINDING' has no tools/refresh.conf" >&2; exit 2; } +[ -f "$MEOSAPI/tools/provision-meos.sh" ] || { + echo "refresh-binding.sh: no tools/provision-meos.sh under --meos-api '$MEOSAPI'" >&2; exit 2; } +: "${WORK_DIR:=$BINDING/.meos-chain}" +mkdir -p "$WORK_DIR" +WORK_DIR="$(cd "$WORK_DIR" && pwd)" + +# Read the binding's last-leg description; apply defaults, then let CLI flags override. +ENGINE=""; BUILD_DIR="."; BUILD_LIBMEOS="true"; MDB_REF="master" +FAMILIES="-DALL=ON"; CATALOG_DEST="tools/meos-idl.json"; JMEOS_COORDS=""; BUILD_CMD="" +# shellcheck source=/dev/null +. "$BINDING/tools/refresh.conf" +[ -n "$BUILD_CMD" ] || { echo "refresh.conf: BUILD_CMD is required" >&2; exit 2; } +[ -n "$MDB_REF_CLI" ] && MDB_REF="$MDB_REF_CLI" +[ -n "$FAMILIES_CLI" ] && FAMILIES="$FAMILIES_CLI" + +# A sibling is either an existing checkout (used as-is, any branch) or a fresh clone of the +# given ref into the work dir, kept current on re-runs. +resolve_repo() { # name url ref existing_path -> echoes absolute path + local name="$1" url="$2" ref="$3" existing="$4" dir + if [ -n "$existing" ]; then + (cd "$existing" && pwd); return + fi + dir="$WORK_DIR/$name" + if [ -d "$dir/.git" ]; then + git -C "$dir" fetch --quiet "$url" "$ref" + git -C "$dir" checkout --quiet FETCH_HEAD + else + git clone --quiet --depth 1 --branch "$ref" "$url" "$dir" 2>/dev/null \ + || git clone --quiet "$url" "$dir" # fall back to full clone for a non-branch ref + fi + (cd "$dir" && pwd) +} + +step "Resolving MobilityDB @ ${MDB:-$MDB_REF} (MEOS-API $MEOSAPI)" +MDB="$(resolve_repo MobilityDB https://github.com/MobilityDB/MobilityDB "$MDB_REF" "$MDB")" +MDB_COMMIT="$(git -C "$MDB" rev-parse HEAD 2>/dev/null || echo unknown)" +echo "MobilityDB $MDB ($MDB_COMMIT)" >&2 + +PREFIX="$WORK_DIR/prefix" +CATALOG="$WORK_DIR/meos-idl.json" +LIBMEOS="$PREFIX/lib/libmeos.so" +STAMP="$WORK_DIR/.mdb-commit" + +step "Python dependencies for the catalog parse" +python3 -m pip install --quiet -r "$MEOSAPI/requirements.txt" + +# The catalog (and, for native/FFI bindings, libmeos) — the slow leg. Skip it when the commit, +# the family flags and the build-libmeos choice are all already current, unless --force. +STAMP_KEY="$MDB_COMMIT $FAMILIES libmeos=$BUILD_LIBMEOS" +provision_current=0 +if [ "$FORCE" = 0 ] && [ -f "$CATALOG" ] && [ "$(cat "$STAMP" 2>/dev/null)" = "$STAMP_KEY" ]; then + if [ "$BUILD_LIBMEOS" != "true" ] || [ -f "$LIBMEOS" ]; then provision_current=1; fi +fi +if [ "$provision_current" = 1 ]; then + step "catalog + libmeos already current for $MDB_COMMIT ($FAMILIES) (use --force to rebuild)" +else + step "Deriving the catalog from MobilityDB $MDB_COMMIT ($FAMILIES, build-libmeos=$BUILD_LIBMEOS)" + if [ "$BUILD_LIBMEOS" = "true" ]; then + "$MEOSAPI/tools/provision-meos.sh" --mdb-src "$MDB" --build-libmeos \ + --families "$FAMILIES" --parse-prefix "$PREFIX" --catalog-out "$CATALOG" + else + "$MEOSAPI/tools/provision-meos.sh" --mdb-src "$MDB" --catalog-out "$CATALOG" + fi + echo "$STAMP_KEY" > "$STAMP" +fi +[ "$BUILD_LIBMEOS" = "true" ] || { PREFIX=""; LIBMEOS=""; } + +# JVM only: build + install the JMEOS jar the consumers bind. +JAR="" +if [ -n "$JMEOS_COORDS" ]; then + JMEOS="$(resolve_repo JMEOS https://github.com/MobilityDB/JMEOS "$JMEOS_REF" "$JMEOS")" + step "Building and installing the JMEOS jar ($JMEOS_COORDS) from the catalog" + CATALOG="$CATALOG" LIBMEOS="$LIBMEOS" "$JMEOS/tools/regen-from-catalog.sh" + IFS=: read -r G A V <&2 +else echo " catalog : $CATALOG" >&2; fi +[ -n "$LIBMEOS" ] && echo " libmeos : $LIBMEOS" >&2 +[ -n "$JAR" ] && echo " jmeos : $JAR ($JMEOS_COORDS)" >&2 diff --git a/tools/refresh-jvm-chain.sh b/tools/refresh-jvm-chain.sh index 5619010..02064fa 100755 --- a/tools/refresh-jvm-chain.sh +++ b/tools/refresh-jvm-chain.sh @@ -1,164 +1,10 @@ #!/usr/bin/env bash -# refresh-jvm-chain.sh — refresh one JVM consumer (MobilityFlink / MobilitySpark / -# MobilityKafka) against the latest MEOS API, end to end, with one command. -# -# It is the whole chain -# MobilityDB -> MEOS-API (catalog + libmeos) -> JMEOS (jar) -> consumer (facades) -# composed from the pieces that already define each leg — this adds no derivation logic of -# its own, only the sequencing and the sibling checkouts, so it cannot drift from CI: -# 1. tools/provision-meos.sh builds libmeos and the catalog from a MobilityDB checkout; -# 2. JMEOS tools/regen-from-catalog.sh builds the jar the consumers bind; -# 3. the consumer's own Maven build regenerates its facades at generate-sources. -# -# By default every sibling is taken at its latest upstream default branch, so the refresh -# tracks the newest MEOS API. Point --mdb / --jmeos (or $MDB / $JMEOS) at an existing checkout -# to refresh against in-flight work on any branch instead — e.g. a MEOS function you have not -# merged yet. -# -# The consumer describes its last leg in tools/refresh.conf (sourced), a few shell lines: -# ENGINE=flink # flink | spark | kafka -# BUILD_DIR=flink-processor # Maven module dir, relative to the repo root ("." = root) -# JMEOS_COORDS=org.jmeos:meos:1.0 # groupId:artifactId:version the build resolves the jar as -# BUILD_CMD='mvn -B -Dmeos.lib.dir="$PREFIX/lib" -Dmeos.enabled=true clean test' -# BUILD_CMD runs from / with $PREFIX (the libmeos install prefix), -# $CATALOG and $JAR exported. -# -# Usage: -# tools/refresh-jvm-chain.sh --consumer [options] -# -# Options: -# --consumer The JVM consumer repo to refresh (has tools/refresh.conf). Default: $PWD. -# --work-dir Scratch for sibling clones and the libmeos prefix. -# Default: /.meos-chain. -# --mdb Existing MobilityDB checkout (any branch), used as-is. Env: MDB. -# Otherwise MobilityDB is cloned at --mdb-ref into the work dir. -# --jmeos Existing JMEOS checkout, used as-is. Env: JMEOS. -# Otherwise JMEOS is cloned at --jmeos-ref into the work dir. -# --meos-api MEOS-API repo (holds provision-meos.sh). Default: this script's repo. -# --mdb-ref Ref to clone MobilityDB at when --mdb is not given. Default: master. -# --jmeos-ref Ref to clone JMEOS at when --jmeos is not given. Default: main. -# --families cmake family flags for the libmeos build. Default: -DALL=ON. -# --skip-tests Build the consumer without running its test suite (SKIP_TESTS=1 for BUILD_CMD). -# --force Rebuild libmeos even when the MobilityDB commit is unchanged. -# -h, --help -set -euo pipefail - -usage() { sed -n '2,/^set -euo/{/^set -euo/d;s/^# \{0,1\}//;p;}' "$0"; } - -CONSUMER="$PWD" -WORK_DIR="" -MDB="${MDB:-}" -JMEOS="${JMEOS:-}" -MEOSAPI="${MEOSAPI:-$(cd "$(dirname "$0")/.." && pwd)}" -MDB_REF="master" -JMEOS_REF="main" -FAMILIES="-DALL=ON" -SKIP_TESTS=0 -FORCE=0 - -while [ $# -gt 0 ]; do - case "$1" in - --consumer) CONSUMER="$2"; shift 2 ;; - --work-dir) WORK_DIR="$2"; shift 2 ;; - --mdb) MDB="$2"; shift 2 ;; - --jmeos) JMEOS="$2"; shift 2 ;; - --meos-api) MEOSAPI="$(cd "$2" && pwd)"; shift 2 ;; - --mdb-ref) MDB_REF="$2"; shift 2 ;; - --jmeos-ref) JMEOS_REF="$2"; shift 2 ;; - --families) FAMILIES="$2"; shift 2 ;; - --skip-tests) SKIP_TESTS=1; shift ;; - --force) FORCE=1; shift ;; - -h|--help) usage; exit 0 ;; - *) echo "refresh-jvm-chain.sh: unknown argument '$1'" >&2; usage >&2; exit 2 ;; - esac -done - -step() { echo; echo "==> $*" >&2; } - -CONSUMER="$(cd "$CONSUMER" && pwd)" -[ -f "$CONSUMER/tools/refresh.conf" ] || { - echo "refresh-jvm-chain.sh: '$CONSUMER' has no tools/refresh.conf" >&2; exit 2; } -[ -f "$MEOSAPI/tools/provision-meos.sh" ] || { - echo "refresh-jvm-chain.sh: no tools/provision-meos.sh under --meos-api '$MEOSAPI'" >&2; exit 2; } -: "${WORK_DIR:=$CONSUMER/.meos-chain}" -mkdir -p "$WORK_DIR" -WORK_DIR="$(cd "$WORK_DIR" && pwd)" - -# A sibling is either an existing checkout (used as-is, any branch) or a fresh clone of the -# upstream default branch into the work dir, kept current on re-runs. -resolve_repo() { # name url ref existing_path -> echoes absolute path - local name="$1" url="$2" ref="$3" existing="$4" dir - if [ -n "$existing" ]; then - (cd "$existing" && pwd); return - fi - dir="$WORK_DIR/$name" - if [ -d "$dir/.git" ]; then - git -C "$dir" fetch --quiet "$url" "$ref" - git -C "$dir" checkout --quiet FETCH_HEAD - else - git clone --quiet --depth 1 --branch "$ref" "$url" "$dir" 2>/dev/null \ - || git clone --quiet "$url" "$dir" # fall back to full clone for a non-branch ref - fi - (cd "$dir" && pwd) -} - -step "Resolving siblings (MobilityDB @ ${MDB:-$MDB_REF}, JMEOS @ ${JMEOS:-$JMEOS_REF}, MEOS-API $MEOSAPI)" -MDB="$(resolve_repo MobilityDB https://github.com/MobilityDB/MobilityDB "$MDB_REF" "$MDB")" -JMEOS="$(resolve_repo JMEOS https://github.com/MobilityDB/JMEOS "$JMEOS_REF" "$JMEOS")" -MDB_COMMIT="$(git -C "$MDB" rev-parse HEAD 2>/dev/null || echo unknown)" -echo "MobilityDB $MDB ($MDB_COMMIT)" >&2 - -PREFIX="$WORK_DIR/prefix" -CATALOG="$WORK_DIR/meos-idl.json" -LIBMEOS="$PREFIX/lib/libmeos.so" -STAMP="$WORK_DIR/.mdb-commit" - -step "Python dependencies for the catalog parse" -python3 -m pip install --quiet -r "$MEOSAPI/requirements.txt" - -# libmeos + catalog: the slow leg. Skip it when the MobilityDB commit and the built library -# are both already current, unless --force. The stamp carries the commit AND the family flags, -# so changing --families rebuilds rather than reusing a library with a different surface. -STAMP_KEY="$MDB_COMMIT $FAMILIES" -if [ "$FORCE" = 0 ] && [ -f "$LIBMEOS" ] && [ -f "$CATALOG" ] \ - && [ "$(cat "$STAMP" 2>/dev/null)" = "$STAMP_KEY" ]; then - step "libmeos + catalog already current for $MDB_COMMIT ($FAMILIES) (use --force to rebuild)" -else - step "Building libmeos + catalog from MobilityDB $MDB_COMMIT ($FAMILIES)" - "$MEOSAPI/tools/provision-meos.sh" \ - --mdb-src "$MDB" \ - --build-libmeos \ - --families "$FAMILIES" \ - --parse-prefix "$PREFIX" \ - --catalog-out "$CATALOG" - echo "$STAMP_KEY" > "$STAMP" -fi - -# Read the consumer's last-leg description. -ENGINE=""; BUILD_DIR="."; JMEOS_COORDS=""; BUILD_CMD="" -# shellcheck source=/dev/null -. "$CONSUMER/tools/refresh.conf" -[ -n "$JMEOS_COORDS" ] || { echo "refresh.conf: JMEOS_COORDS is required" >&2; exit 2; } -[ -n "$BUILD_CMD" ] || { echo "refresh.conf: BUILD_CMD is required" >&2; exit 2; } - -step "Building and installing the JMEOS jar ($JMEOS_COORDS) from the catalog" -CATALOG="$CATALOG" LIBMEOS="$LIBMEOS" "$JMEOS/tools/regen-from-catalog.sh" -IFS=: read -r G A V <&2 -echo " libmeos : $LIBMEOS" >&2 -echo " jmeos : $JMEOS/jar/JMEOS.jar ($JMEOS_COORDS)" >&2 +# refresh-jvm-chain.sh — compatibility shim for refresh-binding.sh. +# +# The JVM consumers (MobilityFlink / MobilitySpark / MobilityKafka) call this through their +# tools/refresh-from-master.sh wrapper. The generic entry point for every binding is now +# tools/refresh-binding.sh; a JVM consumer is just a binding whose tools/refresh.conf sets +# JMEOS_COORDS (which triggers the JMEOS-jar leg). This forwards unchanged — --consumer is an +# accepted alias for --binding — so those wrappers keep working; new bindings call +# refresh-binding.sh directly. +exec "$(dirname "$0")/refresh-binding.sh" "$@"