[R] Vendor trimmed thirdparty sources so offline (CRAN) source builds can include Parquet #55478

This is an analysis I ran with Claude to investigate what we might be able to vendor so that we could it on in an offline installation. This is a follow on to the work in #55461 which added the initial vendor-some-dependencies-necessary to even run. With that in place, I wondered: could we support just enough of Parquet to make an Arrow package, built from source, without access to the internet more useful?

TL;DR, yes, though it means going from a ~5MB tarball to a ~14MB tarball (+ maintaining these trimming scripts, testing them, etc)

Describe the enhancement requested

When the R package builds libarrow from source without network access (e.g. on CRAN), nixlibs.R calls turn_off_all_optional_features(). That build has no Parquet, Dataset, JSON, compression codecs, RE2 or utf8proc. #55461 (GH-55454) added a way to ship thirdparty source tarballs inside the R package (tools/vendored, populated by tools/vendor_dependencies.sh during make sync-cpp) for xsimd and uriparser, which even a minimal build needs. The same mechanism could ship everything a full non-cloud build needs, including Parquet, and keep the package small.

Summary: trimmed tarballs for every non-cloud dependency (Parquet, Dataset, Acero, JSON, all codecs, RE2, utf8proc, mimalloc) add ~9.9 MB to the package (~14 MB total, compared with 4.2 MB today). Untrimmed they would add ~157 MB, almost all of it Boost (136 MB). I built libarrow from the trimmed tarballs with no downloads on macOS arm64 (details below).

Current size

• arrow 26.0.1 on CRAN: 4.2 MB
• + xsimd and uriparser vendored by GH-55454: [R][C++] Vendor URI parser for the R library #55461: 0.57 MB, so ~4.8 MB

Size by feature (.tar.gz, versions from cpp/thirdparty/versions.txt at the time of writing)

Dataset and Acero need no further thirdparty dependencies. abseil isn't needed: the pinned re2 (2023-03-01) is older than re2's abseil dependency.

Resulting package size, including the 4.2 MB package itself:

Boost is the only dependency that must be trimmed. Trimming the others saves ~16 MB.

Trimming steps

Boost (136 MB to 4.9 MB)

1. Configure Boost standalone with the BOOST_INCLUDE_LIBRARIES list from build_boost() in ThirdpartyToolchain.cmake. For a threaded build with Parquet, that's algorithm;crc;numeric/conversion;scope_exit;throw_exception;tokenizer;locale;thread;uuid. Boost's CMake resolves the transitive dependencies, which come to 54 libraries. The libraries it adds appear as subdirectories of <build>/libs.
2. Keep only the top-level files, tools/cmake, and those 54 libs/* directories.
3. Delete top-level doc/, status/, more/, and everything in tools/ except tools/cmake.
4. Inside each kept library, delete test, doc, example(s), bench(mark), perf, performance, meta and tools directories.

If multiprecision is added (only needed when the compiler has no native int128, see ARROW_BOOST_NEED_MULTIPRECISION), the trimmed tarball is 10.5 MB instead. gcc/clang/mingw all have native int128, so the smaller list should be enough for R. The library list depends on Arrow's CMake configuration, so it needs re-checking whenever Boost or build_boost() changes. The old cpp/build-support/trim-boost.sh (removed in #50885) did something similar for the b2-based Boost tarball.

Everything else

The generic step deletes test, tests, testdata, doc, docs, example(s), benchmark(s), bench and fuzz directories. On top of that:

Each trimmed tarball no longer matches the pinned ARROW_*_BUILD_SHA256_CHECKSUM in versions.txt, and the bundled build checks those hashes (URL_HASH). The R package has its own copy of the file, so sync-cpp would rewrite the checksums in r/tools/cpp/thirdparty/versions.txt to match the trimmed tarballs, the same way it already seds tools/cpp/CMakeLists.txt. (Alternatively, trim only Boost and ship the rest as-is: ~30 MB package.)

The scripts I used (directory names hard-code the current versions):

#!/usr/bin/env bash
# Trim the Boost CMake source tarball to the libraries libarrow's bundled
# Boost build uses.
# Usage: trim-boost.sh boost-1.92.0-cmake.tar.gz OUT.tar.gz
set -euo pipefail
IN="$(cd "$(dirname "$1")" && pwd)/$(basename "$1")"
OUT="$(cd "$(dirname "$2")" && pwd)/$(basename "$2")"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
cd "$WORK"
tar xzf "$IN"
SRC="$(ls -d boost-*)"
SRC_ABS="$WORK/$SRC"

# BOOST_INCLUDE_LIBRARIES from build_boost() in ThirdpartyToolchain.cmake for a
# threaded build with Parquet (thrift) and native int128. Add multiprecision if
# ARROW_BOOST_NEED_MULTIPRECISION would be true.
LIBS="algorithm;crc;numeric/conversion;scope_exit;throw_exception;tokenizer;locale;thread;uuid"

# Let Boost's CMake resolve transitive dependencies; each library it adds
# shows up as a directory under build/libs.
cmake -S "$SRC" -B build -DBOOST_INCLUDE_LIBRARIES="$LIBS" > /dev/null
KEEP=$(cd build/libs && find . -maxdepth 2 -type d ! -name CMakeFiles | sed 's|^\./||' |
  while read -r d; do [ -f "$SRC_ABS/libs/$d/CMakeLists.txt" ] && echo "libs/$d" || true; done)

mkdir trimmed
(cd "$SRC" && tar cf - $(ls | grep -v '^libs$') $KEEP) | (mkdir "trimmed/$SRC" && tar xf - -C "trimmed/$SRC")
cd "trimmed/$SRC"
rm -rf doc status more
(cd tools && ls | grep -v '^cmake$' | xargs rm -rf)
find libs -depth -type d \( -name test -o -name doc -o -name example -o -name examples \
  -o -name bench -o -name benchmark -o -name perf -o -name performance -o -name meta \
  -o -name tools \) -prune -exec rm -rf {} +
cd ..
tar czf "$OUT" "$SRC"

#!/usr/bin/env bash
# Trim thirdparty source tarballs down to what libarrow's bundled build uses.
# Usage: trim.sh SRC_TARBALL_DIR OUT_DIR
set -euo pipefail
IN="$(cd "$1" && pwd)"
OUT="$(mkdir -p "$2" && cd "$2" && pwd)"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT

rmd() { for p in "$@"; do rm -rf "$p"; done; }

trim_generic() {
  # dirs no bundled build of these deps compiles; pass names to keep as args
  local keep=("$@")
  find . -depth -type d \( -name test -o -name tests -o -name testdata \
    -o -name doc -o -name docs -o -name example -o -name examples \
    -o -name benchmark -o -name benchmarks -o -name bench -o -name fuzz \) \
    -prune -print | while read -r d; do
      for k in "${keep[@]+"${keep[@]}"}"; do [ "$d" = "./$k" ] && continue 2; done
      rm -rf "$d"
    done
}

pack() { # dir, output name
  tar czf "$OUT/$2" "$1"
}

cd "$WORK"
for t in "$IN"/*; do tar xf "$t"; done

# thrift: only the C++ library is built (BUILD_COMPILER=OFF, WITH_*=OFF)
( cd thrift-0.25.0 && rmd tutorial test compiler contrib doc debian \
    && (cd lib && ls | grep -v -E '^(cpp|CMakeLists.txt)$' | xargs rm -rf) && trim_generic )
pack thrift-0.25.0 thrift-0.25.0.tar.gz

# simdjson: built from src/ + include/; singleheader/ is a 20 MB amalgamation
( cd simdjson-4.6.4 && rmd singleheader images jsonexamples tools scripts && trim_generic )
pack simdjson-4.6.4 simdjson-v4.6.4.tar.gz

# zlib builds test/example.c by default (ZLIB_BUILD_EXAMPLES isn't turned off)
( cd zlib-1.3.1 && rmd contrib && trim_generic test )
pack zlib-1.3.1 zlib-1.3.1.tar.gz

( cd snappy-1.2.2 && trim_generic )
pack snappy-1.2.2 snappy-1.2.2.tar.gz

( cd lz4-1.10.0 && trim_generic && (cd build && ls | grep -v '^cmake$' | xargs rm -rf) )
pack lz4-1.10.0 lz4-v1.10.0.tar.gz

( cd zstd-1.5.7 && rmd contrib zlibWrapper && trim_generic \
    && (cd build && ls | grep -v '^cmake$' | xargs rm -rf) )
pack zstd-1.5.7 zstd-1.5.7.tar.gz

# brotli installs man pages from docs/
( cd brotli-1.2.1 && rmd go python java csharp js research && trim_generic docs )
pack brotli-1.2.1 brotli-v1.2.1.tar.gz

( cd bzip2-1.0.8 && rm -f manual.* sample* *.html && trim_generic )
pack bzip2-1.0.8 bzip2-1.0.8.tar.gz

( cd re2-2023-03-01 && rmd benchlog python app && trim_generic )
pack re2-2023-03-01 re2-2023-03-01.tar.gz

( cd utf8proc-2.10.0 && trim_generic )
pack utf8proc-2.10.0 utf8proc-v2.10.0.tar.gz

R=rapidjson-232389d4f1012dddec4ef84861face2d2ba85709
( cd $R && rmd bin thirdparty && trim_generic )
pack $R $R.tar.gz

( cd mimalloc-3.4.1 && rmd ide && trim_generic )
pack mimalloc-3.4.1 mimalloc-v3.4.1.tar.gz

( cd xsimd-14.3.0 && trim_generic )
pack xsimd-14.3.0 xsimd-14.3.0.tar.gz

( cd uriparser-1.0.2 && trim_generic )
tar cjf "$OUT/uriparser-1.0.2.tar.bz2" uriparser-1.0.2

Verification

On macOS arm64 (Apple clang), I ran r/inst/build_arrow_static.sh with LIBARROW_MINIMAL=false ARROW_DEPENDENCY_SOURCE=BUNDLED ARROW_S3=OFF ARROW_GCS=OFF ARROW_AZURE=OFF ARROW_JEMALLOC=OFF. Every ARROW_*_URL pointed at a trimmed local tarball, and the checksums in versions.txt were rewritten to match. The FetchContent/ExternalProject logs confirm that every dependency came from a local file. libarrow, libarrow_compute, libarrow_acero, libarrow_dataset, libparquet and libarrow_bundled_dependencies all built and installed.

Not yet tested: Linux, Windows (Rtools), and actually installing and checking the R package.

What would need to change

1. tools/vendor_dependencies.sh: vendor the dependencies above, apply the trimming, and rewrite the checksums in r/tools/cpp/thirdparty/versions.txt.
2. nixlibs.R: offline with no thirdparty_dependency_dir, the build currently calls turn_off_all_optional_features() (around L630-644). It should instead turn off only the features whose tarballs aren't vendored. S3 and Azure need care, because they default to ON on macOS. The existing set_thirdparty_urls() / create_package_with_all_dependencies() code already handles "build from a directory of tarballs" and could probably be reused.
3. LICENSE/NOTICE: add entries for the newly shipped sources (Boost Software License, thrift, simdjson, the codecs, etc.).

Other considerations

• CRAN policy: CRAN asks packages to be "of the minimum necessary size", so a ~14 MB source tarball may need justifying.
• Build time: building thrift, Boost.Locale/Thread and Parquet adds noticeable compile time to CRAN source builds.

Cloud filesystems (numbers only, not proposed)

For reference, the cloud filesystems are much more expensive and also need system libraries that an offline build can't get:

Component(s)

R