From 86544a8b0ffb8feafabaffc6dd98dc6cb071c21a Mon Sep 17 00:00:00 2001 From: Raman Gupta Date: Fri, 30 Oct 2026 16:37:06 -0400 Subject: [PATCH] config: substitute with Hoplite's resolvers, not preprocessors Hoplite has deprecated preprocessors in favour of resolvers since 2023. Resolvers resolve a value only when it is bound, so `${...}` in exported shell variables such as PS1 no longer fails loading, and the isKnownNonConfigEnvVar filter is removed. - `${{ env:name }}` reads the environment that hopliteConfigBinder's environmentVariables supplies, rather than System.getenv as Hoplite's EnvVarContextResolver does. - Test resolvers, HOCON's own substitutions, and pin down that resolvers leave `${{ ... }}` in map values unresolved. - Document the syntax change and its gaps in the CHANGELOG and the module README. --- CHANGELOG.adoc | 49 +++++++++ README.adoc | 1 + boot-config-hoplite/README.adoc | 29 +++++ .../bootable/config/HopliteConfigBinder.kt | 55 ++++++---- .../config/HopliteConfigBinderTest.kt | 101 ++++++++++++++++-- 5 files changed, 202 insertions(+), 33 deletions(-) diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index c587838..d11a4a9 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -89,6 +89,55 @@ Bootable now uses Hoplite 3.0.5, where an environment variable selects a sealed For example, `tiebreaker { _type = Shuffled, seed = 42 }` in config, or `BOOT_SERVICE_ORDER_TIEBREAKER=Shuffled` and `BOOT_SERVICE_ORDER_TIEBREAKER_SEED=42`. An environment variable that selects a subtype overrides the one config selects, and config can still set the subtype's properties. +=== Config substitutions use Hoplite's resolvers + +`hopliteConfigBinder` now substitutes values with Hoplite's resolvers, `${{ context:path }}`, instead of its preprocessors, `${VAR}` and `{{key}}`. +Hoplite has deprecated preprocessors in favour of resolvers since 2023, and its maintainer has confirmed resolvers are the intended default. +Resolvers resolve a value only when it is bound, so a `${...}` elsewhere, such as in an exported shell variable like `PS1`, no longer fails loading. +That makes Bootable's filter of such environment variables unnecessary, so it is removed. + +The substitutions map as follows: + +[cols="1,1"] +|=== +|Preprocessor |Resolver + +|`${VAR}` (environment variable or system property) +|`${{ env:VAR }}`, or `${{ sysprop:name }}` for a system property + +|`${VAR:-default}` +|`${{ env:VAR:-default }}` + +|`{{key}}` or `${key}` (another config value) +|`${{ ref:key }}` + +|`${random.int}`, `${random.int(10)}`, `${random.uuid}`, ... +|`${{ random:int }}`, `${{ random:int(10) }}`, `${{ random:uuid }}`, ... +|=== + +`${{ env:name }}` reads the environment that `hopliteConfigBinder`'s `environmentVariables` supplies, as the environment variables source does. + +HOCON's own substitutions, unquoted `${VAR}`, `${?VAR}`, and `${a.b}`, are unaffected: HOCON resolves them when it parses the file, falling back to the environment for `${VAR}` and `${?VAR}`. +They are the recommended way to reference environment variables, and the only one inside maps and lists. + +==== Breaking changes + +* Preprocessor substitutions are no longer expanded. +They are not an error: a `${VAR}` or `{{key}}` is left in the value as literal text. +That includes a `${VAR}` inside a quoted HOCON string, which Hoplite's preprocessors used to expand and HOCON doesn't. +* A resolver resolves only a data class field whose value is a single string. +A `${{ ... }}` inside a map, list, or set, such as `dataSourceProperties`, Flyway's `placeholders`, `locations`, or `schemas`, or telemetry's `otlp.headers` or `resourceAttributes`, is left as literal text; use HOCON's `${?VAR}` there. +* A `${{ ... }}` that resolves to nothing fails the bind, and doesn't fall back to the data class's default value (https://github.com/sksamuel/hoplite/issues/460[sksamuel/hoplite#460]). +Give an optional value a fallback, e.g. `${{ env:DB_HOST:-localhost }}`. +* Hoplite's module resolvers use different names from its module preprocessors: `awssm://` (and `${awssecret:...}`, `secretsmanager://`) is `aws-secrets-manager://`, `azurekeyvault://` is `azure-key-vault://`, and `gcpsm://` is `gcp-secrets-manager://`. +`vault://` and `consul://` are unchanged. +There is no resolver for AWS Parameter Store's `${ssm:key}` or `paramstore://`. +* Hoplite's module resolvers, for Vault, AWS, GCP, Azure, and Consul, can now be added in the `withConfigStageN` hooks with `addResolver`, which used to throw because Bootable had added preprocessors. +A preprocessor added there still runs, but an unresolved substitution is no longer reported, since Hoplite skips that check once there are resolvers. +* The config report shows values before resolvers run, so it shows `${{ env:DB_HOST }}` rather than the value it resolves to. +Resolvers don't mark values as secrets in the report the way some preprocessors did, but since the report shows the expression, it doesn't print the secret either. +* `isKnownNonConfigEnvVar` is removed. + === A failed start shuts down the services that started When a service's `start()` throws, Bootable now runs the normal shutdown before exiting with -1, instead of exiting at once. diff --git a/README.adoc b/README.adoc index 057a2c2..ae6c13b 100644 --- a/README.adoc +++ b/README.adoc @@ -64,6 +64,7 @@ In your own config classes, declare secrets with your config library's type, suc See https://github.com/sksamuel/hoplite[hoplite]. Add this module to your project to use Hoplite for configuration. +A config value can take an environment variable with HOCON's own `${?VAR}`, or with Hoplite's `${{ env:VAR }}` resolver, which works only in a field that holds a single string; see its documentation's substitutions section. xref:./boot-config-hoplite/README.adoc[Documentation] diff --git a/boot-config-hoplite/README.adoc b/boot-config-hoplite/README.adoc index 04d3f6f..05df040 100644 --- a/boot-config-hoplite/README.adoc +++ b/boot-config-hoplite/README.adoc @@ -37,6 +37,35 @@ Bootable injects a Hoplite `ParameterMapper` that maps configuration properties See also https://github.com/sksamuel/hoplite/issues/410[hoplite issue #410]. +## Substitutions + +A config value can reference an environment variable, a system property, or another config value with Hoplite's https://github.com/sksamuel/hoplite#resolvers[resolvers]: + +```hocon +app.name = "orders" +db { + host = "${{ env:DB_HOST }}" + port = "${{ env:DB_PORT:-5432 }}" + user = "${{ sysprop:db.user }}" + name = "${{ ref:app.name }}" +} +``` + +A reference that resolves to nothing, without a `:-fallback`, fails the bind. +`${{ env:name }}` reads the environment variables passed to `hopliteConfigBinder(environmentVariables = ...)`, which default to the process environment. +Hoplite modules add resolvers for secret stores, such as Vault, AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault; add them with `addResolver` in a `withConfigStageN` hook. + +Resolvers resolve only a data class field whose value is a single string, not the entries of a map, list, or set. +There, and anywhere else, use HOCON's own substitutions, which HOCON resolves when it parses the file, falling back to the environment: + +```hocon +db.dataSourceProperties { + sslmode = ${?DB_SSLMODE} +} +``` + +Hoplite's deprecated preprocessor syntax, `${VAR}` in a quoted string or `{{key}}`, is not expanded and is left as literal text. + ## Custom Types Hoplite `Decoder` instances may be registered via the Java ServiceLoader framework, or passed to the `hopliteConfigBinder(...)` factory. diff --git a/boot-config-hoplite/src/main/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinder.kt b/boot-config-hoplite/src/main/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinder.kt index f483032..d7808f4 100644 --- a/boot-config-hoplite/src/main/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinder.kt +++ b/boot-config-hoplite/src/main/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinder.kt @@ -3,6 +3,9 @@ package com.github.rocketraman.bootable.config import com.sksamuel.hoplite.* import com.sksamuel.hoplite.ConfigBinder as UpstreamConfigBinder import com.sksamuel.hoplite.decoder.Decoder +import com.sksamuel.hoplite.fp.valid +import com.sksamuel.hoplite.resolver.context.ContextResolver +import com.sksamuel.hoplite.resolver.context.EnvVarContextResolver import com.sksamuel.hoplite.secrets.CompositeSecretsPolicy import com.sksamuel.hoplite.secrets.Obfuscator import com.sksamuel.hoplite.secrets.SecretsPolicy @@ -19,24 +22,10 @@ class HopliteConfigBinder(private val binder: UpstreamConfigBinder): ConfigBinde binder.bindOrThrow(type, prefix) } -/** - * Whether the environment variable [name] is set by the shell or other tooling and is never application config. - * These are excluded from the environment source because their values are shell code that routinely contains - * `${...}`, which Hoplite would otherwise fail on as an unresolved substitution. - * - * TODO remove once Hoplite excludes these itself: https://github.com/sksamuel/hoplite/issues/602 - */ -fun isKnownNonConfigEnvVar(name: String): Boolean = - // exported bash functions (`export -f`, or tools like Lmod), named `BASH_FUNC_name%%` - name.startsWith("BASH_FUNC_") || - name in shellPromptEnvVars - -private val shellPromptEnvVars = setOf("PS1", "PS2", "PS4", "PROMPT_COMMAND") - /** * Builds a default Hoplite-based [ConfigBinder], reading properties in the following order, from highest priority * to lowest: - * - environment variables (supports upper case, convert `_` to `.`; see [isKnownNonConfigEnvVar] for exclusions) + * - environment variables (supports upper case, convert `_` to `.`) * - system properties (with `config.override.` prefix) * - `application-local.conf` in resources * - `application.conf` in resources @@ -54,11 +43,17 @@ private val shellPromptEnvVars = setOf("PS1", "PS2", "PS4", "PROMPT_COMMAND") * - stage3 * * If report is true (the default), a report of all config keys and values is enabled. All sensitive values should - * be of type [Masked] (or Hoplite's `Secret`) to avoid leaking them in the report, named with "secret", - * "credential", or "pass" in the name, or have pre-processor metadata that identifies the value as a secret. - * Configure this via [secretsPolicy]; a [Masked] value is masked whatever the policy. + * be of type [Masked] (or Hoplite's `Secret`), or named with "secret", "credential", or "pass" in the name, to + * avoid leaking them in the report. Configure this via [secretsPolicy]; a [Masked] value is masked whatever the + * policy. The report shows values as loaded, before resolvers run, so it shows `${{ env:DB_PASSWORD }}`, not the + * password. + * + * Config values may reference other values with Hoplite's resolvers, e.g. `${{ env:DB_HOST }}`, + * `${{ env:DB_HOST:-localhost }}`, or `${{ sysprop:name }}`. These resolve only in a field whose value is a single + * string, not in a map, list, or set; use HOCON's own `${?DB_HOST}` substitutions there. * - * [environmentVariables] supplies the environment variables to read (defaults to [System.getenv]). + * [environmentVariables] supplies the environment variables to read, both as a config source and for + * `${{ env:name }}` (defaults to [System.getenv]). * * A sealed type is chosen by its subtype's name, in a `_type` key next to the subtype's properties, e.g. * `tiebreaker { _type = Shuffled, seed = 42 }`, or, in the environment, as the sealed type's own value, e.g. @@ -89,13 +84,13 @@ fun hopliteConfigBinder( .addDefaultDecoders() .addDecoder(MaskedDecoder()) .addDecoders(decoders) - .addDefaultPreprocessors() + .addResolvers(defaultResolvers().map { + if (it == EnvVarContextResolver) EnvironmentVariablesContextResolver(environmentVariables) else it + }) .addDefaultNodeTransformers() .addDefaultParamMappers() .withConfigStage1() - .addPropertySource(EnvironmentVariablesPropertySource({ - environmentVariables().filterKeys { !isKnownNonConfigEnvVar(it) } - })) + .addPropertySource(EnvironmentVariablesPropertySource(environmentVariables)) .addPropertySource(SystemPropertiesPropertySource) // requires config.override. prefix .addPropertySource(UserSettingsPropertySource) .addPropertySource(XdgConfigPropertySource) @@ -121,3 +116,17 @@ fun hopliteConfigBinder( return HopliteConfigBinder(upstream) } + +/** + * Hoplite's [EnvVarContextResolver], reading [environmentVariables] instead of [System.getenv], so that + * `${{ env:name }}` sees the same environment as the environment variables source. + */ +private class EnvironmentVariablesContextResolver( + private val environmentVariables: () -> Map, +) : ContextResolver() { + override val contextKey = EnvVarContextResolver.contextKey + override val default = EnvVarContextResolver.default + + override fun lookup(path: String, node: StringNode, root: Node, context: DecoderContext): ConfigResult = + environmentVariables()[path].valid() +} diff --git a/boot-config-hoplite/src/test/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinderTest.kt b/boot-config-hoplite/src/test/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinderTest.kt index e769249..02afed9 100644 --- a/boot-config-hoplite/src/test/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinderTest.kt +++ b/boot-config-hoplite/src/test/kotlin/com/github/rocketraman/bootable/config/HopliteConfigBinderTest.kt @@ -21,6 +21,9 @@ enum class Speed { Fast, Slow } // public: hoplite constructs it reflectively data class SpeedConfig(val speed: Speed) +// public: hoplite constructs it reflectively +data class PropertiesConfig(val values: Map) + private fun binder(env: Map): ConfigBinder = hopliteConfigBinder(report = false, environmentVariables = { env }) @@ -52,6 +55,11 @@ private object NothingSecret : SecretsPolicy { override fun isSecret(node: Node, type: KType?) = false } +private fun hocon(conf: String, env: Map = emptyMap()): ConfigBinder = + hopliteConfigBinder(report = false, environmentVariables = { env }, withConfigStage2 = { + addPropertySource(PropertySource.string(conf, "conf")) + }) + val HopliteConfigBinderTest by testSuite { test("decodes Masked values in strict mode, maps' included") { val binder = hopliteConfigBinder( @@ -81,24 +89,97 @@ val HopliteConfigBinderTest by testSuite { assert("https://api.example" in printed && "key-123" !in printed && "tenant-789" !in printed) } - test("exported bash functions in the environment are ignored") { - // what `export -f` / Lmod put in the environment - val env = mapOf("BASH_FUNC_module%%" to "() { if [ -z \"\${LMOD_SH_DBG_ON+x}\" ]; then :; fi\n}") + test("shell code in exported environment variables doesn't fail binding") { + // https://github.com/sksamuel/hoplite/issues/602: Debian's default prompt, and what `export -f` / Lmod put in + // the environment + val env = mapOf( + "PS1" to "\${debian_chroot:+(\$debian_chroot)}\\u@\\h:\\w\\$ ", + "BASH_FUNC_ml%%" to "() { if [ -z \"\${LMOD_SH_DBG_ON+x}\" ]; then :; fi\n}", + ) assert(binder(env).config("server") == ServerConfig("localhost", 8080)) } - test("shell prompt variables in the environment are ignored") { - val env = mapOf("PS1" to "\${debian_chroot:+(\$debian_chroot)}\\u@\\h:\\w\\$ ") + test("an env resolver in a string field resolves from the environment variables") { + val binder = hocon("server.host = \"\${{ env:BOOTABLE_TEST_HOST }}\"", mapOf("BOOTABLE_TEST_HOST" to "example.com")) - assert(binder(env).config("server") == ServerConfig("localhost", 8080)) + assert(binder.config("server") == ServerConfig("example.com", 8080)) + } + + test("an unset env resolver fails") { + val binder = hocon("server.host = \"\${{ env:BOOTABLE_TEST_HOST }}\"") + + val e = runCatching { binder.config("server") }.exceptionOrNull() + assert(e is ConfigException && "Could not resolve 'BOOTABLE_TEST_HOST'" in e.message.orEmpty()) + } + + test("an unset env resolver with a fallback is the fallback") { + val binder = hocon("server.host = \"\${{ env:BOOTABLE_TEST_HOST:-fallback.example.com }}\"") + + assert(binder.config("server") == ServerConfig("fallback.example.com", 8080)) + } + + test("a ref resolver resolves another config value, and an env resolver resolves into a non-string field") { + val binder = hocon( + "app.host = example.com\nserver.host = \"\${{ ref:app.host }}\"\nserver.port = \"\${{ env:BOOTABLE_TEST_PORT }}\"", + mapOf("BOOTABLE_TEST_PORT" to "9090"), + ) + + assert(binder.config("server") == ServerConfig("example.com", 9090)) } - test("unresolved substitutions in config still fail") { - val env = mapOf("SERVER_HOST" to "\${BOOTABLE_TEST_UNSET_VARIABLE}") + test("an env resolver resolves into a Masked field") { + val binder = hopliteConfigBinder( + report = false, + withConfigStage3 = { + addPropertySource(PropertySource.string(apiConf.replace("key-123", "\"\${{ env:BOOTABLE_TEST_KEY }}\""), "conf")) + }, + environmentVariables = { mapOf("BOOTABLE_TEST_KEY" to "key-456") }, + ) + + assert(binder.config("api").apiKey == Masked("key-456")) + } + + test("preprocessor substitutions are left as literal text") { + val binder = hocon("server.host = \"\${BOOTABLE_TEST_HOST}\"", mapOf("BOOTABLE_TEST_HOST" to "example.com")) + + assert(binder.config("server") == ServerConfig("\${BOOTABLE_TEST_HOST}", 8080)) + } + + // HOCON resolves its own substitutions at parse time, falling back to the process environment, which the + // environmentVariables parameter doesn't reach; so these reference PATH, which is set, and a variable that isn't. + test("HOCON substitutions resolve, in strings and in maps") { + val path = System.getenv("PATH") + val binder = hocon( + """ + base = "example.com" + server.host = ${'$'}{base} + server.host = ${'$'}{?BOOTABLE_TEST_UNSET_VARIABLE} + properties.values { + path = ${'$'}{PATH} + optionalPath = ${'$'}{?PATH} + base = ${'$'}{base} + } + """.trimIndent() + ) + + assert(binder.config("server") == ServerConfig("example.com", 8080)) + assert( + binder.config("properties").values == + mapOf("path" to path, "optionalPath" to path, "base" to "example.com") + ) + } + + test("an env resolver in a map value is not resolved") { + // Hoplite's resolvers only resolve a data class field whose value is a string, not the values of a map, list, + // or set; see ContextResolver.resolve in hoplite-core. If this fails, Hoplite resolves them now: update this + // test, the README, and the CHANGELOG. + val binder = hocon( + "properties.values.host = \"\${{ env:BOOTABLE_TEST_HOST }}\"", + mapOf("BOOTABLE_TEST_HOST" to "example.com"), + ) - val e = runCatching { binder(env).config("server") }.exceptionOrNull() - assert(e is ConfigException && "Unresolved substitution" in e.message.orEmpty()) + assert(binder.config("properties").values == mapOf("host" to "\${{ env:BOOTABLE_TEST_HOST }}")) } test("enum values match the entries' names in any case") {