Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions CHANGELOG.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,55 @@ Bootable now uses Hoplite 3.0.5, where an environment variable selects a sealed
For example, `tiebreaker { _type = Shuffled, seed = 42 }` in config, or `BOOT_SERVICE_ORDER_TIEBREAKER=Shuffled` and `BOOT_SERVICE_ORDER_TIEBREAKER_SEED=42`.
An environment variable that selects a subtype overrides the one config selects, and config can still set the subtype's properties.

=== Config substitutions use Hoplite's resolvers

`hopliteConfigBinder` now substitutes values with Hoplite's resolvers, `${{ context:path }}`, instead of its preprocessors, `${VAR}` and `{{key}}`.
Hoplite has deprecated preprocessors in favour of resolvers since 2023, and its maintainer has confirmed resolvers are the intended default.
Resolvers resolve a value only when it is bound, so a `${...}` elsewhere, such as in an exported shell variable like `PS1`, no longer fails loading.
That makes Bootable's filter of such environment variables unnecessary, so it is removed.

The substitutions map as follows:

[cols="1,1"]
|===
|Preprocessor |Resolver

|`${VAR}` (environment variable or system property)
|`${{ env:VAR }}`, or `${{ sysprop:name }}` for a system property

|`${VAR:-default}`
|`${{ env:VAR:-default }}`

|`{{key}}` or `${key}` (another config value)
|`${{ ref:key }}`

|`${random.int}`, `${random.int(10)}`, `${random.uuid}`, ...
|`${{ random:int }}`, `${{ random:int(10) }}`, `${{ random:uuid }}`, ...
|===

`${{ env:name }}` reads the environment that `hopliteConfigBinder`'s `environmentVariables` supplies, as the environment variables source does.

HOCON's own substitutions, unquoted `${VAR}`, `${?VAR}`, and `${a.b}`, are unaffected: HOCON resolves them when it parses the file, falling back to the environment for `${VAR}` and `${?VAR}`.
They are the recommended way to reference environment variables, and the only one inside maps and lists.

==== Breaking changes

* Preprocessor substitutions are no longer expanded.
They are not an error: a `${VAR}` or `{{key}}` is left in the value as literal text.
That includes a `${VAR}` inside a quoted HOCON string, which Hoplite's preprocessors used to expand and HOCON doesn't.
* A resolver resolves only a data class field whose value is a single string.
A `${{ ... }}` inside a map, list, or set, such as `dataSourceProperties`, Flyway's `placeholders`, `locations`, or `schemas`, or telemetry's `otlp.headers` or `resourceAttributes`, is left as literal text; use HOCON's `${?VAR}` there.
* A `${{ ... }}` that resolves to nothing fails the bind, and doesn't fall back to the data class's default value (https://github.com/sksamuel/hoplite/issues/460[sksamuel/hoplite#460]).
Give an optional value a fallback, e.g. `${{ env:DB_HOST:-localhost }}`.
* Hoplite's module resolvers use different names from its module preprocessors: `awssm://` (and `${awssecret:...}`, `secretsmanager://`) is `aws-secrets-manager://`, `azurekeyvault://` is `azure-key-vault://`, and `gcpsm://` is `gcp-secrets-manager://`.
`vault://` and `consul://` are unchanged.
There is no resolver for AWS Parameter Store's `${ssm:key}` or `paramstore://`.
* Hoplite's module resolvers, for Vault, AWS, GCP, Azure, and Consul, can now be added in the `withConfigStageN` hooks with `addResolver`, which used to throw because Bootable had added preprocessors.
A preprocessor added there still runs, but an unresolved substitution is no longer reported, since Hoplite skips that check once there are resolvers.
* The config report shows values before resolvers run, so it shows `${{ env:DB_HOST }}` rather than the value it resolves to.
Resolvers don't mark values as secrets in the report the way some preprocessors did, but since the report shows the expression, it doesn't print the secret either.
* `isKnownNonConfigEnvVar` is removed.

=== A failed start shuts down the services that started

When a service's `start()` throws, Bootable now runs the normal shutdown before exiting with -1, instead of exiting at once.
Expand Down
1 change: 1 addition & 0 deletions README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ In your own config classes, declare secrets with your config library's type, suc
See https://github.com/sksamuel/hoplite[hoplite].

Add this module to your project to use Hoplite for configuration.
A config value can take an environment variable with HOCON's own `${?VAR}`, or with Hoplite's `${{ env:VAR }}` resolver, which works only in a field that holds a single string; see its documentation's substitutions section.

xref:./boot-config-hoplite/README.adoc[Documentation]

Expand Down
29 changes: 29 additions & 0 deletions boot-config-hoplite/README.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,35 @@ Bootable injects a Hoplite `ParameterMapper` that maps configuration properties

See also https://github.com/sksamuel/hoplite/issues/410[hoplite issue #410].

## Substitutions

A config value can reference an environment variable, a system property, or another config value with Hoplite's https://github.com/sksamuel/hoplite#resolvers[resolvers]:

```hocon
app.name = "orders"
db {
host = "${{ env:DB_HOST }}"
port = "${{ env:DB_PORT:-5432 }}"
user = "${{ sysprop:db.user }}"
name = "${{ ref:app.name }}"
}
```

A reference that resolves to nothing, without a `:-fallback`, fails the bind.
`${{ env:name }}` reads the environment variables passed to `hopliteConfigBinder(environmentVariables = ...)`, which default to the process environment.
Hoplite modules add resolvers for secret stores, such as Vault, AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault; add them with `addResolver` in a `withConfigStageN` hook.

Resolvers resolve only a data class field whose value is a single string, not the entries of a map, list, or set.
There, and anywhere else, use HOCON's own substitutions, which HOCON resolves when it parses the file, falling back to the environment:

```hocon
db.dataSourceProperties {
sslmode = ${?DB_SSLMODE}
}
```

Hoplite's deprecated preprocessor syntax, `${VAR}` in a quoted string or `{{key}}`, is not expanded and is left as literal text.

## Custom Types

Hoplite `Decoder` instances may be registered via the Java ServiceLoader framework, or passed to the `hopliteConfigBinder(...)` factory.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ package com.github.rocketraman.bootable.config
import com.sksamuel.hoplite.*
import com.sksamuel.hoplite.ConfigBinder as UpstreamConfigBinder
import com.sksamuel.hoplite.decoder.Decoder
import com.sksamuel.hoplite.fp.valid
import com.sksamuel.hoplite.resolver.context.ContextResolver
import com.sksamuel.hoplite.resolver.context.EnvVarContextResolver
import com.sksamuel.hoplite.secrets.CompositeSecretsPolicy
import com.sksamuel.hoplite.secrets.Obfuscator
import com.sksamuel.hoplite.secrets.SecretsPolicy
Expand All @@ -19,24 +22,10 @@ class HopliteConfigBinder(private val binder: UpstreamConfigBinder): ConfigBinde
binder.bindOrThrow(type, prefix)
}

/**
* Whether the environment variable [name] is set by the shell or other tooling and is never application config.
* These are excluded from the environment source because their values are shell code that routinely contains
* `${...}`, which Hoplite would otherwise fail on as an unresolved substitution.
*
* TODO remove once Hoplite excludes these itself: https://github.com/sksamuel/hoplite/issues/602
*/
fun isKnownNonConfigEnvVar(name: String): Boolean =
// exported bash functions (`export -f`, or tools like Lmod), named `BASH_FUNC_name%%`
name.startsWith("BASH_FUNC_") ||
name in shellPromptEnvVars

private val shellPromptEnvVars = setOf("PS1", "PS2", "PS4", "PROMPT_COMMAND")

/**
* Builds a default Hoplite-based [ConfigBinder], reading properties in the following order, from highest priority
* to lowest:
* - environment variables (supports upper case, convert `_` to `.`; see [isKnownNonConfigEnvVar] for exclusions)
* - environment variables (supports upper case, convert `_` to `.`)
* - system properties (with `config.override.` prefix)
* - `application-local.conf` in resources
* - `application.conf` in resources
Expand All @@ -54,11 +43,17 @@ private val shellPromptEnvVars = setOf("PS1", "PS2", "PS4", "PROMPT_COMMAND")
* - stage3
*
* If report is true (the default), a report of all config keys and values is enabled. All sensitive values should
* be of type [Masked] (or Hoplite's `Secret`) to avoid leaking them in the report, named with "secret",
* "credential", or "pass" in the name, or have pre-processor metadata that identifies the value as a secret.
* Configure this via [secretsPolicy]; a [Masked] value is masked whatever the policy.
* be of type [Masked] (or Hoplite's `Secret`), or named with "secret", "credential", or "pass" in the name, to
* avoid leaking them in the report. Configure this via [secretsPolicy]; a [Masked] value is masked whatever the
* policy. The report shows values as loaded, before resolvers run, so it shows `${{ env:DB_PASSWORD }}`, not the
* password.
*
* Config values may reference other values with Hoplite's resolvers, e.g. `${{ env:DB_HOST }}`,
* `${{ env:DB_HOST:-localhost }}`, or `${{ sysprop:name }}`. These resolve only in a field whose value is a single
* string, not in a map, list, or set; use HOCON's own `${?DB_HOST}` substitutions there.
*
* [environmentVariables] supplies the environment variables to read (defaults to [System.getenv]).
* [environmentVariables] supplies the environment variables to read, both as a config source and for
* `${{ env:name }}` (defaults to [System.getenv]).
*
* A sealed type is chosen by its subtype's name, in a `_type` key next to the subtype's properties, e.g.
* `tiebreaker { _type = Shuffled, seed = 42 }`, or, in the environment, as the sealed type's own value, e.g.
Expand Down Expand Up @@ -89,13 +84,13 @@ fun hopliteConfigBinder(
.addDefaultDecoders()
.addDecoder(MaskedDecoder())
.addDecoders(decoders)
.addDefaultPreprocessors()
.addResolvers(defaultResolvers().map {
if (it == EnvVarContextResolver) EnvironmentVariablesContextResolver(environmentVariables) else it
})
.addDefaultNodeTransformers()
.addDefaultParamMappers()
.withConfigStage1()
.addPropertySource(EnvironmentVariablesPropertySource({
environmentVariables().filterKeys { !isKnownNonConfigEnvVar(it) }
}))
.addPropertySource(EnvironmentVariablesPropertySource(environmentVariables))
.addPropertySource(SystemPropertiesPropertySource) // requires config.override. prefix
.addPropertySource(UserSettingsPropertySource)
.addPropertySource(XdgConfigPropertySource)
Expand All @@ -121,3 +116,17 @@ fun hopliteConfigBinder(

return HopliteConfigBinder(upstream)
}

/**
* Hoplite's [EnvVarContextResolver], reading [environmentVariables] instead of [System.getenv], so that
* `${{ env:name }}` sees the same environment as the environment variables source.
*/
private class EnvironmentVariablesContextResolver(
private val environmentVariables: () -> Map<String, String>,
) : ContextResolver() {
override val contextKey = EnvVarContextResolver.contextKey
override val default = EnvVarContextResolver.default

override fun lookup(path: String, node: StringNode, root: Node, context: DecoderContext): ConfigResult<String?> =
environmentVariables()[path].valid()
}
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ enum class Speed { Fast, Slow }
// public: hoplite constructs it reflectively
data class SpeedConfig(val speed: Speed)

// public: hoplite constructs it reflectively
data class PropertiesConfig(val values: Map<String, String>)

private fun binder(env: Map<String, String>): ConfigBinder =
hopliteConfigBinder(report = false, environmentVariables = { env })

Expand Down Expand Up @@ -52,6 +55,11 @@ private object NothingSecret : SecretsPolicy {
override fun isSecret(node: Node, type: KType?) = false
}

private fun hocon(conf: String, env: Map<String, String> = emptyMap()): ConfigBinder =
hopliteConfigBinder(report = false, environmentVariables = { env }, withConfigStage2 = {
addPropertySource(PropertySource.string(conf, "conf"))
})

val HopliteConfigBinderTest by testSuite {
test("decodes Masked values in strict mode, maps' included") {
val binder = hopliteConfigBinder(
Expand Down Expand Up @@ -81,24 +89,97 @@ val HopliteConfigBinderTest by testSuite {
assert("https://api.example" in printed && "key-123" !in printed && "tenant-789" !in printed)
}

test("exported bash functions in the environment are ignored") {
// what `export -f` / Lmod put in the environment
val env = mapOf("BASH_FUNC_module%%" to "() { if [ -z \"\${LMOD_SH_DBG_ON+x}\" ]; then :; fi\n}")
test("shell code in exported environment variables doesn't fail binding") {
// https://github.com/sksamuel/hoplite/issues/602: Debian's default prompt, and what `export -f` / Lmod put in
// the environment
val env = mapOf(
"PS1" to "\${debian_chroot:+(\$debian_chroot)}\\u@\\h:\\w\\$ ",
"BASH_FUNC_ml%%" to "() { if [ -z \"\${LMOD_SH_DBG_ON+x}\" ]; then :; fi\n}",
)

assert(binder(env).config<ServerConfig>("server") == ServerConfig("localhost", 8080))
}

test("shell prompt variables in the environment are ignored") {
val env = mapOf("PS1" to "\${debian_chroot:+(\$debian_chroot)}\\u@\\h:\\w\\$ ")
test("an env resolver in a string field resolves from the environment variables") {
val binder = hocon("server.host = \"\${{ env:BOOTABLE_TEST_HOST }}\"", mapOf("BOOTABLE_TEST_HOST" to "example.com"))

assert(binder(env).config<ServerConfig>("server") == ServerConfig("localhost", 8080))
assert(binder.config<ServerConfig>("server") == ServerConfig("example.com", 8080))
}

test("an unset env resolver fails") {
val binder = hocon("server.host = \"\${{ env:BOOTABLE_TEST_HOST }}\"")

val e = runCatching { binder.config<ServerConfig>("server") }.exceptionOrNull()
assert(e is ConfigException && "Could not resolve 'BOOTABLE_TEST_HOST'" in e.message.orEmpty())
}

test("an unset env resolver with a fallback is the fallback") {
val binder = hocon("server.host = \"\${{ env:BOOTABLE_TEST_HOST:-fallback.example.com }}\"")

assert(binder.config<ServerConfig>("server") == ServerConfig("fallback.example.com", 8080))
}

test("a ref resolver resolves another config value, and an env resolver resolves into a non-string field") {
val binder = hocon(
"app.host = example.com\nserver.host = \"\${{ ref:app.host }}\"\nserver.port = \"\${{ env:BOOTABLE_TEST_PORT }}\"",
mapOf("BOOTABLE_TEST_PORT" to "9090"),
)

assert(binder.config<ServerConfig>("server") == ServerConfig("example.com", 9090))
}

test("unresolved substitutions in config still fail") {
val env = mapOf("SERVER_HOST" to "\${BOOTABLE_TEST_UNSET_VARIABLE}")
test("an env resolver resolves into a Masked field") {
val binder = hopliteConfigBinder(
report = false,
withConfigStage3 = {
addPropertySource(PropertySource.string(apiConf.replace("key-123", "\"\${{ env:BOOTABLE_TEST_KEY }}\""), "conf"))
},
environmentVariables = { mapOf("BOOTABLE_TEST_KEY" to "key-456") },
)

assert(binder.config<ApiConfig>("api").apiKey == Masked("key-456"))
}

test("preprocessor substitutions are left as literal text") {
val binder = hocon("server.host = \"\${BOOTABLE_TEST_HOST}\"", mapOf("BOOTABLE_TEST_HOST" to "example.com"))

assert(binder.config<ServerConfig>("server") == ServerConfig("\${BOOTABLE_TEST_HOST}", 8080))
}

// HOCON resolves its own substitutions at parse time, falling back to the process environment, which the
// environmentVariables parameter doesn't reach; so these reference PATH, which is set, and a variable that isn't.
test("HOCON substitutions resolve, in strings and in maps") {
val path = System.getenv("PATH")
val binder = hocon(
"""
base = "example.com"
server.host = ${'$'}{base}
server.host = ${'$'}{?BOOTABLE_TEST_UNSET_VARIABLE}
properties.values {
path = ${'$'}{PATH}
optionalPath = ${'$'}{?PATH}
base = ${'$'}{base}
}
""".trimIndent()
)

assert(binder.config<ServerConfig>("server") == ServerConfig("example.com", 8080))
assert(
binder.config<PropertiesConfig>("properties").values ==
mapOf("path" to path, "optionalPath" to path, "base" to "example.com")
)
}

test("an env resolver in a map value is not resolved") {
// Hoplite's resolvers only resolve a data class field whose value is a string, not the values of a map, list,
// or set; see ContextResolver.resolve in hoplite-core. If this fails, Hoplite resolves them now: update this
// test, the README, and the CHANGELOG.
val binder = hocon(
"properties.values.host = \"\${{ env:BOOTABLE_TEST_HOST }}\"",
mapOf("BOOTABLE_TEST_HOST" to "example.com"),
)

val e = runCatching { binder(env).config<ServerConfig>("server") }.exceptionOrNull()
assert(e is ConfigException && "Unresolved substitution" in e.message.orEmpty())
assert(binder.config<PropertiesConfig>("properties").values == mapOf("host" to "\${{ env:BOOTABLE_TEST_HOST }}"))
}

test("enum values match the entries' names in any case") {
Expand Down
Loading