Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions crates/chunk-backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,14 @@ retains its bundle before its functions become callable; retained bundles reload
commits are outstanding or another deployment change is in progress. At most 16 deployments are resident, and they all
serve against the one database.

Schema changes are additive. Table names are the keys of the schema the project's `server/schema/index.ts` exports, not
file paths. A deployment's tables are merged into the store's installed schema: new tables, optional new fields and new
indexes are accepted; changing an existing field or index is rejected. Tables starting with `chunk_` are reserved.
Table names are the keys of the schema the project's `server/schema/index.ts` exports, not file paths. A deployment's
tables are merged into the store's installed schema: new tables, optional new fields and new indexes are accepted, and
deployments that declare one index name with different fields each read their own. Removing or renaming a field, or
adding a required one, needs a migration in the deployment's journal (see the SDK's
[migrations](../chunk-build/sdk/README.md#migrations)); a field's type can't change in place. While an expand migration
is active the store keeps both shapes, and a resident deployment that carries it runs its `to` on each write from the
old shape, and its `back`, where it has one, on each write from the new. Finishing it drops the old shape. Tables
starting with `chunk_` are reserved.

Release a deployment once its mutations and subscriptions have drained; `release` returns `Busy` while an action still
uses it. Releasing removes the bundle and permanently retires the ID; data, schema and operation outcomes stay. If a
Expand Down
21 changes: 16 additions & 5 deletions crates/chunk-bots/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -63,20 +63,31 @@ pub struct Config {
impl Config {
pub fn validate(&self) -> Result<()> {
ensure!(self.bots > 0 && self.bots <= 100_000, "bots must be 1..=100000");
let longest = self.prefix.len() + (self.first + self.bots - 1).to_string().len();
let last = self.first.checked_add(self.bots - 1);
ensure!(last.is_some(), "first and bots must leave every index within a u32");
let longest = self.prefix.len() + last.map_or(0, |last| last.to_string().len());
ensure!(
!self.prefix.is_empty()
&& longest <= 16
&& self.prefix.bytes().all(|byte| byte.is_ascii_alphanumeric() || byte == b'_'),
"prefix must be alphanumeric or '_', and names at most 16 characters"
);
ensure!(self.login_rate > 0.0 && self.login_rate <= 10_000.0, "login rate must be in (0, 10000]");
ensure!(self.max_pending > 0, "max pending must be positive");
ensure!((0.001..=10_000.0).contains(&self.login_rate), "login rate must be in [0.001, 10000]");
ensure!((1..=100_000).contains(&self.max_pending), "max pending must be 1..=100000");
ensure!(self.move_hz <= 20, "move rate must be at most 20 Hz, the client tick rate");
ensure!(self.ping_interval >= 0.0 && self.command_interval > 0.0, "invalid ping or command interval");
ensure!(
(0.0..=86_400.0).contains(&self.ping_interval)
&& self.command_interval > 0.0
&& self.command_interval <= 86_400.0,
"ping interval must be in [0, 86400] and command interval in (0, 86400]"
);
ensure!(self.command.as_ref().is_none_or(|command| command.len() < 200), "command too long");
ensure!(self.view_distance >= 2, "view distance must be at least 2");
ensure!(self.stats_interval > 0 && self.login_timeout > 0, "intervals must be positive");
ensure!(
(1..=86_400).contains(&self.stats_interval) && (1..=86_400).contains(&self.login_timeout),
"stats interval and login timeout must be in [1, 86400]"
);
ensure!(self.threads.is_none_or(|threads| (1..=1024).contains(&threads)), "threads must be 1..=1024");
Ok(())
}

Expand Down
3 changes: 1 addition & 2 deletions crates/chunk-environment/src/services.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,13 @@ impl Default for Services {
impl FromStr for Services {
type Err = io::Error;

/// Accepts `core,gateway`, `core` and `gateway`, in any order. A legacy `exec` is ignored with a warning.
/// Accepts `core,gateway`, `core` and `gateway`, in any order.
fn from_str(value: &str) -> io::Result<Self> {
let mut services = BTreeSet::new();
for name in value.split(',').map(str::trim) {
match name {
"core" => _ = services.insert(Service::Core),
"gateway" => _ = services.insert(Service::Gateway),
"exec" => tracing::warn!("ignoring exec in CHUNK_SERVICES; core runs backend functions"),
name => return Err(io::Error::other(format!("unknown service {name:?} in CHUNK_SERVICES"))),
}
}
Expand Down
7 changes: 3 additions & 4 deletions crates/chunk-environment/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -132,12 +132,11 @@ fn authorized<T>(message: T, token: &str) -> tonic::Request<T> {
}

#[test]
fn services_accept_core_and_gateway_and_ignore_a_legacy_exec() {
fn services_accept_core_and_gateway() {
assert_eq!("gateway, core".parse::<Services>().unwrap(), Services::default());
assert_eq!("core,gateway,exec".parse::<Services>().unwrap(), Services::default());
let core = "core,exec".parse::<Services>().unwrap();
let core = "core".parse::<Services>().unwrap();
assert!(core.contains(Service::Core) && !core.contains(Service::Gateway));
for rejected in ["", "exec", "core,jvm", "core,,gateway"] {
for rejected in ["", "core,exec", "core,jvm", "core,,gateway"] {
assert!(rejected.parse::<Services>().is_err(), "{rejected:?} was accepted");
}
let gateway = "gateway".parse::<Services>().unwrap();
Expand Down
11 changes: 7 additions & 4 deletions crates/chunk-jvm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,10 @@ replaced, not restarted: a restarted runner exits with code 77.

Empty values count as unset. Cached releases and AOT caches are verified again before reuse.

The JVM runs as `java -Xmx<heap>m -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError [AOT flags] -jar <app jar>` in a fresh
working directory under `/tmp`. The heap is the lowest of the cgroup v2 memory limit, the machine's memory and the
The JVM runs as
`java -Xms<heap>m -Xmx<heap>m -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError --enable-native-access=ALL-UNNAMED [AOT flags] -jar <app jar>`
in a fresh working directory under `/tmp`. On one CPU it runs `-XX:+UseSerialGC` instead of G1, and a JVM recording its
AOT cache starts without `-Xms`. The heap is the lowest of the cgroup v2 memory limit, the machine's memory and the
launch profile's memory, less 200 MiB and a tenth of that memory for everything outside the heap. Besides the runner's
own environment, the JVM gets what the Java runtime reads: `CHUNK_PROCESS_TOKEN` (the machine credential),
`CHUNK_DEPLOYMENT`, `CHUNK_CORE_ENDPOINT`, `CHUNK_PROCESS_ID`, `CHUNK_PROCESS_GENERATION`, `CHUNK_MACHINE_PROFILE`,
Expand All @@ -48,8 +50,9 @@ The runner reports its Java runtime to core in `chunk:launch` (`IMPLEMENTOR`, `J
`-XX:AOTMode=record -XX:AOTConfiguration=<file>`. Once it exits cleanly (0, or on a forwarded SIGTERM or SIGINT), the
runner creates the cache with `-XX:AOTMode=create`, for up to 45 seconds, and uploads it (`chunk:aot-write`) for up to
30 seconds more; core holds the host's release for up to 90 seconds meanwhile. If anything fails, the runner tells
core no cache came, and another host may record it. A machine with less than 768 MiB of memory doesn't record, since
creating the cache needs a few hundred MiB beyond the heap, but it still uses a cache a larger machine made.
core no cache came, and another host may record it. A machine with less than 768 MiB of memory, or 448 MiB on one CPU,
doesn't record, since creating the cache needs a few hundred MiB beyond the heap, but it still uses a cache a larger
machine made.

The cache never changes the runner's exit code. Java checks the JAR's path, size and modification time against the
cache; unpacked releases keep the archive's zero timestamps under the same `CHUNK_CACHE`, so a cache fits every machine
Expand Down
3 changes: 2 additions & 1 deletion crates/chunk-management/src/methods.rs
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,8 @@ unary! {
/// Starts deleting an environment, which finishes in the background; `get_environment` is not found once it has.
delete_environment:
"ProjectService" / "DeleteEnvironment" (DeleteEnvironmentRequest) -> DeleteEnvironmentResponse;
/// Creates an environment from a snapshot of another, deploying the source's active release to it.
/// Creates an environment from a snapshot of another. It starts without a deployment, and gets the release its
/// restored data was serving once its core first attaches.
fork_environment: "ProjectService" / "ForkEnvironment" (ForkEnvironmentRequest) -> ForkEnvironmentResponse;
/// Lists the snapshots stored in an environment's log, newest first.
list_snapshots: "ProjectService" / "ListSnapshots" (ListSnapshotsRequest) -> ListSnapshotsResponse;
Expand Down
2 changes: 1 addition & 1 deletion crates/chunk-store/src/model.rs
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ impl IndexRange {
}
}

/// The backend hashes deployment, contract, function, caller and validated arguments.
/// The backend hashes function, caller and validated arguments, so the identity outlasts a redeployment.
/// Stable identity lets a lost acknowledgement be recovered without reexecution.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Operation {
Expand Down
Loading